Title: Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Meteorological Variable Modeling

Federared Learning

On-device Intelligence

Foundation Model
Abstract: This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variable modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

Abstract

This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-WEATHER, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, lowrank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-WEATHER outperforms the state-of-the-art results by a large margin across various tasks ( e.g. , forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-WEATHER can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios. Code available on https://github.com/shengchaochen82/LM-Weather .

1 Introduction

Accurately modeling weather variation pattern from large amount of meteorological variables sequences is increasingly vital for providing efficient weather analysis support for disaster warning. Recently, the promise of learning to understand weather pattern from data via deep learning (DL) has led to an ongoing paradigm shift apart from the long-established physics-based methods [1, 2].

Mining potential patterns from meteorological sequences that collected from different regions, including forecasting and imputation, is one of the most important problems in meteorology. Significant progress has been made by several latest time series approaches [1, 3, 4]. These approaches formulate meteorological variable modeling as an end-to-end spatio-temporal learning problem. This overlooks the reality that ground weather devices distributed globally gather vast amounts of data quickly. The sheer volume of data, coupled with limited network capacity, necessitates local processing on the devices, making centralised learning challenging [5]. On-device intelligence enables edge devices to compute independently, offering a primary solution to the problem.

Federated Learning (FL) [7] is a promising on-device intelligence implementation that collaboratively train a uniform model across devices without exchanging raw data. However, the model often underperform due to data heterogeneity among clients. Personalized FL (PFL) provides new insights for on-device intelligence that allows each device obtains customized models for providing personalized

️

�

Low-rank Matrices

Full-rank Matrices

❄️

�

Low-rank Matrices

Full-rank Matrices

❄️

Average

Client 1

Client N

Server

Communication

Communication

Task Adapter Generation

PLM

Prediction

Trend

Seasonal

Generator

Generator

Residual

Generator

Reversible

Normalization

FFN

Channel-Independent

Patching

Decomposition

Position Embedding

Temporal Embedding

Reversible

Normalization

Channel-Independent

Patching

......

......

......

Multivariate Weather Series

(Task Adapter Generation)

LoRA

Personalized

Adapter

Local Weather Data

Personalized

Adapter

Local Weather Data

a

b

c

d

e

Information Flow

Trainable

Frozen

Plus

Concatenate

insights [8, 9]. Albeit PFL methods showing revolutionized capability in this field, we argue that the current advancements are not necessarily at their best in on-device meteorological variable modeling as three major obstacles remain and hinder further progress:

(i) Challenge of Heterogeneity. Weather data's heterogeneity, unlike that of images or text, arises mainly from the unique characteristics of data collected by weather devices in various regions, such as tropical or arid areas. Furthermore, sensor malfunctions or extreme events can lead to collection disruptions or inconsistent missing data, which significantly increase the differences in data distribution across devices.

(ii) Underperformed Shallow Network Structures. The vast and varied data gathered by weather devices challenge simpler neural network models to generalize effectively. Furthermore, the frequent updates of weather data (hourly or by the minute) require neural models on devices to train and infer more often. This demand is hard to meet with deeper models that, while more performant, are also more resource-intensive.

(iii) Resource-constrained Weather Devices. From a computation perspective, weather devices cannot afford of training complex neural models from scratch, especially for foundation models [4]. From a communication perspective, transmitting complete model during the

aggregation phase in FL/PFL significantly increases communication overhead, which is impractical for real-time weather modeling.

Therefore, a compact foundational model (FM) is crucial for personalized on-device weather modeling. Yet, there's a gap in FMs for observational data. Models trained on large-scale simulation data struggle in practical applications because of notable differences in data formats and parameter scales [1, 4].

Inspired by the impressive progress of large language models (LLMs) in natural language processing, recent literature in time series analysis research has also demonstrated that pre-trained LMs provide excellent performance over dedicated models for time series analysis with tuning [10] or reprogramming [11]. This comprehensive and thorough sequence knowledge from language models can be effortlessly transferred across domains without large-scale parameter tuning. Thus, an exciting research question naturally arises:

Question: Since PLMs are powerful sequence modelers, can we leverage PLMs as foundation models to achieve personalized on-device meteorological variable modeling?

In this paper, we show that pre-trained language models (PLMs) can as outstanding foundation models that tuned on each device with low cost can achieve personalized on-device weather pattern modeling. We propose LM-WEATHER, a generic approach to taming PLMs to understand heterogeneity ondevice weather data. As shown in Fig.1a , we conduct a local tuning on an uniform PLM (e.g., GPT2), where lightweight personalized adapters are implanted to endow PLMs with weather pattern awareness by decomposing weather sequence to implicit knowledge (e.g., seasonal, trend). During communication between client and server, fewer parameters are shared globally while locally retained adapters are enforced to resist heterogeneity and facilitate privacy-assured fusion of global knowledge.

We highlight our contributions and findings as follows:

We introduce LM-WEATHER, a generic approach that transforms Pre-trained Language Models as the foundation model to customized on-device meteorological variable modeling via personalized adapter . LM-WEATHER yields preferable meteorological variable sequences modeling, while being parameter-, communication-, and data-efficient.

We collect and compile four real-world versatile datasets for on-device meteorological variable modeling across regions. As opposed to simulated datasets such as ERA5 [12], our datasets are all real-time observations. These datasets based on real-world practice and challenging, provide a pioneer in the field of on-device meteorological variable modeling.

Experiments show that LM-WEATHER advances the state-of-the-art methods by a large margin across various setting while keeping 3.7% of parameters communication. LMWEATHER also demonstrates superior communication efficiency in the context of meteorological variable modeling, beating FL baselines tailored to reduce communication overhead.

In particular, we find that LM-WEATHER can accurately handle structurally nondeterministic sequences (e.g., differences in time or variable dimensions across devices) thanks to the learned sequences knowledge from pre-trained LMs. We also find that LMWEATHER can indeed be spatio-temporal sequences sensitive, thereby better modeling the weather pattern specificity of those high distribution similarity.

We find that LM-WEATHER can work well in data-limited environments across various few-shot settings. We further evaluate zero-shot generalizability of LM-WEATHER in modeling complex weather patterns of unseen data, including different group of datasets and other devices, and observe superb performance.

We highlight that the goal of this study is not to compete but instead to complement current on-device meteorological variables modeling framework. Today's climate foundation models are typically trained from scratch, utilizing exceptionally large datasets (nearly 100TB [4, 13]) and incurring substantial computational costs [1]. We hope that LM-WEATHER offers a cost-effective alternative for modeling meteorological variables on-device, thereby enabling accurate regional weather trend analysis. In addition, the dataset we complied can be the important resource to provide exploring chances for this field, facilitating future research.

2 Preliminaries

2.1 On-device Meteorological Variable (Sequence) Modeling

The on-device meteorological variable (sequence) modeling challenge involves predicting future sequences from past observations for forecasting or predicting missing values for imputation on each device. While traditional physics-based approach this as a complex problem of solving multilevel atmospheric equations [14], recent deep learning techniques have shown significant potential in uncovering patterns for better weather prediction [4, 2].

Problem Formulation On-device meteorological variable modeling can be formulated as an endto-end sequence-to-sequence learning problem for each device without exchange raw data. Formally, a parameterized local model for i -th device M i θ is tasked with predicting the weather sequence,

where the X i ∈ R L × C and ˆ X i ∈ R L ′ × C ′ denote the input and output sequences on i -device, L and L ′ is the input length and output length, C and C ′ is the number of input and output variable. Note that the L ′ → L when performing imputation. The local learning objective on each device is to find the model parameter θ that minimize the distance between ˆ X i and X i given sufficient weather sequence data. The overall optimization objective is based on FedAvg,

where n i and n is the number of samples held by the i -th client and all clients 1 , respectively, F ( θ |{ D } ) denotes the local objective function, { D } is the local data.

2.2 Language Models in Time Series

Language models (LMs) trained on large-scale sequence data have shown extraordinary advances and led to a significant paradigm shift in NLP, boosting machines in understanding human languages (BERT/MLM-style) and synthesizing human-like text (GPT/CLM-style [15]). Analogies between time series and human languages have long been noted [16]. Recent advancements in time series analysis have demonstrated the effectiveness of PLMs in modeling time series [17, 11]. Although some of those have shown that PLMs can beat time series-specific models in updating a minor fraction of parameters [18]. As such, it is exciting to expect cutting-edge techniques of language modeling can tackle weather variables sequence-related problems rather than considering train climate foundation models [4, 1] from scratch that are heavy and expensive, and are trained from simulated data.

3 Taming PLMs for On-device Meteorological Variable (Sequence) Modeling

Overview We proposed a generic framework named LM-WEATHER that encouraging PLMs to yield accurate prediction while keeping high efficiency for each device. The architecture is illustrated in Fig. 1 . To endow PLMs with weather pattern awareness, we introduce a lightweight personalized adapter into PLMs (e.g., GPT2 [15]) such that the emergent ability of sequence modeling that transferred from text into weather is activated. To achieve cross-domain knowledge transfer with minimal effort while maintaining the sequence modeling capabilities of PLMs as intact as possible, we introduce lightweight operations in it enables both clients and servers to achieve a good trade-off between performance and efficiency (e.g., computation and communication).

3.1 Local Training on Each Device

Our LM-WEATHER refines PLMs for personalized weather sequence modeling on heterogeneous devices using a modular, plug-and-play architecture. Specifically, we introduce personalized adapter consists of (1) Task Adapter from latent weather knowledge and (2) Parameter Adapter that converts representation from the PLM into into weather forecasts. In addition, we employ lightweight operations in local training to boost computational efficiency.

Task Adapter. To provide PLMs with richer effective information to activate their sequence modeling capabilities in the target knowledge domain, similar to text-based prompts in language to LLMs in NLP, we constructed task adapters by decomposing the input weather sequences into multimodal latent statistical information,

where X k ∈ R L × 1 denote the k -th variable in weather sequence X ∈ R L × C , the trend component X Trend and the seasonal component X Seasonal captures the underlying long-term weather pattern and encapsulates the repeating short-term weather cycles, respectively. Furthermore, the residual component X Residual represents the remainder of the sequence after the trend and seasonality have been extracted. Note that X Trend , X Seasonal, and X Residual have the same shape as X . This decomposition explicitly enables the identification of unusual observation and shifts in seasonal patterns or trends. The X Trend , X Seasonal , X Residual are used to generate Task Adapter via an unified generator as Fig. 1c & Fig. 1d that consisting of Token Embedding, Position Embedding, and Temporal Embedding. Specially, we use one-dimensional convolution operation to map each each specific sample X k while keeping raw shape to generate Token Adapter P TO. Additionally, we use a trainable lookup table to map each point's explicit position in the entire sequence, to generate Position Adapter P PO. Furthermore, we separately encode different time attributes such as minutes, hours, days, weeks, and months, via trainable parameters to dynamically model complex temporal shifts, to generate Temporal Adapter P TE. Finally, for each decomposition components, corresponding generated adapters can be obtained by aggregating Token Adapter P TO ∈ R L × C , Position Adapter P PO ∈ R L × C , and Temporal Adapter P TE ∈ R L × C as P d = P d TO + P d PO + P d TE , where d ∈ { Trend , Seasonal , Residual } , this means that we can obtain P Trend , P Seasonal , P Residual. Details about the generator in Appendix B.5 .

Lightweight Operations. To enhance the PLMs' ability to represent complex inputs while reducing the computational burden to adapt to low-resource devices, we introduce lightweight operations, which includes channel-independent patching (CIP, Fig. 1e ) [6] for input and efficient tuning of parameters for PLMs. Among them, CIP splits the multivariate sequence into separate univariate sequences, each processed by a single model with length L p . This approach outperforms the original method of mixing channels by treating the variables as independent. It enables the model to capture channel interactions indirectly through shared weights, leading to improved performance without directly modeling the complexity of multiple data channels. The total number of inputs patches is P = ( T -L p ) S +2 , where S denotes the horizontal sliding stride. Given these patches X i P ∈ R P × L p , we use rearrange operation and a trainable FFN embed them as ˆ X i P ∈ R P × d m , where d m is dimensions created by the FFN. We also introduce a low-rank adaptation (LoRA) [19] inside PLMs aiming at language modeling for lightweight fine-tuning of attention layers to achieve cross-modal/-domain knowledge transfer from text sequences to weather sequences with minimal effort.

Parameter Adapter. To adapt PLM outputs for downstream weather sequence modeling, we introduce Parameter Adapter , a simple FFN with a single linear layer positioned after the PLM. This adapter transforms the PLM's output to match the prediction horizon, formalized as follows:

where the ˆ P Trend , ˆ P Seasonal , ˆ P Residual, and ˆ X are obtained from CIP based on P Trend , P Seasonal , P Residual , and X . The key objectives are twofold: (1) to enrich the PLM's cross-modal representations by incorporating task-specific knowledge, and (2) to enhance the PLM's output accuracy while preserving its inherent knowledge through the integration of weather data for cross-domain knowledge transfer.

3.2 High-efficiency Communication Between Clients and Server

To avoid data silos and counteract the performance disparities caused by data heterogeneity while ensuring efficient communication, we update personalized adapters locally and share low-rank parameters globally in each round. Specifically, the local PLM M θ can be formulated as below:

where M θ,t denotes the trainable parameter from the low-rank matrices of query and value in attention modules, M θ,f is the frozen parameter (mainly the PLM backbone) and other trainable ones (primarily for the personalized adapter). During client-server communication, only M θ,t is transmitted and averaged using FedAvg [7]. At the start of the next training round, the updated M θ,t is broadcast to clients for further updates. Privacy is further protected by sending minimal parameters.

4 Main Theorems

Theorem 4.1 ( Decomposition Rationality from Time Series). Given a weather series X = X Trend ,t + X Seasonal ,t + X Residual ,t , t ∈ [ t 1 , t n ] . Let E = { e 1 , e 2 , ..., e n } denotes a set of orthogonal bases. Lets E Seasonal ⊆ E denote the subset of E on which X Seasonal ,t has non-zero eigenvalues and E Trend ⊆ E denote the subset of E on which X Trend ,t has non-zero eigenvalues. If X Trend ,t and X Seasonal ,t are not orthogonal, i.e., ∑ n i =1 X i Trend ,t X i Seasonal ,t = 0 , then E Trend ⋂ E Seasonal = 0 , i.e., E can not disentangle the two signals onto two disjoint set of bases.

̸

̸

Theorem 4.2 ( Exchange Low-Rank Matrices Ensures Privacy). Given a on-device weather modeling framework based on federated learning that gloabl optimization object is F ( θ ) = ∑ i =1 n p i f ( { D i } ; θ ) , where f ( x ; θ ) is the loss function of i -th client, { D i } is dataset of i -th client, and p i and θ denote the data distribution weight of client i and the model parameters, respectively. Given that the parameters θ of the PLM M θ broadcasted by the server consist of two parts: a frozen part M θ,f and a trainable part M θ,t , interacting only the low-rank matrix parameters M θ,l ⊂ M θ,t is a subset of trainable part M θ,t during each round ensures privacy.

5 Experiments

In this section, we first present the real-world datasets that we have collected and compiled for ondevice meteorological variable modeling, and second, we evaluate LM-WEATHER on these datasets, which involves normal scenario, a data-limited few-shot scenario, and a zero-shot scenario with no training data (OOD). Please refer to Appendix for more detailed information about proposed datasets and additional results of all evaluations (e.g., full results, additional findings & experiments).

5.1 Datasets

Despite the proliferation of reanalysis data aimed at building frameworks for global climate analysis, these datasets often struggle to model regional weather trend due to: (1) they depend on numerous simulations of atmospheric equations, introducing biases inconsistent with real observations, and (2) they face challenges in refining their scale to suit specific regional applications. Hence, we collected real observational data from various weather stations across different regions. We then organized this data into two series, each comprising two distinct datasets, to underscore the heterogeneity inherent in real-world settings. For detailed information on these datasets, please see the Appendix B.1 .

On-device Weather Series 1# (ODW1). The dataset gathered from 15 ground weather stations across China, Japan, and South Korea, encompasses over 20 variables. It has been divided into two subsets: ODW1T has a heterogeneous time span, meaning the data collection start and end times vary by location. and ODW1V extends ODW1T by adding variability in the observed variables; while one variable remains constant at each station, the others vary.

On-device Weather Series 2# (ODW2). This dataset consists of data from 36 weather observation stations in the United States, Canada, and Israel, covering 5 different variables with a temporal resolution of 1 hour. Following the dataset setting of ODW1 , the dataset was also subdivided into two different dataset, including ODW2T and ODW2V .

5.2 Setup

Baseline. Since our framework is based on a language model, we compare with DL-based SOTA time series models, including Transformer-based methods: Transformer [20], Informer [3], Reformer [21], Pyraformer [22], iTransformer [23], and PatchTST [6], and recent competitive models: GPT4TS [17], DLinear [24] and LightTS [25]. Note that our setting is FL-based, so we place them in FL and rename them FL-( baseline ) like FL-Transformer, etc., and all aggregation methods used in above models is FedAvg [7]. In addition, we report a variants of LM-WEATHER, LM-WEATHER-AVE that based on FedAvg without personalization. Detailed information are in Appendix B.2 .

Basic Setup. We focus on on-device meteorological variable forecasting and imputation tasks. For forecasting, we create scenarios for predicting a single variable (multivariate-univariate) and for predicting all variables (multivariate-multivariate). The main text only includes multivariate-tomultivariate forecasting results due to page constraints. For multivariate-to-univariate forecasts, refer

to the Appendix E. In imputation, we use sequence lengths of { 96 , 192 , 336 , 720 } and apply three different masking probabilities { 25% , 35% , 50% } to represent missing data. The main manuscript shows imputation results for a 50% masking ratio. For more details on the setup, please refer to Appendix B.3 . All our experiments are repeat five times and we report the averaged results.

5.3 Main Results

In this section, we evaluate LM-WEATHER and baseline methods on four on-device meteorological variable modeling datasets in general experiments to validate its effectiveness.

Setups & Results of Forecasting Tasks. Input length L b is fixed to 192, and we use four different prediction horizons L f ∈ { 96 , 192 , 336 , 720 } . Evaluation metrics include mean absolute error (MAE) and root square mean error (RMSE). The brief results is shown in Tab. 1 , where our LMWEATHER outperforms all baselines in most cases and significantly so to the majority of them. Particularly noteworthy is the comparison with GPT4TS that involves fine-tuning PLMs, where LM-WEATHER has an average 9.8% improvement over FL-GPT4TS (MAE reported), and even the variant LM-WEATHER-AVE has an average 4% improvement over FL-GPT4TS. In addition, LM-WEATHER shows significant average performance gains of 11.2% and 19% w.r.t. MAE relative to other SOTA such as FL-DLinear and FL-PatchTST.

Method

LM-WEATHER-AVE

LM-WEATHER

FL-GPT4TS

FL-Reformer

FL-Pyraformer

FL-DLinear

FL-PatchTST

FL-iTransformer

FL-LightTS

FL-Transformer

FL-Informer

Dataset

Length

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

ODW1T

96

44.1

74.8

42.3

71.1

46.3

78.5

70.7

92.9

67.2

86.1

49.7

78.6

45.0

77.0

48.4

80.2

54.8

85.6

50.7

82.1

51.9

83.2

192

46.3

77.5

44.4

73.6

48.6

81.3

75.1

98.3

70.0

90.9

52.3

81.8

47.3

79.8

51.8

84.3

59.5

90.6

52.1

84.0

52.9

84.6

336

47.9

79.3

45.8

75.2

50.3

83.2

79.8

100.5

74.1

92.8

53.9

83.7

49.0

81.7

54.5

87.3

64.0

94.6

52.9

85.2

53.5

85.6

720

51.8

83.0

49.2

78.5

54.4

87.2

87.1

102.9

80.5

95.2

57.2

87.3

53.3

85.6

60.1

93.1

72.4

102.7

55.4

87.6

55.3

87.4

47.5

78.7

45.4

74.6

49.9

82.5

78.2

98.7

73.0

91.3

53.3

82.8

48.6

81.0

53.7

63.7

62.7

93.4

52.8

84.7

53.4

85.2

ODW1V

96

42.7

69.5

42.3

69.6

44.0

71.4

42.9

67.8

57.7

67.2

46.4

73.3

44.3

69.6

56.8

76.8

48.0

75.1

67.0

89.4

59.0

80.3

192

45.5

72.6

44.4

71.7

47.0

75.8

48.4

75.4

59.2

69.4

47.9

75.1

46.8

72.1

55.0

75.0

49.1

79.2

69.9

93.0

61.2

82.8

336

47.2

74.3

46.0

72.4

48.8

77.7

51.0

77.0

63.4

73.3

49.1

76.9

48.5

74.8

62.4

83.7

50.8

77.9

71.4

94.8

63.7

85.8

720

51.2

78.2

49.7

74.0

53.3

81.7

54.5

82.3

67.3

76.1

52.5

80.3

54.3

79.1

72.1

96.2

54.7

82.7

76.2

87.3

68.4

91.8

46.6

73.6

45.6

71.9

48.3

76.7

49.2

75.6

61.9

71.5

49.0

76.4

48.5

73.9

58.1

85.0

50.7

78.7

71.1

91.1

63.1

85.2

ODW2T

96

64.3

88.2

62.8

85.5

66.8

91.7

100.3

126.3

95.0

120.3

67.9

84.7

70.2

88.1

68.6

86.5

68.4

85.4

85.0

103.0

84.7

102.7

192

67.7

91.5

66.2

89.1

71.1

96.1

102.1

130.3

99.9

125.8

71.4

88.1

72.2

90.7

71.1

88.9

71.9

88.9

85.0

103.0

84.9

102.8

336

69.5

93.7

67.9

91.1

72.9

98.4

104.2

130.0

102.0

128.5

73.0

89.5

73.0

91.9

71.8

89.6

73.7

90.5

82.6

100.5

84.8

102.9

720

72.6

97.3

70.7

94.6

76.2

101.2

107.3

134.2

104.2

131.4

76.1

92.9

75.1

93.3

72.9

91.0

76.7

93.7

84.1

105.1

85.4

103.8

68.5

92.7

66.9

90.1

71.8

96.9

103.5

130.2

100.3

126.5

72.1

88.8

72.6

91.0

71.1

89.0

72.7

89.6

84.2

102.9

84.9

103.1

ODW2V

96

76.8

99.7

65.1

88.4

78.5

102.7

89.6

112.7

89.1

112.5

74.8

96.8

76.3

99.9

73.5

97.7

92.2

117.7

77.0

100.1

77.4

100.4

192

77.9

100.8

68.3

91.4

79.7

103.8

90.5

114.2

96.4

120.1

76.6

98.9

79.9

103.3

78.8

103.6

100.5

128.1

78.3

101.8

78.0

101.1

336

78.5

101.5

69.9

93.0

80.3

104.5

94.2

119.3

98.4

122.2

77.6

100.2

81.8

105.3

82.1

107.5

105.5

134.4

79.4

103.3

78.7

102.0

720

79.9

103.6

72.9

96.5

82.0

106.7

97.4

120.4

100.5

125.0

79.6

103.0

86.2

100.2

86.2

112.7

111.0

141.3

86.1

112.3

81.3

105.6

78.3

101.4

69.0

92.3

80.1

104.4

92.9

116.6

96.1

120.0

77.2

99.7

81.1

102.2

80.2

105.4

102.3

130.4

80.2

104.4

78.8

102.2

1

st

Count

0

29

0

0

4

0

0

0

0

0

0

Setups & Results of Imputation Tasks. Our brief results are in Tab. 2 , where LM-WEATHER consistently surpasses all baselines, outperforming FL-GPT4TS by 5.7% . LM-WEATHER remains competitive even when compared with the SOTA, FL-PatchTST, FL-LightTS, and FL-DLinear.

Method

LM-WEATHER-AVE

LM-WEATHER

FL-GPT4TS

FL-Reformer

FL-Pyraformer

FL-DLinear

FL-PatchTST

FL-iTransformer

FL-LightTS

FL-Transformer

FL-Informer

Dataset

Length

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

MAE

RMSE

ODW1T

96

22.4

43.5

21.7

41.8

23.3

45.2

63.7

88.4

62.2

85.9

29.2

50.8

28.9

54.6

22.8

44.5

24.4

43.7

58.3

82.8

70.8

99.6

192

23.4

43.7

22.6

42.0

24.6

45.9

67.2

91.2

65.5

88.5

28.7

50.2

47.5

77.3

23.8

44.1

25.7

45.3

57.3

82.4

66.3

92.1

336

24.1

44.1

23.2

42.4

25.3

46.3

70.4

93.4

68.5

90.6

28.3

49.4

48.6

77.0

27.2

47.7

26.9

46.6

58.4

83.5

36.9

55.3

720

26.0

45.1

24.9

43.3

27.3

47.4

77.9

96.8

75.8

93.9

28.0

49.0

56.6

85.1

36.5

56.2

27.2

47.4

56.6

80.4

71.7

96.7

24.0

44.1

23.1

42.4

25.1

46.2

69.8

92.5

68.0

89.7

28.5

49.9

45.4

73.5

27.6

48.2

26.1

45.7

57.6

82.3

61.4

85.9

ODW1V

96

42.1

62.0

41.1

60.4

42.9

63.8

43.8

64.9

42.3

53.0

43.0

63.0

53.6

77.1

38.7

58.2

41.5

61.5

37.8

56.9

41.1

59.2

192

43.9

64.5

42.8

62.8

45.6

66.9

45.8

67.6

44.7

56.2

49.3

71.2

57.5

81.5

49.3

68.9

41.9

62.0

44.1

57.4

48.8

66.8

336

45.7

66.6

44.6

64.9

47.5

69.2

47.6

69.8

54.6

65.7

53.4

76.6

60.7

85.0

60.0

79.8

47.3

64.6

48.5

68.0

50.2

67.1

720

47.5

68.7

46.3

66.9

49.4

71.4

49.6

72.0

59.2

73.5

56.8

80.7

63.3

87.4

61.6

80.4

52.5

72.9

52.7

70.1

60.3

77.2

44.8

65.5

43.7

63.8

46.4

67.8

46.7

68.6

50.2

62.1

50.6

72.9

58.8

82.7

52.4

71.8

45.8

65.3

45.8

63.1

50.1

67.6

ODW2T

96

38.0

56.6

36.9

54.9

39.1

58.3

50.3

70.3

95.4

120.8

40.8

60.0

38.4

58.6

39.1

58.3

38.8

57.8

65.5

86.6

51.7

72.0

192

38.3

56.6

37.2

54.9

39.8

58.9

52.1

74.2

96.2

122.3

42.9

62.7

66.7

87.8

39.4

58.3

39.5

58.4

71.4

92.8

55.0

75.7

336

43.5

65.5

42.2

63.5

44.8

68.1

56.6

78.9

97.8

125.5

46.0

67.7

68.7

90.1

44.8

67.5

47.8

65.3

66.8

88.8

51.5

72.8

720

47.9

68.8

46.5

66.7

49.8

71.5

64.3

87.7

99.1

129.9

52.8

76.1

70.4

93.5

49.3

71.0

48.0

68.0

67.4

89.2

51.5

73.0

41.9

61.9

38.8

61.7

43.4

64.2

55.8

77.8

97.1

124.6

45.6

66.6

61.1

82.5

43.2

63.8

43.5

62.4

67.8

89.4

52.4

73.4

ODW2V

96

28.1

45.3

27.5

44.0

28.4

45.8

50.3

70.3

53.2

72.4

72.1

92.0

39.8

58.4

72.7

94.7

96.4

123.5

52.7

73.2

54.8

76.9

192

28.6

45.3

28.0

44.0

29.2

46.1

51.0

71.1

46.1

65.2

75.7

95.9

44.9

63.7

79.1

102.0

98.6

125.8

53.9

74.7

56.2

78.8

336

33.7

49.8

32.7

48.4

34.9

51.8

54.2

76.6

74.2

97.3

77.3

97.8

50.9

70.1

82.6

106.1

101.2

128.8

54.4

75.4

56.8

79.7

720

37.1

53.1

36.0

51.5

39.3

56.3

59.4

81.7

82.4

100.9

77.1

97.3

59.2

79.3

83.0

106.0

98.5

124.3

55.4

77.5

56.4

78.6

31.9

48.4

31.1

47.0

33.0

50.0

53.7

74.9

64.0

84.0

75.5

95.8

48.7

67.9

79.4

102.2

98.7

125.6

54.1

75.2

56.0

78.5

1

st

Count

0

30

0

0

0

0

0

0

0

0

0

5.4 Few-Shot Learning Experiments

PLMs have demonstrated remarkable few-shot learning capabilities [26]. In this subsection, we assess whether LM-WEATHER retains this ability in both forecasting and imputation tasks, based on FL for resource-constrained on-device weather modeling environments.

Setups and Results of Forecasting & Imputation. For both forecasting and imputation tasks, we evaluate the few-shot learning capability in scenarios using limited data, specifically, we use training ratios of 5% and 15% (Our full few-shot learning results (training ratio of 5% and 15%) can be found at Appendix E.2 ). The brief 5% few-shot learning results on forecasting and imputation tasks are depicted in Tab. 3 and Tab. 4 , respectively. LM-WEATHER remarkably excels over all baseline methods, and we attribute this to the successful cross-domain knowledge activation in our local dual fine-tuning for the PLM. In addition, our LM-WEATHER's communication mechanism also reduces the impact of data heterogeneity on performance, which is reflected in the fact that LM-WEATHER has an average 14.7% and 20% improvement relative to LM-WEATHER-AVE, in the forecasting and imputation, respectively. In relation to recent SOTA methods such as FL-PatchTST, FL-LightTS, and FL-DLinear, our LM-WEATHER enhancements surpass 78% , 14.3% , and 72.8% for forecasting, and 102.1% , 122.1% , and 96.35% for imputation. This means that heterogeneity poses challenge to baseline and they struggle to understand weather patterns with limited data. Moreover, it implies that LM-WEATHER can effectively achieve cross-domain knowledge transfer to PLMs. This benfits from the personalized adapter we integrated into the PLM, coupled with lightweight operations.

5.5 Zero-Shot Learning (Out of Distribution Modeling) Experiments

Beyond few-shot learning, PLMs hold potential as effective zero-shot reasoners. We evaluate the zero-shot learning capabilities of LM-WEATHER within the framework of cross-domain adaption. Specifically, we examine how well a method performs on a dataset when it is optimized on another dataset, where the model has not encountered any data samples from the original dataset. We use forecasting/imputation protocol and evaluate on various cross-domain scenarios. Note that we choose LM-WEATHERAVE rather than LM-WEATHER for

comparison due to it can obtain an unified model for zero-shot experiments whereas LM-WEATHER is obtain multiple personalized models. The results are in Tab. 5 . LM-WEATHER-AVE consistently outperforms the most competitive baselines by a large margin, over 14.2% and 14.2% w.r.t the second-best in MAE reduction, in forecasting and imputation, respectively. We attribute this to our personalized adapter that we implant in PLMs being better at activating the PLM's knowledge transfer and domain-adaption capabilities in a resource-efficient manner when modeling weather variables.

5.6 Framework Analysis Experiments

We demonstrate the effectiveness of LM-WEATHER through experiments focused on ablation studies, computational/communication comparison, and robustness evaluation. For detailed results and further analysis, please refer to the Appendix D and Appendix E .

Ablation Study. Follow the setting of main experiments, we report our brief ablation results in Tab. 6 , please refer to Appendix E.3 for full results. The results indicate a notable drop in performance when we omit the weather decomposition components (LM-WEATHER-A/B/C/D). Additionally, keeping the decomposition term but removing the associated generator leads to a 14.5% average performance decline. This suggests that our personalized adapter effectively leverages the PLM's modeling of weather data. Conversely, when we alter the personalized approach by changing the shared low-rank matrix to other trainable parameters (LM-WEATHER-F), we observe a significant performance drop and increased communication costs. Furthermore, moving from LoRA to fully fine-tuning the attention parameters results in a slight performance gain but incurs over four times the parameter count and a massive increase in communication overhead, which is inefficient for us. These outcomes highlight the benefits of the personalized adapter.

Parameter Comparison. The results are shown in Tab. 7 . LM-WEATHER ensures top while only communicate about 3.7% of the trainable parameters, compared to the baseline that communicates the full model parameters. When compared with competitive methods, FL-DLinear and FL-LightTS, LMWEATHER's communication overhead is just 35.9% and 22.6% of theirs, respectively, highlighting LM-WEATHER's superior communication efficiency.

Communication Efficiency. To further validate the excellent communication efficiency of LMWEATHER, we introduce quantitative comparisons by including FL methods tailored to improve communication efficiency (FedKD [27], FedPAQ [28], FedBF [29], FedAP [29], PromptFL [30]) as baselines 2 . The results is shown in Table 8 , which demonstrate our LM-WEATHER achieves a significant improvement in communication efficiency while maintaining excellent performance. Additionally, LM-WEATHER significantly outperforms baseline in terms of both communication efficiency and performance across different tasks. Even when compared to lightweight baselines

(i.e., FL-LightTS/DLinear), LM-WEATHER continues to outperform them. This underscores LMWEATHER's superiority in both communication efficiency and performance.

Robustness to Number of Devices. To evaluate LM-WEATHER's robustness against device count variations, we assessed the percentage change in performance relative to the default device number. Our results ( Tab. 9 ) reveals that LM-WEATHER maintains robustness across different device counts due to several factors: (1) Increasing device numbers during training typically yields slight performance improvements within a stable range, applicable in both regular and few-shot scenarios. (2) Additional devices can sometimes impair performance due to imbalances in data distribution, highlighting non-proportional gains. (3) Adding more devices increases communication overhead, which may not justify minor improvements, especially in resource-limited settings. These findings underscore LM-WEATHER's relative resilience to device count variations and its ability to strike an optimal balance between performance enhancement and communication overhead.

6 Conclusion and Future Works

This paper demonstrate that pre-trained language models (PLMs) are strong foundation models for personalized on-device meteorological variable modeling. We propose LM-WEATHER, a generic framework to taming PLMs to acquire highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalize adapter into PLMs and endow it with weather pattern awareness. Experiments on real-world datasets demonstrate that LM-WEATHER outperforms the SOTA results by a large margin across various tasks. In addition, extensive analyses indicate that LM-WEATHER can (1) effectively achieve crossdomain knowledge transfers, (2) render device with highly customized model while keeping high efficiency, and (3) generalize under few-shot and zero-shot scenario. In future work, we plan to extend LM-WEATHER to multimodal weather data (text/image/time-series) and to finer scales.

References

Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343 , 2023.

Ryan Keisler. Forecasting global weather with graph neural networks. arXiv preprint arXiv:2202.07575 , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature , 619(7970):533-538, 2023.

Shengchao Chen, Guodong Long, Jing Jiang, Dikai Liu, and Chengqi Zhang. Foundation models for weather and climate data understanding: A comprehensive survey. arXiv preprint arXiv:2312.03014 , 2023.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 , 2021.

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Promptbased generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948 , 2023.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023.

Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society , 146(730):1999-2049, 2020.

Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems , 12(11):e2020MS002203, 2020.

Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature , 525(7567):47-55, 2015.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2023.

Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Universal time series analysis by pretrained lm and specially designed adaptors. arXiv preprint arXiv:2311.14782 , 2023.

Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469 , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125 , 2022.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

David Campos, Miao Zhang, Bin Yang, Tung Kieu, Chenjuan Guo, and Christian S Jensen. Lightts: Lightweight time series classification with adaptive ensemble distillation. Proceedings of the ACM on Management of Data , 1(2):1-27, 2023.

Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525 , 2023.

Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-efficient federated learning via knowledge distillation. Nature communications , 13(1):2032, 2022.

Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International conference on artificial intelligence and statistics , pages 2021-2031. PMLR, 2020.

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics 2023 , pages 9963-9977. Association for Computational Linguistics (ACL), 2023.

Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing , 2023.

Shengchao Chen, Ting Shu, Huan Zhao, Guo Zhong, and Xunlai Chen. Tempee: Temporal-spatial parallel transformer for radar echo extrapolation beyond auto-regression. IEEE Transactions on Geoscience and Remote Sensing , 2023.

Shengchao Chen, Ting Shu, Huan Zhao, Qilin Wan, Jincan Huang, and Cailing Li. Dynamic multiscale fusion generative adversarial network for radar image extrapolation. IEEE Transactions on Geoscience and Remote Sensing , 60:1-11, 2022.

Shengchao Chen, Guodong Long, Tao Shen, Tianyi Zhou, and Jing Jiang. Spatial-temporal prompt learning for federated weather forecasting. arXiv preprint arXiv:2305.14244 , 2023.

Shengchao Chen, Guodong Long, Tao Shen, and Jing Jiang. Prompt federated learning for weather forecasting: Toward foundation models on meteorological data. arXiv preprint arXiv:2301.09152 , 2023.

Xin Man, Chenghong Zhang, Jin Feng, Changyu Li, and Jie Shao. W-mae: Pre-trained weather model with masked autoencoder for multi-variable weather forecasting. arXiv preprint arXiv:2304.08754 , 2023.

Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212.12794 , 2022.

Sojung An, Junha Lee, Jiyeon Jang, Inchae Na, Wooyeon Park, and Sujeong You. Self-supervised pre-training for precipitation post-processor. arXiv preprint arXiv:2310.20187 , 2023.

Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems , 33:2304-2315, 2020.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning , pages 6357-6368. PMLR, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning , pages 2089-2099. PMLR, 2021.

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565 , 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems , 33:3557-3568, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381 , 2022.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures, 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2021.

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8432-8440, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Jiˇ rí Matoušek. On variants of the johnson-lindenstrauss lemma. Random Structures & Algorithms , 33(2):142-156, 2008.

Jun Luo and Shandong Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning. In IJCAI: proceedings of the conference , volume 2022, page 2166. NIH Public Access, 2022.

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818 , 2019.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11237-11244, 2023.

Yang Cao, Haolong Xiang, Hang Zhang, Ye Zhu, and Kai Ming Ting. Anomaly detection based on isolation mechanisms: A survey. arXiv preprint arXiv:2403.10802 , 2024.

APPENDIX: Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

The appendix includes missing information from our main text, including: Appendix A More Related Work; Appendix B Experimental Details; Appendix C Theorems and Proof; Appendix D Additional Finding & Experiment & Discussion; Appendix E Full Experimental Results and Appendix F Additional Statements.

Appendix A More Related Work

In this section, we will discuss in detail advances relevant to our work, which include weather variable modeling, personalized federated learning, universal time series learning, and large language models (LLMs) in time series.

From Meteorological Variable Modeling to Weather Forecasting. Weather conditions play a crucial role in sectors such as transportation, tourism, and agriculture. Meteorological factors, including temperature, humidity, and precipitation, provide essential support and historical insights that enable researchers to analyze weather trends. For decades, Numerical Weather Prediction (NWP) [14] has been the prevalent method, employing physical models to simulate and forecast atmospheric dynamics. However, the accuracy of NWP can be compromised by the uncertainty of initial conditions in differential equations [31, 32], particularly in complex atmospheric processes, and it requires significant computational resources [5, 33, 34].

The recent exponential growth in weather data has prompted a shift from traditional physics-based methods to data-driven approaches using machine learning (ML) and deep learning (DL), which bypass physical constraints in meteorological variables [5]. DL strategies, with their deeper representational capabilities, generally surpass ML methods. Various deep network architectures have been employed to perform extensive weather modeling using large-scale reanalysis data [1, 4, 35, 36, 37]. Yet, these methods tend to focus on global weather patterns, often overlooking the specifics of regional weather variables, and thus fail to offer detailed regional analyses. Moreover, these models require extensive datasets and substantial computational resources-for example, some need to train on 192 NVIDIA Tesla V100 GPUs for 16 days [4]. Additionally, prevailing models assume centralized data storage, which contrasts with the decentralized data collection from diverse ground weather stations. Our research addresses these challenges by focusing on regional meteorological variables in lowresource settings, aiming to provide reliable analytical support for weather pattern modeling and understanding.

Personalized Federated Learning. Federated learning (FL) [7] is a distributed learning paradigm that facilitates the collaborative training of models without exposing data from each participant. Personalized FL (PFL) aims to train a personalized model for each client. Existing PFLs are based on various techniques. Refs. [38, 39, 40] add a regularization term that benefits decomposing the personalized model optimization from global model learning. Refs. [8, 41] share part of the model and keep personalized layers private to achieve personalization. Ref. [42] enables a more flexible personalization by adaptive weighted aggregation. Ref. [43] study PFL from a Model-Agnostic Meta-Learning where a meta-model is learned to generate the initialized local model for each client. This paper tackles on-device meteorological variable modeling from PFL perspective.

Universal Time Series Learning. On-device meteorological variable modeling addresses time series analysis of complex weather patterns on diverse, low-resource devices. We have expanded this to include taskspecific time series learning. Recent advancements have enhanced Transformer [44] for time series forecasting by integrating signal processing techniques such as patching [6], exponential smoothing [45], decomposition [24], and frequency analysis [46]. Among them, PatchTST [6] improves the accuracy of long-term forecasting compared to other Transformer models. ETSFormer [47] applies principles of power series smoothing within the Transformer framework to boost efficiency. Similarly, FEDformer [46] merges the Transformer with seasonal & trend decomposition, offering improved performance and efficiency. Autoformer [45] leverages sequence periodicity for better dependency discovery and representation, excelling in both efficiency and accuracy.

While these methods excel in efficiency and accuracy, they are typically tailored for narrow-range forecasting on select classical time series datasets. Real-world weather data, however, often displays more complex patterns and interconnected variable relationships. Furthermore, weather modeling extends beyond forecasting, rendering these methods less effective for weather sequences. To improve modeling for intricate weather sequences, models need the flexibility to adjust to complex distributions and various tasks with minimal training. The ideal weather models would capture weather patterns accurately, facilitating knowledge transfer, such as between regions. However, creating versatile weather models remains a challenging endeavor. Recent studies have started to examine the potential of large-scale climate models [1, 4], utilizing simulated datasets advances. Yet, their generalizability is hindered by data differences, complex architectures, and the vast number of model parameters.

LLMsinTimeSeries. Large language models (LLMs) have spurred advances in natural language processing (NLP). Although time series modeling hasn't seen similar leaps, the impressive capabilities of LLMs have led to their use in this field. In general, pre-trained LLMs are often fine-tuned or reprogrammed to model

time series [18, 11, 10, 17]. Among them, PROMPTCAST [16] and HEALTHLEARNER [26] treat time series as "text sequences," inputting them directly into LLMs and using prompts for forecasting. [17] dencodes time series as embeddings for LLM output, showing LLMs' strength in time series analysis. LLM4TS [18] uses a two-stage fine-tuning approach to adapt LLMs to time series data. TEMPO [10] breaks down time series features to leverage LLMs in prediction tasks, while TIME-LLM [11] fine-tunes LLMs with multimodal data, integrating relevant text prompts for efficient analysis. However, these approaches focus on centralized time series modeling and overlook the complexities of real-world distributed settings. Weather data, in particular, has unique challenges like heterogeneity from geographic factors and privacy concerns, making central training methods both risky and difficult.

Appendix B Experimental Details

B.1 Datasets

Despite the proliferation of reanalysis data aimed at building frameworks for global climate analysis, these datasets often struggle to model regional weather trend due to: (1) they depend on numerous simulations of atmospheric equations, introducing biases inconsistent with real observations, and (2) they face challenges in refining their scale to suit specific regional applications. Hence, we collected real observational data from various weather stations across different regions. We then organized this data into two series, each comprising two distinct datasets, to underscore the heterogeneity inherent in real-world settings.

On-device Weather Series 1# (ODW1). The dataset gathered from 15 ground weather stations across China, Japan, and South Korea, encompasses over 20 variables 3 . It has been divided into two subsets: ODW1T has a heterogeneous time span, meaning the data collection start and end times vary by location. and ODW1V extends ODW1T by adding variability in the observed variables; while one variable remains constant at each station, the others vary. The temporal resolution of the dataset is 1h. Details are presented in Tab. 10 and Tab. 11 .

Table 10: Details about ODW1T dataset, where Start and End indicate the respective beginning and ending timestamps of data collected at a specific weather station, Samples denotes the count of weather sequence samples gathered at that station, and Variables refers to the weather variables included in the data from each station (For the full names of these variables, please refer to Tab. 14 ).

On-device Weather Series 2# (ODW2). This dataset consists of data from 36 weather observation stations in the United States, Canada, and Israel, covering 5 different variables with a temporal resolution of 1 hour. Following the dataset setting of ODW1 , the dataset was also subdivided into two different dataset, including ODW2T and ODW2V . Detailed information are presented in Tab. 12 and Tab. 13 .

Remark. Four standard steps were performed during the collection and compilation of these dataset, as shown below:

Collection of Raw Meteorological Data. Raw data collection represents the foundational and initial step in constructing our dataset. We procure open-source raw data from various national meteorological centers and data repositories, including the National Meteorological Science Data Center of China 4 , Korea Meteorological Administration 5 , Global Surface Meteorological Observations

Historical Dataset 6 , Canadian Meteorological Data Center 7 , and World Weather Data Repository

from Kaggle 8 . This process ensures that the collected weather data from these sources are consistent in terms of temporal resolution and variable dimensions. All raw data are open-source and can be freely utilized or modified.

Selection of Critical Meteorological Variables. To support personalized on-device meteorological variable modeling and enhance regional weather forecasting reliability, we selected twenty representative meteorological variables. These variables, including temperature, barometric pressure, relative humidity, and precipitation, were chosen based on their significant impact on weather conditions. Detailed definitions, physical descriptions, and units of these selected variables are provided in Table 14 .

Ensuring Completion of Meteorological Time Series. In this step, we primarily focus on ensuring the completeness of weather time series data collected from ground weather stations. Incomplete weather time series can generate unreliable predictions, potentially leading to significant unforeseen losses. Most ground weather stations are susceptible to unpredictable events such as power outages and equipment damage, which may result in data gaps. To enhance dataset completeness, we meticulously examined the raw data for missing values across various timestamps and employed a linear interpolation strategy to fill these gaps.

Handling of Outliers. Outliers are common in weather time series data. We distinguish between factual outliers, typically caused by extreme weather events (e.g., heavy rainfall, typhoons, thunderstorms), and non-factual outliers, often due to observational device anomalies or sensor malfunctions

at weather stations. We identify significant deviations in a weather variable-for instance, a sudden increase from an average rainfall of 2 mm to 200 mm-as outliers. These are manually corrected; initially, the values are set to zero and then replaced using linear interpolation, reflecting the gradual nature of weather phenomena.

Visualisation. We hope to deepen the reader's understanding of the datasets we have collected and compiled by providing standard visualizations. Considering the overall size of the datasets and the large number of meteorological variables, we have provided visualisations of representative variables here for reference. The visualisation of OWD1 is shown in Fig. 2. Due to the number of devices involved in the OWD2 dataset, we have divided it into two consecutive images for presentation, as shown in Fig 3 and Fig. 4.

B.2 Baselines

We compare with state-of-the-art time series analysis models and put them into Federated Learning environments, including Transformer-based methods like Transformer [44], Informer [3], Reformer [21], Pyraformer [22], iTransformer [23], and PatchTST [6], and recent competitive models including GPT4TS [17], DLinear [24] and LightTS [48], detailed information about baselines is below:

Transformer. [44] This model uses a self-attention mechanism, popular for time series prediction tasks, to efficiently and accurately learn relationships within a sequence and contextual information.

Informer. [3] An optimized Transformer-based model for long-range time series prediction. It uses ProbSparse self-attention for efficiency, processes long inputs effectively, and employs a fast prediction decoder.

Reformer. [21] This model improves Transformer efficiency by using locality-sensitive hashing for attention and reversible residual layers. It offers better memory efficiency and speed for lengthy sequences without sacrificing performance.

Pyraformer. [22] It features hierarchical pyramidal attention modules with binary trees to capture temporal dependencies across different ranges efficiently, both in time and memory complexity.

iTransformer. [23] The iTransformer adds attention and feedforward networks to the inverse dimension. It embeds time points as variable tokens, using attention to capture multivariate correlations and feedforward networks for nonlinear representation of each token.

PatchTST. [6] This method divides the time series into patches at the subseries level for input to the Transformer. Each channel holds a univariate time series, sharing the same embedding and Transformer weights across all series.

Hua-Nan

Temperature

Humidity

Precipitation in 1h / 12h

Wind Direction

E-Min

Huhehaote

Hai-Dian

Seoul

Temperature

Humidity

Precipitation in 1h / 12h

Wind Direction

Xin-Du

Lin-Zhi

Tokyo

Nagoya

Hiroshima

Busan

3 2 3 3 . 2 8 8

0'0

0.04

2003000102

员员

2.5

0.0

0.02

1.00

0.04

0.04

9.0

2.5

0.2

0.5

1.0

2.5

DLinear. [24] DLinear integrates decomposition schemes from Autoformer and FEDformer with linear layers to model time series data tables. It effectively summarizes trend and seasonal components, enhancing performance on datasets rich in trends.

LightTS. [48] A lightweight structure based on a simple MLP. It utilizes two downsampling strategies-spaced and sequential sampling-on the MLP structure, capitalizing on the fact that downsampled time series generally maintain most of their original information.

GPT4TS. [17] This model is designed for time series analysis across various scenarios, achieved by fine-tuning a pre-trained language model, specifically GPT2, for the time series domain. It's important to note that for a fair comparison, our baseline setup differs from the original publication's configuration. Instead of using the first six layers of GPT2 as the backbone, we align with our approach and utilize only the first five layers.

In addition, pre-trained language models (PLMs) are the key component of our LM-Weather , we use different PLMs as the backbone to demonstrate the PLM can as the strong weather foundation model for on-device weather modeling. We use GPT-2 as the default setting, and BERT [49], LLaMA [50] as the alternatives.

BERT. [49] BERT, short for Bidirectional Encoder Representations from Transformers, is a deep learning model that uses the Transformer architecture. It understands the context of words by analyzing

Albuquerque

Humidity

Wind Direction

Precipitation

Wind Speed

Atlanta

Beersheba

Boston

Charlotte

Chicago

Dallas

Detroit

Eilat

Haifa

Houston

Indianapoils

Jacksonvile

Jerusalem

Denver

Temperature

Humidity

Wind Direction

Precipitation

Wind Speed

Temperature

Kansas City

Tel Aviv

District

1035

1030

70

1025

1020

1015

200

5C

1010

100

30

00t

020

1010

980

582

970

082

100

300

6101

50

05

40

V%

SCOT

1030

1025

1020

1015

1010

20

1005

270

100

A

00

010.0

279

[035.0

BO

(032.5

[030.0

300

027.5

025.0

40

022.5

(020.0

100

90

80

032.5

40

0'0Z0

022.5

0

00

8

17.5

15.0

7.5

200

52

10

994

988

200

29

52

20

1005

582

1030

1025

1020

1015

50

100

80

text in both directions. When used as a baseline for evaluation, we only employ the first 5 layers of the pre-trained BERT.

GPT-2. [15] Developed by OpenAI, GPT-2 is a language model that can generate coherent and diverse text based on a given prompt. In our research, we utilize the first 5 layers of the pre-trained GPT-2-base.

LLaMa. [50] LLaMa stands for Large Language Model Meta AI and is a series of cutting-edge language models with sizes ranging from 7B to 65B parameters. They offer top-notch performance with less computational power and resources. In our research, we utilize the first 4 layers of the 3B LLaMa model.

A brief description of these FL methods tailored to improve communication efficiency is as follows.

FedKD : This parameter-efficient PFL method integrates knowledge distillation within a single client and employs a parameter aggregation strategy using Singular Value Decomposition (SVD). For the purposes of this section, which focuses solely on comparing communication efficiency, we incorporate only the SVD-based client-server communication strategies into LM-WEATHER as a baseline.

Los Angeles

Humidity

Wind Direction

Precipitation

Wind Speed

Miami

Minneapolis

Montreal

Nahariyya

Nashville

New York

San Fancisco

Philadelphia

Temperature

Humidity

Wind Direction

Precipitation

Wind Speed

Temperature

Seattle

Phoenix

Pittsburgh

Portland

Saint Louis

San Antonio

San Diego

Toronto

Vancouver

100

882

40

1020

265

100

250

10

100

O

1020

101:

1010

10

0.30

025

50

60

1015

020

40

1010

10

0.30

08

0

020

1015

40

1010

00

1020

0

EVV

1040

80

60

50

100

17.5

15.0

12.5

10.0

5.0

2.5

SEOT

DEO1

300

285

250

70

1020

60

Q1

150

100

50

80

270

950

10

70

250

200

150

40

100

50

269

12

10

00

50

1010

05T

0

80

g020.0

022

00T

1035

2.5

80

1030

200

250

1025

30

1020

285

0.5

15.0

1010

10.0

7.5

5.0

2.5

0.0

FedPer : This PFL approach maintains a personalized layer while sharing the remaining base layers during communication. This enhances communication efficiency by transmitting only a portion of the parameters.

FedBF : This fine-tuning method enhances parameter efficiency by sharing only the biases of the local model during global aggregation, thereby reducing communication overhead. To integrate this method into LM-WEATHER as a baseline, we adjusted all biases in LM-WEATHER to be unfrozen.

FedAP : A parameter-efficient fine-tuning method in FL, which involves sharing only adapters during global aggregation.

PromptFL : This parameter-efficient FL method enables participants to cooperatively train lightweight prompts without sharing the entire model, significantly accelerating both local training and global aggregation. In our experiments, we treat the adapter generated on clients as the prompt to facilitate the incorporation of this baseline.

B.3 Task Setups

We evaluate our proposed LM-WEATHER using four distinct on-device weather modeling datasets, each with tailored settings for various tasks. The specific task settings for these datasets are detailed in Tab. 15 .

Additionally, the specific tasks and scenarios for the on-device weather forecasting/imputation vary by dataset, as outlined in Tab. 16 .

B.4 Implementation

We mainly follow the experimental configurations across all baselines within a unified evaluation pipeline in https://github.com/thuml/Time-Series-Library for fair comparison. Specially, we use GPT-2-base as the default backbone model unless state otherwise. All our experiments are repeat five times and we report the averaged results. Our detailed model configurations are in Appendix B.8. All the algorithm implementations and designs in this study are based on Py torch and the algorithms are run on two RTX3090 GPUs 24GB.

B.5 Technical Details

Reversible Normalization. In time series analysis, statistical properties like mean and variance often shift over time, indicating distributional changes in the data. To address this, we've incorporated Reversible Normalization (RevIn) [51] into our LM-WEATHER. Specifically, we've integrated RevIn into our Task Adapter Generation. This introduces two dynamic factors that adaptively normalize segments of the meteorological variable sequence X , or their decomposed components (Trend X Trend, Seasonal X Seasonal, Residual X Residual ), enhancing the accuracy of meteorological variable modeling. Specifically, for the trend component of X , i.e., , X Trend, its transformed value X ′ Trend can be given by:

where E [ X Trend ] and Var [ X Trend ] are the instance-specific mean and variance, respectively. γ T and β T are the trainable parameters for this component. This transformation is also applied to both the seasonal and residual components.

Pre-trained Language Model (PLM). In LM-WEATHER, we do not change the main architecture of the PLM, but use the parameter-efficient fine-tuning (PEFT) strategy to avoid large-scale parameter variations to ensure high efficiency on resource-constrained weather devices, and in this way, to achieve more reliable cross-domain knowledge transfer. Specifically, we introduce LoRA in the local PLM, which allows only 1.5% of the PLM parameters to be trained while the rest remain frozen, as shown in Fig. Note that LoRA is only applied to the query and value of each Attention in the PLM, and the resulting low-rank matrices are used for global sharing between the client and the server.

Pre-trained Language Model

️

Word Embedding

Latent Representation

N-heads

Wq

Wv

Wk

Query

Value

Key

Attention

Add & Layer Norm

Feed Forward

Add & Layer Norm

LoRA

LoRA

×

N

Task Adapter Generation

PLM

Prediction

LoRA

Low-Rank Adaption (LoRA). To achieve more reliable cross-domain knowledge transfer (i.e., from natural language to complex weather sequences) while guaranteeing higher efficiency, we introduce LoRA [19], a parameter-efficient fine-tuning method for large language models, into PLM. Specifically, LoRA is applied to the Query and Value of each Attention layer by creating low-rank matrices for two pre-trained parameters W q and W k :

where X denote the latent representation from input weather sequences through PLM's word embedding layer, A q ∈ R d × r and B q ∈ R r × d are low-rank matrices created from W q ∈ R d × d , A v ∈ R d × r and B v ∈ R r × d are low-rank matrices created from W v ∈ R d × d , d is the number of dimensions, r is the rank, and r ≪ d . It's important to note that only the low-rank matrices A q , B q , A v , B v are trainable; the others remain fixed during training. Initially, A q and A v are set with random Gaussian values, and B q and B v start as zero at the beginning of training.

Task Adapter Generation. The X Trend , X Seasonal , X Residual obtained from decomposition are used to generate Task Adapter via an unified generator as Fig. 1B that consisting of Token Embedding, Position Embedding, and Temporal Embedding. Specially, we use one-dimensional convolution operation to map each each specific sample X k ∈ R T × 1 while keeping raw shape to generate TOKEN ADAPTER P TO ∈ R T × C , as

Additionally, we use a trainable lookup table to map each point's explicit position in the entire sequence, to generate POSITION ADAPTER P PO ∈ R T × C , as:

where E ( · ) is the trainable lookup table, and INDEX ( · ) is a function that achieve the indices of each point's locations of weather sequence X . Furthermore, we separately encode different time attributes such as minutes, hours, days, weeks, and months, using trainable parameters to dynamically model complex temporal shifts, to generate TEMPORAL ADAPTER P TE, as

where α represents different temporal attributes, E α denotes the trainable lookup table for each temporal attributes. Finally, for each decomposition components, corresponding generated adapters can be obtained by aggregating Token Adapter P TO ∈ R L × C , Position Adapter P PO ∈ R L × C , and Temporal Adapter P TE ∈ R L × C as P d = P d TO + P d PO + P d TE, where d ∈ { Trend , Seasonal , Residual } , this means that we can obtain P Trend , P Seasonal , P Residual .

B.6 Theoretical Insights on Personalized Adapter

The effectiveness and superiority of our proposed Personalized Adapter have been demonstrated in sufficient ablation studies (please refer to Table 6 in the main text). Here, we will discuss theoretical insights that further supports the effectiveness of Personalized Adapter. Personalized Adapter can effectively capture potential pattern in meteorological variable time series, which comprises both the Task Adapter and the FFN-based Parameter Adapter. Specifically, our focus primarily revolves around the Task Adapter active in extracting representations, which including Token/Positional/Temporal Embedding for transforming meteorological variable time series.

Let the weather sequence be X = ( x 1 , x 2 , · · · , x T ) , where x t ∈ R d Is the observed value with d variables at t moment. Let X represent the function space to which x t of a weather sequence belongs, and Z denote the function space to which the implicit representation z t belongs. Token Embedding can be interpreted as a mapping f θ : X → Z . According to the Kolmogorov-Arnold representation theorem, for any continuous function f ∈ C ([0 , 1] d ) , there exist 2 d +1 continuous functions ϕ q ∈ C ([0 , 1]) and ψ q ∈ C ([0 , 1]) such:

which means Token Embedding can construct a high-dimensional nonlinear mapping from multiple onedimensional functions, capturing complex patterns within weather sequences. Positional Embedding introduces a vector p t for each step, enabling the model to differentiate between observations at different time steps. For any two steps t 1 and t 2 , their position vectors p t 1 and p t 2 satisfy:

The growing distance between t 1 and t 2 with an increasing time gap mirrors the relative positioning of time steps, aiding the model in grasping temporal dependencies. Additionally, the sine-cosine function's periodicity resonates with weather data's cyclical behavior, helping the model to learn from these recurrent patterns.

Finally, consider the role of Temporal Embedding from the view of matrix decomposition. Suppose the temporal matrix T has a rank of r , it can be decomposed as z t ⊗ T = ∑ r i =1 ( z t ⊗ u i ) v T i . Temporal Embedding transforms the original sequence by scaling and rotating it to represent different interaction patterns. Using singular value decomposition, the top r singular vectors distill the core structure of the time-based matrix. This allows Temporal Embedding to intuitively learn a compact representation of weather sequences, highlighting the primary interactions between variables. In optimizing Personalized Adapter within LM-WEATHER, the focus lies solely on Personalized Adapter and the attention layer influenced by LoRA during local updates. As Personalized Adapter undergoes solely local updates while sharing low-rank matrices globally, akin to layer-wise optimization in PLM, the efficacy of its optimization process can be theoretically substantiated by the theoretical analysis provided in [52].

B.7 Evaluation Metrics

For evaluation metrics, as [34], we utilize the mean absolute error (MAE) and root mean square error (RMSE) for both forecasting and imputation. The calculation of these metrics are as follows:

where T denotes the number of data points ( i.e. , prediction horizon in our cases), Y i and ˆ Y i are the i -th ground truth and prediction where i ∈ { 1 , ..., T } .

B.8 Model Configurations

The configurations of our LM-WEATHER for different tasks and datasets are summarized in Tab. 17 . We consistently use the AdamW [53] optimizer in all experiments.

Appendix C Theorems and Proofs

̸

Theorem C.1 ( Decomposition Rationality from Time Series). Given a weather time series X = X Trend ,t + X Seasonal ,t + X Residual ,t , t ∈ [ t 1 , t n ] . Let E = { e 1 , e 2 , ..., e n } denotes a set of orthogonal bases. Lets E Seasonal ⊆ E denote the subset of E on which X Seasonal ,t has non-zero eigenvalues and E Trend ⊆ E denote the subset of E on which X Trend ,t has non-zero eigenvalues. If X Trend ,t and X Seasonal ,t are not orthogonal, i.e., ∑ n i =1 X i Trend ,t X i Seasonal ,t = 0 , then E Trend ⋂ E Seasonal = 0 , i.e., E can not disentangle the two signals onto two disjoint set of bases.

̸

̸

Proof. We decompose X Seasonal ,t and X Trend ,t onto E and acquire that X Seasonal ,t = ∑ a i e i and X Trend ,t = ∑ b i e i . Then it is obvious that e i ∈ X Seasonal ⇔ a i = 0 and e i ∈ X Trend ⇔ b i = 0 . Now, let us consider the inner product of X Seasonal ,t and X Trend ,t :

̸

̸

̸

Note that ∑ n i =1 X i Trend ,t X i Seasonal ,t = 0 . Thus, there must be at least one i such that a i = 0 and b i = 0 . Thus. e i ∈ E Seasonal and e i ∈ E Trend, in other words, E Trend ∩ E Seasonal = 0 . The theorem demonstrates that if X Trend ,t and X Seasonal ,t are not orthogonal, orthogonal bases that separate X Trend ,t and X Seasonal ,t into two distinct sets cannot exist. Typically, periodic and non-periodic signals are not orthogonal because the periodic signal has a discrete spectrum, while the non-periodic signal has a continuous one, leading to potential overlaps at non-zero frequencies. Principal Component Analysis (PCA) seeks to find orthogonal bases in data, but it cannot split these two signals into separate bases. Citing Theorem 1 from [17], we understand that self-attentive mechanisms in pre-trained large models function similarly to PCA. Thus, without manual intervention, the self-attentive mechanism is unable to automatically divide a time series into trend and seasonal components.

̸

Theorem C.2 ( Exchange Low-Rank Matrices Ensures Privacy: Parameter Interaction Perspective). Given a on-device weather modeling framework based on federated learning that global optimization object is F ( θ ) = ∑ i =1 n p i f ( { D i } ; θ ) , where f ( x ; θ ) is the loss function of i -th client, { D i } is dataset of i -th client, and p i and θ denote the data distribution weight of client i and the model parameters, respectively. Given that the parameters θ of the PLM M θ broadcasted by the server consist of two parts: a frozen part M θ,f and a trainable part M θ,t , interacting only the low-rank matrix parameters M θ,l ⊂ M θ,t is a subset of trainable part M θ,t during each round ensures privacy.

Proof. We assume that f ( x ; θ ) is a convex function with respect to θ , i.e., for any θ 1 and θ 2 and λ ∈ [0 , 1] , we have

Since only low-rank matrices parameter M θ,l parameterized by θ l is exchanged, we can convert θ to θ ′ = [ θ ′ l , θ o ] , where θ ′ l is the embedding parameter after the server update. Since we only update on θ l , θ o remains unchanged. Thus, data privacy can be ensured, as θ o contains parameters that reveal user-specific information. Furthermore, the low-rank matrices applied to the PLM M θ using LoRA are initialized with a random Gaussian distribution and all-zero values, respectively, before training. This global information sharing approach also helps to enhance privacy.

Theorem C.3 ( Exchange Low-Rank Matrices Ensures Privacy: Model Indistinguishability Perspective). Our LM-Weather can hide key features of local model when sharing low-rank parameters, even external attacker gains access to a shared low-rank update, it is difficult to reconstruct or differentiate between the original models of different participants. Support client i and client j get two different local model due to updated on heterogeneity weather time series, as M i and M j parametered by θ i and θ j , L ( M ) denotes the low-rank matrix generated by model M , where B ∈ R d × r and A ∈ R r × d , and r ≪ d , and define Adv as the adversary. Conditions exist such that for any polynomial-time attacker Adv , | Pr [ Adv ( L ( M i )) = 1] -Pr [ Adv ( L ( M j )) = 1] | ≤ ϵ holds,

where ϵ is a small positive number, which implies that even if the attacker acquires a shared low-rank update, it is difficult to reconstruct or distinguish between the original models of the different participants.

Proof. The goal of LoRA is to find the low-rank approximation: min || θ -θ 0 -BA || F , where θ 0 is the initial weight, || · || F is the Frobenius paradigm. Consider two models M i and M j , whose weights differ by ∆ θ = θ i -θ j . The corresponding LoRA matrix is:

According to matrix approximation theory, for the best approximation with rank r , the upper bound on the error is:

where σ r +1 (∆ θ i ) is the r+1-st singular value of ∆ θ i . By Johnson-Lindenstrauss Lemma [54], for any ϵ > 0 , there exists a mapping f : R d → R k , where k = O ( log ( n ) /ϵ 2 ) , such that for any x, y ∈ R d . In our case, LoRA can be regarded as such a degenerate mapping. Assume || ∆ θ i -∆ θ j || F ≤ δ and according to the trigonometric inequality:

For any polynomial-time attacker Adv ,its ability to distinguish between L i and L j is restricted to the difference in their Frobenius paradigms. We can define a function f such that:

where f is a monotonically increasing function that represents the attacker's capability with respect to the matrix difference. Combining Eq. 16 and Eq. 17:

when δ and ε ′ are small enough, we can make sure that the right-hand side is smaller than the intended ε . This means that an attacker cannot reverse-engineer personalised local parameters and data to ensure privacy through the low-rank matrix of communication across clients.

Appendix D Additional Finding & Experiment & Discussion

In this section, we explore and discuss potential research findings and questions for our LM-WEATHER via conducting additional experiments. These potential research questions are as follows:

RQ1. How does LM-WEATHER compare to Personalized Federated Learning (PFL) baselines in terms of trade-offs in personalization and global model performance?

RQ2. How does LM-WEATHER perform compared to centralized and local-only training modes?

RQ3. How does the pre-trained language model contribute in LM-WEATHER?

RQ4. What is the resource utilization and training & inference cost of LM-WEATHER?

RQ5. Can LM-WEATHER be used for other tasks?

D.1 Trade-offs in Personalization and Global Model Performance (RQ1)

Our LM-WEATHER builds on the assumption that the foundation model already exists, treating pre-trained language models (PLMs) as such and broadcasting it to each client to achieve local updates. Our aim is to employ device information-specific (e.g., geographic/atmospheric patterns) adapter, to promote the local PLM in achieving cross-domain knowledge transfer from language to meteorological sequences. This approach yields highly customized models for individual devices while achieving global knowledge to avoid data silo, thereby supporting diverse analyses of heterogeneous weather data. Alternative PFL methods do not match the efficiency and flexibility of our personalized adapter in this context, making them less suitable. By incorporating PFL baselines (Per-FedAvg [43], APPLE [55], FedPer [56], and FedALA [57]), we provide quantitative results that substantiate our claims, experiment setting is consistent with the manuscript on ODW1T.

PFL Baseline. A brief description of PFL baselines used in this section of the experiment is as follows.

Per-FedAvg : Allowing for personalized model updates for each client by adding client-specific parameters to the global model and optimizing them during FL training.

APPLE : Tackling statistical heterogeneity in FL by automatically capturing information required by clients from global models using adaptive local aggregation methods.

FedPer : Dividing the model into a base layer and a personalization layer, only the base layer is uploaded during aggregation while keeping the personalization layer to combat statistical heterogeneity.

FedALA : Tackling statistical heterogeneity in FL by automatically capturing information required by clients from global models using adaptive local aggregation methods.

Personalized Performance Comparison. The performance quantification of our LM-WEATHER and PFL baselines under personalized performance for different tasks and scenarios is shown in Table 18. Our LM-WEATHER outperform other PFL baselines across different tasks (forecasting/imputation) and scenarios (regular/few-shot learning) by a wide margin. This supports our finding that in the scenario of on-device weather variable modeling, PFL methods is not appropriate.

Global Model Performance Comparison. The comparison on global model performance across client between our LM-WEATHER and PFL baselines are shown in Table 18. Our LM-WEATHER outperforms PFL baselines in terms of global model performance, as demonstrated by the fact that its global model performs more stable across client with heterogeneous data.

Personalization and Global Model Performance Trade-offs. We consider the trade-off between personalization performance and global model performance for our LM-WEATHER and PFL baselines, the results are shown in Table 20. Compared with PFL baselines, our LM-WEATHER maintains the best trade-off between personalization performance and global model performance, i.e. , the personalization performance does not significantly exceed the global model performance while the performance far exceeds PFL methods, which means that the LM-WEATHER can be flexibly applied to different practical scenarios, including personalised analysis of regional weather trends as well as comprehensive analysis of weather trends over large-scale regions. This means that LM-WEATHER can be flexibly applied to different practice scenarios, including the personalised analysis of regional weather trends and the comprehensive analysis of weather trends over large scale areas.

Performance and Adapter Updating Trade-offs. Furthermore, we investigated the effect of varying the number of local update rounds in adapters across clients on the performance of LM-WEATHER regrading personalization. The results are presented in Table 21 . We observed that increasing the local update rounds from the default five to fifteen leads to smoother and enhanced personalization performance across heterogeneous clients. However, this increase in local update rounds also incurs additional computational and communication costs, which, in our assessment, do not justify the modest performance improvements.

D.2 Centralised and Local-only Training (RQ2)

The ordinary centralised training strategy (all data were aggregated into a single server) exhibits learning efficiency that an ordinary distributed learning strategy. The ultimate goal of FL is to achieve performance close to that of centralised training and to ensure privacy across data sources. Table 22 illustrates that our LM-WEATHER achieves comparable effectiveness to Non-FL (centralised) training, with only a 2.04% disparity. Compared to LM-WEATHER-Local, which lacks interaction between devices, LM-WEATHER performs better due to overcoming data silos.

D.3 Contributions of Pre-trained Language Model in LM-WEATHER (RQ3)

Our LM-WEATHER significantly outperforms time series-specific models trained from scratch under centralised setup. Centralised training aims to acquire an excellent pre-trained model, where PLMs possess inherent advantages due to their prior sequence modeling capabilities. Moreover, various parameter-efficient fine-tuning (PEFT) strategies enable PLMs to adapt to new domain knowledge cost-effectively. FL-based aggregation facilitates a stable on-device fine-tuning process, with LM-Weather enabling highly customized on-device fine-tuning of PLMs with greater efficiency. This highlights the substantial contribution of PLMs in this task.

D.4 No Free Lunch in Performance Improvement (RQ4)

The remarkable capabilities of cutting-edge DL models across various domains and tasks, such as LLMs, and VLMs, can be attributed to their extensive parameters and training on large datasets. Currently, a perfect balance among performance, model size, and cost does not exist. Despite numerous studies focusing on reducing training and inference costs while maintaining superior performance, there is no one-size-fits-all solution. This is also true for our LM-WEATHER, which demonstrates exceptional performance across diverse tasks and scales on real-world datasets with significant heterogeneity, significantly outperforming comparable DL methods. In this context, we analyze the costs associated with training and inference for LM-WEATHER and its baselines, exploring and discussing the trade-offs between cost-effectiveness and performance in practical applications.

The quantification and comparison of computational costs against LM-WEATHER and baseline are shown in Table 24 . We discuss this results from two perspectives as follow.

Communication and Performance. LM-WEATHER outperforms baseline in these two key metrics, which is critical for practical meteorological variable modeling and analysis, a bandwidth-sensitive and high accuracy demanding application.

Trade-offs between Resource Consumption and Performance. In terms of training and inference time and memory usage, LM-WEATHER is less efficient than lightweight baselines such as FL-DLinear, LightTS, and iTransformer, due to its use of a Pretrained Language Model (PLM) as a backbone. Although LM-WEATHER demands more resources, the trade-off is justified by its cost-effective performance gains. Its slightly increased memory requirements for training and inference are manageable on most devices. In the context of weather analysis, where precision is critical, prioritizing performance improvements over minimal resource consumption is essential. Additionally, LM-WEATHER capitalizes on the knowledge-rich PLM and requires only minimal, low-cost fine-tuning on devices to achieve superior performance. This strategy not only enhances performance but also reduces the frequency of future model updates, thereby lowering long-term costs compared to developing a baseline model from scratch.

Table 25: Comparison between LM-Weather and baseline in terms of model size on the device and performance of forecasting (multivariate-to-multivariate), and imputation (50% masking rate) on ODW1T (MAE/RMSE report), where Bold and Underline denote the best and the second best.

In addition, we further provide a comparison of model sizes and performance between LM-WEATHER and baseline, as shown in Table 25 . The difference in model size between our LM-WEATHER and baseline can be deemed acceptable for the following reasons.

Trade-offs between Performance and Size. While LM-WEATHER may not be as compact in terms of model size or resource efficiency as lightweight baselines, it offers significant advantages in various analysis tasks. Its high performance is particularly valuable in practical applications. Moreover, with a model size of 304.19 M, LM-WEATHER is still accessible for devices with limited resources. This contrasts sharply with many large foundation models, which typically comprise several hundred million parameters. The trade-off between performance and size is justified, especially considering the critical nature of accurate weather data analysis.

Efficient Parameter Update and Communication. LM-WEATHER implements efficient on-device fine-tuning of the pretrained language model. Unlike baselines that require training from scratch, LM-WEATHER only needs fine-tuning of a relatively small number of parameters (10.38 M) on each device, with minimal device-to-server communication overhead (0.38 M). This approach facilitates highly personalized cross-domain knowledge transfer, significantly reducing the ongoing costs associated with processing the ever-changing streams of weather data. These aspects highlight the pragmatic considerations that have shaped the design of LM-WEATHER. The model's capabilities to deliver exceptional performance, combined with its efficient parameter tuning and communication strategies, offer a cost-effective solution for advanced weather data analysis in resource-constrained environments.

D.5 Additional Tasks for Potential Applications (RQ5)

Given datasets we proposed in this paper focus on forecasting and imputation tasks, we broaden its scope briefly to explore its potential application by integrating anomaly weather detection tasks. This involves relabeling the dataset to identify intervals with anomalous meteorological variables as instances of abnormal weather processes. Specifically, we label original datasets via Isolation Forecast [58], the main process as follows: (1) We set the cut length to 100, using this metric to segment each channel (variable) and construct several random trees that collectively form a forest. (2) The 'isolation' degree of each data point is quantified by the average path length from the root node to the terminal node. (3) Data points with shorter path lengths are more easily isolated and

thus more likely to be outliers. (4) We establish a threshold based on the average path length; data points falling below this threshold are classified as anomalies.

Evaluation Metrics. We used Precision (P), Recall (R), and F1-Score (F1) to simply quantify the performance of LM-WEATHER and baselines on the weather anomaly detection task, these can be formulated as:

where TP (True Positives), FP (False Positives), and FN (False Negatives) represent the number of samples correctly labeled as anomalous, the number of samples incorrectly labeled as anomalous, and the number of samples that were not labeled as anomalous by the model but were actually anomalous, respectively.

Experiments and Results. We set the input time series length to 100, and other settings (e.g., baselines, hyper-parameters, local updating steps and federated communication rounds, etc.) are consistent with those in the main text, and we conduct experiments on OWD1T and OWD2T to briefly show the results. The performance quantification of our proposed LM-WEATHER and baseline on weather anomaly detection tasks is shown in Table 26 (ODW1T results) and Table 27 (ODW2T results).The findings underscore LM-WEATHER's robust applicability and its Moreover, LM-WEATHER's superior performance over baselines in both regular and fewshot tasks reaffirms its effectiveness and overall superiority. Moreover, LM-WEATHER's superior performance over baselines in both normal and few-shot tasks reaffirms its effectiveness and overall superiority.

Appendix E Full Experiment Results

In this section, we provide the full experimental results not included in the main manuscript. This includes the main experiments ( Appendix E.1 ), few-shot learning experiments ( Appendix E.2 ), and ablation studies ( Appendix E.3 ), as well as extra analysis of our framework, covering hyperparameter sensitivity ( Appendix E.4 ) and its performance with different PLMs.

E.1 Full Main Results

In this section, we show detailed and full experimental results including:

Forecasting ( Tab. 28 ) and imputation ( Tab. 29 ) across different scenes and settings on the ODW1T dataset.

Forecasting ( Tab. 30 ) and imputation ( Tab. 31 ) across different scenes and settings on the ODW1V dataset.

Forecasting ( Tab. 32 ) and imputation ( Tab. 33 ) across different scenes and settings on the ODW2T dataset.

Forecasting ( Tab. 34 ) and imputation ( Tab. 35 ) across different scenarios and settings on the ODW2V dataset.

Note that we only show the comparison between the proposed LM-WEATHER and the time series-specific baseline in the full experimental results. Our LM-WEATHER outperforms specialized time-series analysis models on on-device weather datasets across various environments. Unlike these models, our method doesn't require training from scratch but only minor adjustments to a small number of parameters. This validates the effectiveness and superiority of our proposed framework in on-device weather modeling practice.

E.2 Full Few-Shot Learning Experiments

In this section, we show detailed and full few-shot learning experimental results including:

Forecasting ( Table. 36 for 5% training data, Table. 37 for 15% training data) and imputation ( Table. 38 for 5% training data, Table. 39 for 15% training data) across different scenes and settings on the ODW1T dataset.

Forecasting ( Table. 40 for 5% training data, Table. 41 for 15% training data) and imputation ( Table. 42 for 5% training data, Table. 43 for 15% training data) across different scenes and settings on the ODW1V dataset.

Forecasting ( Table. 44 for 5% training data, Table. 45 for 15% training data) and imputation ( Table. 46 for 5% training data, Table. 47 for 15% training data) across different scenes and settings on the ODW2T dataset.

Forecasting ( Table. 48 for 5% training data, Table. 49 for 15% training data) and imputation ( Table. 50 for 5% training data, Table. 51 for 15% training data) across different scenarios and settings on the ODW2V dataset.

Experimental results indicate that our LM-WEATHER significantly outperforms the baseline in resourceconstrained situations, such as few-shot learning environments with limited training data. This suggests that LM-WEATHER effectively leverages PLMs for sequential data modeling and achieves commendable performance without requiring extensive data for training.

E.3 Full Ablation Experiments

In this subsection, we show the results of the complete ablation experiment, both in the forecasting ( Table. 52 ) and in imputation ( Table. 53 ).

E.4 Hyper-parameter Sensitivity

The impacts of rank on performance are detailed in Table. 54 . As the rank goes up, there's a consistent improvement, reaching its best at r = 8 . However, when r = 12 , there's a drop in performance. This happens because a higher rank means the local model has more trainable parameters, which can improve performance empirically. While a higher rank can cause increased communication cost and introduce more uncertainty.

E.5 Pre-trained Language Model Variants

We compare three representative PLM backbones with varying capacities, the result is shown in Table. 55 . Under the proposed LM-WEATHER framework, it's evident that various PLM backbones maintain strong sequence modeling capabilities. Moreover, the lightweight personalized adapter in LM-WEATHER enhance the PLM's

ability to transfer knowledge from natural language sequences to complex weather sequences. This further validates the superiority and versatility of our LM-WEATHER.

Appendix F Additional Statements

F.1 Impact Statements

We highlight that the goal of this study to proposed LM-WEATHER is not to compete but instead to complement current on-device meteorological variable modeling framework. Today's climate foundation models are typically trained from scratch, utilizing exceptionally large datasets (nearly 100TB) and incurring substantial computational costs. We hope that LM-WEATHER offers a cost-effective alternative for modeling meteorological variables on-device, thereby enabling accurate regional weather trend analysis. In addition, the dataset we complied can be the important resource to provide exploring chances for this field, facilitating future research.

This research seeks to make on-device meteorological variable modeling more efficient and adaptable. By using a PLM as a foundation model instead of training large foundation models from scratch, it eliminates the need for large-scale real weather data and extensive computational resources. Additionally, it supports a variety of devices, enabling everything from advanced smartphones to basic IoT sensors to perform meteorological variable modeling. The method is also designed to be stable in environments with limited data and those outside of typical distribution ranges, providing credible analytical support for further weather trend analyses.

F.2 Limitations

Although our LM-WEATHER significantly outperforms models trained from scratch for time series analysis across various tasks and scenarios with minimal parameter tweaks, it still faces two primary limitations:

Limited Dataset Scale : Due to constraints on computational resources and operational costs, we evaluated the performance of LM-WEATHER using the real-world datasets that did not approach the scale of tens of terabytes often required for training large-scale meteorological models. This limitation does not affect LM-WEATHER to be extended as a general framework for regional weather trend analysis. This framework supports the analysis of on-device meteorological variables and can be further developed and adapted for additional applications.

Dependence on PLMs' Quality and Performance : Although LM-WEATHER leverages PLMs to achieve high efficiency and customization on heterogeneous devices, this dependency means that the quality and the performance of LM-WEATHER are intrinsically tied to the underlying PLMs. Should there be inherent limitations or biases within the PLMs, these could translate to the meteorological modeling performance. Conversely, if conditions allow the use of a more powerful LLM, LMWEATHER's performance can be significantly improved. This might give the community more opportunities to explore the future road-map.

F.3 Future Works

In future work, we aim to broaden the use of LM-WEATHER across more on-device variable modeling applications. We also plan to incorporate additional types of data, including satellite and radar imagery, as well as textual weather descriptions, to advance towards a more generalized approach to on-device meteorological variable modeling.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the main abstract and introduction has accurately reflected the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This paper has discussed the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: This paper provide the full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper has fully disclosed all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: This paper has provided access to the code, but due to the size of the dataset, additional releases are required.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper has specified all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiments in this paper were conducted under five different random seeds and averaged to from the final reprot.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This paper has provided sufficient information on the computer resource.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research is conducted in the paper conform, in every respect, with the NeurIPS code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper has discussed both potential positive societal impacts and negative societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: This paper has described the relevant safegurads adopted.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper has noted the original owners of the assets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper introduces new datasets and has been described in detail.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach called LM-WEATHER that utilizes pre-trained language models (PLMs) as foundation models for on-device modeling of heterogeneous meteorological variables. LM-WEATHER enhances PLMs-equipped devices with local weather knowledge through a lightweight personalized adapter. Additionally, it leverages low-rank based transmission to fuse global knowledge among devices, enabling high-efficiency communication.This approach provides an effective solution for modeling real-world, heterogeneous, and continuously updated weather data, without requiring high resource demands. I believe this is a meaningful contribution to the field.\n\nSTRENGTHS:\nThis paper extensively compares various time series models across different meteorological variables and weather station.\n\nThe appendix section of the paper is comprehensive.\n\nWEAKNESSES:\nMy first concern is that since the personalized adapter is already being used to adapt to the modeling of heterogeneous weather data collected from different weather station devices, what is the point of using the averaging operations during inter-device communication? This requires further explanation.\n\nThe definitions and formulations of symbols and equations throughout the sections, from Section 3.1 Local Training to Section 4 Theorems, are disorganized and complex. Additionally, there are numerous instances of incorrect singular and plural word usage. It is recommended to rewrite these sections with increased clarity and accuracy, simplifying the content for better comprehension.\n\nQUESTIONS:\nThis paper uses GPT-2 as the PLM, which is a very basic pre-trained language model. I am curious why more advanced PLMs like GPT-4 [1], Llama [2], and Vicuna [3] were not considered for better sequence modeling performance.\n\nReference: \n[1] Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.\n[2] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.\n[3] Chiang W L, Li Z, Lin Z, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LM-WEATHER, which builds upon previous work to explore further the powerful capabilities of PLMs in modelling meteorological variables. By learning sequence modelling capabilities from natural databases and applying them to on-device heterogeneity meteorological variable modelling, a lightweight adapter was developed to provide weather pattern awareness. Additionally, two real-world meteorological station datasets ODW1 & ODW2 support regional weather forecasting. Abundant experiments were designed to verify the feasibility of LM-WEATHER in modelling meteorological variables.\n\nThe article work is more valuable to further validate the superiority of PLMs in modelling meteorological variables and the provision of weather station data that can be used as a baseline dataset for weather station forecasting.\n\nSTRENGTHS:\n1.This work is more valuable to further validate the superiority of PLMs in modelling meteorological variables.\n2.Two real-world meteorological station datasets ODW1 & ODW2, which were collected and compiled from four real-world versatile datasets for on-device meteorological variable benchmark.\n3. Exploring the spatio-temporal sequence modeling of weather pattern specificity with high distributional similarity to provide a viable solution paradigm for sparse and heterogeneous weather station data.\n4. From the experimental results, this method can further enhance the possibility of maintaining confidence in its future development in the presence of unconstrained data and PLM models.\n\nWEAKNESSES:\n1.\tFigure 1 does not intuitively reflect the contributions of the paper. The Fig.1F mentioned in line 68 seems to be missing; additionally, the explanation for Figure C should be Task Adapter Generation, which is hard to understand. It is recommended to keep the labels a, b, … in the figure consistent with A, B, … in the text.\n2.\tThe paper discusses sequence modeling in the time dimension of stations. Can this method be validated for spatiotemporal modeling?\n3.\tThe experiments in the paper mainly focus on the accuracy of forecasting time. Can further validation be done on the timeliness and stability of the forecasts?\n4.\tTo better present the experimental data, some statistical charts can be drawn to visually demonstrate the advantages of LM-WEATHER in meteorological variable forecasting.\n\nQUESTIONS:\n1.\tIn the Zero-Shot Learning (Out of Distribution Modeling) experiments mentioned in line 282, the domain transfer experiments for regional forecasting on the OWD1 dataset showed that the transfer performance was not as good as the forecasting performance of GPT-4. Can you explain the reason for this?\n2.\tIn Theorem 4.2, how is it demonstrated that Low-Rank Matrices ensure privacy? Are there any related research papers on this topic? Can you provide a detailed explanation?\n3.\tCurrently, there are many sparse station forecasts. Can interpolation be performed based on sparse station data to further extend the model to spatial dimension forecasting and enhance its generalizability?\n4.\tLM-WEATHER seems to outperform current methods. Can it maintain stability in long-term forecasting, and what is the maximum forecasting time it can achieve?\n5.\tIn Table 6, line 305, it appears that having more trainable parameters is better. If the comparison experiment groups had the same number of trainable parameters, what would the results be?\n\nLIMITATIONS:\nRegarding limitation 1, further validating the impact of increased data scale on model performance using large-scale ERA5 data would be beneficial. Utilizing ERA5 as pre-training data to model invariance in meteorological forecasting and learning fundamental meteorological variable patterns, followed by fine-tuning with real-world data, could be a highly valuable endeavor.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to develop weather foundation models by leveraging the on-device data on many distributed sensors. Specifically, a federated learning and low-rank adaption mechanism have been applied to a time-series-based foundation model training on Meteorology data.\n\nSTRENGTHS:\n1) The paper proposes a new way to develop weather foundation models by leveraging many distributed devices with collected Meteorology data. It is worth noting that weather foundation models are an emerging research domain critically important to geoscience and tackling climate change. \n\n2) The proposed solution is technique sounds. The procedure of time series processing, LoRA-based adaptation, and federated learning are seamlessly integrated to implement its goal. \n\n3) The paper provides sufficient details with an appendix and open-sourced codes. It is an essential part to ensure the reproducibility of this work. \n\n4) The paper conducted a comprehensive experiment for comparison and evaluation of the few/zero-shot learning scenarios on the foundation model.\n\nWEAKNESSES:\n1) The paper needs a strong justification to describe the motivation for using a pre-trained language model as a basis for the weather foundation model. Are there any texts in the Meteorology dataset? \n\n2) The paper’s contents organization could be improved. For example, the figure 1 is too complex to understand. In Eq 2, it would be better to write the function as F(theta, D). Because F(theta | D) is easily confusing with the conditional distribution. Moreover, some technique details should not be placed in the appendix. It is a usual assumption that the main paper is self-contained so that the readers can understand the paper without reading the appendix. \n\n3) In the contribution part (lines 73 - 94), the last two items look like the advantages of the proposed method rather than contributions.\n\nQUESTIONS:\n1) Does the proposed method only focus on time series data?\n\n2) Would you please highlight the main research question in plain language? It seems there are many components mixed in the paper. \n\n3) In Section 3, what’s the federated aggregation mechanism? Is there any pseudo code to describe the algorithm?\n\n4) Can you please explain the token embedding for time series data? Is the token to be described as a special pattern of a subsequence of the time series? How to define the vocabulary of the tokens?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LM-WEATHER, a framework leveraging pre-trained language models (PLMs) for on-device meteorological variable modeling. The framework integrates personalized adapters into PLMs, enhancing their ability to handle heterogeneous weather data efficiently. Key contributions include superior performance in forecasting and imputation tasks, minimal communication overhead, and maintaining privacy during client-server interactions. Experiments on real-world datasets demonstrate LM-WEATHER's effectiveness across various scenarios, including few-shot and zero-shot learning.\n\nSTRENGTHS:\nS1. LM-WEATHER introduces a novel approach by integrating personalized adapters into PLMs, allowing for efficient and accurate on-device meteorological modeling.\n\nS2. The framework addresses heterogeneity in weather data and demonstrates robust performance in various tasks and scenarios.\n\nS3.The approach offers a promising solution for personalized weather modeling on resource-constrained devices, potentially benefiting various applications in meteorology and related fields.\n\nWEAKNESSES:\nW1.In the communication section, it is mentioned that only the low-rank matrix parameters are transmitted between the client and server, but specific details about the transmission mechanism and strategy are missing. This includes transmission frequency, bandwidth consumption, and potential issues in practical applications. These missing details may prevent readers from fully understanding the feasibility and effectiveness of the communication strategy.\n\nW2. In the parameter adapter section, it describes integrating the generated adapters with the original input through FFN to produce the final weather sequence predictions. However, the specific architecture of the FFN, how its parameters are configured, and why this particular architecture was chosen are not detailed. The lack of these key details might leave readers questioning the mechanism and effectiveness of the parameter adapter.\n\nW3. In Section 4, while Theorems 4.1 and 4.2 propose the rationality of time series decomposition and privacy assurance through low-rank matrix exchange. There is no introduction and description about the background and application of these Theorems.  Also, the descriptions are too brief, making it difficult for readers to fully understand the theoretical basis.\n\nW4. The paper's experimental results are heavily supplemented by a 37-page appendix, which can hinder readability and accessibility of the core content. Important experimental details and results should be better integrated into the main body of the paper to facilitate easier comprehension and evaluation by the readers.\n\nQUESTIONS:\nQ1.Could you provide more detailed proofs and theoretical validation for the decomposition rationality and privacy assurances mentioned in the theorems?\n\nQ2.Could you clarify the potential confusion in the notations n¬i and nk in Section 2.2, and ensure the formulas are correctly presented?\n\nQ3.Could you provide detailed information on the architecture and parameter configuration of the FFN used in the parameter adapter, and explain why this architecture was chosen?\n\nQ4.Could you provide more detailed information about the datasets, particularly whether data from extreme climatic regions(e.g., tropical rainforests and deserts) are included? If not, how do you plan to expand these datasets to cover more global climatic conditions?\n\nLIMITATIONS:\n1 The authors acknowledge the limitations in the diversity of the datasets and the potential challenges in modeling specific climatic conditions not covered in the current study. Future work should focus on expanding dataset coverage and providing more detailed theoretical and empirical validation. Additionally, the practical applications and limitations of few-shot and zero-shot learning capabilities should be more thoroughly explored and documented.\n\n2 It would be much better integrate the essential experimental details and results into the main body of the paper to improve readability and comprehension?\n\n3 I have questioned whether there will be a powerful foundation model for time series prediction. If the assumption is not true, the application of the proposed solution is limited.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}