Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
Authors: Shengchao Chen
Guodong Long
Jing Jiang
Chengqi Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Meteorological Variable Modeling
Federared Learning
On-device Intelligence
Foundation Model
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: Taming pre-trained language models as foundation models for personalized on-device meteorological variables modeling.

Abstract: This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variable modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
Shengchao Chen', Guodong Long", Jing Jiang", and Chengqi Zhang
Australian Artificial Intelligence Institute, University of Technology Sydney Department of Data Science and AI, The Hong Kong Polytechnic University shengchao. chen. uts@gmail. com, {guodong long, jing jing.jinnn)uuu edu au chengqi zhang@polyu edu.
Abstract
This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present L.M WEATHER, generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low- rank- based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments real-wold dataset show that LM-W WEATHER outperforms the taee-off-hhe-rr results by large margin across various tasks (e.g. forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that A.M-EEATIER can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogene- ity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios. Code available on /tops:://githnn wwtelb.oooo/heeeeeeeeeeeeeeeeeee chtm/Shhnngchocccees3//111 Meethhh
Introduction
Accurately modeling weather variation pattern from large amount of meteorological variables se- quences is increasingly vital for providing efficient weather analysis support for disaster warning. Recently, the promise of learning understand weather pattern from data via deep learning (DL) has led to an ongoing paradigm shift apart from the cotg-sstablished physics-based methods 21.
Mining potential patterns from metcorological sequences that collected from differrnnt regions, includ- ing forecasting and imputation, is one of the most important problems in meteorology. Significant progress has been made by several latest time series approaches B 4 These approaches formulate meteorological variable modeling as an end-to-ent spatio-temporal learning problem. This overlooks the reality that ground weather devices distributed globally gather vast amounts of data quickly. The sheer volume of data, coupled with limited network capacity, necessitates local processing on devices, making centralised learning challenging [5]. On-device intelligence enables edge devices to compute independently, offering a primary solution to the problem.
Federated Learning (FL) (7) is promising on-device intelligence implementation that collaboratively train a uniform model across devices without exchanging raw data. However, the model often under- perform due to data heterogeneity among clients. Personalized FL (PFL) provides new insights for on-device intelligence that allows each device obtains customized models for providing personalized
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Client 1
Low-rank Matrices
Communication
Local Weather Data
Personalized Adapter
Server
Pre-trained Language Model (PLM)
Average
Client N
Full-rank Matrices
Local Weather Data
Communication
Personalized Adapter
Low-rank Matrices
Pre-trained Language Model (PLM)
b
FFN
Prediction
Generator
Generator
Generator
Channel-Indeperedee Patching
Reversible
Parameter Adapter
Secsonal
Residuel
LoRA
Decomposition
(Task Adapter Generation)
PLM
d
Task Adapter ter Generation
Charnal Indipandent
Norm aliza tion
Token Embedding
Reversible
Position Embedding
Temporal Embedding
Multivariate Weather Series
Information Flow
Trainable
Frozen
Plus
Concatenate
Figure 1: Framework Overview. (a) Schematic of LM- WEATHER, each client using personalized
adapter to endow the PLM for local weather awareness, only low-rank matrices transmitted to enhance efficiency during communication; (b) Brief structure of PI.M on each client, detailed architecture be found in Appendix; (c) Task Adapter Generation, the multivariate weather series input splits into two paths. The first path isolates the trend, seasonal, and residual elements, which each go through independent generator produce specific adapters; (d) Architecture of the generator for each decomposed element; (e) Schematic diagram of Channel-Independen Patching (6).
insights 8 Albeit PFL methods showing revolutionized capability in this field, we argue that the current advancements are not necessarily at their best in on-device meteorological variable modeling as three major obstacles remain and hinder further progress:
Challenge of Heterogeneity. Weather data's heterogeneity, unlike that of images or text, arises mainly from the unique characteristics of data collected by weather devices in various regions, such as tropical arid areas. Furthermore, sensor malfunctions extreme events can lead to collection disruptions or inconsistent missing data, which significantly increase the differences in data distribution across devices.
(ii) Underperformed Shallow Network Structures. The vast and varied data gathered by weather devices challenge simpler neural network models to generalize effectively, Further- more, the frequent updates of weather data (hourly or by the minute) require neural models on devices to train and infer more often. This demand is hard to meet with deeper models that, while more performant, are also more resource-intensive.
(iii) Resource-constrainee Weather Devices. From computation perspective, weather devices cannot afford of training complex neural models from scratch, especially for foundation models [ From communication perspective, transmitting complete model during the

---

aggregation phase in FL/PFL significantly increases communication overhead, which is impractical for real-time weather modeling.
Therefore, a compact foundational model (FM) is crucial for personalized on-device weather modeling. Yet, there's a gap in FMs for observational data. Models trained on large-scale simulation data struggle in practical applications because of notable differences in data formats and parameter scales [T 4].
Inspired by the impressive progress of large language models (LLMs) natural language process- ing, recent literature in time series analysis research has also demonstrated that pre-trained LMs provide excellent performance over dedicated models for time series analysis with tuning [00 or reprogramming [T.. This comprehensive and thorough sequence knowledge from language models can be effortlessly transferred across domains without large-scale parameter tuning. Thus, an exciting research question naturally arises:
Question: Since PLMs powerful sequence modelers, leverage PLMs foundation models to achieve personalized on-device meteorological variable modeling?
In this paper, we show that pre-trained language models (PLMs) can as outstanding foundation models that tuned on each device with low cost can achieve personalized on-device weather pattern modeling. We propose LM- WEATHER, a generic approach to taming PLMs to understand heterogeneity on- device weather data. As shown in Fig.la, we conduct local tuning on an uniform PLM (e.g.. GPT2), where lightweight personalized adapters are implanted to endow PLMs with weather pattern awareness by decomposing weather sequence implicit knowledge (c.g., seasonal, trend). During communication between client and server, fewer parameters are shared globally while locally retained adapters are enforced to resist heterogeneity and facilitate privacy-assura fusion of global knowledge.
We highlight our contributions and findings as follows:
We introduce LM- WM-WERTHER, a generic approach that transforms Pre-trained Language Models as the foundation model to customized on-device meteorological variable model- ing via personalized adapter. LM-WE.ATHER yields preferable meteorological variable sequences modeling, while being parameter-, communication-, and data-cfficient.
We collect and compile four real-world versatile datasets for on-device metcorological variable modeling across regions. As opposed simulated datasets such as ERAS [12), our datasets are all real-time observations. These datasets based on real-world practice and challenging, provide a pioneer in the field of on-device meteorological variable modeling.
Experiments show that LM- WEATHER advances the state-of-the thetthe-rt methods by large margin across various setting while keeping 3.7% of parameters communication. LM- WEATHER also demonstrates superior communication efficiency in the context of meteoro- logical variable modeling, beating FL. baselines tailored to reduce communication overhead.
In particular, we find that LM-WEATHER can accurately handle structurally non- deterministic sequences (e.g.. differences in time or variable dimensions across devices) thanks to the learned sequences knowledge from pre-trained LMs. We also find that LM- WEATHER can indeed be spatio-temporal sequences sensitive, thereby better modeling the weather pattern specificity of those high distribution similarity.
We find that LM-WEATHER can work well in data-limited environments across various few-shot settings. We further evaluate zero-shot generalizability of LM- MM-WEATUER in modeling complex weather patterns of unseen data, including different group of datasets and other devices, and observe superb performance.
We highlight that the goal of this study is not to compete but instead to complement current on-device meteorological variables modeling framework. Today's 's climate foundation models are typically trained from scratch, utilizing exceptionally large datasets (nearly 100TB 4 [3] and incurring substantial computational costs Α.. We hope that LM- WEATHER offers a cost-effective alternative for modeling meteorological variables on-device, thereby enabling accurate regional weather trend analysis. addition, the dataset we complied can be the important resource to provide exploring chances for this field, facilitating future research.
3

---

2 Preliminaries
2.1 On-device Meteorological Variable (Sequence) Modeling
The on-device meteorological variable (sequence) modeling challenge involves predicting future sequences from past observations for forecasting predicting missing values for imputation each device. While traditional physics-based approach this as a complex problem of solving multilevel atmospheric equations [14), recent deep learning techniques have shown significant potential in uncovering patterns for better weather prediction [4.
Problem Formulation On-device meteorological variable modeling can be formulated end- to-end ecquence-to-sequeee to-sequnnce learning problem for each device without exchange raw data. Formally, parameterized local model for i-th device M2 is tasked with predicting the weather sequence,
(I) where the X E RLXC and & E LC'''" denote the input and output sequences on i-device, L and L' is the input length and output length, C'and C' is the number of input and output variable. Note that the 1' + when performing imputation. The local learning objective on each device is to find the model parameter / that minimize the distance between X; and X; given sufficient weather sequence data. The overall optimization objective is based on FedAvg.
Ma : X; + +
F(0): arg min Σ n4 ((OD)((D.)..
(2)
where n, and is the number of samples held by the i-th client and all clients" respectively, F() denotes the local objective function, {D} the local data.
2.2 Language Models in Time Series
Language models (LMs) trained on large-scale sequence data have shown extraordinary advances and led to significant paradigm shift in NLP, boosting machines in understanding human languages (BERT/MLM--tylee) and synthesizing human-like text (GPT/CLL-ssyyle (I5)). Analogies between time series and human languages have long been noted [16]. Recent advancements in time series analysis have demonstrated the effectiveness of PLMs in modeling time series [17] [. Although some of those have shown that PLMs can beat time series-specific models updating a minor fraction of parameters (18). As such, it is exciting to expect outting-edge techniques of language modeling can tackle weather variables sequence-related problems rather than considering train climate foundation models 1 from scratch that are heavy and expensive, and are trained from simulated data.
3 Taming PLMs for On-device Metcorological Variable (Sequence) Modeling
Overview We proposed generic framework named L.-WEATHER that encouraging PLMS yield accurate prediction while keeping high efficiency for each device. The architecture is illustrated Fig. D To endow PLMs with weather pattern awareness, we introduce a lightweight personalized adapter into PLMs (c.g., GPT2 (15)) such that the emergent ability of sequence modeling that transferred from text into weather is activated. To achieve cross-domain knowledge transfer with minimal effort while maintaining the sequence modeling capabilities of PLMs as intact as possible, we introduce lightweight operations in it enables both clients and servers achieve good trade-off between performance and efficiency (e.g., computation and communication).
3.1 Local Training on Each Device
Our LM- WEATHER refines PLMs for personalized weather sequence modeling on heterogencous devices using a modular, plug-and-play architecture. Specifically, we introduce personalized adapter consists of (1) Task Adapter from latent weather knowledge and (2) Parameter Adapter that converts representation from the PLM into into weather forecasts. In addition, we employ lightweight operations in local training to boost computational efficiency.
'The words 'cientt and 'devcce' have the same meaning paper.

---

Task Adapter: To provide PLMs with richer effective information to activate their sequence modeling capabiliiiess in the target knowledge domain, similar to text-based prompts in language to L.LMs in NLP, we constructed task adapters by decomposing the input weather sequences into multimodal latent statistical information,
Iren + Sessonal + 2 Residual - Decomp (2*)
where 2 R denote the k-th variable in weather sequence X' E R the trend component Trend and the seasonal component XSessonal captures the underlying long-term weather pattern and encapsulates the repeating short-term weather cycles, respectively. Furthermore, the residual component Residiul represents the remainder of the sequence after the trend and seasonality have been extracted. Note that Xirend, X and KResidual have the same shape as X. This decomposition explicitly enables the identification of unusual observation and shifts in seasonal patterns or trends. The Krrend ASscasonal KResidual are used to generate Task Adapter via an unified generator as Fig. [ & Fig. [id that consisting of Token Embedding, Position Embedding, and Temporal Embedding. Specially, we use one-dimensional convolution operation to map each each specific sample x² while keeping raw shape to generate Token Adapter Pro- Additionally, we use trainable lookup table to map each point's 's explicit position in the entire sequence, to generate Position Adapter Pro- Furthermore, we separately encode different time attributes such minutes, hours, days, weeks, and months, via trainable parameters to dynamically model complex temporal shifts, to generate Temporal Adapter PTE. Finally, for each decomposition components, corresponding generated adapters can be obtained by aggregating Token Adapter Pro E RLXC Position Adapter Pro E TR LxC and Temporal Adapter PTB IR L as Pa- Pio PP + PiE. where d E (Trend, Seasonal, Residual), this means that we can obtain Preedd P Sensonal, PR Residual. Details about the generator in Appendix B.5
Lightweight Operations. To enhance the PLMs' ability represent complex inputs while reducing the computational burden to adapt to low-resource devices, we introduce lightweight operations, which includes channel-independen patching (CIP, Fig. Te) [6] for input and efficient tuning of parameters for PLMs. Among them, CIP splits the multivariate sequence into separate univariate sequences, each processed by single model with length Ly. This approach outperforms the original method of mixing channels by treating the variables as independent. It enables the model to capture channel interactions indirectly through shared weights, leading improved performance without directly modeling the complexity of multiple data channels. The total number of inputs patches is P L2) +2, where S denotes the horizontal sliding stride. Given these patches X'. E RRPL we
use rearrange operation and trainable FFN embed them as Xÿ.. E R Pxdi,, where dm, is dimensions created by the FFN. We also introduce low-rank adaptation (LoRA) 19 inside PL.Ss aiming at language modeling for lightweight fine-tuning of attention layers to achieve pross-modal/-domee knowledge transfer from text sequences to weather sequences with minimal effort.
To adapt PLM outputs for downstream weather sequence modeling, we introduceParammetr Adapter, a simple FFN with a single linear layer positioned after the PLM. This adapter transforms the PLM's output to match the prediction horizon, formalized as follows:
Parameter Adapter.
x = FFN(MM(( (Cncat| (FOoMoCC P Prendd PSessonal, PResidual, 2))):
where the Pirend. P Pseasonal, P PResidual. and & are obtained from CIP based on Pirend- Pseusonal P PResidual-
and X. The key objectives incorporating task- specifi knowledge, and (2) to enhance the PLM's output accuracy while preserving its inherent knowledge through the integration of weather data for cross-domain knowledge transfer.
twofold: (1) to enrich the PLM's cross modsl representations by
3.2 High-efficiency Communication Between Clients and Server
To avoid data silos and counteract the performance disparities caused by data heterogeneity while ensuring efficient communication, we update personalized adapters locally and share low-rank parameters globally in each round. Specifically, the local PLM Ma can be formulated as below:
(5) where Ma. denotes the trainable parameter from the low-rank matrices of query and value in attention modules, Me.r is the frozen parameter (mainly the PLM backbone) and other trainable ones (primarily for the personalized adapter). During client-server communication, only Ma.: is transmitted and averaged using FedAvg (7]. At the start of the next training round, the updated Ma, is broadcast to clients for further updates. Privacy is further protected by sending minimal parameters.
Mg + Me (Commmmicattioon + Me.
5

---

4 Main Theorems
Theorem 4.1 (Decomposition Rationality from Time Series). Given weather series X Vrennd + Seasomalt + Residuull E 1 1n Let E = {er, 02,-., denotes a set of orthogonal bases. Lets Eseasomal S E denote the subset of E on which XSeasomal. has non-zero eigenvalues and Errend S E denote the subset of E on which XTrendt t has non-zero eigenvalues. If XTrend,t and Neasomal. are not orthogonal, i.e., Li=1 Xreend X Seasonal,3 & O, then Etrend Essassnal F O, i.e., E can not disentangle the two signals onto two disjoint set of bases.
Theorem 4.2 (Exchange Low-Rank Matrices Ensures Privacy). Given a on-device weather modeling framework based on federated learning that gloabl optimization object is F())
Enst D=f((D));)) where {x; 8) the loss function i-th client, {D:} is dataset of i-th client, and P; and 0 denote the data distribution weight of client and the model parameters, respectively. Given that the parameters of the PLM Ma broadcasted by the server consist of two parts: a frozen part Ma and trainable part Ma.. interacting only the low-rank matrix parameters Ma.1 C MMa is a subset of trainable part Ma,, during each round ensures privacy.
Experiments
In this section, we first present the real-world datasets that we have collected and compiled for on- device meteorological variable modeling, and second, we evaluate LM-W WEATHER on these datasets, which involves normal scenario, a data- lata-limited few-shot scenario, and zero-shot scenario with no training data (OOD). Please refer to Appendix for more detailed information about proposed datasets and additional results of all evaluations (e.g., full results, additional findings & experiments).
5.1 Datasets
Despite the proliferation of reanalysis data aimed at building frameworks for global climate analysis, these datasets often struggle to model regional weather trend due to: (1) they depend on numerous simulations of atmospheric equations, introducing biases inconsistent with real observations, and (2) they face challenges in refining their scale to suit specific regional applications. Hence, we collected real observational data from various weather stations across different regions. We then organized this data into two series, each comprising two distinct datasets, to underscore the heterogeneity inherent in real-world settings. For detailed information these datasets, please see the Appendix B.1
On-device Weather Series 1# (ODW1). The dataset gathered from 15 ground weather stations across China, Japan, and South Korea, encompasses over 20 variables. It has been divided into two subsets: ODW1T has heterogeneous time span, meaning the data collection start and end times vary by location. and ODWIV extends ODWIT by adding variability in the observed variables; while one variable remains constant at each station, the others vary.
On-device Weather Series 2# (ODW2). This dataset consists of data from 36 weather observation stations in the United States, Canada, and Israel, covering 5 different variables with a temporal resolution of hour. Following the dataset setting of ODW1, the dataset was also subdivided into two different dataset, including ODW2T and ODW2V.
5.2 Setup
Baseline. Since our framework is based on a language model, we compare with DL-based SOTA time series models, including Transformer-basee methods: Transformer (20), Informer 33 Re- former (21). Pyraformer [22), iTrrasformmer
(23), and PatchTST [6], and recent competitive models: GPT4TS [77, DLinear (24) and LightTS (25]. Note that our setting is FL-based, so we place them in FL and rename them FL-(baseline) like FL-Transformer, etc., and all aggregation methods used in above models is FedAvg 77]. In addition, we report a variants LM- WEATHER, LM-WEATHER-A AVE that based on FedAvg without personalization. Detailed information are in Appendix B.2
Basic Setup. We focus on on-device meteorological variable forecasting and imputation tasks. For forecasting, we create scenarios for predicting a single variable (multivariate-uiiariiae, and for predicting all variables multivariate forecasting results due to page constraints. For citivvaraatet--uivaatt forecasts, refer
-uuliaaattemmliiiiaaaee).. The main text only includes multivariate-to-

---

to the Appendix E In imputation, we use sequence lengths of (96, 192, 336, 720} and apply three different masking probabilities f25%, 35%, 50%) to represent missing data. The main manuscript shows imputation results for a 50% masking ratio. For more details on the setup, please refer to Appendix B.3
All our experiments are repeat five times and we report the averaged results.
5.3
Main Results
In this section, we evaluate H-M-WEATIER and baseline methods on four on-device meteorological variable modeling datasets in general experiments to validate its effectiveness.
Setups & Results of Forecasting Tasks. Input length L,, is fixed to 192, and we use four different prediction horizons L, E {96, 192, 336, 720}. Evaluation metrics include mean absolute error (MAE) and root square mean error (RMSE). The brief results is shown in Tab. I where our LM- WEATHER outperforms all baselines in most cases and significantly so to the majority of them. Particularly noteworthy is the comparison with GPT4TS that involves fine-tuning PLMs, where LM- L.ATEET has an average 9.8% % improvement over FL-GPT4TS (MAE reported), and even the variant LM- WEATHER- AVE has an average 4% improvement over FL-GPT4TS. In addition, LM- WEATHER shows significant average performance gains of 11.2% % and 19% w.r.t. MAE relative to other SOTA such as FL.-DLineaa inear and FL-PPtthTTTT
Table 1: Results under on-device meteorological variable forecasting task (multivariatee-o- multivariate). A lower value indicates better performance. Bold: the best, Underline: the second best. Complete results can be found at Appendix E due to page limitation.
Miched
LLM Waucsen J0W|
EMM WasneEs
FI,GFITS
R. Releew
RR.
R.DEAN
PL.PATST
[R. dhaTeedee
R.
[R. Traneeeen
Fl. fnfweee
1462
BMSE
MEAE
R35E
[MASE
RMSSE]
MAAE
RNGE
[MAE
MAAE
MAA
3ASE
IMAE
RAGE
MAA
835E
[HAE
135E
|MAE
RNSE
182
66.3
17.3
46.4
13.6
45.5
11.3
13.
983
700
2003
52.3
11.5
43.3
71.8
51.8
14.3
91.5
13.5
+2.1
14.0
$2.5
14.6
00W11
730
11.8
83.0
48.2
78.3
544
97.3
833
352.9
80.5
05.3
57.3
87.
5333
15.6
68.1
13.3
124
100.7
654
67.6
59.3
$7.4
192
15.0
12.6
38.0
12.3
15.8
48.4
754
56.5
69.8
67.8
15.1
45.5
12.1
35.0
15.0
811
18.3
68.
61.0
813
80.6
QUNIYY
131
11.2
18.3
14.
55.5
$1.7
545
323
50.5
281
52.5
88.3
34.3
10.1
12.1
15.2
580
82.
19.3
30.3
18.0
11.3
Arp.
23.5
66.3
18.7
493
13.6
615
71.5
16.4
46.5
13.5
36.1
532
73.1
53.4
$8.
555
06.5
41.7
181.
10.11
1511
67.9
153
15.
69.6
$5.5
14.1
15.4
2010
10001
541
DBW2T
131
15.3
033
01.5
562
36.4
197.
13.55
105.3
131.4
25.0
655
73.0
43.3
718
15.
63.3
536
105.1
546
1029
Aeg.
00.7
555
10.55
1962
1953
1365
12.6
542
96
76.8
10.3
65.1
$8.4
76.5
100.
59.
1127
8011
113.5
36.
96.5
76.3
66.6
23.5
53.5
63.3
117.7
27.0
1001
174
2014
00933
136
16.3
101.5
6631
11.
91.3
194.
542
1965
19.4
1222
775
1003
12.0
K0.1
1913
1403
134.
194
188.5
19.1
3520
Agg
16.3
1014
92.3
91.1
1944
925
1866
16.1
135.0
222
222
60.3
162
1104
1453
1304
602
1984
193
Setups & Results of Imputation Tasks, Our brief results are in Tab. 2 where A--WERTHER consistently surpasses all baselines, outperforming FL-GPT4TS by 5.7% 3.% L.M-W M--EEATHER remains competitive even when compared with the SOTA, FL-PatchTST, FL-LightTS, and FL-DLinear.
Table 2: Results under on-device meteorological variable imputation task, where random masking ratio is 50%. A lower value indicates better performance. Bold: the best, Underline: the second best. Complete results can be found at Appendix EE due to page limitation.
RL.GiTTT F..
Langl.
MAA
33/55
MAA
S3SE
|MA
BNSEE
MAE
105SE
MAE
MAA
RMSS.
MAA
MAA
33456
344
RASSE
| MAA
RMSE
MAE
RNGB
-0.
23.4
43.3
21.7
41.5
23.3
45.2
633
88.4
63.2
$2.0
21.2
50.5
35.5
34.5
22.8
44.3
344
43.7
28.2
$3.5
11.3
0BWIT
330
24.1
44.1
25.2
42.4
15.3
46.3
T24
63.4
08.5
906
36.5
40.4
48.5
17.0
21.2
43.3
35.9
65.5
18.4
$3.5
16.5
553
Arg
25.2
34.1
25.1
42.4
25.1
68.2
653
923
06.0
861
36.5
4119
43-4
133
21.6
48.2
25.1
45.3
31.
12.3
61.6
10.5
43
$2.5
11.1
68.1
12.
63.
43.
64.9
43.5
808
13.0
$3.0
53.6
31.1
16.1
583
11.5
11.5
135
16.5
41:1
3.0
380
64.9
02.
41.
08.3
34,
15.8
76.
6033
55.0
60.0
413
14.
48:
68.3
18.1
12.0
03.0
32.
11.
46.4
$76
46-1
362
82.
300
723
827
15.1
453
45.3
312
24.
39.
56.
23.1
142
429
62.
663
53.
36.4
25.0
15.1
DBWIT
18.5
19.5
76.
64.7
53.
96-1
97.8
1355
1349
53.8
761
35.4
455
46.3
11.
67.5
16.5
47.1
18.
05.
034
665
562
15.5
51.5
73.
185
36.
153
360
14.8
363
662
510
11.
46.1
65.
35.
85.9
46.
637
16.
100.0
55.5
1256
23.6
163
0BW3V
131
171
233
360
115
333
35.
35.4
8333
42.4
1919
12.5
392
333
83.0
11600
25.5
1243
254
115
1644
13.6
Coas
5.4 Few-Shot Learning Experiments
PLMs have demonstrated remarkable few-shot learning capabilities 266 In this subsection, we assess whether L.M WEATHER retains this ability in both forecasting and imputation tasks, based on FL for resource-constraaied on-device weather modeling environments.
7

---

Setups and Results of Forecasting & Imputation. For both forecasting and imputation tasks, we evaluate the few-shot learning capability in scenarios using limited data, specifically, we use training ratios of 5% and 15% (Our full few-shot learning results (training ratio of 5% and 15%) can be found at Appendix F..2. The brief 5% few-shot learning results on forecasting and imputation tasks are depicted in Tab. 3 and Tab. + respectively. LM- WEATHER remarkably excels over all baseline methods, and we attribute this to the successful cross-domain knowledge activation in our local dual fine-tuning for the PLM. In addition, our LM- WEATHER'S 's communication mechanism also reduces the impact of data heterogeneity on performance, which is reflected in the fact that LM- WEATHER has an average 14.7% % and 20% improvement relative to LM- WEATHER- AVE, in the forecasting and imputation, respectively. In relation to recent SOTA methods such as FL-PatchTST, FL-LightTS, and FL-DLinear, our LM- WEATHER enhancements surpass 78 % 14.3%. and 72.8% % for forecasting, and 102.1 %, 122.1 %, and 96.35 % for imputation. This means that heterogeneity poses challenge to baseline and they struggle understand weather patterns with limited data. Moreover, it implies that LM- AA-HERTHER can effectively achieve cross-domain knowledge transfer to PLMs. This benfits from the personalized adapter integrated into the PLM, coupled with lightweight operations.
Table 3: Few-Shot learning results on forecasting task (5% training data). A lower value indicates better performance. Bold: the best, Underline: the second best," '-" denotes insufficient data. Com- plete results can be found at Appendix E.2
Midedd
E. Miuccas Jn
L35 Wicmen
FL-GFEIIS
FC
PL J [Liaaaddoees
FL-Liga:
Merinn
Lmph
MAA:
05465
5465
15600
1355
MAR
85456
MMA5
$355
MAAS
1905
5445
01.9
00.1
143.7
113.1
185.
142.5
1931
0012
164.8
1849
20.3
178.
3850
102.4
202
201.
2067
21007
1881
3360
16.9
1559
101.1
12.3
90.
108.0
285.3
99.3
1832
95.4
96.7
55.6
154
1627
28.5
175.7
381.7
94.3
195.
967
1810
236.
3761
187
13.0
56.9
1960
50b
1996
20.6
353
96.1
023
108.5
181.
138.2
11.5
1853
127.
3255
3124
12.00
1119
-416
112.
1858
1115
1447
115.5
02.
168.
130.9
1879
1989
1418
413.4
112.5
3402
1214
16-11
116.
1824
1830
107.0
142.4
118
1300
341.5
122.1
185.5
90.4
U111
826
1186
693
135.5
1600
136.
45.6
131.3
1967
1467
5565
194.2
157.
1183
1963
1245
1964
1168
1969
1:00
910
1355
1274
1782
1983
2451
1828
1699
1159
1836
2773
-43
1956
158.
2013
00947
1520/472
1:10
1594
960
1355
123.
1962
1863
3412
1123
3461
1528
16.1
1688
1156
1963
1856
275-3
1423
1956
158.
201.5
135.
1221
1163
1554
1915
1563
1123
1102
1621
3125
1963
1466
1882
01092V
192036778
185
966
1221
1102
138.
131.5
1402
1159
3458
3102
145
1821
212
108
1965
140
1852
Table 4: Few-Shot learning results on imputation task (5% training data), where random masking ratio is 50%. A lower value indicates better performance. Bold: the best, Underline: the second best, *.' denotes insufficient data. Appendix E.2
shows our full results.
HL-Lghts
km
MAR
MCR
MAR
MAR
MAA
J MAR
1958
MAR
MAA
MAR
41.
121.2
510
138.5
62.4
135.
1474
2013
14033
2364
1000
30.3
36.2
3400
1150
13.4
1350
3108
2040
142.5
2647
Wi3
110
714
1962
72.3
1421
346
1927
1855
1824
765
1365
1153
31.6
1663
2764
1345
106.
1327
2118
1018
191.
9677
18L5
Avp.
662
42%
116.
11.
142.
1966
196
16.1
1167
112.7
2063
1661
1769
1210
219/1
121.2
2717
101
1969
95.
1968
182.5
196.
914
155.
1124
1572
1862
1013
1243
1856
1257
1193
1153
2567
1213
2866
1355
2003
1243
1825
1083
HL.
00W22
Ang.
122.
166.1
1954
115.0
1922
1947
1613
124.
1856
12577
118.
119
1967
1213
1966
1358
300
124
166
106
254
867
105
56.1
1958
196
335
96.6
1555
1685
1934
1965
1015
1964
196
155.
06
10.6
244
1990
0UWW2V
-24
357
1121
364
773
1968
1365
15.5
155.
1480
153.
1995
2018
1964
1061
14.1
00.1
1366
1900
5.5 Zero-Shot Learning (Out of Distribution Modeling) Experiments
Beyond few-shot learning, PLMs hold potential effective zero-shot reason- ers. We evaluate the zero-shot learn- ing capabilities of L.M-WEATHER within the framework of cross-domain adaption. Specifically, we examine how well a method performs on a dataset when it is optimized on an- other dataset, where the model has not encountered any data samples from the original dataset. We use fore- /assing/impttaiion protocol and evalu- ate on various cross-domain scenarios. Note that we choose LM- WEATHER- AVE rather than LM- WEATHER for
Table 5: Results on Zero-Shot Learning (ave. MAE on forecast- ing/imputation tasks report). Bold: the best, Underline: the second best, +o: domain transferring between datasets.
Setting IT 0t IV IT es 2V
LM- AVE
EL-GPTTTS
FL-DLinear
FL-PaachTST
54. 24.2/48.9
59.4/48.4
50.2/54.9
67.4/99.5
9/33/33.2
89,9/34.2
99.4/80.4
76.7/56.4
IT o+ 2T
80.4/48.4
87.4/53.5
94.8/67.2
$6.5/71.1
2T =t 2V 2T ts IV
84.9/33.3
88.2/36.4
106.5/99.1
92. 0255/5
$7.7/49.6
58.3/47.5
69.1/75.3
74,2/71.1
2T + IT
59,5/25.5
63. 1/21/
78.3/57.1
61.2/38.6
IV c 2V
90.1/36.9
96.7/27.2 22
114.2/101,2
104,7/59.2 2
IV 45 2T
79.3/46.1
84.5/47.5
96.9/71.2
89.7/77.4
IV 69 II
51.2/25,8
53.8/27.2
67.7/54.0
56.4/32.7
2V 6 IV
56.0/51.8
58.5/54.2
70.4/74.2
72.7/69.9 .9
2V 69 II
59.5/29.6
63.1/30.9
72.9/59.8
60.4/39.9
2V + 2T
72.1/44.3
76.9/41.2
87.4/66.7
80.5/65.7
1dt Count
18
8

---

comparison due to it can obtain an unified model for zero-shot experiments whereas LM- WEATHER obtain multiple personalized models. The results are in Tab. 5 AAM-EATHER-ANE consistently outperforms the most competitive baselines by large margin, over 14.2% % and 14.2% w.r. the second-best in MAE reduction, in forecasting and imputation, respectively. We attribute this to our personalized adapter that we implant in PLMs being better at activating the PLM's s knowledge transfer and domain- -amaainn capabilities in a resource-efficient manner when modeling weather variables.
5.6 Framework Analysis Experiments
We demonstrate the effectivvneess of LM-WEATHER through experiments focused on ablation studies, computational/commniaaatii comparison, and robustness evaluation. For detailed results and further analysis, please refer to the Appendix D and Appendix E
Ablation Study. Follow the setting of main experiments, we report our brief ablation results in Tab. 6 please refer to Appendix E.3 for full results. The results indicate a notable drop in WEATHER-THEBBBEEE
performance when we omit the weather decomposition components Additionally, keeping the decomposition term but removing the associated generator leads to a 14.5% average performance decline. This suggests that our personalized adapter effectively leverages the PLM's modeling of weather data. Conversely, when we alter the personalized approach by changing the shared low-rank matrix to other trainable parameters (.M-WEATHER-F) we observe a significant performance drop and increased communication costs. Furthermore, moving from LoRA to fully fine-tuning the attention parameters results in slight performance gain but incurs over four times the parameter count and a massive increase in communication overhead, which is inefficient for us. These outcomes highlight the benefits of the personalized adapter.
Table 6: Ablation results forecasting (multivariate multivariate) and imputation (50% masking ratio, OWDIT dataset). A lower value indicates better performance. Bold: the best, Underline: the second best, and denote performance degradation and performance improvement, respectively.
Task
AMaltio Penjectise
Ave. Variations
Paramu
Method
Feccestig
Iegaratios
Mode Comppeess
Pascealued Melsol
Impataios.
Trsiss'
1.M MM-WEaTHER
45.77
1400
Original
Orginal
10.39
0.38
LM WesemmnAA
$0,897
041.
11.5%
12.6%
0.35 XI
LM Wismmm- 8
$0.935
45.447.1
no Tixal Campasss
Oigind
12.1%
15.0%
10.37
0.35
INt-WechnsseC
90 8
vo Scarnal Exerponuze
Onggndl
16.75
8.25
0138
512095
vo Proegt Gonenate
Onggnal
16.%
18.4%
10.25
LM WEATHER-F
23432
1524
Ougisal
LoRK, Local Los Ruck Mitris, Gited: dee trinkke
138
11.68
10.35
1300 M
Origial Orignal
LoRA, Aicetios eolokk, Alcetho Paunn Globel buicalle
Allettin Pacan
L. We..mmmm
43,5713
23,336.3
M
Parameter Comparison. The results are shown in Tab. 7 LM- LEATHER ensures top while only communicate about 3.7% of the trainable parame- ters, compared to the baseline that com- municates the full model parameters. When compared with competitive meth- ods, FL-DLinear and FL-Ligh(TS, LM- WEATHER'S 's communication overhead is just 35.9% and 22.6% of theirs, respec- tively, highlighting A.M-WEATHERS 's su- perior communication efficiency.
Table 7: Experiment results on parameter comparison (ave. MAE/RMSE report). Bold: the best.
Task
Params
Method
Forecasting
Imputation
Train.
Comm
L.M. WEATHER
45.4/74.6
23.1/42.2
10.38 M
0.38 M
3.70%
EL.GPTTTS
49.9/82.5
1/46.2
12.42
12.42
FL-Refonmer
78. 78.299
69.8 8/9.5
19.74 M
19.74 M
100%
FL-Pyrafommr
73.0/91.3
68.0/89.7
153.32 M
153.32 M
100%
FL-DLinear
63. 63.3882.8
28.5/49.9
1.06 M
1,06 M
100%
FL. PL-PaIchT
48.6/81.0
45.4/73.5
74.74 M
74.74 M
100%
F. Irrnnffomerr
53.7/63.7
27.6/48.2
26.74
26.74
100%
FL-LighTTS
62.7/93.4
1/45.7
1.68 M
1.68 M
100%
FL Transfomee
$2.8/84.7
57.6/82.3
45.55
45.55 M
100%
Informrrerr
$3.4/85.2
61.4/85.
52.31 M
52.31
100%
Communication Efficiency. To further validate the excellent communication efficiency of LM-
WEATHER, we introduce quantitative comparisons by including FL methods tailored to improve communication efficiency (FedKD 277. FedPAQ (28), FedBF (29] FedAP [29]. PromptFL (30)
as b The results shown Table & which demonstrate our LM- WEATHER achieves significant improvement in communication efficiency while maintaining excellent performance. Additionally, LM- WEATHER significantly outperforms bascline in terms of both communication efficiency and performance across different tasks. Even when compared to lightweight baselines
Duue to scenario and model differences, we modified these baselines for LM- M--TEETHER by applying solely their strategies to improve communication efficiency, as detailed in Appendix B.2

---

(i.e.. FL-LighTTS/DLinear) LM- WEATHER continues to outperform them. This underscores LM- WEATHER'S 's superiority in both communication efficiency and performance.
Table 8: Comparison of LM- WEATHER and baseline that tailored to improve communication effi- ciency in terms of forecasting amaltivariate-mllivvriiee / (50% masking rate) performance as well as communication efficiency, with x denotes the improvement in communication efficiency relative to the standard line
MAE/RMSE report. Bold: the best.
Method
Forceasting
Imputation
Train.
Comm. Params.
Comm
Pyrffooore
73.0/91.3
68.0/89.7
153.32
153.32 M
0.07 x
FL-PatchTST
48.6/8100
45.4/73.5
74.74 M
74.74 M
0.14x
FL.-LighrTS
62.7/93.4
16.1/
1.68 M
1.68 M
6.2x
FL-DLincar
63.3/82.8
28.5/49.9
1.06 M
1,06 M
9.8x
LM- MEN-WEATHE-AVE
47.5/78.7
24.0/44.1
10.38 M
10.38 M
1x
LM- ME-WETTHER (Ours)
45.4/74.6
23.1/42.4
10.38 M
0,38 M
27.3x
LM-WEATHER (w FedKD)
49.6/76.2
27.5/43.6
10.38 M
1.68 M
6.2x
L.M WEATHER (w FedPer)
$2.1/79.0
25.1/44.3
10.38 M
8,46 M
1.2x
L.M-WEATHER (wr FedBF)
46.2/78.1
23.7/44.0
10.49 M
10.49 M
0.9X
LM-WEATHES WEATHER (w FedAP)
47.4/79.2
24.3/44.7
10.38 M
9.6 M
1.1X
LM WEATHER (w PromptFF.)
46.0/78.4
23.8/45.1
10.38 M
8.4
1.2x
Robustness to Number of Devices. To evaluate LM- SMM-EEATER'S 's robustness against device count variations, we assessed the percentage change performance relative to the default device num- ber. Our results (Tab. 9) reveals that L.M A.M-EAATHER maintains robustness across different device counts due to several factors: (1) Increasing device numbers during training typically yields slight performance improvements within stable range, applicable in both regular and few-shot scenarios. (2) Additional devices can sometimes impair performance due to imbalances in data distribution, highlighting non-proportional gains. (3) Adding more devices increases communication overhead, which may not justify minor improvements, especially in snource-limitee settings. These findings underscore LM-W Ai-WEAATHER 's relative resilience device count variations and its ability to strike an optimal balance between performance enhancement and communication overhead.
Table 9: Results of (50% masking rate) at different device participation rates [0.1, 0.3, 0.5, 0.7, 0.9]. 1/4 implies an increase/decrease in performance relative to the original setting (0.1), MAE/RMSE report, where 15% represents the proportion of data on each client involved in training.
under forecasting
and imputation
Normal
Few-Shot (15%)
Dataset Rate/ / Devices
Forecasting
Imputation
Forecasting
Imputation
0.1 (2/round)
44.4/73.6
22.6/42.0
64.7/100.4
40 40,2/68.2
0.3 (S/round)
43.7/72.5 (1 L.55)
24.2/43.7 (L 5.55)
63.4/99.7 (f 1.40)
41.4/68.7 7 () 1.85)
ODWIT
0.5 (8/round)
42.9/72.0 (1 2.85)
21.0/4.1 (f 3,90)
63.7/99.2 (f 1.40)
42.3/68.5 (1 2.8)
0.7 (1 l/tound)
43,9/74.1 (1 0.25)
21.8/41.2 (1 2.80)
64.5/101.0 (? 0.10)
39,5/66.7 ( 2.00)
1.0 (16/round)
44.2/74.0 (0 -)
21.3/41.6 (1 3,10)
63.6/100.2 (1 0.95)
40.4/68.0 (L 0.1)
0.1 (4/round)
66.2/89.1
37.2/54.9
89.7/131.8
77.2/112.6
0.3 (1 (1roud)
68,2/89.7 (L 1.85)
36.5/53.1 (1 2.65)
90.2/132.5 0.55)
15.4 (1 2.25)
ODW2T
0.5 (18/roun)
65.4/89.2 (1 0.55)
36.7/53.4 (t 2.05)
89. 1.111/1144444 (* 0.50)
76.5/1111.2 (1 1.10)
0.7 (25fround)
65,7/88.8 8 (1 0.90)
36.1/53.9 (1 2.45)
88.9/130.9 0.80)
76.9/112.3 0.35)
1.0 (36/round)
65.9/89.0 (1 0.25)
36.9/55.0 (1 0.30)
89.1/130.7 (7 0.75)
76.7/112.1 (1 0.50)
6
Conclusion and Future Works
This paper demonstrate that pre-trained language models (PLMs) are strong foundation models for personalized on-device metcorological variable modeling. We propose LM- WEATHER, a generic framework taming PLMs to acquire highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalize adapter into PLMs and endow with weather pattern awareness. Experiments on real-world datasets demonstrate that LM- WEATHER outperforms the SOTA results by large margin across various tasks. In addition, extensive analyses indicate that LM- WEATHER can (1) effectively achieve cross- domain knowledge transfers, (2) render device with highly customized model while keeping high efficiency, and (3) generalize under few-shot and sero-shot scenario. In future work, we plan to extend LM- WEATHER to multimodal weather data (tex/image/time-srriss and to finer scales.
10

---

References
[1] Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: foundation model for weather and climate. arXiv preprint urXiv:2301.0333, 2023.
[2] Ryan Keisler. Forecasting global weather with graph neural networks. arXiv preprint arXiv:2202.07557 2022.
[3] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 11 1100-1111,, 2021.
[4] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970):533-555 2023.
[5] Shengehao Chen, Guodong Long, Jing Jiang. Dikai Liu, and Chengqi Zhang. Foundation models for weather and climate data understanding: A comprchensive survey, arXiv preprint arXiv:23/2.03014 2023.
[6] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.
[7] Brendan McMahan, Eider Moore. Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficcee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273- 1282. PMLR, 2017.
[8] Xiaoxiao Li. Meirui Jiang, Xiaofei Zhang. Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
[9] Alysa Ziying Tan. Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. JEEE Transactions on Neural Nenworks and Learning Systems, 2022.
[10] Defu Cao, Furong Jia, Serean O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt- based generative pre-trained transformer for time series forecasting, arXiv preprint arXiv:2310.0444 2023.
[11] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang. Yuan-Fang Li, Shirui Pan, et al. Time-Ilm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
[12] Hans Hershach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Sabatz-Sabter, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The eraS global reanalysis. Quarterly Journal of the Royal Meteorological Society, 1967730):19999220
2020.
[13] Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn. Soukayna Mouatadid, and Nils Thuerey. Weatherbench:
benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12020022222,,,,,2,23 2020.
[14] Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 55577577):777555,, 2015.
[15] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[16] Hao Xue and Flora D Salim. Promptcast: A new prompt-hased learning paradigm for time series forecasting. IEFE Transactions on Knowledge and Data Engineering, 2023.
[17] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Universal time series analysis by pretrained Im and specially designed adaptors. arXiv preprint arXiv:2311.14782. 2023.
[18] Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with tre-trained Ilms. arXiv preprint Xii:23008008466 2023.
[19] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 4rXiv:2106.09685, 2021.
[20] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziging Ma, Junchi Yan, and Liang Sun. Transform- in time series: A survey. arXiv preprint arXiv:2202.071225.000 2022.

---

[21] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint urXiv:2001.04451, 2020.
[22] Shizhan Liu. Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference learning representations, 2021.
[23] Yong Liu, Tengge Hu, Haoran Zhang. Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itrans- former: Inverted transformers are effective for time series forecasting. arXiv preprint 11X.v223,000665, 2023.
[24] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[25] David Campos, Miao Zhang. Bin Yang. Tung Kieu, Chenjuan Guo, and Christian Jensen. Lightts: Lightweight time series classification with adaptive ensemble distillation. Proceedings of the ACM on Management of Data, 1(2):127,, 2023.
[26] Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learnrrs. arXiv preprint 2.305 15525, 2023.
[27] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu. Yongfeng Huang, and Xing Xic. Communication federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[28] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-ellicee federated learning method with periodic averaging and quantization. In International conference artificial intelligence and statistics, pages 2021-2031. PMLR, 2020.
[29] Zhuo Zhang. Yuanhang Yang. Yong Dai, Qifan Wang. Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficiee tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational I Linguistics 2023, pages 9963-9977, Association for Computational Linguisties (ACI.), 2023.
[30] Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023.
[31] Shengchao Chen, Ting Shu, Huan Zhao, Guo Zhong, and Xunlai Chen. Tempee: Temporal-spatial parallel transformer for radar echo extrapolation beyond auto-regression. IEEE Transactions on Geoscience and Remote Sensing, 2023.
[32] Shengchao Chen, Ting Shu, Huan Zhao. Qilin Wan. Jinean Huang. and Cailing Li. Dynamic multiscale fusion generative adversarial network for radar image extrapolation. IEEE Transactions on Geoscience and Remote Sensing, 60:1-11, 2022.
[33] Shengchao Chen, Guodong Long, Tao Shen, Tianyi Zhou, and Jing Jiang. Spatial-temporal prompt learning for federated weather forecasting. arXiv preprint arXSv:2305. 14244, 2023.
[34] Shengchao Chen, Guodong Long, Tao Shen, and Jing Jiang. Prompt federated learning weather forecasting: Toward foundation models on meteorological data, arXiv preprint arXiv:2301.0915: 2023.
[35] Xin Man. Chenghong Zhang, Jin Feng, Changyu Li, and Jie Shao. W-mae: Pre-trained weather model with masked autoencoder for multi-variable weather forecasting, arXiv preprint arXiv:2304.0875. 2023.
[36] Remi Lam, Alvaro Sanchez- Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton- Rosen, Weihua Hu, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:22 /2794, 2022.
[37] Sojung An, Junha Lec, Jiycon Jang, Inchae Na, Wooycon Park, and Sujeong You. Self-supervised pre-training for precipitation post-processor. arXiv preprint arXiv:2310.20182 2023.
[38] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárík. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304-2315, 2020.
[39] Tian Li. Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
12

---

[40] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[41] Liam Collins, Hamed Hassami, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference machine learning, pages 2089-2099 PMLR, 2021.
[42] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.00 08265.88...88 2020.
[43] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Nearal Information Processing Systems, 33:3557-3568,, 2020.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
[45] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419 22430,223300200 2021.
[46] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced dccomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages -72266. PMLR, 2022.
[47] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar. and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
[48]
Tianping Zhang. Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-orice oriented mip structures, 2022.
[49] Jacob Devlin. Ming-Was Ming-WWii Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18X0::8100004005 2018.
[50] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[51] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ilo Choi, and Jaegul Choo. Reversible instance normaiizaiion for accurate time- seriss forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[52] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1432-84400, 2022.
[53] Ilya Loshchilov and Frank Hutter. arXiv:771.05101.2000 2017.
Decoupled weight decay regularization. arXiv preprint
[54] Jiří Matoušck. On variants of the 33(2):142-156, 2008
lemma. Random Structures & Algorithms,
[55] Jun Luo and Shandong Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning. In IJCAI: proceedings the conference, volume 2022, page 2166. NIH Public Access, 2022.
[56] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.0m 00818, 2019.
[57] Jianqing Zhang, Yang Hua, Hao Wang. Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedingy of the AAAI Conference on Artificial Intelligence, volume 37, pages 1237-11244 2023.
[58] Yang Cao, Haolong Xiang, Hang Zhang, Ye Zhu, and Kai Ming Ting. Anomaly detection based on isolation mechanisms: survey. arXiv preprint arXiv:2403. 10802, 2024.
13

---

APPENDIX: Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
The appendix includes missing information from our main text, including: Appendix A More Related Work; Appendix [B] Experimental Details; Appendix C
Theorems and Proof; Appendix [D Additional Finding & Experiment & Discussion; Appendix [ Full Experimental Results and Appendix F Additional Statements.
Appendix A More Related Work
this section, we will discuss in detail advances relevant to our work, which include weather variable modeling. personalized federated learning, universal time series learning, and large language models (LLMs) in time series.
From Meteorological Variable Modeling to Weather Forecasting. Weather conditions play a crucial role in sectors such as transportation, tourism, and agriculture. Metcorological factors, including temperature, humidity, and precipitation. provide essential support and historical insights that enable researchers to analyze weather trends. For decades, Numerical Weather Prediction (NWP) [4 has been the prevalent method, employing physical models to simulate and forecast almospheric dynamics. However, the accuracy of NWP can be compromised by the uncertainty of initial conditions in differential equations 31 32 particularly in complex atmospheric processes, and it requires significant computational resources 5 33
34
The recent exponential growth in weather data has prompted shift from traditional physics based methods to data-driven approaches using machine learning (ML) and deep learning (DI.) which bypass physical constraints in meteorological variables [5] DI strategies, with their deeper representational capabilities, generally surpass MI. methods. Various deep network architectures have been employed to perform extensive weather modeling using large-scale reanalysis data [4 35 [6 37). Yet, these methods tend to focus on glohal weather patterns, often overlooking the specifics of regional weather variables, and thus fail to offer detailed regional analyses. Moreover, these models require extensive datasets and substantial computational resources for example, need to train on 192 NVIDIA Tesla V100 GPUs for 16 days [. Additionally, prevailing models assume centralized data storage, which contrasts with the decentralized data collection from diverse ground weather stations. Our rescarch addresses these challenges by focusing on regional metcorological variables in low- resource settings, aiming to provide reliable analytical support for weather pattern modeling and understanding.
Personalized Federated Learning. Federated learning (FL) 7 is a distributed learning paradigm that facilitates the collaborative training of models without exposing data from each participant. Personalized FL (PFL) aims to train personalized model for each client. Existing PFLs are based on various techniques. add a regularization term that benefits decomposing the personalized model optimization from global model learning. Refs. S 41 share part of the model and keep personalized layers private to achieve personalization. Ref. [42] enables more flexible personalization by adaptive weighted aggregation. Ref. (43 study PFL from Model-Agnostic Meta-Learning where meta-model is learned generate the initialized local model for each client. This paper tackles on-device meteorological variable modeling from PFL perspective.
Refs. (38 (3 40)
Universal Time Series Learning. On-device meteorological variable modeling addresses time series analysis of complex weather patterns on diverse, low-resource devices. We have expanded this to include task- specific time series learning. Recent advancements have enhanced Transformer 44 for time series forecasting by integrating signal processing techniques such as patching (6), exponential smoothing [45), decomposition [24]. and frequency analysis [46). Among them, PatchTST 66 improves the accuracy of long-term forecasting compared other Transformer models. ETSFormer 44 applies principles power series smoothing within the Transformer framework to boost efficiency. Similarly, FEDformer (46) merges the Transformer with seasonal & trend decomposition, offering improved performance and efficiency. Autoformer [5] leverages sequence periodicity for better dependency discovery and representation, excelling in both efficiency and accuracy.
While these methods excel in efficiency and accuracy, they are typically tailored for narrow- range forecasting on select classical time series datasets. Real-world weather data, however, often displays more complex patterns and interconnected variable relationships. Furthermore, weather modeling extends beyond forecasting, rendering these methods less effective for weather sequences. To improve modeling for intricate weather sequences, models need the ficxibility to adjust to complex distributions and various tasks with minimal training. The ideal weather models would capture weather patterns accurately, facilitating knowledge transfer, such between regions. However, creating versatile weather models remains a challenging endeavor. Recent studies have started to examine the potential of large-scale climate models [ 4. utilizing simulated datasets advances. Yet, their generalizability is hindered by data differences, complex architectures, and the vast number of model parameters.
LLMs in Time Series. Large language models (LLMs) have spurred advances in natural language processing (NLP). Although time series modeling hasn't seen similar leaps, the impressive capabilities of LLMs have led to their use in this field. In general, pre-trained LLMs are often fine-tuned reprogrammed to model
14

---

Among them, PROMPTCAST [66 and HEALTHLEARNER [26] treat time series "text sequences, inputting them directly into LLMs and using prompts for forecasting. [77 dencodes time series as embeddings for LLM oulput, showing LLMs' strength in time series analysis. LLM4TS [8] uses a two-stage fine-tuning approach to adapt LLMs to time series data, TEMPO [10] breaks down time series features to leverage LLMs in prediction tasks, while TIME-LLM I1 fin-tunes LLMs with multimodal data, integrating relevant text prompts for efficient analysis. However, these approaches focus centralized time series modeling and overlook the complexities of real-world distributed settings. Weather data, in particular, has unique challenges like heterogeneity from geographic factors and privacy concerns, making central training methods both risky and difficult.
time series [88 11 17
Appendix B Experimental Details
B.1 Datasets
Despite the prolifcration of reanalysis data aimed at building frameworks for global climate analysis, these datasets often struggle to model regional weather trend due to: (1) they depend numerous simulations of atmospheric equations, introducing biases inconsistent with real observations, and (2) they face challenges refining their scale to suit specific regional applications. Hence, we collected real observational data from various weather stations across different regions. We then organized this data into two series, each comprising two distinct datasets, underscore the heterogeneity inherent in real-world settings.
On-device Weather Series ## (ODW1). The dataset gathered from 15 ground weather stations across China, Japan, and South Korea, encompasses over 20 variables has heen divided into two subsets: ODWIT has a heterogencous time span, meaning the data collection start and end times vary by location. and ODWIV extends ODWIT by adding variability in the observed variables; while one variable remains constant at each station, the others vary. The temporal resolution of the dataset is 1h. Details presented in Tab, I0 and Tab.
Table 10: Details about ODWIT dataset, where Start and End indicate the respective beginning and ending timestamps of data collected specific weather station, Samples denotes the count of weather sequence samples gathered at that station, and Variables refers to the weather variables included the data from each station (For the full names of these variables, please refer to Tab. [4].
Weather Station
Start (UTC++)
End (UTC++0)
Num. Samples
Variables
Huz-Nin
06/11/2018
16:00
00/19/2021 00:00
230,230
E-Mia
01/23/2019
06/17/2024 04:00
240.220
Hubdhoote
09/12/1028
06/11/20211 07.00
Yin-Chuan
05/06/2020 07:00
Shen-Yang
04/99/2018
06/30/2019
06/29/2017
12/05/2018 11:00
206.340
01/29/2020
06/29/2011
Las-Zhi
10113/2019
03/3112021
mst, mns. di, rh, wap. pl,p?. ph wd, was med, hel, hv2,
Kun- Ming
08/28/2018
06/13/2021
22:00
314,740
Wis-on
05/30/2019 01-00
02/17/20011 21:00
247.160
Tokyu
01/17/2017 23:00
01/27/2019
14:00
248.150
Napoya
05/10/2017 15:00
02/14/2019
17:00
309,550 ,50
Hicehtima
05/27/2019 10:00
1AA11/20211
03.00
251, 520
Secul
04/28/201
04:000
11/20/2088
00:00
274,040
Rusen
02/24/1020
18:00
03/1 773000 17:00
244, 340
On-device Weather Series 2# (ODW2). This dataset consists of data from 36 weather observation stations in the United States, Canada, and Israel, covering 5 different variables with a temporal resolution of hour. Following the datasct setting of ODW1, the dataset was also subdivided into two different dataset, including ODW2T and ODW2V, Detailed information are presented in Tab. 12 and Tab. 13
Remark. Four standard steps were performed during the collection and compilation of these dataset, as shown below:
[1] Collection of Raw Meteorological Data. Raw data collection represents the foundational and initial step in constructing our dataset. We procure open-source raw data from various national metcorological centers and data repositories, including the National Metcorological Science Data Center of China") Korca Metcorological Administrationf Global Surface Meteorological Observations
We treat the value of each metcorological variable at each timestamp as independent sample uniformly. httpa://data. cma. cn/
https: : ://www .mmaa gr.
15

---

Table 11: Details about ODWIV dataset, where Start and End indicate the respective beginning and ending timestamps of data collected specific weather station, Num. Samples # denotes the count of weather sequence samples gathered at that station, and Fixed Variables refers to the shared variables among different weather stations, and Other Variables is the remain weather variables in each weather station (For the full names of these variables, please refer to Tab. 14).
Weather Station
Start (UTC+0)
End (UTC+0)
Num. of Samples
Fixed Variable
Other Variables
Hua-Nan
06/1 1/2018 16:00
03/19/2020
00:00
69.084
ws, p4. pl. p5, pl. mwd. p2, wvp, ws vs, p3, dt, dt, mnt. p3, wd, st, hv mwd, pS, ap, mxt, pl, hyl, wyp, mxt, p5
E-Min
01/23/2019 16:00
06/07/2024 04:00
72,072
Hubehaote
09/02/1028 01:00
06/01/2020 07:00
91,920
pl
Yin-Chuan
10/31/2018 20:00
05/06/2020 07:00
79,566
Shen Yang
04/19/208 01:00
D8/30/2019
06:00
73,194
Hai- Dian
05/29/2017 16:00
12/05/2018
11:00
79,902
Xin-Du Du
01/29/2020 08:00
06/29/2021
011:00
74,412
Lin -hi
10/13/2019 08:00
03/30/2021
09:00
76,914
Temperature
ws, vv, pl, hvl. hv2, mws, mnt, p2, mwd st, ws, p3. p5. th p5, st, hv ws, mwd, mws, p4, mwd, wd, hvl, dt lvv,, mwd, vv, p4 p4. p3. wep, pl. vv
Kun-Ming
08/28/2018 07:00
06/13/2020
22:00
94,422
Wu-Han
05/20/2019 01:00
02/17/2021
21:00
92,148
Tokyo
07/17/2017 23:00
01/27/2019 14:00
80,454
Nagnya
05/10/2017 15:00
02/14/2019
17:00
92,904
Hiroshima
05/27/2019 10:00
11/01/2020
03:00
75.426
Scoul
82,212
Busan
02/24/1029 18:00
07/17/2020 17:00
73,302
Table 12: Details about ODW2T dataset, where Start and End indicate the respective beginning and ending timestamps of data collected at specific weather station, Num. of Samples # denotes the count of weather sequence samples gathered at that station, and Variables refers the weather variables included in the data from each station (For meaning of variables, please refer to Tab. 14).
Weather Station
Start (UTC++)
End (UTC+0)
Num. Samples #
Variables
Albuquerque
02/26/2016 01:00
11/16/2016
19:00
31,780
Atlanta
03/24/2013 21:00
12/05/2013
18:00
30.715
Beersheba
05/06/2014 06:00
03/26/2015
18:00
35,350
Boston
04/06/2015 10:00
03/23/2016
00:00
35,120
Charlotte
08/10/2013 09:00
04/24/2014
14:00
30.875
Chicago
06/09/2015 17:00
03/22/2016
10:00
34,415
Dullas
04/11/2013 00:00
08/26/2014
11:00
35,463
Denver
06/23/201, 22:00
05/18/2016
02:00
39,510
Detroit
10/30/2012 06:00
08/22/2013
22:00
35,610
Eilat
11/24/2012 19:00
08/02/2013 05:00
30,060
Haifa
05/18/2013 21:00
04/16/201 14:00
39,915
Houston
12/19/2013 02:00
11/13/2014 02:00
39,490
Indianapoils
01/10/2016 23:00
10/07/2016 04:00
32,435
Jacksonvile
11/11/2012 16:00
09/28/2013 01:00
38,455
Jerusalem
04/22/2015 03:00
02/28/2016 01:00
37,440
Kansas City
06/03/2015 12:00
03/08/2016
21:00
33,535
Las Veges
11/25/2014 09:00
10/19/2015 15:00
39,400
Los Angeles
07/04/2013 08:00
04/28/2014
18:00
35,820
Miami
02/10/2014 10:00
12/05/2014
10:00
35,770
h. 1, wd.
Minncapolis
10/17/2013 03:00
08/31/2014
15:00
38.230
Montreal
08/14/2015 07:00
04/26/2016
06:00
30,725
Nahariyya
12/03/2013 02:00
09/18/2014
01:00
34,685
Nashville
02/08/2017 03:00
11/13/2017
15:00
33,430
New York
09/27/2013 20:00
06/14/2014
15:00
31,185
Philadelphia
12/08/2012 05:00
09/13/2013
20:00
33,565
Phoenix
07/25/2013 15:00
06/05/2014
09:00
37,780
Pittsburgh
10/16/2015 06:00
07/19/2016
16:00
33,300
Portland
10/12/2013 12:00
06/27/2014
18:00
31,000
Saint Louis
09/16/2014 13:00
07/21/2015
06:00
36,935
Antonio
03/15/2014 01:00
12/03/2014
20:00
31.665
San Diego
07/02/2015 11:00
03/18/2016
00:00
31.155
San Francisco
05/10/2013 06:00
01/21/2014
12:00
30,760
Seattle
08/06/2014 22:00
06/28/2015
05:00
39,045
Tel Aviy District
09/23/2013 06:00
06/27/2014
18:00
33,310
Toronto
12/06/2016 16:00
10/19/2017
15:00
38,045
Vancouver
08/26/2015 10:00
06/02/2016 20:00
33,780
Historical Dataset
Canadian Metcorological Data Center
World Weather Data Repository
https: :/k data. htt.://ww.....eeeaaaaceeeeeaaaaa
16

---

Table 13: Details about ODW2V dataset, where Start and End indicate the respective beginning and ending timestamps of data collected specific weather station, Num. of Samples # denotes the count of weather sequence samples gathered at that station, and Fixed Variables refers to the shared variables among different weather stations, and Other Variables is the remain weather variables in each weather station (For the full names of these variables, please refer to Tab. 14).
Weather Station
Start (UTC+0)
End (UTC+0)
Num. of Samples
Fixed Variable
Other Variables
Albuquerque
02/26/2016
01:00
11/16/2016
19:00
19,068
ws, wd wd.p
Atlanta
03/24/2013 21:00
12/05/2013 18:00
18,429
Beersheha
05/06/2014 06:00
03/26/2015
18:00
21,210
ws
Boston
04/06/2015 10:00
03/23/2016
00:00
21,072
ws, wd
Charlotte
08/10/2013 09:00
04/24/2014
14:00
18,525
Lp
Chicago
06/09/2015 17:00
03/22/2016
10:00
20,649
t.
Dallas
04/11/2013 00:00
08/26/2014 11:00
21.279
p. wd
Denver
06/23/2015 22:00
05/18/2016 02:00
23,706
Detroit
10/30/2122 06:00
08/22/2013
22:00
21.366
p.t
Eilat
11/24/2012 19:00
08/02/2013 05:00
18,036
ws.
Haifa
05/18/2013 21:00
04/16/2014 14:00
23,949
ws,
Houston
12/19/2013 02:00
11/13/2014 02:00
23,694
p.
Indianapoils
01/10/2016 23:00
10/07/2016 04:00
19,461
Jacksonvile
11/11/2012 16:00
09/28/2013 3 01:00
23,073
ws,
Jerusalem
04/22/2015 03:00
02/28/2016 01:00
22,464
ws. P
Kansas City
06/03/2015 12:00
03/08/2016 21:00
20,121
wd. ws
Las Veges
11/25/2014 09:00
10/19/2015 15:00
23,640
p.
Los Angeles
07/04/2013 08:00
04/28/2014 18:00
21,492
Miami
02/10/2014 10:00
12/05/2014 10:00
21.462
Humidity
wd,
Minncapolis
10/17/2013 03:00
08/31/2014 15:00
22,938
t.
Montreal
08/14/2015 07:00
04/26/2016 06:00
18,435
wd, p
Nahariyya
12/03/2013 02:00
09/18/2014
01:00
20,811
P. ws
Nashville
02/08/2017 03:00
11/13/2017 15:00
20,058
wd,
New York
09/27/2013 20:00
06/14/2014
15:00
18.711
P
Philadelphia
12/08/2012 05:00
09/13/2013 20:00
20,139
ws, wd,
Phoemix
07/25/201 15:00
06/05/2014
09:00
22,668
ws, wd
Pittsburgh
10/16/2015 06:00
07/19/2016
16:00
19,980
wd, ws
Portland
10/12/2013 12:00
06/27/2014 18:00
18,600
ws. P
Saint Louis
09/16/2014 13:00
07/21/2015
06:00
22.161
t. wd
San Antonio
03/15/2014 01:00
12/03/2014
20:00
18,999
t. wd
San Diego
07/02/2015 11:00
03/18/2016 00:00
18,693
f, P
San Francisco
05/10/201 06:00
01/21/2014 12:00
18,456
p.
Scattle
08/06/2014 22:00
06/28/2015
05:00
23,427
wd
Tel Aviv District
09/23/2013 06:00
06/27/2014
18:00
19,986
t.
Toronto
12/06/2016 16:00
10/19/2017
7 15:00
22,827
wd.
Vancouver
08/26/2015 10:00
06/02/2016 20:00
20,268
wd, ws
from Kaggle 8 This process ensures that the collected weather data from these sources are consistent in terms of temporal resolution and variable dimensions. All raw data are open-source and can be freely utilized or modified.
[2] Selection of Critical Meteorological Variables. To support personalized on-device meteorological variable modeling and enhance regional weather forecasting reliability, we selected twenty representa- tive meteorological variables. These variahles, including temperature, barometric pressure, relative humidity, and precipitation, were chosen hased on their significant impact weather conditions, Detailed definitions, physical descriptions, and units of these selected variables are provided in Table 14
[3] Ensuring Completion of Meteorological Time Series. In this step, we primarily focus on ensuring the completeness of weather time series data collected from ground weather stations. Incomplete weather time series can generate unreliable predictions, potentially leading to significant unforeseen losses. Most ground weather stations susceptible to unpredictable events such power outages and equipment damage, which may result data gaps. To enhance dataset completeness, we meticulously examined the raw data for missing values across various timestamps and employed lincar interpolation strategy to fill these gaps.
[4] Handling of Outliers. Outliers are common in weather time series data. We distinguish between factual outliers, typically caused by extreme weather events (e.g.. heavy rainfall, typhoons. thunder- storms), and non-factual outliers, often due observational device anomalies or sensor malfunctions
httpp::://eeaahrr.gcco
attps://wwe www..ka...l.....aaeeee.....o
com cttps//...ccmmtatattttt datasets
17

---

Table 14: Abbreviations, full names and corresponding units of the different variables in our proposed datasets.
Abbreviation
Full name
Unit
ap mxt mnt
Air Pressure
hpa
Air Temperature
C
Maximum Temperature
C
Minimum Temperature
C
dt
Dewpoint Temperature
C
rh
Relative Humidity
wvp
Water Vapor Pressure Precipitation 1h
hpa
pl
mm
p2
Precipitation in 3h Precipitation 6h
p3
mm
p4
Precipitation in 12h
p5
Precipitation in 24h Wind Direction
mm
wd
"C
ws
Wind Speed
ms 1
mwd
Maximum Wind Direction
st
Land Surface Temperature
C
hvl
Horizontal Visibility min
hv2
Horizontal Visibility in 10 min
VV
Vertical Visibility
m
at weather stations. identify significant deviations weather variable instance, sudden increase from an average rainfall of 2 mm to 200 mm as outliers. These are manually corrected; initially, the values are set to zero and then replaced using linear interpolation, reflecting the gradual nature of weather phenomena.
Visualisation. hope deepen the reader's understanding of the datasets have collected and compiled by providing standard visualizations. Considering the overall size of the datasets and the large number of meteorological variables, we have provided visualisations of representative variables here for reference. The visualisation of OWDI is shown in Fig. [ Due to the number of devices involved in the OWD2 dataset, we have divided into consecutive images for presentation, shown in Fig and Fig.
B.2 Baselines
We compare with state-of-the-ar time series analysis models and put them into Federated Learning environments. including Transformer-basee methods like Transformer 444, Informer [3, Reformer (2T). Pyraformer 22
iTransformer [23], and PatchTST (6. and recent competitive models including GPT4TS 17 DLinear [E] and
LightTS 48 detailed information about baselines is below:
This model uses a self- attention mechanism, popular for time series prediction tasks, to efficiently and accurately learn relationships within a sequence and contextual information.
Transformer. 44
Informer. 3 An optimized Transformer based model for long-range time series prediction. It uses ProbSparse self-attention for efficiency, processes long inputs effectively, and employs a fast prediction decoder.
Reformer. 21 This model improves Transformer efficiency by using secasivy-sensitive hashing for attention and reversible residual layers. It offers better memory efficiency and speed for lengthy sequences without sacrificing performance.
Pyraformer. 22 It features hierarchical pyramidal attention modules with binary trees capture temporal dependencies across different ranges efficiently, both in time and memory complexity.
iTransformer. 23 The iTransformer adds attention and feodforward networks the inverse dimen- sion. It embeds time points as variable tokens, using attention to capture multivariate correlations and feedforward networks for nonlinear representation of each token.
This method divides the time series into patches at the subseries level for input to the Transformer. Each channel holds univariate time series, sharing the same embedding and Transformer weights across all series.
PatchTST. 6
18

---

Tangerature
Precipitation 1h Himidtty
Temparahure
Precipitation 12h
Humidity
Direction
MuatMMon
X0n-Du
Lin-ZMi
Yin-Chwan
Shen-Yang
Tokyo
Hal-Dian
Magoya
Seoul
Hiroshime
Busan
Figure Visualisation of partial variables ODWI dataset, where we have selected the first 1,000 time points for presentation. The data distribution from different ground weather stations exhibit significant heterogeneity, and even though the trends of some variables may be similar, there serious differences in magnitudes. The selected variables are, from left to right, temperature, precipitation in 1hhour/12-hous humidity, and wind direction.
DLinear. 24
DLinear integrates decomposition schemes from Autoformer and FEDformer with linear layers to model time series data tables. It effectively summarizes trend and seasonal components, enhancing performance datasets rich in trends.
LightTS. [88
lightweight structure based gies-spacces and sequential sampling-on the MLP structure, capitalizing on the fact that downsam- pled time series generally maintain most of their original information.
simple MLP. utilizes two downsampling strate-
This model is designed for time series analysis across various scenarios, achieved by fine-tuning pre-trained language model, specifically GPT2, for time series domain. important to note that for fair comparison, our baseline setup differs from the original publication's configuration. Instead of using the first six layers of GPT2 the backbone, we align with our approach and utilize only the first five layers.
GPT4TS. [17]
In addition, pre-trained language models (PLMs) are the key component of our LM-Weather, we use different PLMMs as the backbone to demonstrate the PLM can as the strong weather foundation model for on-device weather modeling. We GPT-2 the default setting, and BERT 49 LLaMA 50)
the alternatives.
BERT. 49 BERT, short for Bidirectional Encoder Representations from Transformers, is deep learning model that uses the Transformer architecture. It understands the context of words by analyzing
19

---

Hurldity
Predipitation Temperahure Direction Wind Speed
Humidlty
Precipitation Temparahure Wind Gpeed
Albuquarque
Petrolt
HUM
Attenta
Ellat
Bearsheba
Bos
Charlatte
Indianapolls
Chicago
Jhe dictsomilll
Dallas
Jerusaler
M
AUH!
Danver
Kansas city
Los Veges
Tel Awiiv District
Figure 3: Visualisation of partial variables in ODW2 dataset, where we have selected the first 1,000 time points for presentation. The data distribution from different ground weather stations exhibit significant heterogeneity, and even though the trends of some variables may be similar, there are serious differences in magnitudes. The selected variables are, from left right, humidity, precipitation, temperature, wind direction, and wind speed.
text in both directions, When used as baseline for evaluation, we only employ the first 5 layers of the pre- traitra BERT.
GPT-2. 15 Developed by OpenAI, GPT-2 is a language model that can generate coherent and diverse
text based on given prompt. In our research, we utilize the first 5 layers of the pre-trained GPT-2-base.
LLaMa. (50) LLaMa stands for Large Language Model Meta Al and is a series of cutting-edge language models with sizes ranging from 7B to 65B parameters. They offer top-notch performance with less computational power and resources. our rescarch, we utilize the first 4 layers of the 3B LLaMa model.
A brief description of these FL methods tailored to improve communication efficiency is as follows.
This parameter- efficient method integrates knowledge distillation within a single client and employs a parameter aggregation strategy using Singular Value Decomposition (SVD). For the purposes of this section, which focuses solely on comparing communication efficiency, we incorporate only the syDebased client-server communication strategies into LM- WEATHER baseline.
FedKD:
20

---

Preciiitaioon Temparahure
Wind Speed
Pracipitation Tampereture
Wind Speed
Angalea
Phoenix
Miami
Pittsburghh
Minnoppoiis
Porttand
Montreal
Salnt
Maharlyya
San Antonio
Naahville
New York
Philodelphie
Seattle
Vancouwer
Figure 4: (Figure continudd)) Visualisation of partial variables in ODW2 dataset, where we have selected the first 1,000 time points for presentation. The selected variables are, from left right, humidity, precipitation, temperature, wind direction, and wind speed.
continued)
FedPer: This PFL approach maintains personalized layer while sharing the remaining base layers during communication. This enhances communication efficiency by transmitting only portion of the parameters.
This fine-tuning method enhances parameter efficiency by sharing only the biases of the local model during global aggregation, thereby reducing communication overhead. To integrate this method into LM- WEATHER bascline, adjusted all biases in LM- W WEATHER be unfrozen.
FedBF:
FedAP: A parameter--fflicent fine-tuning method in FL, which involves sharing only adapters during
global aggregation.
This parameter-efficient FL method enables participants to coopcratively train lightweight prompts without sharing the entire model, significantly accelerating both local training and global aggregation. In experiments, treat the adapter generated on clients the prompt facilitate the incorporation of this baseline.
PromptFL:
B.3 Task Setups
We evaluate our proposed LM- MM-WEETHER using four distinct on-device weather modeling datasets, each with tailored settings for various tasks. The specific task settings for these datasets detailed in Tab. 15
21

---

Additionally, the specific tasks and scenarios for the on-device weather forecastingimpuuttioo vary by dataset, outlined in Tab.[6
Table 15: Task setup for different datasets during the evaluation. Note that for the imputation task there are actually no historical observations, but rather they are performed on a single long sequence.
Dataset
Task
Historical Observation Horizon
Prediction Horizon
Random Masking Ratio
Forecasting
192
N
ODWIT
Imputation
Consistent with the prediction horizon
{25%, 35%, 50%
Forceasting
192
N
ODWIV
Imputation
Consistent with the prediction horizon
{25%, 35%, 50%
Forecasting
192
196, 336, 720
N
ODW2T
Imputation
Consistent with the prediction horizon
(25%, 35%, 50%
Forccasting
192
N
ODW2V
Imputation
Consistent with the prediction horizon
{25%, 35%, 50%)
Table 16: Summary of framework evaluation scenarios for various datasets. Scenario 1/2/3/4 (in forecasting) refers to multivariate to univariate forecasting, where all historical variables are used to predict single future variable. All represents multivariate multivariate forecasting, meaning all variables predict all others. The symbol "_" indicates non-existent scenario for that dataset. Scenario 1/2/3 (in imputation) indicates different masking ratios for the original weather sequences.
Forecasting
Imputation
Dataset
Scenario
Scenario 2
Scenario
Scenario
Scenario 5
Scenariu
Scenurio 2 Scenario 3
ODWIT
Temperature
Humidity
Wind Speed
Surface Temperature
ODWIV
Temperature
All
ODW2T
Temperature
Humidity
25%
35%
50%
ODW2V
Humidity
B.4 Implementation
We mainly follow the experimental configurations across all baselines within a unified evaluation pipeline in https: /tgg:://ttttt cwwcttotttaee (eiuumm/iimme-eeeeeeeeeeii Series-Liiie Library for fair comparison. Specially, we use GPT-2-base the default backbone model unless state otherwise. All experiments repeat five times and we report the averaged results. Our detailed model configurations are Appendix [B.8] All the algorithm implementations and designs in this study are based on Py torch and the algorithms are run on two RTX 3090 GPUs 24GB.
B.5 Technical Details
Reversible Normalization. In time series analysis, statistical properties like mean and variance often shift over time, indicating distributional changes in the data. address this, we've incorporated Reversible Normalization (RevIn) [51] into our LM-W MM-WEATHER. Specifically, we've integrated RevIn into our Task Adapter Generation. This introduces two dynamic factors that adaptively normalize segments of the metcorological variable sequence X, or their decomposed components (Trend Xirend Scasonal Xsasonll Residual ARessial), enhancing the accuracy of metcorological variable modeling. Specifically, for the trend component of X,, i.e., Xrrend, its transformed value X'annd can given by:
E. E[Aereand]
X
Xfrend
+ Br
(6)
Var [Kirend] + ET
where (Tee) and Var [Arwna) are instance-specific mean and variance, respectively. "T and Br are trainable parameters for this component. This transformation is also applied to both the seasonal and residual components.
Pre-trained Language Model (PLM). In LM-WEATHER, we do not change the main architecture of the PLM, but use the parameter-efficien fine-tuning (PEFT) strategy to avoid large-scale parameter variations ensure high efficiency on resource-constrainee weather devices, and in this way, to achieve more reliable cross-domain knowledge transfer. Specifically, we introduce LoRA the local PLM, which allows only 1.5 % of the PLM parameters to be trained while the rest remain frozen, as shown in Fig. Note that LoRA is only applied to the query and value of each Attention in the PLM, and the resulting low-rank matrices are used for global sharing between the client and the server.
22

---

Pre-trained Language Model
xN
Add & Layer Norm
Prediction
Feed Forward
Add & Layer Norm
Parameter Adapter
LoRA
Attention
PLM
Query
Value
Key
Task Adapter Generaflen
LoRA
W.
LoRA
W
Latent Representation
N-heads
Word Embedding
Figure 5: Schematic diagram of the PLM in LM- WEATHER, where we introduced LoRA to the PLM, achieve more reliable cross-domain knowledge transfer while at the same time ensuring greater efficiency in adapting to low-resource weather devices.
Low-Rank Adaption (LoRA). To achieve more reliable cross cross-domain knowledge transfer (i.c., from
natural language to complex weather sequences) while guaranteeing higher efficiency, we introduce LoRA [1)]. a parameter-effiient fine-tuning method for large language models, into PLM. Specifically, LoRA is applied to the Query and Value of each Attention layer by creating low-rank matrices for two pre-trained parameters W,, and WN:
(7) where denote the latent representation from input weather sequences through PLM's word embedding layer, re
QUERY W + By VALUE W. + A, B.,
Ag E Rdxr and B, E gexd are low-rank matrices created from W, E Rdxd A, E Rdxr and B, E xxx low-rank matrices created from W. E Raxd d is the number of dimensions, is the rank, and & d. It's important to note that only the low-rank matrices Aq. Bg, Av, B, are trainable; the others remain fixed during training. Initially, A, and A, are set with random Gaussian values, and B, and B, start as zero at the beginning of training.
Task Adapter Generation. The Xmaand Assassonal, Vissuual obtained from decomposition are used to generate Task Adapter via an unified generator as Fig. [ that consisting of Token Embedding, Position Embedding. and Temporal Embedding. Specially, while keeping raw shape to generate TOKEN ADAPTER Pro E Txx
dimensiiion convolution operation to map each each specific
sample x k xx1
P CONVID(X**) Pro CONvID(A)
(8)
Additionally, we use a trainable lookup table to map each point's explicit position in the entire sequence, to generate POSITION ADAPTER Pro E R TTXC as:
Pro E(INDEX(X)),
where E()) is trainable lookup table, and INDEX(-) a function that achieve the indices of cach point's locations of weather sequence X', Furthermore, we separately encode different time attributes such as minutes, hours, days, weeks, and months, using trainable parameters dynamically model complex temporal shifts, to generate TEMPORAL ADAPTER Pte,
PIE
Σ
(x(x)
(10)
a (mins, www....oorr.dyy...ss...oo months)
where represents different temporal attributes. E., denotes the trainable lookup table for each temporal
attributes. Finally, for decomposition components, corresponding generated adapters can be obtained by aggregating Token Adapter Pio E IR LxC Position Adapter Pro E _g_ and Temporal Adapter PIe E RLXC as P. P + Pd + P where d E (Trend, Seasonal, Residual). this means that we can obtain Pieend, P Peevcall PResidul.
23

---

B.6 Theoretical Insights on Personalized Adapter
The effectiveness and superiority of our proposed Personalized Adapter have been demonstrated in sufficient ablation studies (please refer T the main text). Here, we will discuss theoretical insights that further supports effectiveness Personalized Adapter. Personalized Adapter can effectively capture potential pattern in meteorological variable time series, which comprises both the Task Adapter and the -FN-based Parameter Adapter. Specifically, our focus primarily revolves around the Task Adapter active in extracting representations, Embedding for transforming metcorological variable time series.
which including
Let the weather sequence he X [x1, X2. xT), where E Rd Is the observed value with d variables & moment. Let X' represent the function space which x, of weather sequence helongs, and denote the function space to which the implicit representation n, belongs. Token Embedding can be interpreted as a mapping fa : X + Z. According to the Kolmogorov-Arnon representation theorem, for any continuous function f C(O0, 1)"), there exist 2d + continuous functions og E C((0, , 1J) and 1i E C(O, 1J) such:
f(x
=
2 (xp)),
which means Token Embedding can construct high-ding dimenssienii nonlinear mapping from multiple one- dimensional functions, capturing complex patterns within weather sequences. Positional Embedding introduces vector Pe for each step, enabling the model differentiate between ohservations different time steps. For any two steps & and f2, their position vectors Pt, and Pt, satisfy:
||p; Pt2 12 V 2k(1 2 f2)
10000
The growing distance between and la with increasing time gap mirrors the relative positioning of time steps, aiding model in grasping temporal dependencies. Additionally, resonates with weather data's 's cyclical behavior, helping the model to learn from these recurrent patterns.
sine-cosine function's periodicity
Finally, consider the role of Temporal Embedding from the view of matrix decomposition. Suppose the temporal matrix T has a rank of r, it can be decomposed as 2, 8T En (z, & u.)v?' Temporal Embedding transforms the original sequence by scaling and rotating it to represent different interaction patterns. Using singular value dccomposition, the top singular vectors distill the core structure of the time-based matrix. This allows Temporal Embedding to intuitively learn a compact representation weather sequences. highlighting the primary interactions between variables. In optimizing Personalized Adapter within LM-WEATTE WEATHER, the focus lies solely on Personalized Adapter and the attention layer influenced by LoRA during local updates. As Personalized Adapter undergoes solely local updates while sharing low-rank matrices globally, akin to layer- wisc optimization PLM, efficacy of its optimization process can be theoretically substantiated by the thcoretical analysis provided [52].
B.7 Evaluation Metries
For evaluation metrics, as 34). we utilize the mean absolute error (MAE) and root square error (RMSE) for both forecasting and imputation. The calculation of these metrics are as follows:
T Σ Y. Y%b. MAE T
T
RMSE
1
T
D (Y. Ý1)
(1I)
where T denotes the number of data points (i.e., prediction horizon our cases), Y. and Y, the i-th ground truth and prediction where E T........
B.8 Model Configurations
The configurations of our I.M-WAATHER for different tasks and datasets are summarized in Tab. 17 We consistently use the AdamW 5 optimizer in all experiments.
Appendix C Theorems and Proofs
Theorem C.1 (Decomposition Rationality from Time Series), Given weather time series X XTrend + Xscasmal, X. t. [n]. Let E {e1, (2,,,..a denotes
of orthogonal bases. Lets Excaumee C denote the subset of on which Xseaonati has non-zero eigenvalues and Evrent C E de- note the subset of E on which Xyoou, has non-zero eigenvalues. If Xront and Xsssomol.t are not orthogonal, iee. Din Niwad, Xi + O, then Eroud 0 Escaonat f 0, i.e., E can not disentangle the two signals onto disjoint of bases.
24

---

Table 17: An overview of the experimental configuration for LM- AM-WEATHER. LR is the initial learning rate, (FS) denotes the few-shot learning setting.
Medil Hygerpanmeesr
Tharny Franes
Tel.: Dutset/ Configantion
Buckhone (PLM) Layezs
Ingrat Lengch
Paoch Dia
Hensh
Low
Size
Commanccaico Rousi
puttiipation
Feroceing- GLOWIT
192
0:000
MSS
138
20
50
D.1
Feeessting ODWTV
192
16
0,006
MSSE
128
20
50
0.1
Furceainy-
236
Tvereseing- D1WFVV
190
16
[0000
MSS
756
70
50
0.1
Impuratio- OUWFT
336. 720
15
6006
MSB
128
20
50
0.1
Imputamii NDWIV
190,
730
[0006
MS5
138
30
50
0.1
Impurtiie- GUWIT
96. 192, 336, 720
15
00806
MSS
236
20
50
0.1
Impatammn
Forccusiigg OUWIT (PS)
192
0:006
MSE
128
20
90
0.3
Funsaingg (FSIY
192
15
(0005
MSB
128
20
50
0.1
Pewrssiing ODW1T (85)
190
0.009
MSR
756
70
Funcaingg GUWZY (PS)
192
16
60806
MSSB
48
20
50
0.1
Imputeionn D1W11
Impuration OUWIN (2S)
192, 336, 720
6,008
MSB
128
20
80
0.1
Imputation 00W27 (FS)
192,
16
6.000
MSS
256
30
50
MSSE|
20
50
Preef. We decompose XSemomal, and Frennd onto E and acquire that XSesumill Σ ase; and X'rend, Σ bse.. Then it is obvious that e E Assassoua 4> ass A and e; A'rrend <> b, * O. Now, let us consider the inner product of & and 1
Frend Asiasanaal, :
(12)
aces).
E Ebes)
Σ asbseses
Note that Σ e .essooona. O. Thus, there must be at least one i such that a; A 0 and b * 0. Thus. e; E Eseasonul and es E ETient, in other words, Erenn O Esencnal o 0. The theorem demonstrates that if Xrrnnd and X'sessomt are not orthogonal, orthogonal bases that separate Xrrest, and I into two distinct sets cannot cxist. Typically, periodic and non-periodic signals are not orthogonal because the periodic signal has a discrete spectrum, while the non-periodic signal has continuous one, leading to potential overlaps non-zero frequencics. Principal Component Analysis (PCA) seeks find orthogonal bases data, but it cannot split these two signals into separate bases. Citing Theorem from [77] we understand that self-attentive mechanisms in pre-trained large models function similarly to PCA. Thus, without manual intervention, the self-attentive mechanism is unable to automatically divide time series into trend and seasonal components.
O
Theorem C.2 (Exchange Low-Rank Matrices Ensures Privacy: Parameter Interaction Perspective).
Given a on-device weather modeling framework based on federated learning that global optimization object is F())= = = (: 0), where I (ax; 0) is the loss function of i-th client, {Di} is dataset of i-th client, and pt and @ denote the data distribution weight of client i and the model parameters, respectively. Given that the parameters 9 of the PLM Ms broadcasted by the server consist of two parts: frozen part Ma. I and trainable part Ma, u. interacting only the low-rank matrix parameters Ma C Ma. a subset of trainable part Mo.: during each round ensures privacy.
Proof. We assume that f(x; 8) is a convex function with respect to 8, i.c., for any 02 and 02 and À E [0, 1). we have
f(:; 20, + (1 - Xj02) S Af 01) + (1 A)5(2;0))
(13)
Since only low-rank matrices parameter Me. parameterized by 0: is exchanged, we can convert to 0' = [0 0.). where of is the embedding parameter after the server update. Since we only update on A,, A remains unchanged. Thus, data privacy can be ensured, as A. contains parameters that reveal user-specifie information. Furthermore, the low-rank matrices applied to the PLM Ma using LoRA are initialized with a random Gaussian distribution and all-zero values, respectively, before training. This global information sharing approach also helps to enhance privacy.
D
Theorem C.3 (Exchange Low-Rank Matrices Ensures Privacy: Model Indistinguishabiiity Perspective).
Our LM- Weather can hide key features of local model when sharing low-rank parameters, even external attacker gains access to a shored low-rank update, it is difficult to reconstruct or differentiate between the original models of different porticipants. Support client and client get two different local model due to updated on heterogeneity weather time series, as M: and M, parametered by O: and 02,, L.M) denotes the low-rank matrix generated by model M, where B E gdxr and A E g and & d, and define Adu as the adversary. Conditions exist such that for any polynomial-ting time attacker Adv, Pr Pr( (M..) = 1] Pr(Adv(L(M;)) 1]] S holds.
25

---

where is small positive number, which implies that even if the attacker acquires shared low-rank update. it difficult reconstruct distinguish between the original models of the different participants.
Proof. The goal of LoRA is to find the low-rank approximation: min| - 6p - B BAlle, where Bg is the initial weight. || || is the Frobenius paradigm. Consider two models M, and M,, whose weights differ by A 0; 0j. The corresponding LoRA matrix is:
L = B,A, R A8, = 0, 0v.
(14)
I = B R A0 03 Pp.
According to matrix approximation theory, for the best approximation with rank r, the upper bound on the error is:
|| F. S. 0.+1(A6,)
(15)
where + (AO.) (0.) is the r+ I-st singular value of A0, By Johnson-Lindensttnn there exists mapping f : IRd + Rt, where l (llog(n))ee (n)//"), such that for any æ, y E RR In case, LoRA can regarded as such degenerate mapping. Assume AA: A0jlle S and according to the trigonometric inequality:
Lemma 54 for any c > 0.
I|4,
Lglls S I|L, A0:II ||00, - 20,11 + |000 1, 20.4 + 6 (16)
Let =0+1((0.) (A8,)
get:
|| Li || S 2 5
(17)
any polynomial time attacker Adv.its ability distinguish between and L3 restricted to the difference in their Frobenius paradigms. We can define a function f such that:
[P((Av(L(MM.)
Pr(Adv(L(M,)) = 1]l s f([L: Lsllr)
(18)
where f is a monotonically increasing function that represents the attacker's capability with respect to the matrix difference. Combining Eq. [6]and Eq.
[Pr(Adv(L(MM)) = 1] Pr(Adv(L(M,)) = -]] <
(19)
when d and are small enough, we can make sure that the right-hand side smaller than the intended e. This means that an attacker cannot reverse-engineer personalised local parameters and data to ensure privacy through the low-rank matrix of communication across clients.
Appendix D
Additional Finding & Experiment & Discussion
In this section, we explore and discuss potential research findings and questions for our LM- ME--EETHER via conducting additional experiments. These potential research questions as follows:
RQL. How does LM-WEATHER compare to Personalized Federated Learning (PFL) baselines in terms of trade-offss in personalization and global model performance?
RQ2. How does LM-WEATHER WEATHER perform compared to centralized and local-only training modes?
RQ3. How does the pre-trained language model contribute in L.M-WAATHER
RQ4. What the resource utilization and training & inference cost of LM- WEATHER??
RQ5. Can LM- MEATHTE be used for other tasks?
D.1 Trade-offs in Personalization and Global Model Performance (RQ1)
Our LM-WEATHER builds on the assumption that the foundation model already exists, treating pre-trained language models (PLMs) as such and broadcasting it to each client to achieve local updates. Our aim is to employ device information-specificc (e.g., geographii/atmoophher achieving cross-domain knowledge transfer from language to meteorological sequences. This approach yields highly customized models for individual devices while achieving global knowledge avoid data silo, thereby supporting diverse analyses heterogeneous weather data. Alternttiee PFL methods do not match the efficiency and flexibility of our personalized adapter in this context, making them less suitable. By incorporating PFL baselines (Per-Fed FedAvg (43). APPLE (55), FedPer (56), and FedALA (57). we provide quantitative results that substantiate our claims, experiment setting consistent with the manuscript ODWIT.
patterns) adapter, to promote the local PLM in
26

---

PFL Baseline. A brief description of PFL baselines used in this section of the experiment is as follows.
Per-FedAvg:
Allowing for personalized model updates for cach client by adding client-specific parameters the global model and optimizing them during FL training.
Tackling statistical heterogeneity in FL. automatically capturing information required by clients from global models using adaptive local aggregation methods.
APPLE:
FedPer: Dividing the model into base layer and personalization layer, only the base layer is
uploaded during aggregation while keeping the personalization layer to combat statistical heterogeneity.
FedALA:
Tackling statistical heterogeneity in FL by automatically capturing information required by clients from global models using adaptive local aggregation methods.
Table 18:
Comparison on personalized performance between our LM-WEATTIE WEATHER and PFL baselines under forecasting mnultiariiae-muliivrrill and imputation (50% masking rate), where Avg. denotes the average performance of four periods [96, 192, 336, 720), Bold means the best.
Task Method
LM- WEATHER (Ours)
Par-FedAvg
APPLE
FedALA
FedPer
Forecasting (Avg.)
45.4/74.6
48.6/76.7
51.7/79.0
50.4/80.0
12.1/79.0
Imputation (Avg.)
23.1/42.4
27.7/50.1
26.5/49.6
34.4/57.2
30.2/54.0
Forecasting (Avg., 5% Few-Shol)
89.7/96.7
101.4/197.3
99.6/186.2
117.3/214.0
104.9/202.8
Imputation (Avg., 5% Few-Shot)
62.3/124.4
89.2/177.8
92.1/199.2
82.4/153.7
84.9/159.7
Personalized Performance Comparison. The performance quantification of our L.--EATHER and
PFL baselines under personalized performance for different tasks and scenarios shown Table[T8 Our LM-WEATHER outperform other PFL baselines across different tasks (forecasiin/immuuaaiin (regular/few-hhot learning) by wide margin. This supports our finding that in the scenario of on-device weather variable modeling, PFL methods is appropriate.
and scenarios
Global Model Performance Comparison. The comparison global model performance client between our LM- WEATHER and PFL baselines are shown Table Our LM- WEATHER outperforms PFL baselines in terms of global model performance, as demonstrated by the fact that its global model performs more stable across client with heterogeneous data.
Table 19: Comparison on global model performance across client between LM- Weather and PFL baselines multiiariaae-mullivariat forecasting tasks (OWDIT dataset, MAE/RMSE report), Red denotes the original LM-WEATHER 's performance, and Bold means the best among global performance.
Client ID
LM- WEATHER (Ours)
LM WEATHER (Global)
Por-FodAvg (Global)
APPLE (Global)
FedALA (Glebal)
FedPer (Global)
46. 1/75/4
56.8/84.7
61.8/90.1
64.7/88.6
63.2/8..1 1
68. $9.3999
45.2/443
49.3/75.2
54,2/82.6
57.2/81.4
55.9/80.7
57.1/87
66.5/916
4/.1/
48.7/72.1
$2.1/80.3
55, 55.1/78.9
53.6/78.2
549/84466
6
43.3/73.7
$9. 1/87.
63.49117
66.3/90.3
64.8/89.5
70.2/102.1
44.8
47.6/73.8
55.9/84.5
58, 38.9/83.2
$7.3/2,4
59. 19,5/89
1/77.2
53.4800.
09.6788
6/87.4
61.1/7666
7/66
9
42.6/1.7
44 1/70.7
50.9/78.8
538877.6
$2. $2.3/76.9
53.613.5
167/33.3 10
57.9/86.3 2
64,3992.
67,1
65.7/90.
71 71.9/1
S4,7/83.2
15.674.9
55.78829
60.8/89.1
63.9/88.5
62.5/87.7
67.4/6
13
4/77/77.8
43.3/68.5
49.6/77.5
56.3/76.3
50.9/75.5
$1.7/811
14
49.275.5
60.5/88.4
65.794,2
68.7/2.8
67.4/22.4
72.5/104.5
48.7/44.4
56.3/880.2
Total (Avg.)
15/4/44.6
51.2/78.7
$7.0/81.9
59.9/83.2
58/4/82.8
61,4/2277
Personalization and Global Model Performance Trade-offs. We consider the trade-off between personalization performance and global model performance for our LM- WEATHER and PFL baselines, the results are shown in Table 20 Compared with PFL baselines, our LM- M--EEATHER maintains the best trade-off' between personalization performance and global model performance, i.e., the personalization performance does significantly exceed the global model performance while the performance far exceeds PFL methods, which means that the LM WEATHER can flexibly applied to different practical scenarios, including personalised analysis of regional weather trends as well as comprehensive analysis of weather trends over large-scale regions. This means that LM- W EATHER can be flexibly applied to different practice scenarios, including the personalised analysis of regional weather trends and the comprehensive analysis of weather trends over large scale areas.
27

---

Table 20: Comparison of LM- Weeehrre between personalized performance and global model perfor- mance, results are obtained the multivariate- multivariate report), Bold means the best, f represents the improvement (gap) in the personalization performance of the method relative the global model performance.
multivariae-muliivvriaat forecasting task on OWDIT (MAE/RMSE
Method
Personalized Performance
Global Model Performance
Ave. Variation (Personalized vs. Global)
Per-FedAvg APPLE
48.6/76.7
57. 57.0/84.9
13 .9%
51.7/79.0
19.9/83.2
10.58%
FedALA
50.4/80.0
58.4/82.8
1968%
FedPer
52.1/7900
61.4/92.7
1 17 17.59%
LM- L.M-EATHER (Ours)
45,4/7..6
51.2/78.7
1 9.16%
Performance and Adapter Updating Trade-offs. Furthermore, we investigated effect varying the number of local update rounds in adapters across clients on the performance of L.M-EAATHER regrading personalization. The results are presented in T2 We observed that increasing the local update rounds from the default five to fifteen leads to smoother and enhanced personalization performance across heterogeneous clients. However, this increase local update rounds also incurs additional computational communication costs, which, our assessment, not justify the modest performance improvements.
Table 21:
Performance of each client under the multvariate-multivariaee forecasting task on ODWT1 with different adapter local update epoch (MAE/RMSE report), where = 5/110/15 represent the 10, and 15 local training rounds, respectively.
Client ID
LM-WEATHER (E 5)
LM-WEATHER (E 10)
LM-WEATHER (E' 15)
44.8/73.9
41.3/70.5
41.6/69.7
46.1/75.4
47.9/75.8
46.3/74.5
3
45.2/74.3
43.5/72.1
43.1/71.1
4
47.1/79.0
46.2/74.6
45.7/73.6
5
43.1/70.6
42.7/71.3
42,4/70.4
6
43.3/73.7
45.5/74.2
44.9/73.0
44.8/74.1
44.1/73.0
43.5/71.7
8
48.1/77.2
48,6/76.3
47.2/75.2
9
42.6/71.7
40.9/70.1
41.2/69.3
10
46.0/75.3
46.8/75.4
45.3/74.1
45.3/74.4
43.2/71.8
42.8/70.8
12
45.6/74.9
45.7/74.5
44.5/72.7
13
48.7/77.8
42.1/70.9
41.9/70.1
14
49.2/75.5
47.3/75.1
46,8/74.9
15
41.1/73.2
44.8/73.4
43.8/72.3
Total (Avg.)
45.4/74.6
44.6/73.2
44.0/72.2
D.2 Centralised and Local-only Training (RQ2)
The ordinary centralised training strategy (all data aggregated into single server) exhibits learning efficiency that ordinary distributed learning strategy. The ultimate goal of FL achieve performance close to that of centralised training and to ensure privacy across data sources. Table [2] illustrates that our LM-W WEATEETHER achieves comparable effectiveness to Non-FL (ventralised) training, with only a 2.04% disparity. Compared to LM-W WEATHER-Local which lacks interaction between devices, LM- LM-WEATHER performs better due overcoming data silos.
D.3 Contributions of Pre-trained Language Model in LM- L.M-EEAHER (RQ3)
Our LM-WEATHER significantly outperforms time series-specific models trained from scratch under centralised setup. Centralised training aims acquire an excellent pre-trained model, where PLMs possess inherent advantages due to their prior sequence modeling capabilities. Moreover, various parameter-efficiae (PEFT) strategies enable PLMs to adapt to new domain knowledge cost-effectively. FL-based aggregation facilitates stable on on-device fine-tuning process, with LM- Weather enabling highly customized on devieecce fine-tuning of PLMs with greater efficiency. This highlights the substantial contribution of PLMs in this task.
fine-tuning
28

---

Table 22: Comparison of LM- WMM-EEETER'SS 's multivariate-mi muliivvaitatiiiiiaiiat performance in the FL and the Non-FL (centralised) setups, LM- Loaa is the setting in which LMM WEATHER trained locally at each device without communication, and disparity is the difference in performance relative Non-FL.
Dataset
Length
Noa-FL (centralised)
LM-WEXTHER (Ours)
SMM-WEATEER-AN
LM-WEATHER-Locl
96
41.7/70.3
42.3/711
44.1/74.8
45.9/72.9
192
43.5/7166
44.4/73.6
46.3/77.5
46,7/74.3
ODWIT
336
45.2/72.9
8/78/
47.9/79.3
48.2/77.0
720
46.8/73.6
49.2/78.5
51.803.0
50.5/80.5
Avg.
44.3/72.1
45.4/74.6
47,5/78.7
47,8/76.2
96
42.M68.7
42.3/9.6
42.7/69.5
44.5/68.6 6
192
43.9/69.9
44.4771.7
45.5/72.6
1/70//0044
ODWIV
336
45.4/71.2
46.0/72.
47.2/14.3
48.8/73.2
720
46.8/72.6
49.7/74.0
51.2/78.2
53.2/79.4
Avg.
44.6/70.6
45.6/7199
46.6/73.6
48.2/72.9
Communication Param.
None
0.38 M
10.38 M
Not applicable
Disparity
J. 2.04%
4.4.37%
4.5.67%
Table 23: Comparison between fine-tuning PLM with Adapter (LM--EATTHEE and training from scratch using non-PLM. architecture (Pyraformer, Reformer, PatchTST, DLinear, and LightTS) on multivariate- multivariate multivariatemullivvaiiaee forecasting tasks (MAE/RMSE report), Bold means the best.
Dataset
Length
LM WEATHER (Ours)
Pyraformer
Reformer
PatchTST
Dlinear
LightT'S
96
43.0/75.2
66.0/84.2
68.2/89.6
44.7/76.1
48.6/77.6
52.1/83.3
192
44.7/78.4
68.7/87.4
69.3/89.9
46.5/78.4
51.0/79.8
57.5/87.0
ODWIT
336
47.2/80.9
71.3/90.9
71.4/91.7
48.2/80.3
52.3/84.1
62.3/94.2
720
50.4/83.8
76.8/92.3
74.5/93.9
54.2/84.5
55.4/86.3
69.5/98.4
Avg.
46.3/79.6
70.7/88.7
70.9/91.3
48.4/79.8
51.8/82.0
60.4/90.7
96
45.4/71.3
60.4/69.1
45.7/71.8
47.2/73.4
48.5/74.7
50.1/75.3
192
46.9/72.9
61.9/73.2
51.9/75.5
48.4/74.2
49.4/76.6
52.7/78.8
ODWIV
336
49,0/75.5
64.4/76.9
53.4/76.9
48.9/76.7
53.7/78.3
54.0/80.4
720
53.7/79.1
68.2/82.7
56.5/84.9
54.1/77.4
57.2/82.2
57.7/84.8
Avg.
48.8/74.9
63.7/75.5
51.9/77.3
49.7/75.4
52.2/78.0
53.6/79.8
1dt Count
16
3
D.4
No Free Lunch in Performance Improvement (RQ4)
The remarkable capabilities of cutting-ede ge DL models across various domains and tasks, such as LLMs, and VLMs, can be attributed to their extensive parameters and training large datasets. Currently, perfect balance among performance, model size, and does exist. Despite numerous studies focusing reducing training inference costs while maintaining superior performance, there no one-size-fits solution. This is also true for our M--WeATHEE which demonstrates exceptional performance across diverse tasks and scales on real-world datasets with significant heterogeneity, significantly outperforming comparable DL methods. In this context. analyze the costs associated with training and inference for LM- L.M-WEETTEE WEATHER and its baselines. exploring and discussing trade-off's between cost-effectivenes. and performance in practical applications.
Table 24: Comparison of training/inference costs based on ODWIT IT with N 192 under multivariate- multivariate forecasting tasks (MAE/RMSE report), where Bold denotes the best, 'Comm. and 'Perf." denote communication and performance, respectively.
Method
Ave, Traiting Time (per round)
Inforence Time client)
Truniin Memory cllent)
Inference Memory clicat)
Comm.
Timee
Perf
inest
7.93 MB
4.21 MBI
1.44
$2.181
F1.1 iggiTS
4
2
30.94 ki
12.34 MI
9.01
50.50006
BL. PudiTST
124
10's
1260.01
60002
99,62
47.379.8
FL. Thassfomerr
121
5
1700.9511
$41.09
36,445
$2.184.0
[1. Thaasfoorrr
95
1665
708
372.28
22,40 5
$1,58443
Fl-Infomer
110's
1890.00
897.11
42.81
$2,98466
Refoneer
796.96
1880.41
3640.11
48,68113
IM-WEATHER
44.47366
The quantification and comparison of computational costs against LM- SMM-WEATHER and baseline are shown in We discuss this results from perspectives follow.
Table 24
29

---

Communication and Performance. LM-WEATHER outperforms baseline in these two key metrics, which is critical for practical meteorological variable modeling and analysis, sensiivv--enniiivee and high accuracy demanding application.
Trade-offs between Resource Consumption and Performance. In terms of training and inference time and memory usage, A.-WEATHER is less efficient than lightweight haselines such as FI.-DI Linear, LightTS, and iTransformer, due to its use of a Pretrained Language Model (PLM) as backbone. Although LM- WEATHER demands more resources. the trade-off is justified by its cost-effective performance gains. Its slightly increased memory requirements for training and inference are manageable on most devices. In the context of weather analysis, where precision is critical, prioritizing performance improvements over minimal resource consumption is essential. Additionally, LM-W Bi-WEATT WEATHER capitalizes on the knowledge-rieht -ichh PLM and requires only minimal, low-cost fine-tuning on devices achieve superior performance. This strategy not only enhances performance but also reduces the frequency of future model updates, thereby lowering long-term costs compared to developing a baseline model from scratch.
Table 25: Comparison between LM- Weather and baseline in terms of model size the device and performance of forecasting (mulivariate-oommliivvanna to-multivariate), and imputation (50% masking rate) on ODWIT (MAE/RMSE report), where Bold and Underline denote the best and the second best.
Method
Size
Forecasting
Imputation
FL-DLinear
0.28 M
53.3/82.8
28.5/49.9
FL-Ligh
1.10 M
62.7/93.4
26.1/45.7
FL.PPtccTTST
37.61 M
48.6/81.0
45.4/73.5
FI Trannrssforerr
45.55 M
52.8/84.7
57.6/82.3
FL-iTransformer
25.19 M
53.7/63.7
27.6/48.2
FL-Informer
52.31 M
53.4/85.2
61.4/85.9
FL-Reformer
45.59 M
78.2/98.7
69.8/92.5
FL-GPT4TS
321.7 M
49.9/82.5
25.1/46.2
LM-WEATHEER
304.1 M
45.4/74.6
23.1/42.4
addition, we further provide a comparison of model sizes and performance between LM- WEATHER and baseline, as shown in 225 The difference in model size between our LM- WEATHER and baseline can be deemed acceptable for the following reasons.
Trade-offs between Performance and Size. While LM- WEATHER may not be compact in terms of model size or resource efficiency as lightweight baselines, offers significant advantages various analysis tasks. Its high performance is particularly valuable in practical applications. Moreover, with a model size of 304.19 M. LM- WEATHER is still accessible for devices with limited resources. This contrasts sharply with many large foundation models, which typically comprise several hundred million parameters. The trade-off between performance and size is justified, especially considering the critical nature of accurate weather data analysis.
Efficient Parameter Update and Communication. LM-WEATHER implements efficient on-device fine-tuning of the pretrained language model. Unlike baselines that require training from scratch, LM-W WEL-EEATHER only needs fine-tuning of relatively small number of parameters (10.38 M) each device, with minimal device-to-serve se-vve communication overhead (0.38 M). This approach facilitates highly personalized cross doms-n knowledge transfer, significantly reducing the ongoing costs associated with processing the ever ever-changing streams of weather data. These aspects highlight the pragmatic considerations that have shaped the design of L.M. RATHER. The model's capabilities to deliver exceptional performance, combined with its efficient. parameter tuning and communication strategies, offer cost-effective solution for advanced weather data analysis resource-comstrained environments.
D.5 Additional Tasks for Potential Applications (RQ5)
Given datasets we proposed in this paper focus on forecasting and imputation tasks, we broaden its scope briefly to explore its potential application by integrating anomaly weather detection tasks. This involves relabeling the dataset to identify intervals with anomalous meteorological variables instances of abnormal weather processes. Specifically, we label original datasets via Isolation Forecast [58]. the main process as follows: (1) We set the cut length to 100, using this metric to segment each channel (variable) and construct several random trees that collectively form forest. (2) The "isolation" degree of each data point is quantified by the average path length from the root node to the terminal node. (3) Data points with shorter path lengths more easily isolated and
30

---

thus more likely to be outliers. (4) We establish a threshold based on the average path length; data points falling below this threshold classified anomalies.
Evaluation Metries. We used Precision (P), Recall (R), and F1-Score (FI) to simply quantify per- formance of LM-LEMMME WEATHER and baselines on the weather anomaly detection task, these can be formulated as:
TP
TP
FI PxR P+R
P
(20)
+P+ FP
TP FN
where TP (True Positives), FP (False Positives). and FN (False Negatives) represent the number of samples correctly labeled as anomalous, number of samples incorrectly labeled as anomalous, and number of samples that were not labeled as anomalous by the model but were actually anomalous, respectively.
Experiments and Results. We set the input time series length to 100, and other settings (e.g., baselines, pyper-parameters, local updating steps and federated communication rounds, etc.) consistent with those in the main text, and we conduct experiments OWDIT and OWD2T hriefly show the results. The performance quantification of our proposed I.M WEATHER and haseline on weather anomaly detection tasks is shown in Table (able/2((ODWW) (ODWIT IT results) and Table 27 (ODW2T results). The findings underscore LM- WEATHER'S 's robust applicability and its Moreover, LM- W WEATHER's 's superior performance over baselines in both regular and few- shot tasks reaffirms its effectiveness and overall superiority. Moreover, LM-W SM-WEATHER superior performance over baselines both normal few-shot tasks reaffirms its effectiveness and overall superiority.
Table 26: Results of LM- WEATHER and baseline for weather anomaly detection tasks on ODW including regular and few-shot scenarios, where 5% means that 5% of the data is used in training. Bold and Underline denote the best and the second best.
Scenario
Regular
Few-Shot
Method/Metrics
P
R
F1
P
R
F1
FI.-DLinear
82.33
78.54
80.34
69.72
71.22
70.48
FL-LighTTS
86.11
72.89
78.76
70.46
70.86
70.54
FT.PPatcHTST
88.94
82.57
85.55
73.47
71.52
72.41
Fl - Transformer
74.23
76.61
75.36
64.27
66.89
65.65
FL Transformer
83.68
82.96
83.17
67.8
74.87
71.21
FL-Informer
76.87
78.32
77.32
69.24
71.23
70.2
FL-Reformer
78.21
79.76
78.87
70.22
67.43
68.55
FL-Pyraformer
80.28
83.66
81.71
67.46
69.05
68.06
FL-GPT4TS
89.72
85.43
87.44
76.34
77.31
76.76
HIM-WEATER-AAE
90.21
88.14
89.18
80.41
82.68
81.46
L.-WEATHER (Ours)
92.00
90.45
91.15
84.25
86.23
85.09
Table 27: Results of L.M-WEATHER and baseline for weather anomaly detection tasks on ODW2T, including regular and few-shot scenarios, where 5% means that 5% of the data is used in training. Bold and Underline denote the best and the second best.
Scenario
Regular
Few-Shot
Method/Metries
P
R
F1
P
R
Fl
FL-DLinear
80.21
75.88
78.32
72.56
73.75
72.98
FL-LLighTS
84.30
81.74
82.70
68.68
70.24
69.56
FL-PatchTTT
80.65
81.36
80.91
74.23
70.06
72.29
FL- Transformer
76.45
78.11
77.18
68.97
65.34
67.02
FL Transfommee
81.76
69.53
75.07
67.32
70.74
69.02
FL-Informer
78.07
78.38
78.04
68.09
68.84
68.15
FL-Reformer
74.47
84.26
79.04
72.36
77.46
74.61
FL-Pyraformer
77.48
80.23
78.86
66.78
70.90
68.83
FL-GPT4TS
87.37
83.85
85.32
78.48
80.22
79.32
LM- AVV
89.99
89.20
89.55
86,24
84.33
85.10
(Ours)
90.49
95.45
92.98
88.37
87.47
87.64
31

---

Appendix E Full Experiment Results
In this section, we provide the full experimental results not included in the main manuscript. This includes the main experiments (Appendix I.) few-shot learning experiments (Appendix [..)), and ablation studies (Appendix[E.3) F.3), as well as extra analysis of our framework, covering hyperparameter sensitivity (Appendix F..4 and its performance with different PLMs.
E.1
Full Main Results
In this section, we show detailed and full experimental results including:
Forecasting (Tab. 28 and imputation (Tab. across different scenes and settings on the ODWIT
dataset.
Forecasting (Tab. 30 and imputation (Tab. across different scenes and settings on the ODWIV dataset.
Forecasting (Tab. and imputation (Tab. across different scenes and settings on the ODW2T dataset.
Forecasting (Tab.
34 and imputation (Tab. 35 across different scenarios and settings on the ODW2V
dataset.
Note that we only show the comparison between the proposed LM- WEATHER and the time series-specific baseline in the full experimental results. Our LM WEATHER outperforms specialized time-series analysis models on on-device weather datasets across various environments. Unlike these models, our method doesn't 't require training from scratch but only minor adjustments to small number of parameters. This validates the effectiveness and superiority of our proposed framework in on-device weather modeling practice.
Table 28: Comparison of the performance of LM- WEATHER and baselines on the ODWIT under forecasting tasks, Bold: the best, Underline: the second best.
Methel
1.M
Waucrass we
LM
FL 2C DC.ewir
R. PowatST
| [R.Traade oM
FL. TTToeddon
Ft. Re.leeeerr
Vasaaia
XCAB
B335
MA4
E34SE
1MAE
1442
BMSEE
MAB
R34SE
MAAE
RMSE
MAA
R3562
3442.
RMSE
|MAE
150
11.9
15.4
34.4
36.5
75.5
91.5
193
12.1
11.5
43.4
361
46.1
34.
44.
12.3
41.5
312
51.3
43.0
56.1
TespsTRa::
730
30.
48.0
37.
45.
16.
1143
[3.3
800
623
534
460
11.5
4700
43:
56.1
$1.5
64.1
51.5
660
53.3
310
20.3
136
917
192
559
138
18.3
13.
701
$1.
74.3
Haevity
720
13.0
218
73.1
106
13.0
07.2
23.4
18.
751
13.3
13.0
$1.
65.
182
36.5
$5.9
11.
08A
Agg
517
544
01.
51.1
145
06.5
$6.5
13.5
15.
12.5
76.
96
05
63.4
18.5
657
569
103
155
143
V1.1
53.5
18.3
72.5
25.1
738
51.3
63.4
010
51.5
1026
798
182.
Wiltgedl
336
201
03.4
622
857
56.5
188
16.1
1313
11.
053
360
560
153
193
11.
00.2
111
1960
51.0
1343
Agg
967
022
669
853
1385
17.
136.
11.
635
263
47.
746
633
112
60.
1656
585
1859
278
37.
22.0
67.6
365
31A
450
23.0
28.3
400
12.
564
720
Teeperaxee
256
365
311
42.4
479
19.0
16.5
562
750
31.3
43.3
435
438
43.5
57A
124
15.
342
43.4
153
31.
450
451
68.1
581
75.
73
190
463
464
136
25.1
18.3
18.
900
133
81.
47.3
361
21.
14.
125
985
1211
540
236
846
A1
730
113
ST8
P82
785
17.1
100
624
06.1
533
533
53.5
855
66.1
53
1344
554
17.5
463
574
Goem
28
Table 29: Comparison of the performance of the proposed method and the baseline method on the ODWIT under the imputation task, where bold indicates the optimal results and underline indicates the sub-optimal results.
Mathodd
1.M. WILCHER-AV
J.M-WAACHER
HL-Ritvez::
FL-DLiser
IL-PREIST
PL-4ighT'
[L-Tmnaaiemer
P1-bitener
Lengls
KMSEE
KMSE
RMSE
MAE
RMBE
MAE
MAE
MAE
XMSE
MAE
FNSE
RMSE
MAE
06
33.%
39.5
36.3
28.6
300
515
16.3
56.0
23.6
43.8
35.1
45.9
21.0
-0.3
20.5
11.0
56.3
$2.6
-0.2
60.1
302
38.5
30-7
33.0
212
16.3
21.4
41:8
22.6
791
10.1
306
733
13.5
35.0
4310
21.5
39.5
09.
93.1
63.1
18.3
213
412
6511
107.5
27.7
47.6
23.7
42.1
61.
16.0
71.1
$6.6
(3.0)
Arg
355
38.1
36.3
71.
402
23.4
457
23.0
43.2
22.0
82.4
32.5
96
11.0
41.
33.1
205
69
052
67.5
00.5
35.0
15.6
76.
K3.6
932
(71
11.9
15.6
47.
31.7
71.5
08.7
892
33.0
35.5
25.5
25.5
25.6
43.3
23.6
41.0
12.7
42.4
3955
730
11.9
33.2
70.6
77
75.6
06.5
742
15.2
10.3
711
11.
15.1
31.0
113
62.3
10.1
711
38.5
Arp.
31.00
31.9
31.4
95.4
10.5
35.4
21.9
43.5
23.0
42.8
15.3
50.0
96
22.6
$3.5
31.7
41.5
63.
$3.4
62.2
85.9
20.3
$0.6
25.9
54.5
22.5
46.5
24.4
43.7
56.3
$2.5
70.5
39.
143
41.
13.6
$3.0
01.
05.5
10.1
34.1
11.5
17.3
31%
16.1
15.5
12.4
06.3
366
36.5
739
NLB
35.1
34,9
43.3
77.0
95.
75.8
05.9
28.0
49.0
56.6
85.1
365
56
27.2
47.4
56.5
20.4
71.
96.7
Arg
411
3311
12.4
015
6601
19.7
15.4
776
462
36.1
45.7
10.3
614
29
0
0
32

---

Table 30: Comparison of the performance of LM- -.--EATTERR and baselines on the ODWIV under the long-term forecasting task. Bold: the best, Underline: the second best.
Nathed
J LM--EATHES-ATE
LME-WETTEER
PL-ULisar
PL-lizaaioms
FL-bioma
FL-kdfinsr
Vasilee
Letgh
KMSE
MAB
XM466
BMSR
BMSE
BM56
RMSE
33.9
36.3
031-1
190
16.3
91.5
56.7
0:4
18.9
14:1
100
17.3
191.
83.1
87%
Trepecuzn
196
76:
31.6
48.7
11.
16.5
913
569
4733
16.
18.9
116
66.8
1915
87.
093.
87%
[01
Agg
14.3
13.5
18.3
14.3
16.3
68.6
41.5
$3.3
52.4
473
196
48.3
14.0
55.3
17.3
100.
87.
001 5
85.6
19.0
75.3
190
43.4
13.6
45.4
11.3
18.4
16.1
56.3
68.4
(TM
76.
15.3
72.1
15.0
75.0
18.
70.3
08.0
00.0
01.2
$3.8
Al1
720
31.2
48.7
34.0
54.3
52.3
61.3
52.3
50.3
75.1
12.1
15.2
54.1
82.7
16.2
51.3
58.4
91.5
Agg
16.8
110
74.
61.
1911
76-1
71.
850
537
76.7
71
011
43.
1" Ooik
0
Table 31: Comparison of the performance of LM-WEATHER and baselines on the ODWIV dataset under the imputation task. Bold: the best, Underline: the second best.
Mathedd
J.M.
LLM. M.WWcO
Ht-PeceSST
HL.Ligh1
IL draeeteeee
11-bsiemer
HS-Hatommer
B1-ifyralemer
Longts
RM5SE
MAE
RMSE
RASE
MAA
RMIE
NAA
KMSE
MAE
KNSSE
MAE
JMMG
MMAE
RASE
MAA
RMSE
NAA
KNSSE
0g
36.7
56.5
35.0
$5.4
37.5
53.4
16.2
45.6
31.5
30.1
45.2
58.6
36.5
53,3
36.7
55.6
14.5
52.3
33.6
59.3
142
14.3
09.3
31.9
56.2
41.3
63.6
$3.4
51.3
13.2
0013
53.5
54.8
55.4
38.9
$9.3
54.7
41.1
256
730
18.5
06.1
43.4
04.1
16.3
04.3
12:3
66.1
51.
76.0
58.
81.3
65.5
66.3
48.3
65.3
18.4
66.9
50.1
11.4
Arg.
43.5
09.1
402
01.1
645
55.5
42.1
63.9
52.1
400
51.2
41.3
635
40.5
38.9
44.5
96
18.1
58.5
17.5
57.
997
613
10.3
56.3
35.9
55
18.3
31.6
56.1
55.3
18.8
58.3
86.9
51.6
36.5
53.8
092
42.2
01.2
53.
64.
42.5
$3.6
43.1
06.5
53.0
46.3
00.7
33.1
32.4
36.1
35.3
48.4
730
$3.1
45.1
13.1
63.5
68.0
68.
16.
70.1
53.
77.0
93.1
$1.3
601
76.5
29.5
63.0
50.4
09.0
55.5
14.3
Ave.
03.9
39.7
01.4
04.7
4739
50.3
43.3
07.5
34.5
46.3
66.
42.9
03.5
42.3
61.8
45.4
53.7
16
42.1
62.0
4.11
03.4
43.5
64.5
423
55.0
43.0
63.0
53.6
33.1
56.7
56.2
41.5
61.5
57.3
56.3
41.1
59.2
147
13.9
01.5
13.3
07.
15.0
67.6
16.7
19.1
71.3
57.5
19.1
10.4
61.0
0311
18.1
55.
18.
306
730
17.5
16.1
66.9
73.0
162
75.5
90.8
101
63.
61.6
13.5
33.9
53.3
7H1
611
132
Avg
315
853
43.7
616
69.
62.1
91.
73.9
59.
52.1
714
158
65.5
15.4
63.1
911
67.
Table 32: Comparison of the performance of SEM-EEATERR and baselines on the ODW2T under the long-term forecasting task. Bold: the best, Underline: the second best.
TELEIGHTS
Mesto
Rats
MAE
MAA
NAAE
BASSE
SAAR
NAAI:
HNNSE
MAA
R0456
|MABL
EMSSE
MAB
RM5SE:
MAAE
|AA
BARSE
18.3
$1.8
1813
117.0
90.5
115.5
-11
$3.5
53.3
67.5
41.3
51.3
11.5
56.1
-9.7
33.5
941
71.4
222
65.3
35.3
44.6
56.9
07.3
119.3
16.9
116.3
-6.2
25.11
45.0
00.5
46.2
-5.1
45.6
39.4
61.5
13.2
01.5
13.5
Teeqursee
35.3
6912
34.6
68.1
135.5
130.3
122.3
146.1
15.1
13.0
56.2
21.4
55.1
72.5
07:3
13.2
09.1
$2.7
Ang
$2.4
16.8
06.3
164
136.8
105.7
1107
63.7
51.0
05.1
15.7
69.3
03.1
67.0
15.5
67
71.3
66.2
$3.1
85.4
105.0
55.1
105.1
67.5
54.7
30.2
85.1
08.5
96.5
65.6
53.4
85.3
105.0
$4.5
002
91.3
1103
16.7
107.3
18.1
M127
71.1
10.0
11.9
18.0
110.0
$4.0
1,0
Handdry
720
11.5
85.2
90.
114.3
06.00
112.2
78.2
52.5
15.1
05.3
12.3
11.5
16.2
13.3
56.1
106.1
23.4
125
16
66.3
$6.1
53.0
$2.5
1005
135.
15.11
121.5
47.5
30.2
19.1
68.6
96.5
08.2
$5.4
10.0
100.0
96.1
102
336
1933
03.1
63.9
91.1
164.2
1995
102.
126.5
73.0
193
73.0
91.
11.8
30.5
33.7
90.5
82.5
100.5
$4.
102.5
107
946
1995
1342
10-1
191.4
03.9
10.3
72.9
91.3
76.7
1961
45.4
103.
17 Cress
14
Table 33: Comparison of the performance of the proposed method and the baseline method on the ODW2T under the imputation task. Bold: the best, Underline: the second best.
Motial
PLANE-ave
PLANS
FL-Reiieeer
FL. FL-Pardionne
ELPPALTST
RL-iTuasfomse
FL-LighTS
Fl-Iaferecr
Ratio
Length
MIAE
RMSE
MALE
RAGE
MAL
KMSE
MAE
RASI
MAE
R0ASE
BMSS
MAEE
RASSE
MAE
RMSE
MAA
RMSS
XMAE
RMSSE
201.
45.9
48.3
58.5
$5.4
121
3411
52.6
41.5
61.9
155.
51.8
13.4
30.2
$131
21.9
-2.A
62.2
192
50.5
56.9
35.)
13.2
$1.2
71.1
56.7
132.0
31.5
54.7
46.5
16.4
37.5
56.5
13.5
$1.8
62.7
83.6
43.9
G4.0
25%
7.0
06.1
05.5
$3.5
3922
78.5
101.2
126
41.2
631
75.1
16.1
42.5
55.5
43.1
0211
57.3
17.2
41.
0LE
Arg
41.2
01.3
40.3
500
53.4
7330
99.1
125.6
36.99
$7.5
32.9
78.1
41.5
39.5
18.2
36.1
56-7
72.0
42.5
62.2
192
3013
352
16.3
53.5
51.4
11.4
95.2
110.4
31.1
36.5
30.0
71.
37.6
57.0
15.0
54.3
55.0
75.0
47.1
07.6
336
41.9
60.3
40.5
55.5
53.5
34.0
55.0
124.5
39.4
63.7
65.7
50.0
43.2
52.2
433
63.4
56.1
75.5
46.1
66.9
19.
59.2
53.4
132
34.1
59.1
60.4
06.1
96
35.0
16.6
36.
14.9
50.3
10.5
55.4
120.
40.5
00.0
38.6
58.6
39.1
58.3
15.8
57.8
65.5
85.6
51.7
72.0
116
45.5
63.5
35A
75.3
125.5
45.11
06.7
1811
59.5
41.5
65.5
15.3
730
47.
06.1
165
66.7
66.3
177
59.1
179.
53.5
76.1
20.4
95.5
09:3
11.0
45.0
68.0
67.4
89.3
51.5
13.0
Arg
11.0
61.
55.
63.8
67.4
1d CunzA
0
20
0
3
0
E.2 Full Few-Shot Learning Experiments
In this section, we show detailed and full few-shot learning experimental results including:
33

---

Table 34: Comparison of the performance of LLM M-EEHREEER and the baseline method on the ODW2V under the forecasting task. Bold: the best, Underline: the second best.
Mitho
FLANS-
F1ANE
El-RRttemmr
It.PPofomer
T1-18 tsor
11- T1-PachIST
[1-STbentormar
IE-LighTS
[E. TE-Toneforrur
El-trremeer
24.3
07.5
66.3
$4.6
58.5
1060
85.7
100.5
15.2
04.5
53.2
90.5
14.3
92.2
00.3
110.2
6331
101.5
07.5
115.4
376
50.0
110.5
18.5
05.3
91.8
65.0
156.5
109.2
$1.3
101.
136.7
122.9
85.0
101.0
98.5
117.4
720
85.00
10033
72.1
99.5
973
114-2
98.5
111.5
04.5
168
18.3
203
59.2
112.7
194.5
122.5
91.4
105.3
19.3
110/4
Agg.
96
76.8
09.7
66.1
38.4
50.5
115.7
50.1
112.5
74,3
96.
76.3
99.5
12.5
62.3
02.3
117.
77.0
100.
77.4
100.4
192
72.9
11413
91.4
16.3
19-4
12811
78.6
003.3
1021
100L
1263
763
101.
18.0
AI
7231
1940
115.5
72.9
16.5
V7A
12164
191.5
125
1MA
3030
100
16.2
200.2
15.2
112.7
113.1
141.3
86.1
112.3
61.3
108.6
100.0
A0g.
101.
92.3
6239
3156
96.1
1200
9933
80.2
160.3
9912
196.4
1. Couas
0
0
Table 35: Comparison of the performance of M-WEEATHER and the baseline method on the ODW2V under the imputation task, Bold: the best, Underline: the second best.
Methel
[.M. Wiccows--t2 12
[LM WINCTIK
T1
T1-Sotemer:
[T. TT-Pytteemrr
IE-DCJser
TE-PweRIST
IC-ITrachmmer
EI-EighTS
TT. [t-Toreveerrr
Length
06
23.1
45.6
21.5
44.1
43.9
63.2
42:1
35,4
63.4
83.8
35.1
53.5
55.5
78.5
91.2
112.5
45.3
66.5
45.2
56.5
115
15.6
38.3
45.2
68.7
18.3
05.9
31.5
14.5
76.2
106.3
06.0
(7.0
720
14
35.1
13:2
46.7
51.4
76.2
44.3
65.4
71.4
71.7
53.5
13.2
70.5
103.7
93.5
126
45.4
67.5
45.2
55.0
4pg
26.1
20.9
16
26.1
40.5
27.5
34.0
96.8
00.11
64.5
62.0
66.3
35.6
36.7
56.7
66.2
16.3
13.5
1314
46.1
66.3
82.
70%
326
323
53K
12.4
10.4
72.8
66.9
75.11
302.1
1063
19.1
065
527
720
391
48.0
$7.5
56.5
79.6
47.8
6900
73.4
03.6
55.5
75.4
21.5
103.9
991
125.
51.
71.
51.0
10.7
Arg
31.3
70.
65.1
70.5
45.1
74.2
9665
09.
50.1
96
16.1
15.3
27.5
44.0
93
30.5
13.9
13.4
77
67.0
16.8
16.4
73.5
46.1
173.5
63.3
75.
515
76.9
44.0
51.0
71.1
46.1
75.3
79.1
116
65.3
47.6
50.9
156.1
101.
75.
586
720
3711
351
36,0
51.5
39.4
81.7
$2.4
100.
72.3
97.3
55.2
15.3
83.0
305.1
95.5
128.1
154
77.5
55.4
15.6
Arg
1 Cres
30
0
0
0
0
Forecasting (Table. for 5% training data, Table 37 for 15% training data) and imputation (Table. for 5% training data, Table. 33 for 15% training data) across different scenes and settings on the ODWIT dataset.
Forecasting (Table. 40 for 5% training data, Table, for 15% training data) and imputation (Table. for 5% training data, Table. 43 for 15% training data) across different scenes and settings on the ODWIV dataset.
Forecasting (Table. 44 for 5% training data, Table, 15% training data) and imputation (Table. for 5% training data, Table. 47 for 15% training data) across different scenes and settings on the ODW2T dataset.
Forecasting (Table. for 5% training data, Table, 49 for 15% training data) and imputation (Table. for 5% training data, Table. (51 for 15% training data) across different scenarios and settings on the ODW2V dataset.
Table 36: Comparison of the performance of ME-WEETHER with the baseline method on the ODWIT dataset under the long-term forecasting task in a scenario where the proportion of training data is set to be 5% in the shw-shot learning. Bold: the best, Underline: the second best, "_" denotes insufficient training data.
Miched
LEN-EEATEE-WWE
LAS-WEATHER
FL-Dkeee
M--igh
Pi-Trenleree
Lingl
MAG
M145
HMSF
MAF
SEAF
RMASE
MAS
MAA
MAG
RMSF
MAB
MAA
MMA
RMSSE
152
12.3
105.5
95.1
99.2
3014
33000
301,0
132.2
(PL1
125-2
13.5
112.1
100.5
1380
311.5
376.7
1260
199.3
107.5
132.3
TeapaRas:
722
AMg
1063
143
108.0
761
3555
1561
195
176
05.6
533
91.1
31.0
100.5
1153
111.5
137.0
57%
106
Shaukby
136
101.1
396
115.5
194.
1663
116.
116-4
112.0
98.5
111.
A66
06.5
113.3
55.2
104.0
357.3
1265
105.1
125.4
10.6
10555
144
1065
972
109.9
117.
142.5
10.3
129.5
18.5
105.9
19
1953
15.1
1250
115
103
1163
125.
15.2
12035
1115
928
1950
1765
431
135
413
1124
1160
100.
1996
3387
62.1
130.
118.
Wod/spol
733
136
924
114.7
1053
1176
106
16.2
241
152
12.5
127
99.8
185.
2127
1164
1140
1545
15.3
135.1
15.1
1154
Setlee Toegennar
122
16.1
1023
961
1966
2002
182.
213.
11..0
7911
121
196.5
123.4
16.2
122.2
Aag
96.
897
47.
10.9
312
106
87/
113
1413
79666
176
99.
116
96
65.
133
115
156.
3600
173.
208
00.4
1975
85.3
1535
1033
3015
185.
136.5
033
183.6
91.1
1900
116
823
04.
1084
197.5
1776
100
05.0
1453
1954
106
7116
1881
13/66
1002
355
66
1000
122
917
95.3
28.9
175.7
300.7
98.7
1984
106.3
1973
333
55.5
199.0
199.
34

---

Experimental results indicate that our LM- WEATHER significantly outperforms the baseline in resource- constrained situations, such as few-shot learning environments with limited training data. This suggests that L.M-WEATHER effectively leverages PLMs for sequential data modeling and achieves commendable performance without requiring extensive data for training.
Table 37: Comparison of the performance of MMM-EATIIER with baselines on the ODWIT under the forecasting task in scenario where the proportion of training data is set to be 15% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Mathed
120 WLoTHER--W
LM
Referma:
FL-Pyssarmrr
TL-OChue
PL-POUUTT
Te-TTrasSrrarr
FT,Cigho
Verule
Leggl
[ MA5
M4A5
IMMSF
KCAF
BMSF
MAR
RMASF
MAG
$NSSF
MAA
MAP
RMISF
|MAR
RMSSF
MAA
51.8
70.5
52.3
66.6
315-
166.4
191.1
36.4
53.3
03.5
17.5
3/111
170.3
131.4
IG2.
152
90.1
25.5
51.4
11.5
192.
216-7
196-7
206-7
77.3
01-3
R0.
76.7
4912
01.0
15..6
22233
[18-)
191.2
14.3
114.2
Teaperzus
128
90.3
$6.3
68.3
18.5
120
3410
2367
243
06.2
116.7
83.1
104.1
10.6
115.7
236
277.6
110.6
137.3
10877
132.6
Ara
7502
633
885
6333
69-7
$6.0
106.0
163
09-3
87.4
112.3
11.0
00.5
122.8
05.3
111.3
Shaskdy
730
152
68.
11.
07.5
112.9
651
117.4
85.1
1067
13.
912
91.9
166.6
133.8
1367
18.
128
91.3
1102
Ang
95.3
70.1
125
875
15.5
1159
54.
113.0
52.
102.1
12.7
71.6
51.5
100
1012
1254
91.0
1254
18.5
106-
96
19.1
103h
714
196.7
115.1
56.1
15.5
1003
523
1123
01.2
12:57
15.
135.5
57.3
112
81.8
1153
56.
16.1
16.1
1911
Wixlspoo
720
82.3
111.0
113.3
65.6
881
111
10:
60.
1154
996
1151
1315
061
332-5
1318
61.3
$83
115.8
134
56.2
28.1
115.
152
53.4
833
64.3
196
1645
915
06.9
47.
362
1913
155
125.3
20.6
115.2
Serbue Teapurzau
05.3
1710
1410
22.0
670
120
513
182
1855
560
$6.2
1727
13.5
133.3
16.2
112.5
552
46.
645
1405
313
013
605
36.3
1351
125.2
96
56.4
99.4
513
97.
118
1750
125.
182.
76.3
0965
53.3
1863
06.4
106.0
1144
171.6
63.3
111.3
(8.1
106.1
116
66.6
1017
101.5
1305
170
1386
1855
12.3
113.5
66.1
186
770
138
1313
196.7
13.3
134.3
71.3
117.4
1044
943
19211
1750
12.5
185.4
101
756
1164
1913
724
134.5
112-2
Table 38: Comparison of the performance of AM-WEATHER with the baseline on the ODWIT under the imputation task in scenario where the proportion of training data is set to be 5% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Ratio
Lengis
MAE
KM5S
MMAB
335E
MAE
R3GE
| 3AAE
BOSSE
MAA
KNISE
MAX
RMISE:
MAR
X35E
MAA
INSE
MAA
BMSSE
MAA
KNISE
0g
111
342.2
227.0
95.5
155.1
348.0
133.8
253.5
142
63.1
111.3
57.6
$12.1
343.0
318.7
145.6
345.1
175.4
66.5
$48.4
196.5
1971
176.
3166
545.1
355.7
141.1
256.5
256
720
Agg
14.1
113.4
55.1
1186
13477
315.5
1746
335.5
K0.
175.0
63.1
141.7
100.7
198.3
175.8
515.
135.5
3599
137.
251.0
192
614
1763
68.9
13..3
142.6
347.1
141
234.5
97.0
184.3
15.4
1558
110.7
307.7
174.7
307
143.5
355.6
144.5
257.9
3556
739
Arg
61.1
123.8
99.6
323
34.90
346.
1415
233.
10.4
186.3
66.1
1463
11877
208/9
174.5
3114
335M
2541)
143.1
2573
1100
143
63%
136.
62.5
136.1
1515
767.8
157
7681
110.1
155.
178
353.0
175.3
301-1
599
763.4
196.
7603
50%
716
Agg
413
126.2
62.8
134.8
199.
764.6
1908
767.5
1100
206.3
66.7
151.0
1380
335.7
172.6
3661
3450
363.
11T1 3614
Table 39: Comparison of the performance of the LM- MM-WEATHEER with the baseline the ODWIT under the imputation task in a scenario where the proportion of training data is set to be 15% in the sho-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Metuool
LM MM-BUCIAK
Fl-Sateerer
FC-DLaser
Il-Tnzavmerr
FL-booreer
Batn
Tergit
MAF
RMS.
MAE
EMSF
MAF
RMS
MAF
RMSE
MAR
RMSF
MAB
RMSR
MAF
RMSE.
MMA
RMSF
MAF
RMSF
MAP
RMIS
042
23:31
015
61.3
72.4
126.4
15.4
132.1
65.3
11.4
21.5
113.6
117.5
12.6
134.1
21.2
115.6
335
33.31
11.4
41.0
08.6
21.2
150.2
190
126.0
07.5
100.5
43.7
13.8
26.5
115.0
121.1
151.5
12.5
113.0
72.5
113.4
Arg
013
36.7
06.1
795
123.
13.9
1252
06.0
93.1
43.2
11.8
72.6
1155
1167
176.0
13.4
113.4
75.
114.5
892
3530
075
62.5
125
18.1
122.4
313
56.3
12.1
138.2
115.
198.
16.3
135.9
26.1
166
336
41.1
73.7
43.9
70.4
1584
50.5
135
73.
1063
45.8
76.4
00.6
121.2
121.0
151
14.3
1181
75.1
1160
Arg
425
11.9
08.4
123.4
134.9
44.3
77.6
133.7
135.0
76.1
117.4
53.6
73.0
35.7
043
08.3
1967
195
116.3
19.00
111.3
1197
76.4
196.1
56.
335
3330
75.
46.5
75.9
96
136.2
14.0
130.2
91.5
115.
45.7
51.0
87.5
131.1
1204
170.
17.5
133.0
79.1
121.0
Arg
115
150
44.
72.9
135.2
84.5
125.9
793
117.5
41.3
83.4
125.9
111.5
196.3
139.9
90.3
122.5
Chezk
35

---

Table 40: Comparison of the performance of LM- LM-WEATHER with the baseline method on the ODWIV under the long-term forecasting task in a scenario where the proportion of training data set be 5% in the few-shot learning. Bold: the best, Underline: the second best, "_" denotes insufficient training data.
WE
LM-WEATTER RMSSE
PLeadISTT
FL- FL-Innades
Merios
Leugh
| MAR
KMS
MAR
MAR
RMAEE
MAF
RMSE
MAF.
MAT
RMSSF
MAT
RKSST
KSK
RMSSE
MAT
RXSSE
MAA
56
16.1
25.1 111
71.1
91.8
14.0
108.
15.6
06.5
01.5
102.0
500.3
365,3
125.0
135.4
182.5
150.0
100.0
128.5
103.5
122.0
77.3
05.6
08.1
K3-5
112.3
131.0
1096
172-3
131.
105.5
188.8
172.3
111.5
1912
H672
Teepezzz
720
196
205
135
1500
App.
56
20.
100.5
16.1
98.1
101.5
190
81.
10.5
98.5
121.4
521H
392.4
1350
195.3
1110
141.6
1166
135.
111.3
1429
236
103
135.
96.7
135.4
115.
112.4
106.1
137.5
131.
1532
3436
413.
122.1
155.5
136-5
196.7
133.6
161.3
135.3
1574
730
65.6
1166
136.4
02.6
1233
1097
1927
134.5
-03.3
134.2
167.
118.5
1963
1215
1961
1169
1519
Table 41: Comparison of the performance of LM- WEATHER with baselines on the ODWIV under the forecasting task in scenario where the proportion of training data is set to be 15% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
WEATHER
FL.DUeee
EILPAANSST
PL.1ighTT
Rl-lebener
P1.Pyefomer
Mitics
MBAE
BASSE
MMA
BMSE
|MAE
RNSSE
MAA
331SE
|MAE
R35SE
MAAE
BINSE
MAA
X34SE
XAAE
BNSE
SMAE
X34SE
KIAE
BASSE
53.5
67.5
91.3
65.5
15.4
03.2
53.7
06.5
19..3
17.5
141.2
170.7
$3.5
325.1
75.0
$5.3
78.7
312.11
101.
105
192
391
15.3
97.1
14.1
77.3
$1.5
63.3
15.3
6912
$5.5
184.3
220.5
55.4
118.6
85.8
106.2
$7.2
113.3
00.5
114.3
Terermaarz
730
11.6
101.
79.3
58.6
06.2
116.3
83:3
194.7
65.5
115
235.3
278.3
100.5
123.5
104.0
128.3
106.2
133.9
1884
142.0
Aeg.
64.
95.01
02.1
78.6
81.0
112
057
36.1
75.5
95.1
193.
2017
4300
104.4
677
116
65.6
117%
V5.4
121.3
96
$5.0
13.5
54.5
71.3
70.7
101.2
55.0
75.8
66.1
153.4
185.
895
104.5
51.6
192.5
85.0
106-
56.1
113.0
333
01.3
88.7
196.2
08.8
16.2
97.0
297.4
112.4
135-1
115.9
943
06.3
78.4
720
183
1003
85.8
107.4
101-3
127.3
93.6
154.1
103.
130,
253.0
340.5
186.
134.6
5123
1485
1169
1535
119.5
154.
Agg
1 Oous
0
28
0
0
0
0
0
0
Table 42: Comparison of the performance of LM- WEATHER with baselines on the ODW1V under the imputation task in scenario where the proportion of training data is set to be 5% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Metibod
LM WEE
LM
WELCHER
EL-DLIBOE
FL-PachIST
[L-Tinndomm
FL-Ligh15
FL-Thadoee
FL-bomee::
FL-Pyrakenne
Length
RM5E
MAE
R3456
RMSE
RMSE
MAE
RM55
MAA
MA5
RM56
RMSE
MAE
RMSE
MAE
96
51.3
123.3
59.5
1
85.4
149.1
F8.3
131.
91.6
163.
107.6
709-4
58.5
178.1
336
725
Arg.
1636
06
$3.4
118.3
615
139.8
00.9
111.0
37.1
135.3
1008
1900
1623
2853
111.1
2.15
113.4
2152
91.3
183.5
$7.6
180L3
335
735
Arg.
125.5
1024
188.6
162.5
114.6
212.4
$3.9
96
62.2
11.11
63.8
135.5
163.9
158.3
61.5
132.6
113.5
308 4
1611
281.5
117.6
2.955
1196
255.5
96.0
198.5
543
188.1
335
730
Avp.
1" Coes1
14
0
0
0
Table 43: Comparison of the performance of LM-W WEATHER with baselines on the ODWIV under the imputation task in scenario where the proportion of training data is set to be 15% in the sew-shot learning. Bold: the best, Underline: the second best.
Methed
LM.
WILCCHER--AEE ATE
LM,
FL-Diisce
FL-PackIST
FL-TTnaaome:::
FL-LISTS
FL-Toraleemer
FL-hskener
Fi-Refema
FL-Pyrakener
Length
RMS5
MAB
RMSE
EMSE
RMSE
MAA
RM55
MAE
MAS
R3456
RMSE
MAE
RMSE
MAE
96
23.1
56.7
28.7
53.4
41:2
10:4
09.1
107.1
16:11
66.
115.3
17.5
51.%
255
116
11.B
18.6
07.6
60.1
45.4
17.1
T1.3
248
180.
115
1304
667
134
66.5
107.4
66.1
112.1
$1.
$1.5
723
33.1
01.
91.4
95.1
53.2
83.3
14.8
180.2
108.1
101.
62.1
61.9
64.5
33.1
Ave.
96
31.8
$8.8
31.4
27.6
65.
60.2
33:1
50.5
67.5
15595
106.
167.
68.9
113.7
73.0
117.5
753
22.5
57.5
53.6
3566
335
31.0
11.1
43.3
70.0
68.
101.5
44.1
12.6
76.3
15500
1143
173.
07.6
158
69.5
112.2
12.3
113.0
53.6
$7.2
735
$1.
$3.3
12.0
66.
161.
$5.0
20.0
133.4
107.
15.1
130
65.2
100.5
07.4
382
57.6
42.1
Arg.
96
31.
14.1
61.4
75.
111.
63.7
76.3
1310
112.1
1703
72.8
116.5
78.
136.5
81.
1757
67.
H005
315
37.1
151
46.0
34.1
112.5
43.2
15.9
91.3
185.9
114.3
172.0
12.4
134
15.5
116.5
15.4
1346
584
02.5
750
$5.5
26.2
19:0
812
108.
$7.4
9911
195.9
100.
1969
11.0
138
71.
188
73.4
1155
643
102.
Avg.
J Chal
30
0
0
0
0
0
36

---

Table 44: Comparison of the performance of LM- LAM-WEATHER with baselines on the ODW2T under the forecasting task in a scenario where the proportion of training data is set be 5% in the few-shot learning. Bold: the best, Underline: the second best, " denotes insufficient training data.
LM WEATHER-WE
LM--WATTER
Vistade
Leugh
| MAE
KMSE
MAR
RAGE
MAR
RMSE
MAB
RMSB
MAE
KMIE
MAE
RMSE
MAE
KMBE
MAE
RMSE
MAR
MAE
RMSE
56
100.
136.1
923
1163
2565
338.
2513
3166
96.3
133.9
$5.4
106.
153.
1367
3634
3115
1013
1352
2014
3750
196
779
Avg.
1669
1917
56
1182
134.1
12311
1364
1530
158.
106.5
138.7
W6.7
1280
1333
1326
1837
1943
1314
1975
Harkity
296
1162
194.1
22A
1130
1256
153.
1923
1589
196.
125.7
967
1380
133.
135
1630
1153
1950
1434
1314
1975
192
ABI
730
Aeg:
1110
155.
90.
156.5
138.3
241
1753
247.
002.1
150.8
11.1
1472
1159
166.5
183.5
279
1423
1936
1983
291
Cooes
Table 45: Comparison of the performance of LM- WEATHER with baselines on the ODW2T under the forecasting task in scenario where the proportion of training data is set to be 15% in the few-shot learning. Bold: the best, Underline: the second best, "_ . denotes insufficient training data.
Mithed
J LAS WEATHE---E LM--WAAHER
FL
H. HL-Psramma
FL-DLiar
IL-Faciss
PL-thasomms
FL-L-GTTS
TL-Traatoma
FL-Bitema
Yortible
MAT
RMSF
MAAE
RMSSE
M4M
2545E
MAB
MAE
RM45
[MAE:
XIAC
RM450
[MAB
MAR
R3660
MAR
66.3
192
23.1
23.3
98.1
236.3
225.0
347
16.1
106.7
94.5
100.0
256.2
282
121.3
144.
2019
253
Teapesauo
596
91.0
183.0
96.7
106.7
255.3
3555
236.2
355.3
$6.1
116.7
89.5
106.2
25.3
111.7
250.2
355
135.2
193.
20011
270
Ang.
22.4
03.8
76.7
01.1
25677
357.0
235.
345.6
K75
106.5
76.1
97.4
79.5
1003
237.0
3773
117.
342.7
242.5
2559
100
1003
111.8
1964
11601
1051
132
195.
128/2
18.0
111.4
0612
118.
12.6
111.4
116.3
3442
105.3
126.4
1150
134.6
72)
Ang.
96.3
111.5
93.0
1996
109.
330
107.2
129.5
16.2
113.5
912
119.2
113.6
1169
342.4
112.1
123.7
112.5
1344
15
$6.1
126.
$2.5
123.0
142.2
162.0
146.3
208-7
07.4
125.1
55.5
123.7
91.3
152.0
155.1
3225
112.5
142.5
1964
2204
102
133.4
06.4
135.1
161.
123.5
1520
178.3
1969
756
711
720
122.0
235
1" Dxus
2
14
8
Table 46: Comparison of the performance of LM- WEATHER with the baseline on the ODW2T under the imputation task in a scenario where the proportion of training data is set to be 5% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Methool
WEATHER--AE
LM WEATHER
FL-PACTST
|R-Thandonns
FL-L
Fl-iaoen::
FL-Pyrakenssr RL-efanne
Rati
Tergh
MAR
RMSE
MAP
RMSE
MAR
RMSE
MAR
RMSF
MAR
RMSE
MAP
RMSS
MAE.
RMSE
MAR
RMSE
MAR
RMSE
MAP
RMSE
092
2555
736
Agg
93.2
10.4
91.8
1382
96.7
151.2
1152
1076
1016
173A
3605
1157
175.2
131.5
185.
1113
1761
948
197.
193
1566
336
Arg
95.6
147.2
94.
111
199.
141
1163
151.3
112
1605
173.
250
119.
1908
12600
199.5
1159
1763
95.0
151.7
1961
192
405
116
Agg
102.5
1963
99.8
153.
196.2
163.3
1769
1656
1337
1763
175
2567
1933
1936
133.8
300
1313
198
1057
1610
Table 47: Comparison of the performance of LM-WEATEER with the baselines on the ODW2T under the imputation task in scenario where the proportion of training data is set be 15% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
96.
32.5
105.1
31.5
51.1
115.1
B5.1
115.7
76.7
181.8
137.7
305.5
30.9
1411
06.4
106 7
85.6
131.9
76.8
122.8
30,5
T0K3
13.7
106.
165
130
175.01
130.1
150.7
V12
145-4
07.0
145
196.1
71.0
122.2
25%
720
Avg.
14.1
180.4
73.2
100.7
37.0
119
04.4
131.4
90.3
128.1
140.5
221.5
92.9
145.0
56.5
148. 3
87.1
135.7
75.0
122.0
121-7
126.8
192
112
16.6
101.
04.6
120.3
100.5
138.4
99.5
141.0
151.5
234-3
06.2
147.5
101.2
154.9
50.4
142.0
19.3
125.5
35%
336
50.2
19.2
114.3
97.6
133
104.4
142
107.2
150.2
156.2
234.3
99.5
151.0
102.6
155.2
92.2
141.5
79.2
123.8
720
27.4
127.5
134.2
125.3
96
14.8
102-3
7411
1ULT
10000
135.1
105.5
147.7
101.2
147.1
145.4
234.9
96.1
14177
105-4
161
M2.2
143.1
$5.0
1331
78.1
37.2
112.6
105.8
145.3
11237
155.7
18077
1893
152.
224-2
101.
154.4
108.5
165
141.5
83.4
1919
50%
730
Ngg
7900
112.8
78.6
111.3
104.
143.8
113.3
154.0
109.2
156.0
151.8
223.2
101.5
154.2
107.9
164.3
95.9
147.3
83.7
131.1
37

---

Table 48: Comparison of the performance of LM- LEATWEATIER with the baseline on the ODW2V under forecasting tasks in a scenario where the proportion of training data is set to be 5% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Methed
WEATHER-ANT
EM-WRATHER
RuDioxe
R.PxxATST
PL-Toorsoma
Flolghts
Fl-laferer
Vasiabi
Longts
MAE
RASE
MAE
K04SE
MAE
BNSE
MAE
KMSE
MAE
3MSE
XAE
RNSSE]
MAE
KMBE
MAE
RASE
MAE
KM1E
XAE
RNSSE
06
105.
155.5
967
134.
1907
1552
1901
1558
1096
136.7
1367
1410
1133
136.7
1463
1763
1183
144.
1284
1533
Pariiity
130
Are.
180,
135.5
967
124.
1907
1582
1901
1558
1996
136.
1167
1410
1133
1957
1463
1783
1183
1284
9o
1853
1357
96.3
123.1
1513
1967
1863
186.
1122
1412
1355
148.
1962
143A
1621
2123
1964
1758
5496
1961
176
730
Clase
Table 49: Comparison of the performance of LM- WEATHER with baselines on the ODW2V under the long-term forecasting task in a scenario where proportion of training data is set to be 15% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
[1-Pyratenee
11-11pht5
Vastali
Length
MEAE
HNSS:
|MAA
13452
3UAE
34AE
MAA
334SE:
XIA
INNSE:
MAA
K31SE
S4AE
HNSSE
SMAE
3345E
KAA
BASSE
$5.0
16.3
1462
1:2.1
182
201
111.3
23.3
106.2
103.4
130.3
165.5
126.7
91.4
114.4
330.2
110.
$5.7
111.5
113.1
[48.
105.
112.5
1:3.2
134.6
1haviday
120
Agg
01.3
110.
87.3
165-3
104.
196.
105.1
176.7
40.3
111.8
W-5
196.
91.1
310.7
1969
1:3.5
105.
178.9
1:39
1914
16
20.0
106.4
75.6
105.
3354
126.
162.3
995
$18.9
75.5
196
15.7
334
176.
1667
342.1
154.6
171.3
27.8
116.
53.5
186.
3317
131.9
187
K2A
02.7
122.1
141.0
187-3
M554
341.8
134.
AB
130
Arg
85.5
116.
91.
130.
90.9
522
92.2
135.7
141.3
1962
1073
134.
1d Coeoe
13
0
0
0
0
0
Table 50: Comparison of the performance of LM- WEATHER with baselines on the ODW2V under the imputation task in a scenario where the proportion of training data is set to be 5% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Mathhad
1.M- Waoewa--de
LM WH
IU-DUsor
T1-PweeIST
[1
F1-2oveeer
Il-Hetomer
[1-Pyratemerr
Rate
Length
MAE
RMBE
MAE
AMBE
MAE
RMSE
MAA
MAR
AMSE
MAA
RMSE
MAE
RMSE
MAE
R94SE
MAE
IMSE
MAA
KNSE
$2.m
02.9
39.7
00.5
117.
84.9
92.
123.6
155.1
200.
92.3
1252
125.2
504
122.
$3.0
1159
142
776
723
43.9
34.7
005
1178
$1.9
963
138
1961
92.5
135
138.
904
133.4
$7.0
115.9
65.0
38.5
01.1
123.5
2531
18.3
1042
335.7
1347
2013
96.7
129.1
135.
1527
125.6
55.1
192
11b
Arg
41.5
A10
38.5
01.
123,5
18A1
18.9
104.
136.7
1547
201.8
95.7
1201
M59
135.1
125.6
561
117.1
62.9
35.7
112.1
10688
155.5
103
95.3
1133
340.0
153.
1015
136.
106.1
14222
100.1
1346
915
192
116
725
03.9
35.7
113.1
1068
135.5
104
05.5
1159
1080
155,
101.
136.
1120
1916
1190
Czez1
Table 51: Comparison of the performance of LM- WEATHER with baselines on the ODW2V under the imputation task a scenario where the proportion of training data is set to be 15% in the few-shot learning. Bold: the best, Underline: the second best, denotes insufficient training data.
Iannn
Longte
MAB
MAB
RMSI
MAL
MAA.
R3350
MIA
R315I
MI1I
RMISE
MAL:
RMSSI
MAA
RMSE
MAL.
MAI
RMISE
143
36.7
56.3
34.3
63.4
43.9
106.7
17.5
68.1
861
114.0
134.8
179.
77.4
106.9
33.5
144.5
14.6
103.5
61.9
96.5
2556
052
48.0
31.1
11302
19.3
$3.9
96.3
125.5
1443
191.
12.7
133.5
$4.0
115.3
17.3
187.6
71.5
98.7
735
Are.
13.2
36.9
35.9
55.1
13.0
107.4
48.3
71.3
545
112.5
135.1
179.0
16.3
106.1
91.
122.3
1341
194.1
00.3
$2.6
14.5
122.4
143
18.9
$7.0
15.4
56.1
105
114.1
19.5
T1.5
09.3
131.8
155A
1764
794
506
31.
116.
16.6
106
04.1
96.1
3855
716
25.5
11.2
63.5
116.5
13.1
56.4
1000
132.1
114.1
1866
56.0
117.35
87.0
138.5
1077
111.5
71.1
Ang
24:11
99.1
37.5
56.1
10.0
115.3
11.5
72.4
98.6
135
1360
179.
31.0
$13.4
91.3
116.1
17.3
105.9
35.1
115.1
192
31.6
57.
36.7
96.6
07.3
1369
53.9
75.4
101.
133.5
156.
170.3
86.3
115.1
00
134.7
606
110.7
83.6
112.5
5055
116
8XT
41.4
64.4
10077
128
$7.4
M67
107.5
34044
1427
187.5
90.2
125.5
128/
16.2
1142
91.
1061
Arg
23.5
20.0
37.6
88.9
97.0
134.2
55.7
17.7
100.3
133.1
136.
179.7
857
117.0
90.6
124.3
61.5
112.0
82.9
111.1
38

---

E.3 Full Ablation Experiments
In this subsection, we show the results of the complete ablation experiment, both in the forecasting (Table. 52 and imputation (Table.53)
Table 52: Ablation experimental (forecasting) results for both the model composition level and the personalization mechanism level are included, where f represent the degree of performance increase relative the original AM-WEATHER, & represent the degree of performance degradation. and the Comm. Param# represents the number of parameters transferred between client and server communication for the different variants. Bold: the best, Underline: the second best.
Deigital
Vedel Congziiie Preepetiie
Preweelive Pregeetie
LM Wasssso
LM Wicmmne A
EM Weerore B [LM Wesrons
[LM Waomee DJ
[XX Wo.c0e 5.
EM Wiccmie F
L. Waurnss G
1TM Wevress RI
Vaciiin
MEAE
R166
1MAE
3356
14AE
RAGE
MMAE
R366T
MAB
R146.
XAA
R366
|MAE
392
31.4
36.5
14.3
42
11.3
40.2
13.6
13.2
13.0
18.3
35.0
45.8
35.
40.3
30.0
13.5
27.0
12.6
Tarpestez
130
17.3
45.4
43.3
$3.9
4133
10.8
46.7
533
18.3
18.2
44.3
57.8
40.5
56.3
35.1
43.0
13.8
43.6
4677
01.3
0401
78.3
75.1
123
24.1
233
15.1
119
192
51.3
15.5
13.6
75.0
01.7
06.3
11.3
66.
Baskiiy
396
7611
13.5
152
8113
61.0
18.3
60.0
15.1
16.T
072
03.1
96.3
9-.8
$2.0
(8.
Ang.
56.1
06.1
56.6
18.5
(6.0
161
51.4
(65
06
55.1
960
13.8
907
11.3
12.6
71.5
64.
70.8
E2.
76.1
108.
76.
02.6
0622
118
63.9
B1A
Wad sped
136
523
15.
16.
1803
06
74.
12.4
74.
64C
510
63
067
633
454
135
Ang
56.9
85.3
15.
1863
66.5
13.4
65.
133
63.5
56.
852
12.8
66.
124
65.1
K3.1
22.3
17.
430
42.3
31.
24.3
342
36.3
309
43.0
310
63.8
31.
62.5
36.1
361
Tmppesker
136
212
310
16.1
30
26.5
303
21.
4311
351
419
31.0
16.1
21.
192
61.4
254
43.4
06.5
14.3
28.5
12.3
46.
11.4
16.1
463
784
184
ABI
131
011
78.5
12.5
140
14.6
15.2
176
13.2
310
10L3
35.
122
866
143
Armg
43.5
513
51.
720
10.8
30.3
34
1539
20.5
23.2
58
Chirge
13 13.908.
10 - 11.061
14116
1.400
11.08
5.15
6%%
18
9.99
196
4.951
6.06 1 6.961
Goeel Pare
6.6M
0.8M
0.8.1
M
0.3.M
531
10/0
11.90M
1600
Table 53: Ablation experimental results (imputation) for both the model composition level and the personalization mechanism level are included, where f represent the degree of performance increase relative the original LM-WEATHER, + represent the degree of performance degradation, and the Comm. Param# represents the number of parameters transferred between client and server communication for the different variants. Bold: the best, Underline: the second best.
Orginnl
Medel Comsponiion Perpective
Persenakedd Perspetive
Methal
1.M Waoreo
| L.M WeSTern
L.MoWre C
FM WEAMOR--
1.M WesTeen F
M.-Boreeredd G
[M WaAmuss II
Maticc
Lenghh
RASS
MAE
MAB
KMSE
MAE
RNSE
RMSE
BASSE
96
N3
18.6
3.5
43.1
23
12.3
31.2
41.8
21.0
41.5
23.0
13.7
25.1
47.3
15.4
35.0
19.3
35.
25%
356
730
21.0
21.5
18.5
18.4
335
212
66.
43.0
23.5
23
43.6
31.9
12.2
42.1
15.4
21.5
41.2
43
31.
23.2
13.4
42.5
36.
47.9
46.5
206
20.2
36.1
17.1
36.1
35.
16.1
A08.
76
21:3
15.5
23.4
66.5
23.2
46.2
22.4
43.4
22.0
430
21.4
64.2
26.3
2111
20.3
35.5
21,1
165
136
39
14.5
316
317
13.4
35/2
43.
35.0
07
31.5
11.8
760
(5.1
113
36.1
115
720
24.2
428
23.7
44.2
21.
22.4
32.0
14.4
13.0
192
12
47.
345
36.0
45.6
23.9
4500
56.9
16.5
51.4
113
36.
16.9
50%
526
23.2
43.4
25.3
40.3
35.6
471
35.3
46.2
24.5
454
35.7
43.0
26.4
52.
15.3
39.1
22.4
15.5
43.
25.0
47.7
25.4
15.0
24.4
45.
15.5
47.0
52.0
13.8
39.1
15.3
Ausuaga
23.6
24.5
31.2
5307 1 135 L [409 4
11.55 4 386
5.6 1 4.9.
835 1 1008 1 11.8% J 16.9 21.00
4.8.7
1.89 1
4.7% 1
Coner. Parze.
0.3.
036.1
1000
E.4 Hyper-parameter Sensitivity
The impacts of rank on performance are detailed in Table. 54 As the rank goes up, there's a consistent improement, reaching its best at 8. However, when r 12, there's drop in performance. This happens because a higher rank means the local model has more trainable parameters, which can improve performance empirically. While higher rank can cause increased communication cost and introduce more uncertainty.
E.5 Pre-trained Language Model Variants
We compare three representative PLM backbones with varying capacities, the result is shown in 15 Under the proposed LM- ME-WEEATHR framework, it's evident that various PLM backbones maintain strong sequence modeling capabilities. Moreover, the lightweight personalized adapter in LM- MM-WEATHER enhance the PLM's
39

---

Table 54: Results parameter impact study, where Length refers to the length of weather sequences (that is, predicted horizons forecasting and input sequence length imputation). Avg. represents the average value of predicted horizons, encompassing {96, 192, 336, 720).
Forecasting
Imputation
Param.#
Rank
Length
MAE
RMSE
MAE
RMSE
Trainable
Comm.
2
Avg.
47.6
79.8
25.1
46.6
10.14 M
0.12 M
4
Avg.
46.5
78.4
24.2
45.1
10.25 M
0.24 M
6
Avg.
46.5
76.9
24.0
44.6
10.37 M
0.35 M
12
Avg.
45.9
76.1
23.4
43.7
10.72 M
0.70 M
8 (Ori.)
Avg.
45.4
74.6
23.1
42.4
10.49 M
0.47 M
Table 55: Performance statistics for the proposed LM- WEATHER with various PLM backbones are presented, recording only the average performance across all lengths for different datasets (namely, 96/192/336/700 prediction horizons). For the imputation task, results are documented solely for random masking probability of 50%. Bold: the best, Underline: the second best.
Variant
Dataset
Forecasting
Imputation (50%)
ODWIT 45.4/74.6
23.1/42.4
ODWIV
45.6/71.9
43.7/63.8
LM- LE-WEATTIER (GPT2, Original)
ODW2T
66.9/90.1
38.8/61.7
ODW2V
69.0/92.3
31.1/47.0
ODWIT
49.3/82.9
25.0/47.2
ODWIV
49.9/80.0
48.3/71.2
LM- MM-WEATHER (Bert, 5)
ODW2T
73.6/100.4
42.4/68.7
ODW2V
76.2/102.9
34.5/52.6
ODWIT
47.2/77.6
24.0/44.5
ODWIV
47.2/74.2
45.7/67.2
LM- WEATHER (Llama, 4)
ODW2T
69.5/94.4
40.5/65.0
ODW2V
72.3/97.6
32.4/49.8
ability to transfer knowledge from natural language sequences to complex weather sequences. This further validates the superiority and versatility of our L.M WEATHE.
Appendix F
Additional Statements
F.1
Impact Statements
We highlight that the goal of this study proposed LM-WEATHER is not to compete but instead to complement current on-device meteorological variable modeling framework. Today's 's climate foundation models typically trained from scratch, utilizing excoptionally large datascts (ncarly 100TB) and incurring substantial computational costs. We hope that LM- WEATHER offers a cost- cost-effective alternative for modeling metcorological variables on-device, thereby enabling accurate regional weather trend analysis. In addition, the dataset we complied can be the important resource to provide exploring chances for this field, facilitating future research.
This research seeks make on-device meteorological variable modeling more efficient and adaptable. By using PI.M foundation model instead of training large foundation models from scratch, it eliminates the need for large-scale real weather and extensive computational resources. Additionally, supports variety of devices, enabling everything from advanced smartphones to basic loT sensors to perform meteorological variable modeling. The method is also designed to be stable in environments with limited data and those outside of typical distribution ranges, providing credible analytical support for further weather trend analyses.
40

---

F.2 Limitations
Although our LM-WEATHER significantly outperforms models trained from scratch for time series analysis various tasks and scenarios with minimal parameter tweaks, still faces primary limitations:
Limited Dataset Scale: Due to constraints on computational resources and operational costs, we evaluated the performance of LM- WEATHER using the real-world datasets that did not approach the scale of tens of terabytes often required for training large-scale meteorological models. This limitation does not affect LM-W WEATHER be extended general framework for regional weather trend analysis. This framework supports the analysis of on-device metcorological variables and can be further developed and adapted for additional applications.
Dependence PLMs' Quality and Performance: Although LM- IE-WAATHHR leverages PLMs that the
achieve high efficiency and customization heterogeneous devices, this dependency quality and performance LM- WWATHER are intrinsically tied to the underlying PL.Ms. Should there be inherent limitations or biases within the PLMs, these could translate to the meteorological modeling performance. Conversely, if conditions allow the use of more powerful LLM, LM- WEATHER's 's performance can be significantly improved. This might give the community opportunities to explore the future road-map.
E3 Future Works
In future work, we aim to broaden the use of SM-WEATTER across more on-device variable modeling appli- cations. We also plan incorporate additional types of data, including satellite and radar imagery, well as textual weather descriptions, to advance towards more generalized approach to on-device meteorological variable modeling.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made the main abstract and introduction has accurately reflected the
paper's s contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper has discussed the limitations of work.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper provide the full set of assumptions and complete proof.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
42

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper has fully disclosed all the information needed to reproduce the main experi- mental results.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.- with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer:
[Yes]
Justification:
This paper has provided access to the code, but due to the size of the dataset, additional releases required.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //tn://nipp cc/public/ oul.dess/Cooeeuubmiiissooooooo. for more details.
43

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: This paper has specified all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments in this paper were conducted under five different random seeds and averaged to from the final reprot.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: This paper has provided sufficient information on the computer resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
44

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics
Answer:
[Yes]
Justification: This research is conducted in the paper conform, in every respect, with the NeurlPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification: This paper has discussed both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
This paper has described the relevant safegurads adopted.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use usel-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
45

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper has noted the original owners of the assets used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-par re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces new datasets and has been described in detail.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
46

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
47


Reviews:
[{"review_id": "0U3sStMC3j", "reviewer": "Reviewer_Jjnz", "summary": "This paper introduces LM-WEATHER, a framework leveraging pre-trained language models (PLMs) for on-device meteorological variable modeling. The framework integrates personalized adapters into PLMs, enhancing their ability to handle heterogeneous weather data efficiently. Key contributions include superior performance in forecasting and imputation tasks, minimal communication overhead, and maintaining privacy during client-server interactions. Experiments on real-world datasets demonstrate LM-WEATHER's effectiveness across various scenarios, including few-shot and zero-shot learning.", "questions": "Q1.Could you provide more detailed proofs and theoretical validation for the decomposition rationality and privacy assurances mentioned in the theorems?\n\nQ2.Could you clarify the potential confusion in the notations n¬i and nk in Section 2.2, and ensure the formulas are correctly presented?\n\nQ3.Could you provide detailed information on the architecture and parameter configuration of the FFN used in the parameter adapter, and explain why this architecture was chosen?\n\nQ4.Could you provide more detailed information about the datasets, particularly whether data from extreme climatic regions(e.g., tropical rainforests and deserts) are included? If not, how do you plan to expand these datasets to cover more global climatic conditions?", "limitations": "1 The authors acknowledge the limitations in the diversity of the datasets and the potential challenges in modeling specific climatic conditions not covered in the current study. Future work should focus on expanding dataset coverage and providing more detailed theoretical and empirical validation. Additionally, the practical applications and limitations of few-shot and zero-shot learning capabilities should be more thoroughly explored and documented.\n\n2 It would be much better integrate the essential experimental details and results into the main body of the paper to improve readability and comprehension?\n\n3 I have questioned whether there will be a powerful foundation model for time series prediction. If the assumption is not true, the application of the proposed solution is limited.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. LM-WEATHER introduces a novel approach by integrating personalized adapters into PLMs, allowing for efficient and accurate on-device meteorological modeling.\n\nS2. The framework addresses heterogeneity in weather data and demonstrates robust performance in various tasks and scenarios.\n\nS3.The approach offers a promising solution for personalized weather modeling on resource-constrained devices, potentially benefiting various applications in meteorology and related fields.", "weaknesses": "W1.In the communication section, it is mentioned that only the low-rank matrix parameters are transmitted between the client and server, but specific details about the transmission mechanism and strategy are missing. This includes transmission frequency, bandwidth consumption, and potential issues in practical applications. These missing details may prevent readers from fully understanding the feasibility and effectiveness of the communication strategy.\n\nW2. In the parameter adapter section, it describes integrating the generated adapters with the original input through FFN to produce the final weather sequence predictions. However, the specific architecture of the FFN, how its parameters are configured, and why this particular architecture was chosen are not detailed. The lack of these key details might leave readers questioning the mechanism and effectiveness of the parameter adapter.\n\nW3. In Section 4, while Theorems 4.1 and 4.2 propose the rationality of time series decomposition and privacy assurance through low-rank matrix exchange. There is no introduction and description about the background and application of these Theorems.  Also, the descriptions are too brief, making it difficult for readers to fully understand the theoretical basis.\n\nW4. The paper's experimental results are heavily supplemented by a 37-page appendix, which can hinder readability and accessibility of the core content. Important experimental details and results should be better integrated into the main body of the paper to facilitate easier comprehension and evaluation by the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below ***(additional results are in Global Rebuttal PDF file)***:\n\n---\n**Response to Weaknesses and Question (& means joint response)**\n\n**W1. Details about the transmission strategy.**\n\nOur framework employs a synchronous federated learning strategy, where the client and server communicate at a fixed frequency (every 20 iterations). The process of transmission includes: \n* 1. client-side updating: each participant uploads low-rank parameters of their local model.\n* 2. server-side aggregation: the server aggregates uploaded parameters via federated averaging.\n* 3. low-rank parameter broadcasting: the server broadcasts aggregated parameters to each participant for the next round. \n\nBy communicating only low-rank parameters (0.38M parameters), we minimize bandwidth consumption. We acknowledge potential issues in practical applications, including communication overhead, latency, and data privacy. Our method addresses these concerns by transmitting only low-rank parameters, thereby safeguarding privacy and minimizing communication overhead. We will clarify this key aspect of the transmission strategy in the revision to enhance understanding.\n\n**W2 & Q3. Detailed information of the FFN in parameter adapter.**\n\nOur parameter adapter is a straightforward FFN with a single linear layer, converting the PLM's output *(input channels, e.g., 768 for GPT2, 4096 for Llama-3B)* to align prediction horizons of [96, 192, 336, 720] *(output channels)*. We chose this configuration to optimize the balance between performance and computational efficiency. Our additional experiments ***(Table 1, Global Author Rebuttal PDF file)*** show that adding more layers only marginally improves performance (0.21% with two or three layers) and even decreases with four layers, while significantly increasing computational demands. Therefore, the single-layer setup is the most cost-effective, providing an optimal trade-off between efficiency and performance.\n\n**W3 & Q1. Introduction and description of backgournd, and more detailed proofs and theoretical validation for theorems.**\n\nWe provide additional description and proofs for theorems, which can be found in **G1, Global Author Rebuttal**.\n\n**W4. Important details and results should be better integrated into the main body.**\n\nThank you for your constructive feedback. We will revise the layout to highlight the core results in the main body.\n\n**Q2. Potential confusion in the notations $n_i$ and $n_k$.**\n\nWe apologize for the confusion caused by our typo. The correct Eq. 2 is: $n_i$, $F(\\theta) := argmin \\sum_{i=1}^{N} \\frac{n_i}{n} F_i(\\theta_i; \\lbrace D_i \\rbrace)$, where $n_i$ represents the $i$-th client. We will carefully revise and clarify these notations to enhance the readability.\n\n**Q4. Does the dataset have detailed information on extreme climate regions, and if not, what are the plans for expanding the dataset?**\n\nIncluding more details about specific extreme climatic regions indeed benefits the expansion of our dataset's potential application scenarios. Current version comprises real observations from various cities, with limited access to detailed topographic or regional information. Recognizing the potential benefits of expanding this coverage, we are actively working to include a broader range of climatic conditions. Specifically, we plan to introduce multimodal weather data, such as radar echoes and satellite cloud images, into our dataset's corresponding regions. Additionally, by collaborating with climate experts, we will provide textual descriptions of weather trends to create a comprehensive multimodal dataset.\n\n-----------\n**Response to Limitations**\n\n**L1: Future work: expanding the dataset, deepening theoretical and empirical validation, and exploring the applications and limitations of few/shot-learning.**\n\nThank you for your insightful comments, which have provided a clear direction for our future work. **As the response for Q4**, we plan to introduce additional data modalities to enhance the comprehensiveness and applicability of current datasets. In future work, we will conduct more detailed theoretical and empirical analyses based on the newly constructed multimodal dataset. Additionally, we will further explore the limitations of these observations in practical applications and few/zero-shot scenarios.\n\n**L2: It would be much better integrate the essential experimental details and results into the main body.**\n\nWe will integrate essential details and results into the main body in the revision to enhance readability and adjust the layout accordingly.\n\n**L3: Whether there will be a powerful foundation model for time series prediction. If the assumption is not true, the application of the proposed solution is limited.**\n\nOur approach is indeed based on the assumption that there is already a good foundation model. We believe that this assumption is reasonable because there are already many well-established foundation models that can be used in time series forecasting tasks, e.g., MOMENT[1], Timer[2]. However, we also understand the your concern that without a powerful foundation model, our method may suffer. To address this concern, we used several PLMs that weak than GPT-2 (e.g., Bert [3], CTRL [4]) to evaluate the performance of LM-Weather, and the results ***(Table 2, Global Author Rebuttal PDF File)*** show that LM-Weather can achieve good performance when the assumption of a powerful foundation model is not hold, and significantly outperforms other regular model.\n\n---\nReference:\n1. Moment: A family of open time-series foundation models. arXiv 2024.\n2. Timer: Transformers for time series analysis at scale. arXiv 2024.\n3. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018.\n4. Ctrl: A conditional transformer language model for controllable generation. arXiv 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iT0gk2rgAX", "reviewer": "Reviewer_mBbn", "summary": "The paper aims to develop weather foundation models by leveraging the on-device data on many distributed sensors. Specifically, a federated learning and low-rank adaption mechanism have been applied to a time-series-based foundation model training on Meteorology data.", "questions": "1) Does the proposed method only focus on time series data?\n\n2) Would you please highlight the main research question in plain language? It seems there are many components mixed in the paper. \n\n3) In Section 3, what’s the federated aggregation mechanism? Is there any pseudo code to describe the algorithm?\n\n4) Can you please explain the token embedding for time series data? Is the token to be described as a special pattern of a subsequence of the time series? How to define the vocabulary of the tokens?", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1) The paper proposes a new way to develop weather foundation models by leveraging many distributed devices with collected Meteorology data. It is worth noting that weather foundation models are an emerging research domain critically important to geoscience and tackling climate change. \n\n2) The proposed solution is technique sounds. The procedure of time series processing, LoRA-based adaptation, and federated learning are seamlessly integrated to implement its goal. \n\n3) The paper provides sufficient details with an appendix and open-sourced codes. It is an essential part to ensure the reproducibility of this work. \n\n4) The paper conducted a comprehensive experiment for comparison and evaluation of the few/zero-shot learning scenarios on the foundation model.", "weaknesses": "1) The paper needs a strong justification to describe the motivation for using a pre-trained language model as a basis for the weather foundation model. Are there any texts in the Meteorology dataset? \n\n2) The paper’s contents organization could be improved. For example, the figure 1 is too complex to understand. In Eq 2, it would be better to write the function as F(theta, D). Because F(theta | D) is easily confusing with the conditional distribution. Moreover, some technique details should not be placed in the appendix. It is a usual assumption that the main paper is self-contained so that the readers can understand the paper without reading the appendix. \n\n3) In the contribution part (lines 73 - 94), the last two items look like the advantages of the proposed method rather than contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below:\n\n-------\n**Response to Weaknesses**\n\n**W1. Justification for using PLM and clarity on weather dataset.**\n\nAlthough our dataset consists solely of time series data without any textual content, using a PLM as the basis of the weather foundation model offers substantial flexibility and an optimal trade-off between performance and costs. Our motivations can be summarized by: \n* *Limitations of Existing Climate Foundation Models*: Current climate foundation models, such as ClimaX [1] (73B parameters) and Pangu-Weather [2] (up to 100B parameters), demand vast computational resources and extensive reanalysis data volumes (over 100 TB) for pre-training, making them unsuitable for deployment on low-resource meteorological equipment due to prohibitive operational costs and computational demands.\n* *Sequence Modeling Capabilities of PLMs*: Despite being initially trained on text, their advanced sequence modeling capabilities, proven on time series analysis [3]. This eliminates the need for training models from scratch with high-quality weather observational data, which is often scarce and expensive to acquire as opposed to reanalysis data. Consequently, adopting a PLM not only employs its powerful sequence modeling abilities but also drastically cuts down on computational expenses and deployment costs when fine-tuning for weather sequence modeling.\n\n**W2. The paper's organization and clarity can be improved, e.g., notation, figure, and placement of technical details.**\n\nThank you for your careful review and constructive comments. In our revision, we will recheck all notations in the paper to avoid confusion. Additionally, we will adjust the layout of the main text to incorporate important content from the appendix, enhancing readability. Your feedback will significantly improve the quality of our paper!\n\n**W3. The last two items in contribution part appear to be advantages rather than contributions.**\n\nWe will adjust the contribution part to improve clarity of expression.\n\n--------\n**Response to Questions**\n\n**Q1: Does the proposed method only focus on time series data?**\n\nYes, LM-Weather currently specializes in processing weather time series. Utilizing a PLM can bring significant benefits in terms of flexibility and cost-effectiveness, **as detailed in our response for W1**. To further enhance the generality and applicability of LM-Weather, we plan to expand our dataset to include multi-modal data such as descriptive weather texts, satellite cloud imagery, and radar data, ultimately developing a comprehensive multi-modal weather sequence analysis framework.\n\n**Q2. Highlight the main research question.**\n\nThe main research question can be summarized as: **How can we adapt pre-trained language models to efficiently and effectively model weather sequences on local devices in a decentralized manner, while addressing the challenges of data diversity and limited computing resources?** This question seeks to explore:\n* Adaptation: How can complex models originally designed for language processing be tailored to understand weather sequences?\n* Personalization: Can these models provide tailored weather predictions that suit the specific conditions and data available at individual locations and devices?\n* On-device efficiency: Can this approach operate effectively on local devices, ensuring privacy and minimizing resource use despite the constraints of limited computational power and storage?\n\n**Q3. Federated aggregation mechanism and pseudo code.**\n\nOur aggregation mechanism is based on FedAvg, where only low-rank parameters are shared and averaged, weighted by each client's sample size ratio. We provide the pseudo code below, which will be added to the revision for clarity.\n\n*Algorithm 1*. low-rank parameter-based aggregation mechanism (server-side, all clients participate in the training).\n|---|\n*Input*: Low-rank parameters from each client {$\\theta_{l,1}, \\theta_{l,2}, ..., \\theta_{l,k}$}, number of samples per client {$n_1, n_2, ..., n_k$}\n*Output*: Aggregated low-rank parameters $\\theta_{l}$\n\n---\n1. Receiving low-rank parameters from clients\n1. Calculate the total number of samples across clients $N = \\sum_{i=1}^k n_i$\n2. Aggregating low-rank parameters: $\\theta_{l}' = \\sum_{i=1}^k \\frac{n_i}{N} \\theta_{l, i}$\n3. Broadcasting updated $\\theta_{l}'$ to clients to continue next communication round.\n---\n\n**Q4. Explain the token embedding? Is the token to be described as a special pattern of a subsequence of the time series? How to define the vocabulary of the tokens?**\n\nThe token embedding module aims to transform subsequences of weather time series into a higher-dimensional space using a convolutional layer. Each transformed subsequence is represented as a continuous vector, known as a `token`. This process allows the model to capture localized patterns within a specific receptive field. Each token encapsulates patterns from a subsequence of the time series, capturing local dynamics. The kernel size determines the subsequence length, combining each point with its immediate neighbors to form a token. These tokens capture important temporal dependencies. Unlike traditional NLP, the `vocabulary` here is formed by the diverse outputs of the convolutional layer, representing a range of patterns in the input data. This vocabulary is continuously defined, shaped by the convolutional filters that learn to identify and abstract temporal features during training.\n\n---\n**Reference**:\n1. ClimaX: A foundation model for weather and climate. arXiv 2023.\n2. Accurate medium-range global weather forecasting with 3D neural networks. Nature 2023.\n3. \"Large language models for forecasting and anomaly detection: A systematic literature review. arXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "duVQFPxxhA", "reviewer": "Reviewer_tT2F", "summary": "This paper proposes LM-WEATHER, which builds upon previous work to explore further the powerful capabilities of PLMs in modelling meteorological variables. By learning sequence modelling capabilities from natural databases and applying them to on-device heterogeneity meteorological variable modelling, a lightweight adapter was developed to provide weather pattern awareness. Additionally, two real-world meteorological station datasets ODW1 & ODW2 support regional weather forecasting. Abundant experiments were designed to verify the feasibility of LM-WEATHER in modelling meteorological variables.\n\nThe article work is more valuable to further validate the superiority of PLMs in modelling meteorological variables and the provision of weather station data that can be used as a baseline dataset for weather station forecasting.", "questions": "1.\tIn the Zero-Shot Learning (Out of Distribution Modeling) experiments mentioned in line 282, the domain transfer experiments for regional forecasting on the OWD1 dataset showed that the transfer performance was not as good as the forecasting performance of GPT-4. Can you explain the reason for this?\n2.\tIn Theorem 4.2, how is it demonstrated that Low-Rank Matrices ensure privacy? Are there any related research papers on this topic? Can you provide a detailed explanation?\n3.\tCurrently, there are many sparse station forecasts. Can interpolation be performed based on sparse station data to further extend the model to spatial dimension forecasting and enhance its generalizability?\n4.\tLM-WEATHER seems to outperform current methods. Can it maintain stability in long-term forecasting, and what is the maximum forecasting time it can achieve?\n5.\tIn Table 6, line 305, it appears that having more trainable parameters is better. If the comparison experiment groups had the same number of trainable parameters, what would the results be?", "limitations": "Regarding limitation 1, further validating the impact of increased data scale on model performance using large-scale ERA5 data would be beneficial. Utilizing ERA5 as pre-training data to model invariance in meteorological forecasting and learning fundamental meteorological variable patterns, followed by fine-tuning with real-world data, could be a highly valuable endeavor.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.This work is more valuable to further validate the superiority of PLMs in modelling meteorological variables.\n2.Two real-world meteorological station datasets ODW1 & ODW2, which were collected and compiled from four real-world versatile datasets for on-device meteorological variable benchmark.\n3. Exploring the spatio-temporal sequence modeling of weather pattern specificity with high distributional similarity to provide a viable solution paradigm for sparse and heterogeneous weather station data.\n4. From the experimental results, this method can further enhance the possibility of maintaining confidence in its future development in the presence of unconstrained data and PLM models.", "weaknesses": "1.\tFigure 1 does not intuitively reflect the contributions of the paper. The Fig.1F mentioned in line 68 seems to be missing; additionally, the explanation for Figure C should be Task Adapter Generation, which is hard to understand. It is recommended to keep the labels a, b, … in the figure consistent with A, B, … in the text.\n2.\tThe paper discusses sequence modeling in the time dimension of stations. Can this method be validated for spatiotemporal modeling?\n3.\tThe experiments in the paper mainly focus on the accuracy of forecasting time. Can further validation be done on the timeliness and stability of the forecasts?\n4.\tTo better present the experimental data, some statistical charts can be drawn to visually demonstrate the advantages of LM-WEATHER in meteorological variable forecasting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful comments. We've carefully considered and addressed your concerns as follows ***(additional results are in Global Rebuttal PDF file)***:\n\n---\n**Response to Weaknesses**\n\n**W1. Some confusion about Figure 1.**\n\nWe will revise Fig. 1 to clarify the explanation of panel C and unify labels to improve overall clarity and readability, and correct the typo referencing Fig. 1F to Fig. 1a.\n\n**W2. Effectiveness of the method in spatio-temporal modeling.**\n\nTo further demonstrate the effectiveness of LM-Weather for spatio-temporal modeling, we conducted additional experiments with a variant, LM-Weather-ST, which aggregates site data for centralized modeling. We compared its performance to the original LM-Weather and classical spatio-temporal forecasting baselines (STGCN, ASTGCN, TCGN, and CLCRN). The results ***(Table 3, Global Author Rebuttal PDF file)*** show that LM-Weather-ST achieves optimal performance, while the original LM-Weather still significantly outperforms the baselines despite being trained in a decentralized manner. This suggests that LM-Weather has strong potential for spatio-temporal modeling tasks. Future work will explore incorporating server-side graphs to model relationships between devices, further enhancing LM-Weather's benefits for spatio-temporal modeling.\n\n**W3. Timeliness and stability of forecasts**\n\n* **Timeliness:** We provide inference times for various forecasting periods **(Table 4, Global Author Rebuttal PDF file)**. When combined with the communication time of 0.29 s, these results demonstrate that LM-Weather achieves efficient prediction times, ensuring timely delivery of forecasts.\n* **Stability:** All experiment have been conducted five times. We present the standard deviations of LM-Weather and the runner-up model ***(Table 5,  Global Author Rebuttal PDF file)***, which indicate that LM-Weather can maintain stable performance over multiple experiments. Furthermore, we assessed longer-term forecasting tasks (1080, 1440, 1800, and 2160 hours) with a fixed input length of 192 ***(Table 6, Global Author Rebuttal PDF file)***. LM-Weather not only achieves optimal performance but also exhibits relatively stable error variations compared to representative baselines, further validating its stability.\n\n**W4. It is recommended that statistical charts be added.**\n\nWe will add visualizations to each of the core experiment results in the revision.\n\n---\n**Response to Questions**\n\n**Q1.  Why region transfer performance on ODW1 is not as good as that of GPT4.**\n\nLM-Weather outperforms FL-GPT4TS in most zero-shot scenarios, but the specific case of OWD1T -> OWD2V highlights an interesting trade-off. The difference lies in the distinct local tuning and global communication strategies employed by each method. LM-Weather uses learnable low-rank parameters to balance generic sequence modeling with personalized regional insights, whereas FL-GPT4TS unfreezes LayerNorm layers for more direct adaptation of the PLM's knowledge. This allows FL-GPT4TS to update and communicate more parameters, potentially enabling greater knowledge sharing in modeling heterogeneous weather data, but at the cost of increased communication overhead. Additionally, significant non-iid between OWD1T and OWD2V contribute to this difference.\n\n**Q2 How the low rank matrix of Theorem 4.2 ensures privacy.**\n\nWe provide additional description and proofs for this, which can be found in ***G1, Global Author Rebuttal***.\n\n**Q3. Can interpolation based on sparse stations be used to extend the model to spatial dimension prediction and improve generalizability?**\n\nWhile spatial interpolation on sparse stations can increase the scale of training data, it is not an effective approach for extending the model's forecasting range and enhancing its generalizability due to:\n * Spatial interpolation of site data does not ensure the accuracy of the interpolated values, particularly when dealing with very sparse datasets like ODW2.\n * Even when interpolating between geographically proximate stations, the biases introduced by geographical differences cannot be fully compensated by interpolation alone, especially when multiple variables are involved. The complex interactions between variables and the influence of regional meteorological features render spatial interpolation ineffective for sparse station data.\n\nTo further validate our statement, we conducted experiments on OWD1T and OWD2T using three spatial interpolation algorithms (Kriging, IDW, Spline). The results ***(Tables 7&8, Global Author Rebuttal PDF file)*** show that spatial interpolation performs poorly in sparse stations and fails to improve performance.\n\n**Q4. Stability and maximum forecasting time.**\n\nRegarding stability, **please refer to our response for W3**. As for the maximum forecasting time, while it is theoretically possible to extend the forecast horizon indefinitely by increasing computational resources, when the input data length remains fixed, longer forecast periods introduce greater uncertainty, leading to decreased performance. **As discussed in W3**, we observed a significant decline in performance when extending the forecast period from 720 to 2160 hours. This degradation highlights that, although technically feasible, very long forecasting times may not be practically useful due to the increased uncertainty and reduced accuracy.\n\n**Q5. Results when parameters are the same in the ablation experiment.**\n\nWhile it's true that increasing trainable parameters often boosts performance, LM-Weather-G/H's 4.5% gain comes at a steep cost: a five-fold increase in parameters. This trade-off is suboptimal. Our goal is to strike a balance between performance and efficiency. Notably, when parameter of the experiment group are similar, the original LM-Weather consistently outperforms its variants ***(Table 9, Global Author Rebuttal PDF File)***, demonstrating its optimal trade-off between performance and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YfXEQSrfPS", "reviewer": "Reviewer_C54F", "summary": "This paper proposes an approach called LM-WEATHER that utilizes pre-trained language models (PLMs) as foundation models for on-device modeling of heterogeneous meteorological variables. LM-WEATHER enhances PLMs-equipped devices with local weather knowledge through a lightweight personalized adapter. Additionally, it leverages low-rank based transmission to fuse global knowledge among devices, enabling high-efficiency communication.This approach provides an effective solution for modeling real-world, heterogeneous, and continuously updated weather data, without requiring high resource demands. I believe this is a meaningful contribution to the field.", "questions": "This paper uses GPT-2 as the PLM, which is a very basic pre-trained language model. I am curious why more advanced PLMs like GPT-4 [1], Llama [2], and Vicuna [3] were not considered for better sequence modeling performance.\n\nReference: \n[1] Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.\n[2] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.\n[3] Chiang W L, Li Z, Lin Z, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper extensively compares various time series models across different meteorological variables and weather station.\n\nThe appendix section of the paper is comprehensive.", "weaknesses": "My first concern is that since the personalized adapter is already being used to adapt to the modeling of heterogeneous weather data collected from different weather station devices, what is the point of using the averaging operations during inter-device communication? This requires further explanation.\n\nThe definitions and formulations of symbols and equations throughout the sections, from Section 3.1 Local Training to Section 4 Theorems, are disorganized and complex. Additionally, there are numerous instances of incorrect singular and plural word usage. It is recommended to rewrite these sections with increased clarity and accuracy, simplifying the content for better comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below:\n\n---\n**Response to Weaknesses**\n\n**W1**. Further explain the significance of using the averaging operation.\n\nWe appreciate your concern and would like to clarify the importance of the averaging operation. Averaging is the most important operation in FL that knowledge interaction between clients, it serves two key purposes in our framework:\n* **Knowledge sharing:** By averaging learnable generic knowledge across devices, we prevent data silos and enable each device to benefit from the collective insights of the collaboration network, while maintaining local personalization and data privacy when sensitive data are involved.\n\n* **Trade-off between personalization and generalization**: The averaging operation allows devices to learn broader patterns from others, while preserving local adapter personalization. This interplay forms a robust framework for adapting to local characteristics and leveraging global knowledge, striking a balance between personalization and generalization.\n\nOur experiments (**Table 1 below**) show that LM-Weather with averaging outperforms LM-Weather-Local without it, highlighting the importance of averaging operation. We hope these explanations and additional experiments can address your concerns.\n\n*Table 1. Results of LM-Weather and LM-Weather-Local (no averaging operation, i.e., no communication only updating the respective models locally at each client) on forecasting, $\\downarrow$ is the average performance degradation relative to LM-Weather.*\n| ODW1T | LM-Weather | LM-Weather-Local | ODW1V | LM-Weather | LM-Weather-Local |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n| 96    | 42.3   | 45.9     | 96    | 42.3       | 44.5   |\n| 192   | 44.4  | 46.7 | 192   | 44.4       | 46.1  |\n| 336   | 45.8  | 48.2  | 336   | 46.0       | 48.8  |\n| 720   | 49.2   | 50.5  | 720   | 49.7       | 53.2  |\n| Avg.  | 45.4   | 47.8  | Avg.  | 45.6       | 48.2  |\n| **Disparity** | -   | **$\\downarrow$ 5.29%** | **Disparity** | -   | **$\\downarrow$ 5.70%** |\n\n**W2**. Recommended to rewrite Section 3.1-4  with increased clarity and accuracy, simplifying the content for better comprehension.\n\nWe appreciate your careful review and constructive suggestions. We will thoroughly revise these content to streamline the presentation of symbol and equations and simplify the content for better clarity and readability.\n\n---\n**Response to Questions**\n\n**Q.  This paper uses GPT-2 as the PLM, which is a very basic pre-trained language model. I am curious why more advanced PLMs like GPT-4, Llama, and Vicuna were not considered for better sequence modeling performance.**\n\nWe acknowledge the potential benefits of using more advanced PLM. However, Our primary goal is to establish a *general framework* for on-device weather sequence modeling via tuning PLMs. We chose GPT-2 for stability and cost-effectiveness, allowing for a basic yet effective demonstration of our approach. While more advanced PLMs may offer better performance, they require significant computational and storage resources, which can be a challenge for deployment on low-resource weather devices. To explore the LM-Weather performance with more advanced PLMs, we conducted additional evaluations using Llama-3B and Vicuna-7B, which show that more advanced PLMs can indeed improve performance **(Tables 2-5 below)**. Although GPT-4 was not included due to the unavailability of its weights (we are fine-tuning based on weights), our results demonstrate the potential benefits of using more advanced PLMs. Nevertheless, the increased computational demands of these models must be carefully considered for practical deployment. We hope these explanations and additional experiments can address your concerns.\n\n*The following results report the average forecasting (period [96, 192, 336, 720]) and imputation (50%) performance.*\n\n*Table 2. Performance of LM-Weather with different PLM backbone (ODW1T).*\n| PLM Backbone  | Forecasting  | Imputation   |\n|--|--|--|\n| GPT2 (default)| 45.4/74.6    | 23.1/42.4    |\n| LLaMA-3b (4 layers)| 47.2/77.6 | 24.0/44.5 |\n| LLaMA-3b (6 layers)  | 45.0/74.3    | 22.9/42.4    |\n| LLaMA-3b (8 layers)  | 44.0/73.5    | 22.4/42.5    |\n| Vicuna-7b (4 Layers)| 45.2/74.0| 23.0/41.9 |\n| Vicuna-7b (6 Layers) | 44.6/73.9    | 22.7/41.8    |\n| Vicuna-7b (8 Layers) | 43.3/72.8    | 22.0/41.1    |\n\n*Table 3. Performance of LM-Weather with different PLM backbone (ODW1V).*\n| PLM Backbone  | Forecasting  | Imputation   |\n|--|--|--|\n| GPT2 (default)  | 45.6/71.9    | 43.7/63.8    |\n| LLaMA-3b (4 layers)  | 47.2/74.2    | 45.7/67.2    |\n| LLaMA-3b (6 layers)  | 45.5/71.1    | 43.2/63.5    |\n| LLaMA-3b (8 layers)  | 44.4/70.6    | 42.1/62.0    |\n| Vicuna-7b (4 Layers) | 45.0/71.5    | 43.0/63.6    |\n| Vicuna-7b (6 Layers) | 44.3/70.0    | 41.9/61.4    |\n| Vicuna-7b (8 Layers) | 42.5/69.7    | 40.2/60.0    |\n\n*Table 4. Performance of LM-Weather with different PLM backbone (ODW2T).*\n| PLM Backbone  | Forecasting  | Imputation   |\n|--|--|--|\n| GPT2 (default) | 66.9/90.1    | 38.8/61.7    |\n| LLaMA-3b (4 layers) | 69.5/94.4  | 40.5/65.0   |\n| LLaMA-3b (6 layers)  | 67.0/89.2    | 38.2/61.3    |\n| LLaMA-3b (8 layers)  | 65.4/87.5    | 36.7/59.9    |\n| Vicuna-7b (4 Layers) | 66.1/89.2    | 36.5/60.6    |\n| Vicuna-7b (6 Layers) | 64.9/87.1    | 35.4/59.2    |\n| Vicuna-7b (8 Layers) | 62.9/85.0    | 33.6/58.1    |\n\n*Table 5. Performance of LM-Weather with different PLM backbone (ODW2V).*\n| PLM Backbone  | Forecasting  | Imputation   |\n|--|--|--|\n| GPT2 (default)  | 69.0/92.3    | 31.1/47.0    |\n|LLaMA-3b (4 layers)  | 72.3/97.6    | 32.4/49.8    |\n| LLaMA-3b (6 layers)  | 68.4/91.3    | 30.9/46.2    |\n| LLaMA-3b (8 layers)  | 65.4/89.5    | 29.6/45.1    |\n| Vicuna-7b (4 Layers) | 68.8/90.4    | 30.8/46.7    |\n| Vicuna-7b (6 Layers) | 65.6/87.4    | 29.4/45.3    |\n| Vicuna-7b (8 Layers) | 63.2/85.5    | 28.1/44.1    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. ***Additional experiments are included in the attached PDF file***, indexed as follows:
* *Reviewer Jjnz:* Table 1, Table 2
* *Reviewer mBbn:* No additional experiments supplements.
* *Reviewer tT2F:* Table 3 - Table 9
* *Reviewer C54F:* Additional experiments are provided in the box corresponding to Reviewer C54F.

Below are our general replies to the general concerns from the reviewers:

**G1. *(Reviewer Jjnz, Reviewer tT2F)* Detailed introduction, description, and proof for Therorems.**

* **Introduction and Background for Therorem 4.1:** Decomposing time series into trend and seasonal components helps model underlying patterns. This theorem states that non-orthogonal trend and seasonal components cannot be fully separated by orthogonal bases. Since self-attention layers learn orthogonal transformations similar to PCA, applying attention to raw time series is ineffective at separating non-orthogonal components, highlighting the need for manual decomposition.

* **Introduction and Background for Theorem 4.2:** The theorem is built upon federated learning, where an external attacker can exploit client parameters to compromise sensitive information. We demonstrate that LM-Weather ensures privacy by limiting uploaded parameters to a small, low-rank subset, making it challenging for an attacker to reconstruct client data. This guarantees the protection of sensitive information.

**Additional Proof for Theorem 4.1 - Self-attention Layer is a Nonlinear PCA:** The Self-attention can be viewed as a non-linear PCA, with the computation process represented as: $Attention(Q, K, V) = softmax(QK^T / \sqrt{d}) V$.  This can be decomposed into: $Attention(Q, K, V) = \sum_i \alpha_i v_i$, where $\alpha_i$ is the weight coefficient, and $v_i$ is the $i^{th}$ component of the Value. The weight coefficient $\alpha_i$ can be computed as: $\alpha_i = \frac{\exp\left(\frac{QK_i^T}{\sqrt{d}}\right)}{\sum_j \exp\left(\frac{QK_j^T}{\sqrt{d}}\right)}$. This is similar to the SVD of PCA: $X = U\Sigma V^T = \sum_i \sigma_i u_i v_i^T$. where $X$ is the input data, $U$ is the left singular vector, $\Sigma$ is the singular value matrix, $V$ is the right singular vector, $\sigma_i$ is the singular value, and $u_i$ and $v_i$ are the $i^{th}$ components of the left and right singular vectors, respectively. Comparing the two formulas reveals the similarity between Self-Attention and PCA. Combined with Theorem 1 in [1], this suggests that self-attention layer in PLMs functions closely to PCA (only learn orthogonal transformation) and can't automatically decompose time series into trend and seasonal components (non-orthogonal) without manual intervention.

**Additional Proof for Theorem 4.2 - Model Indistinguishability:** Consider two clients with different local models $\mathcal{M}_i$ and $\mathcal{M}_j$ parameterized by $\theta_i$ and $\theta_j$. Let $L(\mathcal{M})$ denote the low-rank matrix from model $\mathcal{M}$, where $B \in \mathbb{R}^{d\times r}$ and $A \in \mathbb{R}^{r\times d}$, with $r\ll d$. Then, for any polynomial-time attacker $\texttt{Adv}$, the following holds: $|Pr[\texttt{Adv}(L(\mathcal{M}_i)) = 1] - Pr[\texttt{Adv}(L(\mathcal{M}_j)) = 1]| \leq \epsilon$, where $\epsilon$ is a small positive number. This implies that the attacker cannot distinguish between client models from a shared low-rank update.

*Proof*. LoRA aims to find the low-rank approximation: $ min ||\theta - \theta_0 - BA||_F$, where $\theta_0$ is the initial weight. Consider two models $\mathcal{M}_i$ and $\mathcal{M}_j$ with weights differing by $\Delta\theta = \theta_i - \theta_j$ from different clients. The corresponding LoRA matrix is:

$$L_i = B_iA_i \approx \Delta \theta_i = \theta_i - \theta_0,  L_j = B_jA_j \approx \Delta \theta_j = \theta_j - \theta_0.$$
According to matrix approximation theory, for the best approximation with rank $r$, the upper bound on the error is: 

$||\Delta \theta_i - L_i||_ F  \leq \sigma_{r+1}(\Delta \theta_i)$ 

where $\sigma_{r+1}(\Delta \theta_i)$ is the $(r+1)$-st singular value of $\Delta \theta_i$. By Johnson-Lindenstrauss Lemma [2], for any $\epsilon > 0$, there exists a mapping $f: \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $k = O(log(n)/ \epsilon^2)$, such that for any $x,y \in \mathbb{R}^d$. LoRA can be regarded as such a mapping. Assuming $||\Delta \theta_i - \Delta \theta_j||_F \leq \delta$ and using the trigonometric inequality:

$$||L_i - L_j||_F \leq ||L_i - \Delta \theta_i||_F + ||\Delta \theta_i - \Delta \theta_j||_F + ||\Delta \theta_j - L_j||_F$$

$$ \leq \sigma_{r+1}(\Delta \theta_i) + \delta + \sigma_{r+1}(\Delta \theta_j).$$

Let $\varepsilon' = \sigma_{r+1}(\Delta \theta_i) + \sigma_{r+1}(\Delta \theta_j)$, we get:

$$||L_i - L_j||_F \leq \delta + \varepsilon'$$

For any polynomial-time attacker $\texttt{Adv}$, its ability to distinguish between $L_i$ and $L_j$ is restricted to the difference in their Frobenius paradigms. We can define a function $f$ such that:

$$\left|Pr[\texttt{Adv}(L(\mathcal{M}_i)) = 1] - Pr[\texttt{Adv}(L(\mathcal{M}_j)) = 1] \right| \leq f(||L_i - L_j||_F),$$

where $f$ is a monotonically increasing function representing the attacker's capability. Finally, we get: $|Pr[\texttt{Adv}(L(\mathcal{M}_i)) = 1] - Pr[\texttt{Adv}(L(\mathcal{M}_j)) = 1]| \leq \epsilon$. When $\delta$ and $\varepsilon'$ are small enough, the right-hand side is smaller than the intended $\varepsilon$, ensuring that an attacker cannot reverse-engineer local parameters and data, thus preserving privacy through low-rank parameters-only communication.

*We hope that the above additions will address the reviewers' concerns. We will add these additional introductions and proofs to the theorems in our paper during the revision to improve readability.*

---
Reference:
1. One fits all: Power general time series analysis by pretrained lm. NeruIPS 2023.
2. On variants of the Johnson–Lindenstrauss lemma. Random Structures & Algorithms 2008.

Meta-review: This paper proposes LM-Weather, which uses pre-trained LMs (PLMs) as foundation models for on-device meteorological variable modeling. LM-Weather integrates personalized adapters into PLMs, enhancing the ability to handle heterogenous weather data. Experiments are conducted on multiple real-world datasets showcasing the effectiveness of LM-Weather, outperforming SoTA results on forecasting and imputation tasks, and having low computation overhead.

Strengths:
Proposed LM-Weather is a novel approach that integrates personalized adapters into PLMs, enabling accurate on-device meteorological modeling. Sound experimentation on real-world data, results showcase robust performance of LM-Weather across various tasks & scenarios, as well as improvements over SoTA. Comparison of various time series models across different meteorological variables and weather stations.


Weaknesses:
Prior to the final submission, authors should address the feedback and questions of the reviewers, as well as clarify: 
(1)  transmission mechanisms and strategy of the low-rank matrix
(2)  motivation behind using FFN architecture, details of the parameter configuration
(3) expanding on the theoretical aspect 
(4) synthesizing the most important results and including them in the main part of the paper to help readers get the main contribution of the work right away
Decision: Accept (poster)