Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

Authors: shengchao  chen
guodong  long
jing  jiang
chengqi  zhang
personalized  adapter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 05:47:34

Abstract: This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variable modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.

Main paper: 
 introduction: 
 Accurately modeling weather variation pattern from large amount of meteorological variables sequences is increasingly vital for providing efficient weather analysis support for disaster warning. Recently, the promise of learning to understand weather pattern from data via deep learning (DL) has led to an ongoing paradigm shift apart from the long-established physics-based methods [1, 2] .Mining potential patterns from meteorological sequences that collected from different regions, including forecasting and imputation, is one of the most important problems in meteorology. Significant progress has been made by several latest time series approaches [1, 3, 4] . These approaches formulate meteorological variable modeling as an end-to-end spatio-temporal learning problem. This overlooks the reality that ground weather devices distributed globally gather vast amounts of data quickly. The sheer volume of data, coupled with limited network capacity, necessitates local processing on the devices, making centralised learning challenging [5] . On-device intelligence enables edge devices to compute independently, offering a primary solution to the problem. Federated Learning (FL) [7] is a promising on-device intelligence implementation that collaboratively train a uniform model across devices without exchanging raw data. However, the model often underperform due to data heterogeneity among clients. Personalized FL (PFL) provides new insights for on-device intelligence that allows each device obtains customized models for providing personalized Figure 1 : Framework Overview. (a) Schematic of LM-WEATHER, each client using personalized adapter to endow the PLM for local weather awareness, only low-rank matrices are transmitted to enhance efficiency during communication; (b) Brief structure of PLM on each client, detailed architecture can be found in Appendix; (c) Task Adapter Generation, the multivariate weather series input splits into two paths. The first path isolates the trend, seasonal, and residual elements, which each go through independent generator to produce specific adapters; (d) Architecture of the generator for each decomposed element; (e) Schematic diagram of Channel-Independent Patching [6] .insights [8, 9] . Albeit PFL methods showing revolutionized capability in this field, we argue that the current advancements are not necessarily at their best in on-device meteorological variable modeling as three major obstacles remain and hinder further progress:(i) Challenge of Heterogeneity. Weather data's heterogeneity, unlike that of images or text, arises mainly from the unique characteristics of data collected by weather devices in various regions, such as tropical or arid areas. Furthermore, sensor malfunctions or extreme events can lead to collection disruptions or inconsistent missing data, which significantly increase the differences in data distribution across devices. (ii) Underperformed Shallow Network Structures. The vast and varied data gathered by weather devices challenge simpler neural network models to generalize effectively. Furthermore, the frequent updates of weather data (hourly or by the minute) require neural models on devices to train and infer more often. This demand is hard to meet with deeper models that, while more performant, are also more resource-intensive. (iii) Resource-constrained Weather Devices. From a computation perspective, weather devices cannot afford of training complex neural models from scratch, especially for foundation models [4] . From a communication perspective, transmitting complete model during the 2 Preliminaries 
on-device meteorological variable (sequence) modeling: 
 The on-device meteorological variable (sequence) modeling challenge involves predicting future sequences from past observations for forecasting or predicting missing values for imputation on each device. While traditional physics-based approach this as a complex problem of solving multilevel atmospheric equations [14] , recent deep learning techniques have shown significant potential in uncovering patterns for better weather prediction [4, 2] .Problem Formulation On-device meteorological variable modeling can be formulated as an endto-end sequence-to-sequence learning problem for each device without exchange raw data. Formally, a parameterized local model for i-th device M i θ is tasked with predicting the weather sequence, M i θ :X i → Xi (1)where the X i ∈ R L×C and Xi ∈ R L ′ ×C ′ denote the input and output sequences on i-device, L and L ′ is the input length and output length, C and C ′ is the number of input and output variable. Note that the L ′ → L when performing imputation. The local learning objective on each device is to find the model parameter θ that minimize the distance between Xi and X i given sufficient weather sequence data. The overall optimization objective is based on FedAvg,EQUATIONwhere n i and n is the number of samples held by the i-th client and all clients 1 , respectively, F (θ|{D}) denotes the local objective function, {D} is the local data.Language models (LMs) trained on large-scale sequence data have shown extraordinary advances and led to a significant paradigm shift in NLP, boosting machines in understanding human languages (BERT/MLM-style) and synthesizing human-like text (GPT/CLM-style [15] ). Analogies between time series and human languages have long been noted [16] . Recent advancements in time series analysis have demonstrated the effectiveness of PLMs in modeling time series [17, 11] . Although some of those have shown that PLMs can beat time series-specific models in updating a minor fraction of parameters [18] . As such, it is exciting to expect cutting-edge techniques of language modeling can tackle weather variables sequence-related problems rather than considering train climate foundation models [4, 1] from scratch that are heavy and expensive, and are trained from simulated data.3 Taming PLMs for On-device Meteorological Variable (Sequence) ModelingOverview We proposed a generic framework named LM-WEATHER that encouraging PLMs to yield accurate prediction while keeping high efficiency for each device. The architecture is illustrated in Fig. 1 . To endow PLMs with weather pattern awareness, we introduce a lightweight personalized adapter into PLMs (e.g., GPT2 [15] ) such that the emergent ability of sequence modeling that transferred from text into weather is activated. To achieve cross-domain knowledge transfer with minimal effort while maintaining the sequence modeling capabilities of PLMs as intact as possible, we introduce lightweight operations in it enables both clients and servers to achieve a good trade-off between performance and efficiency (e.g., computation and communication). 
local training on each device: 
 Our LM-WEATHER refines PLMs for personalized weather sequence modeling on heterogeneous devices using a modular, plug-and-play architecture. Specifically, we introduce personalized adapter consists of (1) Task Adapter from latent weather knowledge and (2) Parameter Adapter that converts representation from the PLM into into weather forecasts. In addition, we employ lightweight operations in local training to boost computational efficiency.Task Adapter. To provide PLMs with richer effective information to activate their sequence modeling capabilities in the target knowledge domain, similar to text-based prompts in language to LLMs in NLP, we constructed task adapters by decomposing the input weather sequences into multimodal latent statistical information,EQUATIONwhere X k ∈ R L×1 denote the k-th variable in weather sequence X ∈ R L×C , the trend component X Trend and the seasonal component X Seasonal captures the underlying long-term weather pattern and encapsulates the repeating short-term weather cycles, respectively. Furthermore, the residual component X Residual represents the remainder of the sequence after the trend and seasonality have been extracted. Note that X Trend , X Seasonal , and X Residual have the same shape as X . This decomposition explicitly enables the identification of unusual observation and shifts in seasonal patterns or trends. The X Trend , X Seasonal , X Residual are used to generate Task Adapter via an unified generator as Fig. 1c & Fig. 1d that consisting of Token Embedding, Position Embedding, and Temporal Embedding. Specially, we use one-dimensional convolution operation to map each each specific sample X k while keeping raw shape to generate Token Adapter P TO . Additionally, we use a trainable lookup table to map each point's explicit position in the entire sequence, to generate Position Adapter P PO . Furthermore, we separately encode different time attributes such as minutes, hours, days, weeks, and months, via trainable parameters to dynamically model complex temporal shifts, to generate Temporal Adapter P TE . Finally, for each decomposition components, corresponding generated adapters can be obtained by aggregating Token Adapter P TO ∈ R L×C , Position Adapter P PO ∈ R L×C , and Temporal Adapter P TE ∈ R L×C as P d = P d TO + P d PO + P d TE , where d ∈ {Trend, Seasonal, Residual}, this means that we can obtain P Trend , P Seasonal , P Residual . Details about the generator in Appendix B.5.Lightweight Operations. To enhance the PLMs' ability to represent complex inputs while reducing the computational burden to adapt to low-resource devices, we introduce lightweight operations, which includes channel-independent patching (CIP, Fig. 1e ) [6] for input and efficient tuning of parameters for PLMs. Among them, CIP splits the multivariate sequence into separate univariate sequences, each processed by a single model with length L p . This approach outperforms the original method of mixing channels by treating the variables as independent. It enables the model to capture channel interactions indirectly through shared weights, leading to improved performance without directly modeling the complexity of multiple data channels. The total number of inputs patches is P =(T -Lp) S+2, where S denotes the horizontal sliding stride. Given these patches X i P ∈ R P ×Lp , we use rearrange operation and a trainable FFN embed them as X i P ∈ R P ×dm , where d m is dimensions created by the FFN. We also introduce a low-rank adaptation (LoRA) [19] inside PLMs aiming at language modeling for lightweight fine-tuning of attention layers to achieve cross-modal/-domain knowledge transfer from text sequences to weather sequences with minimal effort.Parameter Adapter. To adapt PLM outputs for downstream weather sequence modeling, we introduceParameter Adapter, a simple FFN with a single linear layer positioned after the PLM. This adapter transforms the PLM's output to match the prediction horizon, formalized as follows:EQUATIONwhere the PTrend , PSeasonal , PResidual , and X are obtained from CIP based on P Trend , P Seasonal , P Residual , and X . The key objectives are twofold: (1) to enrich the PLM's cross-modal representations by incorporating task-specific knowledge, and (2) to enhance the PLM's output accuracy while preserving its inherent knowledge through the integration of weather data for cross-domain knowledge transfer.To avoid data silos and counteract the performance disparities caused by data heterogeneity while ensuring efficient communication, we update personalized adapters locally and share low-rank parameters globally in each round. Specifically, the local PLM M θ can be formulated as below: M θ → M θ,t (Communication) + M θ,f (Locally) (5) where M θ,t denotes the trainable parameter from the low-rank matrices of query and value in attention modules, M θ,f is the frozen parameter (mainly the PLM backbone) and other trainable ones (primarily for the personalized adapter). During client-server communication, only M θ,t is transmitted and averaged using FedAvg [7] . At the start of the next training round, the updated M θ,t is broadcast to clients for further updates. Privacy is further protected by sending minimal parameters. 
main theorems: 
 Theorem 4.1 (Decomposition Rationality from Time Series). Given a weather series X = X Trend,t + X Seasonal,t + X Residual,t , t ∈ [t 1 , t n ]. Let E = {e 1 , e 2 , ..., e n } denotes a set of orthogonal bases. Lets E Seasonal ⊆ E denote the subset of E on which X Seasonal,t has non-zero eigenvalues and E Trend ⊆ E denote the subset of E on which X Trend,t has non-zero eigenvalues. If X Trend,t and X Seasonal,t are not orthogonal, i.e., n i=1 X i Trend,t X i Seasonal,t ̸ = 0, then E Trend E Seasonal ̸ = 0, i.e., E can not disentangle the two signals onto two disjoint set of bases. Theorem 4.2 (Exchange Low-Rank Matrices Ensures Privacy). Given a on-device weather modeling framework based on federated learning that gloabl optimization object is F (θ) = i=1 n p i f ({D i }; θ), where f (x; θ) is the loss function of i-th client, {D i } is dataset of i-th client, and p i and θ denote the data distribution weight of client i and the model parameters, respectively. Given that the parameters θ of the PLM M θ broadcasted by the server consist of two parts: a frozen part M θ,f and a trainable part M θ,t , interacting only the low-rank matrix parameters M θ,l ⊂ M θ,t is a subset of trainable part M θ,t during each round ensures privacy.On-device Weather Series 1# (ODW1). The dataset gathered from 15 ground weather stations across China, Japan, and South Korea, encompasses over 20 variables. It has been divided into two subsets: ODW1T has a heterogeneous time span, meaning the data collection start and end times vary by location. and ODW1V extends ODW1T by adding variability in the observed variables; while one variable remains constant at each station, the others vary. On-device Weather Series 2# (ODW2). This dataset consists of data from 36 weather observation stations in the United States, Canada, and Israel, covering 5 different variables with a temporal resolution of 1 hour. Following the dataset setting of ODW1, the dataset was also subdivided into two different dataset, including ODW2T and ODW2V.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: This paper has fully disclosed all the information needed to reproduce the main experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
setup: 
 Baseline. Since our framework is based on a language model, we compare with DL-based SOTA time series models, including Transformer-based methods: Transformer [20] , Informer [3] , Reformer [21] , Pyraformer [22] , iTransformer [23] , and PatchTST [6] , and recent competitive models: GPT4TS [17] , DLinear [24] and LightTS [25] . Note that our setting is FL-based, so we place them in FL and rename them FL-(baseline) like FL-Transformer, etc., and all aggregation methods used in above models is FedAvg [7] . In addition, we report a variants of LM-WEATHER, LM-WEATHER-AVE that based on FedAvg without personalization. Detailed information are in Appendix B.2.Basic Setup. We focus on on-device meteorological variable forecasting and imputation tasks. For forecasting, we create scenarios for predicting a single variable (multivariate-univariate) and for predicting all variables (multivariate-multivariate). The main text only includes multivariate-tomultivariate forecasting results due to page constraints. For multivariate-to-univariate forecasts, refer to the Appendix E. In imputation, we use sequence lengths of {96, 192, 336, 720} and apply three different masking probabilities {25%, 35%, 50%} to represent missing data. The main manuscript shows imputation results for a 50% masking ratio. For more details on the setup, please refer to Appendix B.3. All our experiments are repeat five times and we report the averaged results.In this section, we evaluate LM-WEATHER and baseline methods on four on-device meteorological variable modeling datasets in general experiments to validate its effectiveness.Setups & Results of Forecasting Tasks. Input length L b is fixed to 192, and we use four different prediction horizons L f ∈ {96, 192, 336, 720}. Evaluation metrics include mean absolute error (MAE) and root square mean error (RMSE). The brief results is shown in Tab. 1, where our LM-WEATHER outperforms all baselines in most cases and significantly so to the majority of them. Particularly noteworthy is the comparison with GPT4TS that involves fine-tuning PLMs, where LM-WEATHER has an average 9.8% improvement over FL-GPT4TS (MAE reported), and even the variant LM-WEATHER-AVE has an average 4% improvement over FL-GPT4TS. In addition, LM-WEATHER shows significant average performance gains of 11.2% and 19% w.r.t. MAE relative to other SOTA such as FL-DLinear and FL-PatchTST. Setups & Results of Imputation Tasks. Our brief results are in Tab. 2, where LM-WEATHER consistently surpasses all baselines, outperforming FL-GPT4TS by 5.7%. LM-WEATHER remains competitive even when compared with the SOTA, FL-PatchTST, FL-LightTS, and FL-DLinear. PLMs have demonstrated remarkable few-shot learning capabilities [26] . In this subsection, we assess whether LM-WEATHER retains this ability in both forecasting and imputation tasks, based on FL for resource-constrained on-device weather modeling environments.Setups and Results of Forecasting & Imputation. For both forecasting and imputation tasks, we evaluate the few-shot learning capability in scenarios using limited data, specifically, we use training ratios of 5% and 15% (Our full few-shot learning results (training ratio of 5% and 15%) can be found at Appendix E.2). The brief 5% few-shot learning results on forecasting and imputation tasks are depicted in Tab. 3 and Tab. 4, respectively. LM-WEATHER remarkably excels over all baseline methods, and we attribute this to the successful cross-domain knowledge activation in our local dual fine-tuning for the PLM. In addition, our LM-WEATHER's communication mechanism also reduces the impact of data heterogeneity on performance, which is reflected in the fact that LM-WEATHER has an average 14.7% and 20% improvement relative to LM-WEATHER-AVE, in the forecasting and imputation, respectively. In relation to recent SOTA methods such as FL-PatchTST, FL-LightTS, and FL-DLinear, our LM-WEATHER enhancements surpass 78%, 14.3%, and 72.8% for forecasting, and 102.1%, 122.1%, and 96.35% for imputation. This means that heterogeneity poses challenge to baseline and they struggle to understand weather patterns with limited data. Moreover, it implies that LM-WEATHER can effectively achieve cross-domain knowledge transfer to PLMs. This benfits from the personalized adapter we integrated into the PLM, coupled with lightweight operations. Additionally, keeping the decomposition term but removing the associated generator leads to a 14.5% average performance decline. This suggests that our personalized adapter effectively leverages the PLM's modeling of weather data. Conversely, when we alter the personalized approach by changing the shared low-rank matrix to other trainable parameters (LM-WEATHER-F), we observe a significant performance drop and increased communication costs. Furthermore, moving from LoRA to fully fine-tuning the attention parameters results in a slight performance gain but incurs over four times the parameter count and a massive increase in communication overhead, which is inefficient for us. These outcomes highlight the benefits of the personalized adapter. Communication Efficiency. To further validate the excellent communication efficiency of LM-WEATHER, we introduce quantitative comparisons by including FL methods tailored to improve communication efficiency (FedKD [27] , FedPAQ [28] , FedBF [29] , FedAP [29] , PromptFL [30] ) as baselines 2 . The results is shown in Table 8 , which demonstrate our LM-WEATHER achieves a significant improvement in communication efficiency while maintaining excellent performance. Additionally, LM-WEATHER significantly outperforms baseline in terms of both communication efficiency and performance across different tasks. Even when compared to lightweight baselines (i.e., FL-LightTS/DLinear), LM-WEATHER continues to outperform them. This underscores LM-WEATHER's superiority in both communication efficiency and performance. Robustness to Number of Devices. To evaluate LM-WEATHER's robustness against device count variations, we assessed the percentage change in performance relative to the default device number. Our results (Tab. 9) reveals that LM-WEATHER maintains robustness across different device counts due to several factors: (1) Increasing device numbers during training typically yields slight performance improvements within a stable range, applicable in both regular and few-shot scenarios.(2) Additional devices can sometimes impair performance due to imbalances in data distribution, highlighting non-proportional gains.(3) Adding more devices increases communication overhead, which may not justify minor improvements, especially in resource-limited settings. These findings underscore LM-WEATHER's relative resilience to device count variations and its ability to strike an optimal balance between performance enhancement and communication overhead. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: This paper has provided access to the code, but due to the size of the dataset, additional releases are required. 
conclusion and future works: 
 This paper demonstrate that pre-trained language models (PLMs) are strong foundation models for personalized on-device meteorological variable modeling. We propose LM-WEATHER, a generic framework to taming PLMs to acquire highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalize adapter into PLMs and endow it with weather pattern awareness. Experiments on real-world datasets demonstrate that LM-WEATHER outperforms the SOTA results by a large margin across various tasks. In addition, extensive analyses indicate that LM-WEATHER can (1) effectively achieve crossdomain knowledge transfers, (2) render device with highly customized model while keeping high efficiency, and (3) generalize under few-shot and zero-shot scenario. In future work, we plan to extend LM-WEATHER to multimodal weather data (text/image/time-series) and to finer scales.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper has specified all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This paper has provided sufficient information on the computer resource.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. 
 
 Appendix: Towards Weather Foundation Models In this section, we will discuss in detail advances relevant to our work, which include weather variable modeling, personalized federated learning, universal time series learning, and large language models (LLMs) in time series.From Meteorological Variable Modeling to Weather Forecasting. Weather conditions play a crucial role in sectors such as transportation, tourism, and agriculture. Meteorological factors, including temperature, humidity, and precipitation, provide essential support and historical insights that enable researchers to analyze weather trends. For decades, Numerical Weather Prediction (NWP) [14] has been the prevalent method, employing physical models to simulate and forecast atmospheric dynamics. However, the accuracy of NWP can be compromised by the uncertainty of initial conditions in differential equations [31, 32] , particularly in complex atmospheric processes, and it requires significant computational resources [5, 33, 34] .The recent exponential growth in weather data has prompted a shift from traditional physics-based methods to data-driven approaches using machine learning (ML) and deep learning (DL), which bypass physical constraints in meteorological variables [5] . DL strategies, with their deeper representational capabilities, generally surpass ML methods. Various deep network architectures have been employed to perform extensive weather modeling using large-scale reanalysis data [1, 4, 35, 36, 37 ]. Yet, these methods tend to focus on global weather patterns, often overlooking the specifics of regional weather variables, and thus fail to offer detailed regional analyses. Moreover, these models require extensive datasets and substantial computational resources-for example, some need to train on 192 NVIDIA Tesla V100 GPUs for 16 days [4] . Additionally, prevailing models assume centralized data storage, which contrasts with the decentralized data collection from diverse ground weather stations. Our research addresses these challenges by focusing on regional meteorological variables in lowresource settings, aiming to provide reliable analytical support for weather pattern modeling and understanding.Personalized Federated Learning. Federated learning (FL) [7] is a distributed learning paradigm that facilitates the collaborative training of models without exposing data from each participant. Personalized FL (PFL) aims to train a personalized model for each client. Existing PFLs are based on various techniques.Refs. [38, 39, 40] add a regularization term that benefits decomposing the personalized model optimization from global model learning. Refs. [8, 41] share part of the model and keep personalized layers private to achieve personalization. Ref. [42] enables a more flexible personalization by adaptive weighted aggregation. Ref. [43] study PFL from a Model-Agnostic Meta-Learning where a meta-model is learned to generate the initialized local model for each client. This paper tackles on-device meteorological variable modeling from PFL perspective.Universal Time Series Learning. On-device meteorological variable modeling addresses time series analysis of complex weather patterns on diverse, low-resource devices. We have expanded this to include taskspecific time series learning. Recent advancements have enhanced Transformer [44] for time series forecasting by integrating signal processing techniques such as patching [6] , exponential smoothing [45] , decomposition [24] , and frequency analysis [46] . Among them, PatchTST [6] improves the accuracy of long-term forecasting compared to other Transformer models. ETSFormer [47] applies principles of power series smoothing within the Transformer framework to boost efficiency. Similarly, FEDformer [46] merges the Transformer with seasonal & trend decomposition, offering improved performance and efficiency. Autoformer [45] leverages sequence periodicity for better dependency discovery and representation, excelling in both efficiency and accuracy.While these methods excel in efficiency and accuracy, they are typically tailored for narrow-range forecasting on select classical time series datasets. Real-world weather data, however, often displays more complex patterns and interconnected variable relationships. Furthermore, weather modeling extends beyond forecasting, rendering these methods less effective for weather sequences. To improve modeling for intricate weather sequences, models need the flexibility to adjust to complex distributions and various tasks with minimal training. The ideal weather models would capture weather patterns accurately, facilitating knowledge transfer, such as between regions. However, creating versatile weather models remains a challenging endeavor. Recent studies have started to examine the potential of large-scale climate models [1, 4] , utilizing simulated datasets advances. Yet, their generalizability is hindered by data differences, complex architectures, and the vast number of model parameters.LLMs in Time Series. Large language models (LLMs) have spurred advances in natural language processing (NLP). Although time series modeling hasn't seen similar leaps, the impressive capabilities of LLMs have led to their use in this field. In general, pre-trained LLMs are often fine-tuned or reprogrammed to model time series [18, 11, 10, 17] . Among them, PROMPTCAST [16] and HEALTHLEARNER [26] treat time series as "text sequences," inputting them directly into LLMs and using prompts for forecasting. [17] dencodes time series as embeddings for LLM output, showing LLMs' strength in time series analysis. LLM4TS [18] uses a two-stage fine-tuning approach to adapt LLMs to time series data. TEMPO [10] breaks down time series features to leverage LLMs in prediction tasks, while TIME-LLM [11] fine-tunes LLMs with multimodal data, integrating relevant text prompts for efficient analysis. However, these approaches focus on centralized time series modeling and overlook the complexities of real-world distributed settings. Weather data, in particular, has unique challenges like heterogeneity from geographic factors and privacy concerns, making central training methods both risky and difficult.Despite the proliferation of reanalysis data aimed at building frameworks for global climate analysis, these datasets often struggle to model regional weather trend due to: (1) they depend on numerous simulations of atmospheric equations, introducing biases inconsistent with real observations, and (2) they face challenges in refining their scale to suit specific regional applications. Hence, we collected real observational data from various weather stations across different regions. We then organized this data into two series, each comprising two distinct datasets, to underscore the heterogeneity inherent in real-world settings.On-device Weather Series 1# (ODW1). from Kaggle 8 . This process ensures that the collected weather data from these sources are consistent in terms of temporal resolution and variable dimensions. All raw data are open-source and can be freely utilized or modified.[2] Selection of Critical Meteorological Variables. To support personalized on-device meteorological variable modeling and enhance regional weather forecasting reliability, we selected twenty representative meteorological variables. These variables, including temperature, barometric pressure, relative humidity, and precipitation, were chosen based on their significant impact on weather conditions. Detailed definitions, physical descriptions, and units of these selected variables are provided in Table 14 .[3] Ensuring Completion of Meteorological Time Series. In this step, we primarily focus on ensuring the completeness of weather time series data collected from ground weather stations. Incomplete weather time series can generate unreliable predictions, potentially leading to significant unforeseen losses. Most ground weather stations are susceptible to unpredictable events such as power outages and equipment damage, which may result in data gaps. To enhance dataset completeness, we meticulously examined the raw data for missing values across various timestamps and employed a linear interpolation strategy to fill these gaps.[4] Handling of Outliers. Outliers are common in weather time series data. We distinguish between factual outliers, typically caused by extreme weather events (e.g., heavy rainfall, typhoons, thunderstorms), and non-factual outliers, often due to observational device anomalies or sensor malfunctions Vertical Visibility m at weather stations. We identify significant deviations in a weather variable-for instance, a sudden increase from an average rainfall of 2 mm to 200 mm-as outliers. These are manually corrected; initially, the values are set to zero and then replaced using linear interpolation, reflecting the gradual nature of weather phenomena.Visualisation. We hope to deepen the reader's understanding of the datasets we have collected and compiled by providing standard visualizations. Considering the overall size of the datasets and the large number of meteorological variables, we have provided visualisations of representative variables here for reference. The visualisation of OWD1 is shown in We compare with state-of-the-art time series analysis models and put them into Federated Learning environments, including Transformer-based methods like Transformer [44] , Informer [3] , Reformer [21] , Pyraformer [22] , iTransformer [23] , and PatchTST [6] , and recent competitive models including GPT4TS [17] , DLinear [24] and LightTS [48] , detailed information about baselines is below:• Transformer. [44] This model uses a self-attention mechanism, popular for time series prediction tasks, to efficiently and accurately learn relationships within a sequence and contextual information.• Informer. [3] An optimized Transformer-based model for long-range time series prediction. It uses ProbSparse self-attention for efficiency, processes long inputs effectively, and employs a fast prediction decoder.• Reformer. [21] This model improves Transformer efficiency by using locality-sensitive hashing for attention and reversible residual layers. It offers better memory efficiency and speed for lengthy sequences without sacrificing performance.• Pyraformer. [22] It features hierarchical pyramidal attention modules with binary trees to capture temporal dependencies across different ranges efficiently, both in time and memory complexity.• iTransformer. [23] The iTransformer adds attention and feedforward networks to the inverse dimension. It embeds time points as variable tokens, using attention to capture multivariate correlations and feedforward networks for nonlinear representation of each token.• PatchTST. [6] This method divides the time series into patches at the subseries level for input to the Transformer. Each channel holds a univariate time series, sharing the same embedding and Transformer weights across all series. • LightTS. [48] A lightweight structure based on a simple MLP. It utilizes two downsampling strategies-spaced and sequential sampling-on the MLP structure, capitalizing on the fact that downsampled time series generally maintain most of their original information.• GPT4TS. [17] This model is designed for time series analysis across various scenarios, achieved by fine-tuning a pre-trained language model, specifically GPT2, for the time series domain. It's important to note that for a fair comparison, our baseline setup differs from the original publication's configuration. Instead of using the first six layers of GPT2 as the backbone, we align with our approach and utilize only the first five layers.In addition, pre-trained language models (PLMs) are the key component of our LM-Weather, we use different PLMs as the backbone to demonstrate the PLM can as the strong weather foundation model for on-device weather modeling. We use GPT-2 as the default setting, and BERT [49] , LLaMA [50] as the alternatives.• BERT. The data distribution from different ground weather stations exhibit significant heterogeneity, and even though the trends of some variables may be similar, there are serious differences in magnitudes. The selected variables are, from left to right, humidity, precipitation, temperature, wind direction, and wind speed.text in both directions. When used as a baseline for evaluation, we only employ the first 5 layers of the pre-trained BERT.• GPT-2. [15] Developed by OpenAI, GPT-2 is a language model that can generate coherent and diverse text based on a given prompt. In our research, we utilize the first 5 layers of the pre-trained GPT-2-base.• LLaMa. [50] LLaMa stands for Large Language Model Meta AI and is a series of cutting-edge language models with sizes ranging from 7B to 65B parameters. They offer top-notch performance with less computational power and resources. In our research, we utilize the first 4 layers of the 3B LLaMa model.A brief description of these FL methods tailored to improve communication efficiency is as follows.• FedKD: This parameter-efficient PFL method integrates knowledge distillation within a single client and employs a parameter aggregation strategy using Singular Value Decomposition (SVD). For the purposes of this section, which focuses solely on comparing communication efficiency, we incorporate only the SVD-based client-server communication strategies into LM-WEATHER as a baseline. • FedPer: This PFL approach maintains a personalized layer while sharing the remaining base layers during communication. This enhances communication efficiency by transmitting only a portion of the parameters.Humidity• FedBF: This fine-tuning method enhances parameter efficiency by sharing only the biases of the local model during global aggregation, thereby reducing communication overhead. To integrate this method into LM-WEATHER as a baseline, we adjusted all biases in LM-WEATHER to be unfrozen.• FedAP: A parameter-efficient fine-tuning method in FL, which involves sharing only adapters during global aggregation.• PromptFL: This parameter-efficient FL method enables participants to cooperatively train lightweight prompts without sharing the entire model, significantly accelerating both local training and global aggregation. In our experiments, we treat the adapter generated on clients as the prompt to facilitate the incorporation of this baseline.We evaluate our proposed LM-WEATHER using four distinct on-device weather modeling datasets, each with tailored settings for various tasks. The specific task settings for these datasets are detailed in Tab. 15.Additionally, the specific tasks and scenarios for the on-device weather forecasting/imputation vary by dataset, as outlined in Tab. 16. We mainly follow the experimental configurations across all baselines within a unified evaluation pipeline in https://github.com/thuml/Time-Series-Library for fair comparison. Specially, we use GPT-2-base as the default backbone model unless state otherwise. All our experiments are repeat five times and we report the averaged results. Our detailed model configurations are in Appendix B.8. All the algorithm implementations and designs in this study are based on Py torch and the algorithms are run on two RTX3090 GPUs 24GB.Reversible Normalization. In time series analysis, statistical properties like mean and variance often shift over time, indicating distributional changes in the data. To address this, we've incorporated Reversible Normalization (RevIn) [51] into our LM-WEATHER. Specifically, we've integrated RevIn into our Task Adapter Generation. This introduces two dynamic factors that adaptively normalize segments of the meteorological variable sequence X , or their decomposed components (Trend XTrend, Seasonal XSeasonal, Residual XResidual), enhancing the accuracy of meteorological variable modeling. Specifically, for the trend component of X , i.e.,, XTrend, its transformed value X ′ Trend can be given by:EQUATIONwhere E[XTrend] and Var [XTrend] are the instance-specific mean and variance, respectively. γT and βT are the trainable parameters for this component. This transformation is also applied to both the seasonal and residual components.Pre-trained Language Model (PLM). In LM-WEATHER, we do not change the main architecture of the PLM, but use the parameter-efficient fine-tuning (PEFT) strategy to avoid large-scale parameter variations to ensure high efficiency on resource-constrained weather devices, and in this way, to achieve more reliable cross-domain knowledge transfer. Specifically, we introduce LoRA in the local PLM, which allows only 1.5% of the PLM parameters to be trained while the rest remain frozen, as shown in Fig. Note that LoRA is only applied to the query and value of each Attention in the PLM, and the resulting low-rank matrices are used for global sharing between the client and the server.Latent Representation Low-Rank Adaption (LoRA). To achieve more reliable cross-domain knowledge transfer (i.e., from natural language to complex weather sequences) while guaranteeing higher efficiency, we introduce LoRA [19] , a parameter-efficient fine-tuning method for large language models, into PLM. Specifically, LoRA is applied to the Query and Value of each Attention layer by creating low-rank matrices for two pre-trained parameters Wq and W k : Task Adapter Generation. The XTrend, XSeasonal, XResidual obtained from decomposition are used to generate Task Adapter via an unified generator as Fig. 1B that consisting of Token Embedding, Position Embedding, and Temporal Embedding. Specially, we use one-dimensional convolution operation to map each each specific sample X k ∈ R T ×1 while keeping raw shape to generate TOKEN ADAPTER PTO ∈ R T ×C , asW q W v W kEQUATIONEQUATIONAdditionally, we use a trainable lookup table to map each point's explicit position in the entire sequence, to generate POSITION ADAPTER PPO ∈ R T ×C , as:EQUATIONwhere E( The effectiveness and superiority of our proposed Personalized Adapter have been demonstrated in sufficient ablation studies (please refer to Table 6 in the main text). Here, we will discuss theoretical insights that further supports the effectiveness of Personalized Adapter. Personalized Adapter can effectively capture potential pattern in meteorological variable time series, which comprises both the Task Adapter and the FFN-based Parameter Adapter. Specifically, our focus primarily revolves around the Task Adapter active in extracting representations, which including Token/Positional/Temporal Embedding for transforming meteorological variable time series. which means Token Embedding can construct a high-dimensional nonlinear mapping from multiple onedimensional functions, capturing complex patterns within weather sequences. Positional Embedding introduces a vector pt for each step, enabling the model to differentiate between observations at different time steps. For any two steps t1 and t2, their position vectors pt 1 and pt 2 satisfy:||pt 1 -pt 2 ||2 = 2k(1 -cos 2π(t1 -t2) 10000 1/k). The growing distance between t1 and t2 with an increasing time gap mirrors the relative positioning of time steps, aiding the model in grasping temporal dependencies. Additionally, the sine-cosine function's periodicity resonates with weather data's cyclical behavior, helping the model to learn from these recurrent patterns.Finally, consider the role of Temporal Embedding from the view of matrix decomposition. Suppose the temporal matrix T has a rank of r, it can be decomposed as zt ⊗T = r i=1 (zt ⊗ ui)v T i . Temporal Embedding transforms the original sequence by scaling and rotating it to represent different interaction patterns. Using singular value decomposition, the top r singular vectors distill the core structure of the time-based matrix. This allows Temporal Embedding to intuitively learn a compact representation of weather sequences, highlighting the primary interactions between variables. In optimizing Personalized Adapter within LM-WEATHER, the focus lies solely on Personalized Adapter and the attention layer influenced by LoRA during local updates. As Personalized Adapter undergoes solely local updates while sharing low-rank matrices globally, akin to layer-wise optimization in PLM, the efficacy of its optimization process can be theoretically substantiated by the theoretical analysis provided in [52] .For evaluation metrics, as [34] , we utilize the mean absolute error (MAE) and root mean square error (RMSE) for both forecasting and imputation. The calculation of these metrics are as follows:EQUATIONwhere T denotes the number of data points (i.e., prediction horizon in our cases), Yi and Ŷi are the i-th ground truth and prediction where i ∈ {1, ..., T }.The configurations of our LM-WEATHER for different tasks and datasets are summarized in Tab. 17. We consistently use the AdamW [53] optimizer in all experiments. Proof. We decompose XSeasonal,t and XTrend,t onto E and acquire that XSeasonal,t = aiei and XTrend,t = biei. Then it is obvious that ei ∈ XSeasonal ⇔ ai ̸ = 0 and ei ∈ XTrend ⇔ bi ̸ = 0. Now, let us consider the inner product of XSeasonal,t and XTrend,t:EQUATIONNote that n i=1 X i Trend,t X i Seasonal,t = 0. Thus, there must be at least one i such that ai ̸ = 0 and bi ̸ = 0. Thus. ei ∈ ESeasonal and ei ∈ ETrend, in other words, ETrend ∩ ESeasonal ̸ = 0. The theorem demonstrates that if XTrend,t and XSeasonal,t are not orthogonal, orthogonal bases that separate XTrend,t and XSeasonal,t into two distinct sets cannot exist. Typically, periodic and non-periodic signals are not orthogonal because the periodic signal has a discrete spectrum, while the non-periodic signal has a continuous one, leading to potential overlaps at non-zero frequencies. Principal Component Analysis (PCA) seeks to find orthogonal bases in data, but it cannot split these two signals into separate bases. Citing Theorem 1 from [17] , we understand that self-attentive mechanisms in pre-trained large models function similarly to PCA. Thus, without manual intervention, the self-attentive mechanism is unable to automatically divide a time series into trend and seasonal components.Given a on-device weather modeling framework based on federated learning that global optimization object is F (θ) = i=1 n pif ({Di}; θ), where f (x; θ) is the loss function of i-th client, {Di} is dataset of i-th client, and pi and θ denote the data distribution weight of client i and the model parameters, respectively. Given that the parameters θ of the PLM M θ broadcasted by the server consist of two parts: a frozen part M θ,f and a trainable part M θ,t , interacting only the low-rank matrix parameters M θ,l ⊂ M θ,t is a subset of trainable part M θ,t during each round ensures privacy.Proof. We assume that f (x; θ) is a convex function with respect to θ, i.e., for any θ1 and θ2 and λ ∈ [0, 1], we have fEQUATION) Since only low-rank matrices parameter M θ,l parameterized by θ l is exchanged, we can convert θ to θ ′ = [θ ′ l , θo], where θ ′ l is the embedding parameter after the server update. Since we only update on θ l , θo remains unchanged. Thus, data privacy can be ensured, as θo contains parameters that reveal user-specific information. Furthermore, the low-rank matrices applied to the PLM M θ using LoRA are initialized with a random Gaussian distribution and all-zero values, respectively, before training. This global information sharing approach also helps to enhance privacy.Our LM-Weather can hide key features of local model when sharing low-rank parameters, even external attacker gains access to a shared low-rank update, it is difficult to reconstruct or differentiate between the original models of different participants. Support client i and client j get two different local model due to updated on heterogeneity weather time series, as Mi and Mj parametered by θi and θj,L(M) denotes the low-rank matrix generated by model M, where B ∈ R d×r and A ∈ R r×d , and r ≪ d, and define Adv as the adversary. Conditions exist such that for any polynomial-time attacker Adv,|P r[Adv(L(Mi))= 1] -P r[Adv(L(Mj)) = 1]| ≤ ϵ holds,where ϵ is a small positive number, which implies that even if the attacker acquires a shared low-rank update, it is difficult to reconstruct or distinguish between the original models of the different participants.Proof. The goal of LoRA is to find the low-rank approximation: min||θ -θ0 -BA||F , where θ0 is the initial weight, || • ||F is the Frobenius paradigm. Consider two models Mi and Mj, whose weights differ by ∆θ = θi -θj. The corresponding LoRA matrix is:EQUATIONAccording to matrix approximation theory, for the best approximation with rank r, the upper bound on the error is:EQUATIONwhere σr+1(∆θi) is the r+1-st singular value of ∆θi. By Johnson-Lindenstrauss Lemma [54] , for any ϵ > 0, there exists a mapping f : R d → R k , where k = O(log(n)/ϵ 2 ), such that for any x, y ∈ R d . In our case, LoRA can be regarded as such a degenerate mapping. Assume ||∆θi -∆θj||F ≤ δ and according to the trigonometric inequality:EQUATIONLet ε ′ = σr+1(∆θi) + σr+1(∆θj), we get:EQUATION)For any polynomial-time attacker Adv,its ability to distinguish between Li and Lj is restricted to the difference in their Frobenius paradigms. We can define a function f such that:EQUATIONwhere f is a monotonically increasing function that represents the attacker's capability with respect to the matrix difference. Combining Eq. 16 and Eq. 17:EQUATIONwhen δ and ε ′ are small enough, we can make sure that the right-hand side is smaller than the intended ε. This means that an attacker cannot personalised local parameters and data to ensure privacy through the low-rank matrix of communication across clients.In this section, we explore and discuss potential research findings and questions for our LM-WEATHER via conducting additional experiments. These potential research questions are as follows:• RQ1. How does LM-WEATHER compare to Personalized Federated Learning (PFL) baselines in terms of trade-offs in personalization and global model performance?• RQ2. How does LM-WEATHER perform compared to centralized and local-only training modes?• RQ3. How does the pre-trained language model contribute in LM-WEATHER?• RQ4. What is the resource utilization and training & inference cost of LM-WEATHER?• RQ5. Can LM-WEATHER be used for other tasks?Our LM-WEATHER builds on the assumption that the foundation model already exists, treating pre-trained language models (PLMs) as such and broadcasting it to each client to achieve local updates. Our aim is to employ device information-specific (e.g., geographic/atmospheric patterns) adapter, to promote the local PLM in achieving cross-domain knowledge transfer from language to meteorological sequences. This approach yields highly customized models for individual devices while achieving global knowledge to avoid data silo, thereby supporting diverse analyses of heterogeneous weather data. Alternative PFL methods do not match the efficiency and flexibility of our personalized adapter in this context, making them less suitable. By incorporating PFL baselines (Per-FedAvg [43] , APPLE [55] , FedPer [56] , and FedALA [57] ), we provide quantitative results that substantiate our claims, experiment setting is consistent with the manuscript on ODW1T.PFL Baseline. A brief description of PFL baselines used in this section of the experiment is as follows.• Per-FedAvg: Allowing for personalized model updates for each client by adding client-specific parameters to the global model and optimizing them during FL training.• APPLE: Tackling statistical heterogeneity in FL by automatically capturing information required by clients from global models using adaptive local aggregation methods.• FedPer: Dividing the model into a base layer and a personalization layer, only the base layer is uploaded during aggregation while keeping the personalization layer to combat statistical heterogeneity.• FedALA: Tackling statistical heterogeneity in FL by automatically capturing information required by clients from global models using adaptive local aggregation methods. Personalization and Global Model Performance Trade-offs. We consider the trade-off between personalization performance and global model performance for our LM-WEATHER and PFL baselines, the results are shown in Table 20 . Compared with PFL baselines, our LM-WEATHER maintains the best trade-off between personalization performance and global model performance, i.e., the personalization performance does not significantly exceed the global model performance while the performance far exceeds PFL methods, which means that the LM-WEATHER can be flexibly applied to different practical scenarios, including personalised analysis of regional weather trends as well as comprehensive analysis of weather trends over large-scale regions. This means that LM-WEATHER can be flexibly applied to different practice scenarios, including the personalised analysis of regional weather trends and the comprehensive analysis of weather trends over large scale areas. Performance and Adapter Updating Trade-offs. Furthermore, we investigated the effect of varying the number of local update rounds in adapters across clients on the performance of LM-WEATHER regrading personalization. The results are presented in Table 21 . We observed that increasing the local update rounds from the default five to fifteen leads to smoother and enhanced personalization performance across heterogeneous clients. However, this increase in local update rounds also incurs additional computational and communication costs, which, in our assessment, do not justify the modest performance improvements. The ordinary centralised training strategy (all data were aggregated into a single server) exhibits learning efficiency that an ordinary distributed learning strategy. The ultimate goal of FL is to achieve performance close to that of centralised training and to ensure privacy across data sources. Table 22 illustrates that our LM-WEATHER achieves comparable effectiveness to Non-FL (centralised) training, with only a 2.04% disparity.Compared to LM-WEATHER-Local, which lacks interaction between devices, LM-WEATHER performs better due to overcoming data silos.Our LM-WEATHER significantly outperforms time series-specific models trained from scratch under centralised setup. Centralised training aims to acquire an excellent pre-trained model, where PLMs possess inherent advantages due to their prior sequence modeling capabilities. Moreover, various parameter-efficient fine-tuning (PEFT) strategies enable PLMs to adapt to new domain knowledge cost-effectively. FL-based aggregation facilitates a stable on-device fine-tuning process, with LM-Weather enabling highly customized on-device fine-tuning of PLMs with greater efficiency. This highlights the substantial contribution of PLMs in this task. The remarkable capabilities of cutting-edge DL models across various domains and tasks, such as LLMs, and VLMs, can be attributed to their extensive parameters and training on large datasets. Currently, a perfect balance among performance, model size, and cost does not exist. Despite numerous studies focusing on reducing training and inference costs while maintaining superior performance, there is no one-size-fits-all solution. This is also true for our LM-WEATHER, which demonstrates exceptional performance across diverse tasks and scales on real-world datasets with significant heterogeneity, significantly outperforming comparable DL methods. In this context, we analyze the costs associated with training and inference for LM-WEATHER and its baselines, exploring and discussing the trade-offs between cost-effectiveness and performance in practical applications. The quantification and comparison of computational costs against LM-WEATHER and baseline are shown in Table 24 . We discuss this results from two perspectives as follow.Communication and Performance. LM-WEATHER outperforms baseline in these two key metrics, which is critical for practical meteorological variable modeling and analysis, a bandwidth-sensitive and high accuracy demanding application.Trade-offs between Resource Consumption and Performance. In terms of training and inference time and memory usage, LM-WEATHER is less efficient than lightweight baselines such as FL-DLinear, LightTS, and iTransformer, due to its use of a Pretrained Language Model (PLM) as a backbone. Although LM-WEATHER demands more resources, the trade-off is justified by its cost-effective performance gains. Its slightly increased memory requirements for training and inference are manageable on most devices. In the context of weather analysis, where precision is critical, prioritizing performance improvements over minimal resource consumption is essential. Additionally, LM-WEATHER capitalizes on the knowledge-rich PLM and requires only minimal, low-cost fine-tuning on devices to achieve superior performance. This strategy not only enhances performance but also reduces the frequency of future model updates, thereby lowering long-term costs compared to developing a baseline model from scratch. In addition, we further provide a comparison of model sizes and performance between LM-WEATHER and baseline, as shown in Table 25 . The difference in model size between our LM-WEATHER and baseline can be deemed acceptable for the following reasons.Trade-offs between Performance and Size. While LM-WEATHER may not be as compact in terms of model size or resource efficiency as lightweight baselines, it offers significant advantages in various analysis tasks. Its high performance is particularly valuable in practical applications. Moreover, with a model size of 304.19 M, LM-WEATHER is still accessible for devices with limited resources. This contrasts sharply with many large foundation models, which typically comprise several hundred million parameters. The trade-off between performance and size is justified, especially considering the critical nature of accurate weather data analysis.Efficient Parameter Update and Communication. LM-WEATHER implements efficient on-device fine-tuning of the pretrained language model. Unlike baselines that require training from scratch, LM-WEATHER only needs fine-tuning of a relatively small number of parameters (10.38 M) on each device, with minimal device-to-server communication overhead (0.38 M). This approach facilitates highly personalized cross-domain knowledge transfer, significantly reducing the ongoing costs associated with processing the ever-changing streams of weather data. These aspects highlight the pragmatic considerations that have shaped the design of LM-WEATHER. The model's capabilities to deliver exceptional performance, combined with its efficient parameter tuning and communication strategies, offer a cost-effective solution for advanced weather data analysis in resource-constrained environments.Given datasets we proposed in this paper focus on forecasting and imputation tasks, we broaden its scope briefly to explore its potential application by integrating anomaly weather detection tasks. This involves relabeling the dataset to identify intervals with anomalous meteorological variables as instances of abnormal weather processes. Specifically, we label original datasets via Isolation Forecast [58] , the main process as follows: (1) We set the cut length to 100, using this metric to segment each channel (variable) and construct several random trees that collectively form a forest. (2) The "isolation" degree of each data point is quantified by the average path length from the root node to the terminal node. (3) Data points with shorter path lengths are more easily isolated and thus more likely to be outliers. ( 4) We establish a threshold based on the average path length; data points falling below this threshold are classified as anomalies.Evaluation Metrics. We used Precision (P), Recall (R), and F1-Score (F1) to simply quantify the performance of LM-WEATHER and baselines on the weather anomaly detection task, these can be formulated as:EQUATIONwhere TP (True Positives), FP (False Positives), and FN (False Negatives) represent the number of samples correctly labeled as anomalous, the number of samples incorrectly labeled as anomalous, and the number of samples that were not labeled as anomalous by the model but were actually anomalous, respectively.Experiments and Results. We set the input time series length to 100, and other settings (e.g., baselines, hyper-parameters, local updating steps and federated communication rounds, etc.) are consistent with those in the main text, and we conduct experiments on OWD1T and OWD2T to briefly show the results. The performance quantification of our proposed LM-WEATHER and baseline on weather anomaly detection tasks is shown in Table 26 (ODW1T results) and Table 27 (ODW2T results).The findings underscore LM-WEATHER's robust applicability and its Moreover, LM-WEATHER's superior performance over baselines in both regular and fewshot tasks reaffirms its effectiveness and overall superiority. Moreover, LM-WEATHER's superior performance over baselines in both normal and few-shot tasks reaffirms its effectiveness and overall superiority. In this section, we show detailed and full few-shot learning experimental results including: In this subsection, we show the results of the complete ablation experiment, both in the forecasting ( Although our LM-WEATHER significantly outperforms models trained from scratch for time series analysis across various tasks and scenarios with minimal parameter tweaks, it still faces two primary limitations:• Limited Dataset Scale: Due to constraints on computational resources and operational costs, we evaluated the performance of LM-WEATHER using the real-world datasets that did not approach the scale of tens of terabytes often required for training large-scale meteorological models. This limitation does not affect LM-WEATHER to be extended as a general framework for regional weather trend analysis. This framework supports the analysis of on-device meteorological variables and can be further developed and adapted for additional applications.• Dependence on PLMs' Quality and Performance: Although LM-WEATHER leverages PLMs to achieve high efficiency and customization on heterogeneous devices, this dependency means that the quality and the performance of LM-WEATHER are intrinsically tied to the underlying PLMs. Should there be inherent limitations or biases within the PLMs, these could translate to the meteorological modeling performance. Conversely, if conditions allow the use of a more powerful LLM, LM-WEATHER's performance can be significantly improved. This might give the community more opportunities to explore the future road-map.In future work, we aim to broaden the use of LM-WEATHER across more on-device variable modeling applications. We also plan to incorporate additional types of data, including satellite and radar imagery, as well as textual weather descriptions, to advance towards a more generalized approach to on-device meteorological variable modeling.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.The words 'client' and 'device' have the same meaning in our paper.Due to scenario and model differences, we modified these baselines for LM-WEATHER by applying solely their strategies to improve communication efficiency, as detailed in Appendix B.2We treat the value of each meteorological variable at each timestamp as an independent sample uniformly.https://data.cma.cn/https://www.kma.go.kr/https://k.data.cma.cn/mekb/?r=dataService/cdcindex&datacode=A.0020.0002.S001https://weather.gc.ca/https://www.kaggle.com/datasets------------------------------------------------336 --------------------720 ----------------This research seeks to make on-device meteorological variable modeling more efficient and adaptable. By using a PLM as a foundation model instead of training large foundation models from scratch, it eliminates the need for large-scale real weather data and extensive computational resources. Additionally, it supports a variety of devices, enabling everything from advanced smartphones to basic IoT sensors to perform meteorological variable modeling. The method is also designed to be stable in environments with limited data and those outside of typical distribution ranges, providing credible analytical support for further weather trend analyses. 


Decision: This paper proposes LM-Weather, which uses pre-trained LMs (PLMs) as foundation models for on-device meteorological variable modeling. LM-Weather integrates personalized adapters into PLMs, enhancing the ability to handle heterogenous weather data. Experiments are conducted on multiple real-world datasets showcasing the effectiveness of LM-Weather, outperforming SoTA results on forecasting and imputation tasks, and having low computation overhead.

Strengths:
Proposed LM-Weather is a novel approach that integrates personalized adapters into PLMs, enabling accurate on-device meteorological modeling. Sound experimentation on real-world data, results showcase robust performance of LM-Weather across various tasks & scenarios, as well as improvements over SoTA. Comparison of various time series models across different meteorological variables and weather stations.


Weaknesses:
Prior to the final submission, authors should address the feedback and questions of the reviewers, as well as clarify: 
(1)  transmission mechanisms and strategy of the low-rank matrix
(2)  motivation behind using FFN architecture, details of the parameter configuration
(3) expanding on the theoretical aspect 
(4) synthesizing the most important results and including them in the main part of the paper to help readers get the main contribution of the work right away

Reviews: review: paper_summary: paper_summary: This paper introduces LM-WEATHER, a framework leveraging pre-trained language models (PLMs) for on-device meteorological variable modeling. The framework integrates personalized adapters into PLMs, enhancing their ability to handle heterogeneous weather data efficiently. Key contributions include superior performance in forecasting and imputation tasks, minimal communication overhead, and maintaining privacy during client-server interactions. Experiments on real-world datasets demonstrate LM-WEATHER's effectiveness across various scenarios, including few-shot and zero-shot learning.
strength_weakness: strength_weakness: W1.In the communication section, it is mentioned that only the low-rank matrix parameters are transmitted between the client and server, but specific details about the transmission mechanism and strategy are missing. This includes transmission frequency, bandwidth consumption, and potential issues in practical applications. These missing details may prevent readers from fully understanding the feasibility and effectiveness of the communication strategy.

W2. In the parameter adapter section, it describes integrating the generated adapters with the original input through FFN to produce the final weather sequence predictions. However, the specific architecture of the FFN, how its parameters are configured, and why this particular architecture was chosen are not detailed. The lack of these key details might leave readers questioning the mechanism and effectiveness of the parameter adapter.

W3. In Section 4, while Theorems 4.1 and 4.2 propose the rationality of time series decomposition and privacy assurance through low-rank matrix exchange. There is no introduction and description about the background and application of these Theorems.  Also, the descriptions are too brief, making it difficult for readers to fully understand the theoretical basis.

W4. The paper's experimental results are heavily supplemented by a 37-page appendix, which can hinder readability and accessibility of the core content. Important experimental details and results should be better integrated into the main body of the paper to facilitate easier comprehension and evaluation by the readers.
questions: questions: Q1.Could you provide more detailed proofs and theoretical validation for the decomposition rationality and privacy assurances mentioned in the theorems?

Q2.Could you clarify the potential confusion in the notations n¬i and nk in Section 2.2, and ensure the formulas are correctly presented?

Q3.Could you provide detailed information on the architecture and parameter configuration of the FFN used in the parameter adapter, and explain why this architecture was chosen?

Q4.Could you provide more detailed information about the datasets, particularly whether data from extreme climatic regions(e.g., tropical rainforests and deserts) are included? If not, how do you plan to expand these datasets to cover more global climatic conditions?
limitations: limitations: 1 The authors acknowledge the limitations in the diversity of the datasets and the potential challenges in modeling specific climatic conditions not covered in the current study. Future work should focus on expanding dataset coverage and providing more detailed theoretical and empirical validation. Additionally, the practical applications and limitations of few-shot and zero-shot learning capabilities should be more thoroughly explored and documented.

2 It would be much better integrate the essential experimental details and results into the main body of the paper to improve readability and comprehension?

3 I have questioned whether there will be a powerful foundation model for time series prediction. If the assumption is not true, the application of the proposed solution is limited.
paper_summary: paper_summary: The paper aims to develop weather foundation models by leveraging the on-device data on many distributed sensors. Specifically, a federated learning and low-rank adaption mechanism have been applied to a time-series-based foundation model training on Meteorology data.
strength_weakness: strength_weakness: 1) The paper needs a strong justification to describe the motivation for using a pre-trained language model as a basis for the weather foundation model. Are there any texts in the Meteorology dataset? 

2) The paper’s contents organization could be improved. For example, the figure 1 is too complex to understand. In Eq 2, it would be better to write the function as F(theta, D). Because F(theta | D) is easily confusing with the conditional distribution. Moreover, some technique details should not be placed in the appendix. It is a usual assumption that the main paper is self-contained so that the readers can understand the paper without reading the appendix. 

3) In the contribution part (lines 73 - 94), the last two items look like the advantages of the proposed method rather than contributions.
questions: questions: 1) Does the proposed method only focus on time series data?

2) Would you please highlight the main research question in plain language? It seems there are many components mixed in the paper. 

3) In Section 3, what’s the federated aggregation mechanism? Is there any pseudo code to describe the algorithm?

4) Can you please explain the token embedding for time series data? Is the token to be described as a special pattern of a subsequence of the time series? How to define the vocabulary of the tokens?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes LM-WEATHER, which builds upon previous work to explore further the powerful capabilities of PLMs in modelling meteorological variables. By learning sequence modelling capabilities from natural databases and applying them to on-device heterogeneity meteorological variable modelling, a lightweight adapter was developed to provide weather pattern awareness. Additionally, two real-world meteorological station datasets ODW1 & ODW2 support regional weather forecasting. Abundant experiments were designed to verify the feasibility of LM-WEATHER in modelling meteorological variables.

The article work is more valuable to further validate the superiority of PLMs in modelling meteorological variables and the provision of weather station data that can be used as a baseline dataset for weather station forecasting.
strength_weakness: strength_weakness: 1.	Figure 1 does not intuitively reflect the contributions of the paper. The Fig.1F mentioned in line 68 seems to be missing; additionally, the explanation for Figure C should be Task Adapter Generation, which is hard to understand. It is recommended to keep the labels a, b, … in the figure consistent with A, B, … in the text.
2.	The paper discusses sequence modeling in the time dimension of stations. Can this method be validated for spatiotemporal modeling?
3.	The experiments in the paper mainly focus on the accuracy of forecasting time. Can further validation be done on the timeliness and stability of the forecasts?
4.	To better present the experimental data, some statistical charts can be drawn to visually demonstrate the advantages of LM-WEATHER in meteorological variable forecasting.
questions: questions: 1.	In the Zero-Shot Learning (Out of Distribution Modeling) experiments mentioned in line 282, the domain transfer experiments for regional forecasting on the OWD1 dataset showed that the transfer performance was not as good as the forecasting performance of GPT-4. Can you explain the reason for this?
2.	In Theorem 4.2, how is it demonstrated that Low-Rank Matrices ensure privacy? Are there any related research papers on this topic? Can you provide a detailed explanation?
3.	Currently, there are many sparse station forecasts. Can interpolation be performed based on sparse station data to further extend the model to spatial dimension forecasting and enhance its generalizability?
4.	LM-WEATHER seems to outperform current methods. Can it maintain stability in long-term forecasting, and what is the maximum forecasting time it can achieve?
5.	In Table 6, line 305, it appears that having more trainable parameters is better. If the comparison experiment groups had the same number of trainable parameters, what would the results be?
limitations: limitations: Regarding limitation 1, further validating the impact of increased data scale on model performance using large-scale ERA5 data would be beneficial. Utilizing ERA5 as pre-training data to model invariance in meteorological forecasting and learning fundamental meteorological variable patterns, followed by fine-tuning with real-world data, could be a highly valuable endeavor.
paper_summary: paper_summary: This paper proposes an approach called LM-WEATHER that utilizes pre-trained language models (PLMs) as foundation models for on-device modeling of heterogeneous meteorological variables. LM-WEATHER enhances PLMs-equipped devices with local weather knowledge through a lightweight personalized adapter. Additionally, it leverages low-rank based transmission to fuse global knowledge among devices, enabling high-efficiency communication.This approach provides an effective solution for modeling real-world, heterogeneous, and continuously updated weather data, without requiring high resource demands. I believe this is a meaningful contribution to the field.
strength_weakness: strength_weakness: My first concern is that since the personalized adapter is already being used to adapt to the modeling of heterogeneous weather data collected from different weather station devices, what is the point of using the averaging operations during inter-device communication? This requires further explanation.

The definitions and formulations of symbols and equations throughout the sections, from Section 3.1 Local Training to Section 4 Theorems, are disorganized and complex. Additionally, there are numerous instances of incorrect singular and plural word usage. It is recommended to rewrite these sections with increased clarity and accuracy, simplifying the content for better comprehension.
questions: questions: This paper uses GPT-2 as the PLM, which is a very basic pre-trained language model. I am curious why more advanced PLMs like GPT-4 [1], Llama [2], and Vicuna [3] were not considered for better sequence modeling performance.

Reference: 
[1] Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.
[2] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.
[3] Chiang W L, Li Z, Lin Z, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
limitations: limitations: Yes

Comments: comment: Dear Reviewer C54F,

Thank you for raising the rating of our paper. We are happy to have addressed your concerns.

Best regards,

Authors
I appreciate the authors' response, which addresses most of my concerns. I am happy to raise my rating.
Dear Reviewer Jjnz,

Thank you for upgrading your score. We appreciate your constructive suggestions for improving our paper.

Best,

Authors
Thanks a lot for your response and revisions. I have raised my score to reflect such improvement.
Dear Reviewer tT2F,

We appreciate your recognition of our approach's efficiency and significance. Your thoughtful comments also provide us with valuable new insights, for which we are grateful.

Best,

Authors
Thank you for your confirmation. We are pleased that our rebuttals have addressed your concerns and appreciate your continued support for our paper.

Best regards,

Authors
Thank you for your response. This approach offers an efficient solution for modeling heterogeneous and continuously updated real-world weather data, while minimizing resource demands. I believe this is a significant contribution to the field.
Many thanks for your rebuttals. After carefully checking it, my questions have been answered and  I will keep my score.