Title: Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Foundation Model

Cross-Modality Adaption

Time Series Forecasting

Federated Learning
Abstract: Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. 
While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains.
Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances.
To address these issues, we propose Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs.
Specifically, we begin by transforming time series into the modality of text tokens.
To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially.
Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. 
Our comprehensive experiments indicate that Time-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster.
The code is available at https://github.com/CityMind-Lab/NeurIPS24-Time-FFM/tree/main.

TIME-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

Abstract

Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains. Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances. To address these issues, we propose TIME-FFM, a Federated Foundation Model for TIME series forecasting by leveraging pretrained LMs. Specifically, we begin by transforming time series into the modality of text tokens. To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially. Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. Our comprehensive experiments indicate that TIME-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster. The code is available at https://github.com/CityMindLab/NeurIPS24-Time-FFM/tree/main.

1 Introduction

Time series forecasting plays an important role in many real-world application domains [1], such as energy consumption prediction, weather forecasting, and disease transmission. Recently, a multitude of deep learning models have been designed for time series forecasting based on Convolutional Neural Networks [2, 3, 4], Recurrent Neural Networks [5, 6], and Transformers [7, 8, 9, 10]. Inspired by the prominent performance gained by Foundation Models (FMs) in the realms of Natural Language Processing (NLP) [11, 12, 13, 14] and Computer Vision (CV) [15, 16], great research interests have been triggered to build pretrained FMs for time series community [17, 18, 19, 20, 21]. Nonetheless, due to significant time series data scarcity, these FMs are of poor capability to cultivate general representations, failing to promise remarkable fine-tuning or zero-shot performance for diverse downstream forecasting tasks [22, 23]. As a result, a collection of methods have been proposed to borrow the pretrained language FMs to time series community by cross-modality adaption [24, 22, 23], thus unlocking the tapped potential of language models (LMs) for time series modeling.

(a) Domain-Specific Forecasting

(b) Cross-Domain Forecasting

(c) FL In-Modality Adaption

(d) FL Cross-Modality Adaption

Disease

Economics

Model A

Model B

Disease

Economics

Pretrained

LM

Head

Pretrained

LM

Head

server

Source Data Sample

Target Data Sample

Source Task

Target Task

Global parameters

Fine-tune

Frozen

Pretrained LM

Head

Pretrained LM

Head

Server

Cross-Modality

Adaption

Cross-Modality

Adaption

00000

个

雨

While these endeavors provide FMs for time series forecasting, the incorporated cross-modality adaption modules and unfrozen components of pretrained LMs need training from scratch for specific domains, thus restricting the mining of underlying temporal commonality in cross-domain time series data. As is shown in Figure 1(a), disease and economics datasets are employed for training the FM respectively to obtain domain-optimal model parameters, hardly generalizing to other domains. [25] proposes to train a unified model (named UniTime) on the mixture of cross-domain time series data (Figure 1(b)), which ensures the cultivation of general-purpose representations, thus promising the zero-shot performance on unseen domains. Despite its effectiveness, they adopt the centralized training mode, where the historical records of time series across diverse domains are uploaded to a central server for optimizing the unified model. Due to copyright protection and privacy concerns, data owners may hesitate to share the access to these domain-specific raw records.

Federated Learning (FL) [26, 27] provides the mainstream solution for the aforementioned problem, where data owners train prediction models locally and exchange the intermediate model parameters or gradients with the central server, without the disclosure of raw data records. Moreover, in UniTime, a retractable prediction head is introduced to accommodate the heterogeneous output needs whereas FL paradigm makes it possible to construct domain-customized heads. However, current efforts are merely focused on how to fine-tune LMs in federated setting for NLP tasks (i.e., in-modality adaption of LMs for target tasks in Figure 1(c)) [28, 29, 30, 31], rather than cross-modality adaption of LMs for time series forecasting. The realization of this federated FM is non-trivial technically, given the ubiquitous heterogeneity in cross-domain time series data. (1) Heterogeneous inputs : Cross-domain time series data input into the FM are heterogeneous in terms of dimensions and historical readings, posing evident difficulty to modality alignment. (2) Rigid instructions as prompts : Prompts are adopted to bootstrap LMs for time series reasoning hinging on rigid domain-specific instructions [25, 22], rather than the understanding of LMs, exhibiting poor robustness for unseen domains. (3) Conflicts between generalization and personalization : The ideal FM needs to learn the common temporal representations across domains and simultaneously enable the personalized prediction for domain-specific inputs.

To address the challenges, we propose TIME-FFM, a Federated Foundation Model for TIME series forecasting by repurposing LMs (Figure 1(d)). First, we perform modality alignment by transforming time series data into text tokens to empower the pretrained LM for time series reasoning. Second, we design a prompt adaption module to dynamically determine domain-specific prompts, which can bootstrap the LM for cross-domain time series analysis from the perspective of LM itself, rather than from human cognition by employing hand-crafted instructions as prompts. To tackle the data heterogeneity across domains, we introduce a personalized federated training strategy by learning a global encoder and personalized prediction heads, given the shared representations across domains. Our main contributions are summarized as follows.

We present the first attempt to build a federated FM for time series forecasting by exploiting the sequence reasoning potential of LMs, avoiding the disclosure of local data.

We propose TIME-FFM, which firstly aligns the modality from time series data to natural language and adaptively determines prompts to guide the LM for time series reasoning. Moreover, we intro-

duce a personalized FL strategy to strike a balance between sharing common temporal knowledge and ensuring customized prediction results.

The extensive evaluation results demonstrate that TIME-FFM leads to state-of-the-art performance in mainstream forecasting tasks, especially in few-shot or zero-shot forecasting settings.

2 Related Work

FMs for Time Series Forecasting. Recent studies have demonstrated the effectiveness of fine-tuning pretrained FMs for various downstream tasks, such as BERT [11], GPT [12], GPT2 [13], and LLaMA [14] in NLP and DEiT [15] and BEiT [16] in CV. Inspired by the success, some efforts have been focused on developing FMs for time series community, such as [17, 18, 21, 32]. However, due to data deficiencies, these pretrained models cannot guarantee the learning of general-purpose representations for time series analysis and hence fail to apply to a multitude of downstream tasks. Another line of researches attempt to leverage pretrained FMs in NLP or CV for time series analysis by crossmodality adaption strategies [33, 34, 35, 24, 23, 22], such as fine-tuning and model reprogramming, which hinges on the powerful generalization capability of Transformers for sequence tokens. [23] freezes the self-attention modules and feedforward layers of GPT2, and only fine-tunes the positional embedding and normalization layers. The proposed GPT4TS outperforms the relevant models in most time series tasks. On the contrary, [22] freezes the LM as a whole and transforms the modality of time series to natural language by patch reprogramming. These methods enable unified model structure rather than unified parameters for diverse downstream tasks, which makes the proposed FMs learn impaired temporal commonality. [25] proposes to train a unified prediction model for cross-domain time series forecasting, which enables to learn the intrinsic temporal patterns. However, the centralized training mode brings privacy concerns for cross-domain data owners and FL paradigm may provide a promising solution.

Federated Fine-tuning of LMs. Given the exceptional performance of LMs and the emerging privacy preserving resolutions, incorporating LMs with FL is becoming a popular research trend. There have been some implementation frameworks [36, 29, 37, 38, 39, 40] to support fine-tuning LMs in FL setting. Moreover, considering the immense communication cost, some communication-efficient federated fine-tuning methods have been proposed, such as [38, 41, 30, 28]. A few researches aim to investigate the effects of data heterogeneity on fine-tuning performance, and then propose the personalized federated instruction tuning methods, e.g., [42, 29, 31]. Nonetheless, these methods concentrate on fine-tuning or fully-tuning pretrained LMs in FL paradigm for NLP tasks, but fail to cover the cross-modality adaption of LMs for time series forecasting.

3 Methodology

3.1 Problem Definition

Given N domains, let x i,t = { x 1 i,t , · · · , x c i i,t } ∈ R c i denote the observation of domain i at the time step t , where c i represents the number of dimensions (channels). In the context of time series forecasting, we denote X i = { x i, 1 , · · · , x i,L i } ∈ R L i × c i as the input of the prediction model f i ( · ) , where L i represents the domain-variant lookback window. The ground truths can be denoted as Y i = { x i,L i +1 , · · · , x i,L i + F i } ∈ R F i × c i , where F i represents the future prediction window. Let D i = { ( X i ; Y i ) } denote the local data set of i and D i = |D i | the data size. Given the set of personalized model parameters { w i } , the objective of federated FM for cross-domain time series forecasting can be formulated as

3.2 Model Structure

The model architecture is elaborated in Figure 2. Our model encompasses three components: (1) modality alignment and prompt adaption, (2) LM backbone, and (3) prediction head. The modules of modality alignment and prompt adaption are designed for cross-modality alignment and adaptive

Flatten & Linear

De-Normalize

Pre-trained

Word Embeddings

Linear

Multi-Head Attention

Top-M-Score Prototypes

Linear

Prompt Tokens

Normalize

Patching

Linear

Modality Alignment

& Prompt Adaption

Economics

Prediction

Head

LLM

Encoder

Prompt & Patch Embedder

Disease

Prediction

Head

LLM

Encoder

Prompt & Patch Embedder

Weather

Prediction

Head

LM

Backbone

Modality Alignment

& Prompt Adaption

Frozen

Training

Future Predictions

Server

①

②

Modality Alignment

④

⑤

⑥

③

Prompt Adaption

Embedding

Prompt

Frozen prompt determination. We employ the backbone of GPT2 [13] with freezing all parameters. The prediction head enables domain-specific prediction results.

Modality Alignment. Here we transform time series into the modality of text tokens. To accommodate domain-variant channels c i , we adopt the channel-independent strategy [43] to split multivariate time series X i into c i univariate series and individually process each. Let X j i = { x j i, 1 , · · · x j i,L i } ∈ R 1 × L i denote the j -th univariate series from X i . Then we normalize each series X j i to mitigate the effect of distribution diversity [44]. Since each data point of X j i does not have explicit semantic knowledge like words in sentences, we adopt the patching technique [43] to segment X j i into subseries (termed patches ), each of which can aggregate the local information and better retain the temporal knowledge . Specifically, let P denote the patch length and S i denote the stride length of domain i . Hence, the number of patches can be defined as B i = ⌈ L i -P S i ⌉ +1 . We denote X j i,S ∈ R B i × P as the generated patches from X j i . We subsequently employ a linear layer to project the patches into tokens ˆ X j i,S ∈ R B i × D , where D is the input dimension size of the LM backbone. ˆ X j i,S together with prompt tokens (in the next part) will be input into the LM backbone.

Prompt Adaption. In the time series forecasting FMs based on LMs, domain instructions are designed as prompts to complement the patch tokens and inform the LM backbone of domain-specific knowledge [22, 25]. These manually-designed prompts depend completely on experts' knowledge and may vary from each other due to different understandings. Furthermore, according to the results, more detailed instructions can always yield better prediction performance [22], which may make us naturally draw a conclusion that the ultimate performance hinges on the length of prompts. However, longer prompt tokens will present substantial challenge on the computation burden. Different from images [45] or acoustic data [33], which can be 'translated' into natural language seamlessly, the manually-crafted prompts are error-prone to describe the characteristics of the raw time series. To this end, a better way is to design prompts from LM's understandings of the patch tokens rather than human cognition of raw time series data. Here, we propose to adaptively determine prompts

Training

based on patch tokens from the source corpus of pretrained LM (which includes V pretrained word embeddings, denoted as E ∈ R V × D ). Similar to [22], we project E to a smaller collection of text prototypes , denoted as E ′ ∈ R V ′ × D by a linear layer, with V ′ ≪ V , to avoid the potential large parameter space. We adopt a modified multi-head attention layer to obtain the correlation between E ′ and ˆ X j i,S , and subsequently select M mostly related text prototypes as prompts. Concretely, for each head h ∈ { 1 , · · · , H } , we have the query matrix Q j i,h = E ′ W Q h and the key matrix K j i,h = ˆ X j i,S W K h , where W Q h , W K h ∈ R D × d and d = ⌊ D H ⌋ . Since we do not aim to return a weighted value matrix according to the given query but merely evaluate the correlation of text prototypes and patch tokens, we omit the value matrix here. The attention score matrix is denoted as O j i,h ∈ R V ′ × B i and can be calculated as

Weobtain ˆ O j i,h ∈ R V ′ × 1 by calculating the summation of O j i,h per row. Each value in ˆ O j i,h represents the correlation degree of the corresponding text prototype in E ′ to all patch tokens ˆ X j i,S . We select M prototypes from Q j i,h with top attention scores to form the potential prompts Z j i,h ∈ R M × d , i.e., Z j i,h = Q j i,h [ TOPM ( ˆ O j i,h ) ] . We can obtain Z j i ∈ R M × D by aggregating Z j i,h from all H heads. Finally, we employ a linear layer to project Z j i to the prompt tokens ˆ Z j i ∈ R M × D .

Prediction Head. We input the concat of ˆ Z j i and ˆ X j i,S into the LM backbone and obtain the representations R j i ∈ R ( M + B i ) × D , which will be flattened and projected to the final results ˆ Y j i ∈ R 1 × F i by a linear layer.

Personalized Strategy. Time series across different domains could be substantially heterogeneous. Consequently, a generalized global model in FL may fail to capture the disparate temporal patterns and ultimately compromises the prediction performance. Inspired by [46], which indicates that diverse data may share common feature representations, we propose to learn a global encoder (i.e., modality alignment , prompt adaption and LM backbone ) and domain-customized prediction heads . The underlying motivation is to strike a balance between generalization and personalization: (1) increasing the generalization of modality alignment and prompt adaption by access to cross-domain temporal patterns; (2) ensuring prediction results specific for certain domains by personalized heads. Since we keep the LM backbone intact, in each federating round, only the parameters of modality alignment and prompt adaption are communicated . The server performs aggregation by averaging strategy. The training strategy differs from Federated Averaging framework, where the parameters of encoder and decoder are both aggregated at the central server after local optimization.

3.3 Training Process

We denote w g t as the global parameters of modality alignment and prompt adaption at the t -th federated round and w p i,t as prediction head parameters of i at the t -th round. We clarify that ( X i , Y i ) here is reused to represent a training batch. ˆ X i,S , ˆ Z i , R i , and ˆ Y i denote the patch tokens, prompt tokens, representations and prediction results of such batch respectively. The training procedure of TIME-FFM is elaborated in Algorithm 1. (1) In the t -th federated round, the server distributes the global parameters w g t (Line 8 and 9). (2) Each domain loads the global parameters and local head parameters to perform prediction following modality alignment, prompt adaption as well as representation obtaining from LM backbone (Line 12-15) and uploads w g t,i to the server after optimization. (3) Finally, the server aggregates local updated parameters by averaging mechanism to obtain the fresh global parameters w g t +1 for the ( t +1) -th round (Line 6).

4 Experiments

We comprehensively compare the proposed TIME-FFM with state-of-the-art models in FL or centralized settings, especially those by fine-tuning LM for time series forecasting. The numerical results demonstrate the effectiveness of TIME-FFM in time series forecasting. We employ GPT2

Input: Global round number T , local epoch number E , initial global encoder parameters w g 0 , initial personalized head parameters { w p } , local batch number b .

i, 0 i Output: Optimized global encoder parameters w g T , optimized parameters of personalized heads { w p i,T } . 1 SERVEREXECUTE: )

6 // for local training

backbone of the first 6 layers as the default LM backbone and freeze all parameters. To guarantee a fair comparison, we adhere to the experimental configurations in [25].

Baselines. Our baselines cover a board collection of relevant methods, which can be categorised into 3 types: TY1 ( federated fine-tuning methods ): FedIT [31], FedAdapter H [47, 41], and FedAdapter P [48, 41]; TY2 ( across-dataset centralized methods ): UniTime [25], GPT4TS [23], and PatchTST [43]; 2 TY3 ( dataset-specific centralized methods ): TimesNet [4], DLinear [49], FEDformer [50], Autoformer [10], and Informer [9]. We directly cite the results from [25] if applicable.

Setups. We evaluate on 8 benchmark datasets from various domains: ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Weather, Exchange, and ILI, which have been widely adopted for evaluating time series forecasting performance. Each dataset corresponds to a FL participant. Detailed introduction of implementation and datasets can be found in Appendix A. We use Mean Square Error (MSE) and Mean Absolute Error (MAE) as the evaluation metrics.

4.1 Main Results

Main forecasting results are presented in Table 1. TIME-FFM consistently outperforms the other FL methods (in TY1 ) on all datasets, except ETTh2. Specifically, TIME-FFM can improve the performance gains over all datasets by 39.01% in terms of MSE, compared with the second bestperformed FL method. Furthermore, the averaged prediction results of TIME-FFM are even superior to those of the centralized models. When compared with UniTime, the recently-proposed centralized unified model for cross-domain time series forecasting, TIME-FFM can provide more performance gains, which underscores the effectiveness of the proposed cross-modality adaption modules and personalized approach.

4.2 Few-Shot Forecasting

Given the remarkable few-shot learning performance of LMs, we evaluate whether TIME-FFM can retain such capability for time series forecasting. In this section, we compare the prediction performance across TY1 and TY2 in few-shot settings with 10% and 5% time steps adopted as training samples, which is in line with the setups in [23, 22].

Main results of 10% and 5% few-shot forecasting are presented in Table 2 and 3 respectively. TIMEFFM outperforms the other FL methods and even achieves comparable performance in contrast to the centralized methods, which further underscores that TIME-FFM inherits the few-shot capability of LMs and promises proficient FM for time series forecasting. Specifically, TIME-FFM outperforms the centralized methods in the realm of 5% few-shot learning, with 20% reduction in averaged MSE w.r.t UniTime. Interestingly, for all methods except UniTime, results in 10% few-shot learning are worse than those in 5% few-shot learning. We deduce that the pretrained LM is fully-tuned in

UniTime and fewer training samples fail to support optimizing masses of parameters. While in the other methods, the pretrained LMs are frozen or fine-tuned, which can retain the original reasoning capability of LMs even with fewer training instances.

4.3 Zero-Shot Forecasting

Given that language FMs are effective zero-shot forecasters, we evaluate the zero-shot learning capability of TIME-FFM, which is essential for a FM. We adhere to the zero-shot learning settings in [25], where we first train TIME-FFM on ETTh1, ETTm1, and ETTm2, and then evaluate the zero-shot testing performance on ETTh2, Electricity, and Weather.

Since ETTh2 hails from the same domain of ETTh1, we directly reuse the local parameters (including both encoder and head) of ETTh1 for inferring ETTh2. For the other two target datasets from different domains of the source datasets, we successively reuse local parameters of the three source datasets to perform zero-shot testing. The results presented in Table 15 show that local parameters of ETTh1 excel on both target datasets. Hence, we adopt the model parameters of ETTh1 for zero-shot testing on Electricity and Weather. For other methods in TY1 , we train an optimized global model on ETTh1, ETTm1, and ETTm2, and then adopt the obtained global model to conduct zero-shot testing on ETTh2, Electricity, and Weather. The comparison in zero-shot forecasting is presented in Table 4. TIME-FFM consistently ensures significant performance gains on all three datasets, with prediction MSE decreasing by 13.9% w.r.t the second best. It is remarkable that the centralized unified model UniTime exhibits inferior zero-shot testing performance compared to TIME-FFM. We attribute the performance gains of TIME-FFM to the valid knowledge transferability across domains.

4.4 Model Analysis

Model Ablation. We conduct ablation studies on five variants of TIME-FFM and the corresponding results are presented in Table 5 ( A.1 -A.6 ). Thereinto, TIME-FFM-D represents the distirbuted version of TIME-FFM, which ablates the aggregation process. The results demonstrate that ablating either components will compromise the forecasting performance. We have the following key observations: (1) The prompt tokens can bootstrap the LM for target domains. The absence of prompt adaption ( A.2 ) will affect the forecasting performance. When employing instructions in [25] as prompts, A.3 is inferior to TIME-FFM, which underscores the efficacy of prompt adaption. (2) The ablation of personalized heads ( A.4 ) will hurt the performance most. In A.4 , a global prediction head is learned for all domains, hardly ensuring the personalization for cross-domain heterogeneous data. (3) In A.6 , the common temporal knowledge fails to be shared among domains, which makes poorer generalization of cross-modality adaption modules, thus yielding inferior performance. This underscores the significance of building a unified model for cross-domain traffic series forecasting.

Language Model Variants. We investigate the variants of LM, in terms of optimization modes ( B.1 -B.3 ) and backbone layers ( C.1 and C.2 ). Here we train all variants on seven datasets except Electricity, due to GPU memory limitation. In B.3 , the backbones of LM are full-tuned. While in B.2 , we only tune the positional embeddings and layer normalization components of the backbone [23]. Table 6 shows that B.3 performs best, followed by B.2 and B.1 . We argue that the performance remains comparable when we freeze all backbone parameters. This demonstrates that LMs are capable in processing time series tokens by effectively modality alignment. In C.1 and C.2 , 6 and 12 backbone layers are adopted. The results shows that more backbone layers ensure better performance, which indicates the scaling laws of LMs retain in TIME-FFM for time series forecasting [51, 22].

Model Efficiency. Table 7 demonstrates that TIME-FFM can reduce the training parameter quantity and communication overhead with insignificant increase in training time. Moreover, the number of training parameters and communication parameters keeps intact , regardless of backbone layers.

Case Study. We provide a case study on prompt adaption in Figure 3. (a) shows the attention scores between 6 patch tokens and 100 text prototypes for 8 heads on ETTh1 dataset. For each head, only a small set of text prototypes (columns) have remarkable scores, which indicates that each patch token is only related to limited pretrained word embeddings and dynamically prompt adaption is promising. (b)-(d) show top M prototypes of 8 heads on ETTh1, Electricity, and ILI respectively. Darker colors correspond to text prototypes with higher attention scores. From these three subplots,

1

25

50

75

100

0.0

0.12

0.25

0.37

0.50

0.62

(a) Multi-head attention scores on ETTh1

1

25

50

75

100

head_1

head_5

0

7

12

(b) Top M prototypes on ETTh1

1

25

50

75

100

head_1

head_5

(c) Top M prototypes on Electricity

1

25

50

75

100

head_1

head_5

(d) Top M prototypes on Illness

head_1

head_2

head_3

head_4

head_5

head_6

head_7

head_8

we have the following key observations: (1) different datasets correspond to variant text prototypes; (2) the distribution of text prototypes on different datasets has commonality, i.e., gathering in shadow areas. These observations indicate the global prompt adaption module has great generalization for diverse datasets and simultaneously ensures personalization across various domains.

5 Conclusion and Discussion

In this paper, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-

purpose learning and personalized prediction. Specifically, given the differentiation of dimensionality and horizon, we introduce the modality alignment module encompassing the channel-independent and patching techniques, which may follow the track of GPT4TS and Time-LLM. For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions. Due to cross-domain time series heterogeneity, we devise a personalized federated strategy, with global encoder and personalized prediction heads.

Rationale of TIME-FFM. Compared with the modality of text, time series is more domain-specific and copyright-sensitive, i.e., private knowledge may be inferred from historical time series readings, especially in finance and healthcare domain. Hence, it is of great significance to take data privacy into account when constructing time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due to data license restriction, such as Kaggle public datasets. Hence, our work uniquely bridges the gap between foundation models and federated learning, which not only enhances the privacy and applicability of foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in this field.

Limitations and Future Works. We recognize some limitations of our work: the training time is increased compared with the TY1 and the performance in some case is suboptimal. In the future work, we will explore more effective and efficient modality alignment strategies. Moreover, further researches will investigate the correspondence between patch embeddings and word embeddings to explore whether time series data can be seamlessly 'translated' into natural language.

Acknowledgments and Disclosure of Funding

This work is mainly supported by the National Natural Science Foundation of China (No. 62402414). This work is also supported by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2024A03J0620), Guangzhou Municipal Science and Technology Project (No. 2023A03J0011), the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), and a grant from State Key Laboratory of Resources and Environmental Information System, and Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B1212010007).

References

Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4836-4837, 2022.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2022.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations , 2022.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting , 36(3):1181-1191, 2020.

Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting.

In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages 146-156, 2022.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2019.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , pages 11106-11115, 2021.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019 , 2019.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations , 2021.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems , 35:3988-4003, 2022.

Shohreh Deldari, Hao Xue, Aaqib Saeed, Jiayuan He, Daniel V Smith, and Flora D Salim. Beyond just vision: A review on self-supervised representation learning on multimodal and temporal data. arXiv preprint arXiv:2206.02353 , 2022.

Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 , 2023.

Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. arXiv preprint arXiv:2403.14735 , 2024.

Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Y Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, et al. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations , 2024.

Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems , 36, 2024.

Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469 , 2023.

Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024 , 2024.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 9963-9977, 2023.

Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7871-7888, 2023.

Ningxin Su, Chenghao Hu, Baochun Li, and Bo Li. Titanic: Towards production federated learning with large language models. In IEEE INFOCOM 2024 - IEEE Conference on Computer Communications , 2024.

Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6915-6919. IEEE, 2024.

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 , 2024.

Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning , pages 11808-11819. PMLR, 2021.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023.

Pin-Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 22584-22591, 2024.

Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309.00363 , 2023.

Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. arXiv preprint arXiv:2402.06954 , 2024.

Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049 , 2023.

Ronghui Xu, Hao Miao, Senzhang Wang, Philip S Yu, and Jianxin Wang. Pefad: A parameterefficient federated framework for time series anomaly detection. In SIGKDD , pages 3621-3632, 2024.

Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards llm-empowered time series forecasting via cross-modality alignment. arXiv preprint arXiv:2406.01638 , 2024.

Guangyu Sun, Matias Mendieta, Taojiannan Yang, and Chen Chen. Conquering the communication constraints to enable large pre-trained models in federated learning. arXiv preprint arXiv:2210.01708 , 2022.

Pengyu Zhang, Yingbo Zhou, Ming Hu, Junxian Feng, Jiawen Weng, and Mingsong Chen. Personalized federated instruction tuning via neural architecture search. arXiv preprint arXiv:2402.16919 , 2024.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2022.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2021.

Diganta Misra, Agam Goyal, Bharat Runwal, and Pin Yu Chen. Reprogramming under constraints: Revisiting efficient and reliable transferability of lottery tickets. arXiv preprint arXiv:2308.14969 , 2023.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning , pages 2089-2099. PMLR, 2021.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 487-503, 2021.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , pages 11121-11128, 2023.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pages 27268-27286. PMLR, 2022.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations , 2024.

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations , 2020.

Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling crossembodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408.11812 , 2024.

Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. arXiv preprint arXiv:2410.10469 , 2024.

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning , 2024.

A Experimental Details

Implementation. The Adam optimizer with the initial learning rate of 10 -4 is adopted in the training process. The lookback window L i is set to 36 for the ILI dataset, and 96 for the others. The future prediction window F i is set to { 24 , 36 , 48 , 60 } for the ILI dataset, and { 96 , 192 , 336 , 720 } for other ones. We adopt the pretrained GPT2-backbone of the first 6 layers as the LM encoder. The local epoch E is set to 1 for all domains. The global round number T is set to 100. V ′ , M , P and H are set to 100, 12, 16, and 8 respectively for all domains. S i is set to 4 for the ILI dataset, and 16 for other ones. In each round, we calculate the averaged values of validation loss. The round with lowest validation value serves as the optimal round, and then the corresponding model is used for test. All models are implemented on PyTorch with all experiments conducted on NVIDIA A100-80G GPUs.

Training Configurations. The experimental evaluations are conducted on 8 real-world benchmark datasets which include 5 domains. We present the detailed description of these datasets in Table 8. For fair comparison, we perform batch division and oversampling as per [25]. In each federated round, we do not train local models with all training samples, considering large quantity of training samples. Instead, we proportionately calculate the number of batches for each domain in the following steps. (1) We calculate the summation of training batches over all datsets before oversampling. (2) We count training times of each domain after oversampling, i.e., 13 for ILI and 1 for the others, and then we perform normalization to obtain a batch ratio for each domain, i.e., 0.65 for ILI and 0.05 for the others. (3) we can obtain the number of training batches for each domain (denoted as b i ) by multiply the summation (in (1)) and ratios (in (2)) respectively. Actually, for ILI the value is higher than the number of training batches, while the opposite is true for the others. In each round, each local model is trained with training batches sequentially until b i is reached.

We evaluate the effectiveness of oversampling strategy in TIME-FFM and present the results in Table 9. 'w/o OverSampling' represents each local model is trained with all local batches in each FL round. We attribute the performance gains in TIME-FFM to it that the introduction of oversampling strategy can balance the contribution to the global knowledge. For ILI, despite data sparsity, its local knowledge can be augmented in the global encoder. We observe that such local knowledge can enhance forecasting for not only ILI itself but also the other domains.

B Hyperparameter Sensitivity

In this section, we conduct hyperparameter investigation of 3 important hyperparameters, i.e., the number of text prototypes V ′ , the number of prompt tokens M , and the number of self-attention heads H . Figure 4 shows prediction performance on ILI dataset with the variation of the 3 hyperparameters respectively. We have the key observations as follows: (1) When the value of V ′ is lower, word

20

50

100

200

2.0

2.2

2.4

2.6

MSE

V'

(a)

4

8

10

12

2.0

2.4

2.8

MSE

M

(b)

4

8

16

32

2.2

2.4

2.6

MSE

H

(c)

predict-24

predict-36

predict-48

predict-60

predict-24

predict-36

predict-48

predict-60

predict-24

predict-36

predict-48

predict-60

embeddings are projected into less text prototypes. Each prototype will contain both relevant and irrelevant knowledge, which will affect the accuracy of prompt adaption. When text prototypes are more, a stable number of prompt tokens will not cover all relevant knowledge. Hence lower or higher values of V ′ will yield subpar performance. (2) Fewer prompt tokens may not fully cover the useful knowledge. Hence, the best performance is achieved when M is equal to 12. (3) Increasing the number of attention heads cannot always promise better performance because more heads may break the semantic integrity of text prototypes and patch embeddings.

C Additional Results

We compare forecasting performance with PatchTST-FL and DLinear-FL, the federated version of PatchTST and DLinear. As is presented in Table 10, TIME-FFM consistently outperforms the two novel federated methods on all datasets.

We further compare the forecasting performance with three baselines, i.e., iTransformer [52], NBEATS [53], and Crossformer [54]. These three baselines can be categorized into TY3 . The numerical results are presented in Table 11. We have the key observation that TIME-FFM, though trained in federated paradigm, can outperform these three centralized methods.

Some researches delve into training a foundation model from scratch based on the collected time series datasets [55, 32, 56]. We compare our proposed federated foundation model with Moirai [32] and MOMENT [56] in Table 12. Notably, TIME-FFM achieves comparable performance with the two foundation models which are pre-trained firstly on large-scale time series archive.

D Full Results

Full results of forecasting performance comparison on 8 time series benchmarks are presented in Table 13. TIME-FFM exhibits SOTA performance in 32 out of 42 instances , which demonstrates the effectiveness of the cross-modality adaption module, i.e., modality alignment and prompt adaption, as well as the personalized prediction heads.

Our complete results of performance comparison in 10% and 5% few-shot settings are presented in Table 14 and 16 respectively. In both settings, TIME-FFM outperforms the other FL methods in TY1 . In the setting of 10% few-shot forecasting, TIME-FFM achieves comparable performance against methods in TY2 . In the setting of 5% few-shot learning, TIME-FFM attains SOTA performance on 20 out of 48 instances across five time series benchmarks. The results underscore that TIME-FFM promises effective few-shot forecaster.

E Error Bars

Weconduct the experiments of TY1 for three times and report the mean values and standard deviations in Table 18. The results demonstrate the superiority of our proposed TIME-FFM, which agrees with Table 1.

F Border Impacts

In this paper, we propose to build a foundation model for time series forecasting hinging on the impressive capability of pretrained language models for sequence tokens reasoning. The promising advantages are two folds: (1) Data owners do not need to share the access to the private data samples which mitigates the privacy concerns and cater for data protection regulations (say GDPR). (2) The problem of 'data island' can be tackled, which makes it possible to generate satisfactory performance in spite of data scarcity. To the best of our knowledge, our research do not have obvious negative social impacts.

Method

TIME-FFM

FedLoRA

FedAdapter

H

FedAdapter

P

Metric

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

ETTh1

0.446±0.006

0.434±0.001

0.483±0.002

0.462±0.001

0.478±0.020

0.467±0.013

0.496±0.008

0.482±0.004

ETTh2

0.383±0.001

0.407±0.001

0.375±0.001

0.397±0.002

0.375±0.002

0.399±0.001

0.377±0.003

0.401±0.002

ETTm1

0.398±0.001

0.402±0.001

0.667±0.020

0.523±0.006

0.641±0.041

0.673±0.029

0.529±0.012

ETTm2

0.286±0.001

0.331±0.000

0.298±0.001

0.343±0.001

0.298±0.003

0.343±0.003

0.299±0.001

0.344±0.002

Electricity

0.216±0.002

0.299±0.002

0.377±0.012

0.464±0.012

0.359±0.059

0.449±0.050

0.368±0.030

0.457±0.032

Weather

0.274±0.005

0.291±0.004

0.284±0.002

0.310±0.001

0.283±0.002

0.310±0.003

0.285±0.002

0.311±0.002

Exchange

0.349±0.017

0.396±0.008

0.389±0.002

0.423±0.001

0.384±0.002

0.421±0.002

0.380±0.001

0.418±0.001

ILI

2.250±0.146

0.969±0.048

4.712±0.250

1.510±0.054

4.557±0.621

1.516±0.093

4.658±0.518

1.517±0.072

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and evaluation results in abstract and introduction are elaborated in Section 3 and 4 respectively.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and future work in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include the theoretical analysis for our proposed model, just like other relevant works.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We elaborate the our model architecture and how to perform training and inference in Section 3 and provide the experimental details in Section 4

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/CityMind-Lab/NeurIPS24-TimeFFM/tree/main.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the experimental settings and benchmark data in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the error bars in Appendix 18.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the implementation details in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conform with all terms of NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We analyze border impacts of our research in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not pose the risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the Section of Methodology (3) and Experiments (4) we have provided the proper citations for the adopted technologies and results.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have provided detailed information of the model and the available source code at https://github.com/CityMind-Lab/NeurIPS24-Time-FFM/tree/main.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study an important problem of time series forecasting considering the increasing concerns of privacy and copyright. The paper proposes a novel LLM-empowered federated time series forecasting model with three main components, i.e., modality alignment, prompt adaption, and personalized strategy.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The authors propose Time-FFM, a novel LLM-empowered federated foundation model for time series forecasting. Time-LLM encompasses a cross-modality module, a prompt adaption module, and a personalized federated training strategy.\n3. Extensive experiments show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. We often run federated learning on edge devices, which have restrictedly limited data processing capability. Time-FFM is difficult to deploy in real-world scenarios. Additionally, the high latency of LLM during inference may impede its applicability for real-time time series forecasting. This presents a practical challenge for deployment in resource-limited FL environments. Moreover, it requires at least a GPU to run LLM which is impractical and unrealistic in real-world FL scenarios.\n2. TimesFM [r1], a centralized foundation model for time series forecasting from Google, was accepted by ICML 2024. It was released on Arxiv last October. Google possesses various kinds of data from a variety of domains without any copyright and privacy concerns, which conflicts with your motivation. \n\n    [r1]. Das, et al., A decoder-only foundation model for time-series forecasting, ICML, 2024.\n\n3. More detailed explanations are needed regarding how the proposed method addresses data heterogeneity across clients.\n4. It would be better to improve and clarify the technique depth. Patching and channel-independent mechanisms are widely used in existing time series forecasting methods. In addition, the prompt adaption seems to be another version of an existing technique [22]. The projection head is also simple and straightforward. Further, the training process of Time-LLM is based on FedAvg.\n5. It is suggested to illustrate more on personalized strategy including the global encoder. In addition, it is also confusing how to deal with the conflicts between generalization and personalization.\n6. It would be better to convert TimesNet, DLiner, and PatchTST, into their federated version with FedAvg and compare their federated versions with the proposed Time-FFM. Additionally, it would be better to include a case study to intuitively show how accurately the proposed Time-FFM can predict time series.\n\nQUESTIONS:\n1. Are the practical deployment challenges considered?\n2. How to solve the bottleneck problem of limited FL resources? Could you provide more evidence?\n3. How to demonstrate the rationality of the prompt generated by LLM? How to justify the generated prompt is related to the corresponding domain? It would be better to provide more case studies on all datasets for a more comprehensive evaluation. In addition, it is especially encouraged to provide a corresponding theoretical analysis.\n\nLIMITATIONS:\nThe main limitation is the practicality of the proposed Time-FFM. In addition, please see the weaknesses and questions for other limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a federated foundation model for time series forecasting. This foundation model is trained in a distributed setting, with global shared parameters in a server, and domain / dataset specific parameters in clients. This allows the authors to personalize their predictions to local domain-specific data, while still learning general time series patterns. The global parameters belong to the prompt adaptation and modality alignment modules, whereas the local parameters include word embeddings, frozen LLMs, and client specific prediction heads. These methods are then compared with federated fine-tuning, other LLM adaptation methods for time series forecasting, and specialized time series models on standard time series forecasting datasets.\n\nSTRENGTHS:\n1. The paper is well written and easy to understand.\n2. The authors conduct multiple experiments including ablation experiments, which are well executed.\n\nWEAKNESSES:\n1. **Missing literature:** The authors do not cite or discuss two bodies of work which are pertinent to this discussion: (1) literature on time series foundation models, including models such as LagLLama [1], MOIRAI [2], TimesFM [3], Moment [4], Chronos [5] etc., (2) recent time series forecasting methods such as iTransformer [6], or older techniques such as N-BEATs [7] and N-HITS [8] etc. which have been shown to be strong time series forecasters, and (3) literature on federated learning for time series data.\n2. **Rationale and contributions:** I am not sure about the motivation behind the work. I feel that foundation modeling and federated learning seem at odds with each other, at least in this case. The authors argue that data owners may not be willing to share their data. But most time series foundation models, or even LLMs and LLVMs are trained on public data, and later adapted to private data, so I don't see the value proposition of such an approach.  Also I am not sure how this is a foundation model, as it doesn't solve multiple tasks, and is neither trained on multiple large-scale datasets. Secondly, why use large language models for time series forecasting, when studies have shown that there is sufficient data to pre-train foundation models from scratch [4], and that LLMs may not be better at forecasting time series data despite their significant computational cost [9]. Finally, I am not sure what the contributions of the work are. It seems like most of the work is built on architectural design choices from PatchTST, TimeLLM and Federated Averaging. Also, the personalization aspect sounds interesting, but can be addressed by fine-tuning foundation models, or custom built forecasting methods trained on specific datasets. \n3. **Baselines and comparisons:** The authors do not compare their methods with any recent time series foundation model, or recent time series forecasting models such as iTransformer, and old but performant methods such as N-BEATs / N-HITS. Recent studies such as [] have shown that older methods get the better of most recent methods in many settings.  \n\n### References\n1. Rasul, Kashif, et al. \"Lag-llama: Towards foundation models for time series forecasting.\" arXiv preprint arXiv:2310.08278 (2023).\n2. Woo, Gerald, et al. \"Unified training of universal time series forecasting transformers.\" arXiv preprint arXiv:2402.02592 (2024). \n3. Das, Abhimanyu, et al. \"A decoder-only foundation model for time-series forecasting.\" arXiv preprint arXiv:2310.10688 (2023). \n4. Goswami, Mononito, et al. \"Moment: A family of open time-series foundation models.\" arXiv preprint arXiv:2402.03885 (2024). \n5. Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024). \n6. Liu, Yong, et al. \"itransformer: Inverted transformers are effective for time series forecasting.\" arXiv preprint arXiv:2310.06625 (2023).\n7. Oreshkin, Boris N., et al. \"N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.\" arXiv preprint arXiv:1905.10437 (2019).\n8. Challu, Cristian, et al. \"Nhits: Neural hierarchical interpolation for time series forecasting.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 6. 2023.\n9. Tan, Mingtian, et al. \"Are Language Models Actually Useful for Time Series Forecasting?.\" arXiv preprint arXiv:2406.16964 (2024).\n\nQUESTIONS:\nI do not have any specific questions from the authors. But I would appreciate some clarity on the motivation and rationale behind the method. Please see my arguments above.\n\nLIMITATIONS:\nThe authors have discussed some limitations of their work. I believe that the authors should mention the computational cost of their method as a limitation. Each dataset / domain / client has a large language model backbone + other parameters, which makes their method computationally expensive.\n\n----\n\nAfter rebuttal: discussions on legality of using non-permissively licensed datasets to train FMs and notions of privacy need to be discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper introduces TIME-FFM, which is a Federated Foundation Model for Time Series Forecasting. TIME-FFM is comprised of (1) Modality Alignment which aligns time series patches with text tokens; (2) Prompt Adaption which learns the text prompts for an input time series; (3) LM backbone; (4) Prediction Head for domain specific output. Experiments are conducted on several benchmark datasets to demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. Federated learning for time series foundation model is an interesting and promising direction. This paper presents the first attempt in this direction.
2. Compared with SOTA federated methods (the TY1 group), the proposed method TIME-FFM could outperform the baselines. 
3. The writing is clear and easy to follow.

WEAKNESSES:
1. The overall novelty is limited. Modality Alignment, Prompt Adaption and different prediction head has been explored by previous methods. 
2. Compared with SOTA methods from TY2, the proposed TIME-FFM does not have a significant improvement.

QUESTIONS:
1. How do you obtain $E'$ from $E$?
2. Have you tried personalized Modality Alignment and Prompt Adaption for different domains?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TIME-FFM, a federated foundation model aimed at addressing the challenges of time series forecasting due to data scarcity and privacy concerns. The approach involves transforming time series data into text tokens, leveraging pretrained language models (LMs) for analysis, and using a personalized federated training strategy.\n\nSTRENGTHS:\n1.\tInnovative Approach: The idea of transforming time series data into text tokens and leveraging LMs is creative and could potentially address the data scarcity issue effectively.\n2.\tAddressing Privacy Concerns: The use of federated learning helps in alleviating privacy concerns and encourages data sharing without compromising sensitive information.\n3.\tPersonalized Training Strategy: The personalized federated training strategy, which combines global encoders with local prediction heads, is a thoughtful approach to handling data heterogeneity across domains.\n\nWEAKNESSES:\n1.\tAlgorithm Description: On page 6, the description of the algorithm seems to contain an error where Line 6 should actually be Line 5. Please carefully check the manuscript.\n2.\tExperimental Results Discrepancy: On page 6, the value of Baseline PatchTST on ETTm1 in Table 1 differs from the values reported in reference 25. You need to provide a detailed explanation of your simulation results and why these differences occur.\n3.\tDiscussion on Future Directions: The article could benefit from a more in-depth discussion of the future directions and challenges for the proposed method.\n\n[25] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. 369 Unitime: A language-empowered unified model for cross-domain time series forecasting. In 370 Proceedings of the ACM Web Conference 2024, 2024.\n\nQUESTIONS:\nInclude a more detailed discussion on the potential future developments and challenges for TIME-FFM, highlighting areas for further research and potential improvements.\nProvide a detailed explanation of the simulation results in Table 1, particularly regarding the discrepancies with reference 25. This will help in understanding the validity and reliability of your experimental results.\nCorrect the algorithm description to ensure that Line 6 reflects the appropriate content, as it currently appears to match what should be in Line 5.\n\nLIMITATIONS:\nPlease refer to the described weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}