Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting
Authors: Qingxiang Liu
Xu Liu
Chenghao Liu
Qingsong Wen
Yuxuan Liang
Venue: NeurIPS 2024 poster (2024)
Keywords: Foundation Model
Cross-Modality Adaption
Time Series Forecasting
Federated Learning
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We present the first attempt to build a federated foundation model for time series forecasting by exploiting the sequence reasoning potential of language models, avoiding the disclosure of local data.

Abstract: Unlike natural language processing and computer vision, the development of Foundation Models (FMs) for time series forecasting is blocked due to data scarcity. 
While recent efforts are focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains.
Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impossible to simply construct a FM on cross-domain training instances.
To address these issues, we propose Time-FFM, a Federated Foundation Model for Time series forecasting by leveraging pretrained LMs.
Specifically, we begin by transforming time series into the modality of text tokens.
To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially.
Given the data heterogeneity across domains, we design a personalized federated training strategy by learning global encoders and local prediction heads. 
Our comprehensive experiments indicate that Time-FFM outperforms state-of-the-arts and promises effective few-shot and zero-shot forecaster.
The code is available at https://github.com/CityMind-Lab/NeurIPS24-Time-FFM/tree/main.

TIME-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting
Qingxiang Liu 1.2
Xu Liu
Chenghao Liu*
Qingsong Wen
Yuxuan Liang"
The Hong Kong University Science and Technology (Guangzhou) Institute of Computing Technology Chinese Academy of Sciences
3 National University of Singapore 4 Salesforce AI Research 5 Squirrel AI
gingzlanglluu3770gmaall com, liuxu@comp. nus mus.edu..8. sg
chenghao www.oollllssssoro.o.oooooo com, gingsongedu@gmaill com, yuxliang@outlook.com
Abstract
Unlike natural language processing and computer vision, the development of Foun- dation Models (FMs) for time series forecasting is blocked due to data scarcity. While recent efforts focused on building such FMs by unlocking the potential of language models (LMs) for time series analysis, dedicated parameters for various downstream forecasting tasks need training, which hinders the common knowledge sharing across domains. Moreover, data owners may hesitate to share the access to local data due to privacy concerns and copyright protection, which makes it impos- sible to simply construct a FM on cross-domain training instances. To address these issues, propose TIME--FFM, Federated Foundation Model for TIME series forecasting by leveraging pretrained LMs. Specifically, we begin by transforming time series into the modality of text tokens. To bootstrap LMs for time series reasoning, we propose a prompt adaption module to determine domain-customized prompts dynamically instead of artificially. Given the data heterogeneity domains, we design personalized federated training strategy by learning global encoders and local prediction heads. Our comprehensive experiments indicate that TIME-FFM outperforms Nate-of-thhe-rte and promises effective few-shot and zero-shot forecaster. The code available lttpp:///githubbcom Mind Lal/VerrP9224Tiie--HMMMeeeeel
Introduction
Time series forecasting plays an important role in many real-world application domains M such as energy consumption prediction, weather forecasting, and disease transmission. Recently, multitude of deep learning models have been designed for time series forecasting based on Convolutional Neural
Networks 22 3 4. Recurrent Neural Networks [.] and Transformers
7 9 10
Inspired by
the prominent performance gained by Foundation Models (FMs) in the realms of Natural Language
Processing (NLP) [I
12
13 14 and Computer Vision (CV) 15 16 great research interests have
been triggered to build pretrained FMs for time series community 17
18
19
20 21 Nonetheless,
due significant time series data scarcity, these FMs are of poor capability to cultivate general representations, failing to promise remarkable fine-tuning
zero-shot performance for diverse downstream forecasting tasks [22] 23]. As a result, collection of methods have been proposed to borrow the pretrained language FMs to time series community by cross-modality adaption 24 22 thus unlocking the tapped potential of language models (LMs) for time series modeling.
Y. Liang is the corresponding author. This work was done when Q Liu was an intern at HKUST(GZ).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(o) Domain- Specific Forecasting
(b) Cross- Domain Forecasting
(c) FL In- Mo-Mddaltyy Adaption
(d) Gross Prods-MMdaalty Adop
A
Model
Model
Head
Head
Head
Pratroined
Protrained LM
FiA
Doesue
Ecoromics
Stowate
Seanonics
LM
Gress Madally Nappliin
Crocs-Moedily Hdsplion tobdo
Source Data Sample
Target Dato Sample
Source Task
Target Task
porameters
Figure 1:
(a) Specific prediction models are trained for diverse domains. (b) A unified model is
trained for cross-domain time series. (c) The current in in-modality adaption in FL setting fine-tunes LM for NLP tasks, with all the trained parameters are exchanged between clients and the server. (d) Our proposal investigates how to construct FM by unlocking the potential of LM for cross-domain time series forecasting in FL paradigm.
While these endeavors provide FMs for time series forecasting, the incorporated cross-modality adaption modules and unfrozen components of pretrained LMs need training from scratch for specific domains, thus restricting the mining of underlying temporal commonality in cross-domain time series data. As is shown in Figure a), disease and economics datasets are employed for training the FM respectively to obtain Nomainn-ptinne model parameters, hardly generalizing to other domains. (25) proposes to train unified model (named UniTime) on the mixture of cross-domain time series data (Figure [b), which ensures the cultivation of general-purposs representations, thus promising the zero-shot performance on unseen domains. Despite its effectiveness, they adopt the centralized training mode, where the historical records of time series across diverse domains are uploaded to central server for optimizing the unified model. Due to copyright protection and privacy concerns, data owners may hesitate to share the access these domain-specipp raw records.
provides the mainstream solution for the aforementioned problem, where data owners train prediction models locally and exchange the intermediate model parameters or gradients with the central server, without the disclosure of raw data records. Moreover, in UniTime, a retractable prediction head is introduced to accommodate the heterogeneous output needs whereas FL paradigm makes it possible to construct coutain-ustomizzed heads. However, current efforts are merely focused on how to fine-tune LMs in federated setting for NI.P tasks (i.e., in-modality adaption of LMS for target tasks in Figure [28] 29. 30 3T), rather than cross-modality adaption of LMs for time series forecasting. The realization of this federated FM is non-trivial technically, given the ubiquitous beterogeneity in cross-domant time series data. (1) Heterogeneous inputs: Cross-domain time series data input into the FM are heterogeneous in terms of dimensions and historical readings, posing evident difficulty to modality aligmment. (2) Rigid instructions as prompts: Prompts are adopted to bootstrap LMs for time series reasoning hinging rigid comain-specifii instructions rather than the understanding of LMs, exhibiting poor robustness for unseen domains. (3) Conflicts between generalization and personalization: The ideal FM needs to learn the common temporal representations across domains and simultaneously enable the personalized prediction for domain-specific inputs.
Federated Learning (FL) 26 27
25 22
To address the challenges, we propose TIME-FFM, a Federated Foundation Model for TIME series forecasting by repurposing LMs (Figure First, we perform modality alignment by transforming time series data into text tokens to empower the pretrained LM for time series reasoning. Second, we design a prompt adaption module to dynamically determine domain-specift. prompts, which can bootstrap the LM for cross- cross-domain time series analysis from the perspective of LM itself, rather than from human cognition by employing hand-crafted instructions prompts. To tackle the data heterogeneity across domains, we introduce a personalized federated training strategy by learning a global encoder and personalized prediction heads, given the shared representations across domains. Our main contributions are summarized as follows.
We present the first attempt to build federated FM for time series forecasting by exploiting the sequence reasoning potential of LMs, avoiding the disclosure of local data.
We propose TIME-FFM, which firstly aligns the modality from time series data natural language and adaptively determines prompts to guide the LM for time series reasoning. Moreover, we intro-

---

duce a personalized FL strategy to strike balance between sharing common temporal knowledge and ensuring customized prediction results.
The extensive evaluation results demonstrate that TIME-FFM leads state-of-the-art performance in mainstream forecasting tasks, especially in few-shot or zero-shot forecasting settings.
2 Related Work
FMs for Time Series Forecasting, Recent studies have demonstrated the effectiveness fine-tuning pretrained FMs for various downstream tasks, such BERT [1I]. GPT [12), GPT2 [13), and LLaMA 14 in NLP and DEiT [S] and BEiT [16] in CV. Inspired by the success, some efforts have been focused developing FMs for time series community, such 17 18 B2] However, due to data deficiencies, these pretrained models cannot guarantee the learning of ceneral-purpoon representations for time series analysis and hence fail to apply multitude of downstream tasks. Another line of researches attempt to leverage pretrained FMs in NLP or CV for time series analysis by cross- 23 (2), such fine-tuning and model reprogramming, modality adaption strategies [33 34 35 24
which hinges on the powerful generalization capability of Transformers for sequence tokens. [23] freezes the self-attention modules and feedforward layers of GPT2, and only fine-tunes the positional embedding and normalization layers. The proposed GPT4TS outperforms the relevant models in most time series tasks. On the contrary, [22] freezes the LM whole and transforms the modality of time series to natural language by patch reprogramming. These methods enable unified model structure rather than unified parameters for diverse downstream tasks, which makes the proposed FMs learn impaired temporal commonality. [25] proposes to train unified prediction model for cross-domain time series forecasting, which enables to learn the intrinsic temporal patterns. However, the centralized training mode brings privacy concerns for cross-domain data owners and FL paradigm may provide a promising solution.
Federated Fine-tuning of LMs. Given the exceptional performance of LMs and the emerging privacy preserving resolutions, incorporating LMs with FL is becoming a popular research trend. There have 39
40] to support fine-tuning LMs in FL setting. Moreover, considering the immense communication cost, some communication-effin federated
been some implementation frameworks [36] 29 37 38
28 A few researches aim to investigate the effects of data heterogeneity on fine-tuning performance, and then propose the personalized federated instruction tuning methods, e.g., [22 29 31]. Nonetheless, these methods concentrate on fine-tunin or fully-tuning pretrained LMs FL paradigm for NLP tasks, but fail to cover the cross- modality adaption of LMs for time series forecasting.
methods have been proposed, such as [8 41
30
3 Methodology
3.1 Problem Definition
Given N domains, let æ;,, (-),e- 2 E R denote the observation of domain i at the time step 4, where 6; represents the number of dimensions (channels). In the context of time series forecasting, denote X; {ar,,,.... Li.1, E RL.xx. the input of the prediction model fī(-), where 14 represents the domain-variant lookback window. The ground truths can be denoted as Y, {x,, L. F. Rexe,, where F, represents the future prediction window. Let D. = {X,;; Y.) denote the local data set of i and D, = [D.] the data size. Given the set of personalized model parameters }w;}. the objective of federated FM for cross-domain time series forecasting can be formulated as
1
min (wi---WN' WNN
==
Ex,
|| I f((w;; X.) 11..
(I)
N 1=1
D
3.2 Model Structure
The model architecture is elaborated in Figure 2 Our model encompasses three components: (1) modality alignment and prompt adaption, (2) LM backbone, and (3) prediction head. The modules of modality alignment and prompt adaption are designed for cross-modality alignment and adaptive
3

---

Modality Alignment Prompt Adaption
Prompt Tokens
Patch Tokens
Linear
Top Prototypes
Patching
Multi- Attention
Disease
Normalize
Linear
Weather
frainrd
Modality Alignment & Prompt Adaption
Embeddingg
Prompt Adaption
Modality Aligrment
LM Backbone
Prediction Head
Future Predictions
Flatten & Linear
Frozen
Training
Server
De-Normalize
Figure 2: Overall architecture of TIME-FFM. Each round begins with ! downloading global parameters modality alignment and prompt adaption modules. to generate patch tokens and @ adaptively determine prompt tokens. @ The two tokens are input into the LM backbone and optimization, @ the updated parameters of modality alignment and prompt adaption modules uploaded to the server for aggregation.
conduct modality alignment
outputs are projected to generate the prediction results. After local
prompt determination. We employ the backbone of GPT2 13 with freezing all parameters. The prediction head enables domain-specift prediction results.
Modality Alignment. Here we transform time series into the modality of text tokens. To accommodate domain-variant channels c. we adopt the channel-independent strategy 44 to split multivariate time series X; into 6 univariate series and individually process each. Let X (-),)- ... J.,,, E 11XLi denote the j-th univariate series from X,, Then we normalize each series X mitigate the effect of distribution diversity 44]. Since each data point of X does not have explicit semantic knowledge like words in sentences, we adopt the patching technique [43) to segment X into subseries (termed patches), each of which can aggregate the local information and better retain the temporal knowledge. Specifically, let denote the patch length and 5; denote the stride length of domain i. Hence, the number of patches can be defined as B, L,-
1. We 1,5 E B. as the generated patches from X We subsequently employ a linear layer to project the patches into tokens X IR B. where D is the input dimension size the LM backbone. X together with prompt tokens (in the next part) will be input into the LM backbone.
S;
denote X
Prompt Adaption. the time series forecasting FMs based on LMs, domain instructions are designed prompts to complement the patch tokens and inform the LM backbone of domain-specific knowledge [22 225. These maanullly-desgnne prompts depend completely on experts' knowledge and may vary from each other due to different understandings. Furthermore, according to the results, more detailed instructions can always yield better prediction performance [22), which may make us naturally draw conclusion that the ultimate performance hinges on the length prompts. However, longer prompt tokens will present substantial challenge on the computation burden. Different from images 45) or acoustic data [33], which can be "translated" into natural language seamlessly, the manually-crafted prompts are error-prone to describe the characteristics of the raw time series. To this end, a better way to design prompts from LM's understandings the patch tokens rather than human cognition of raw time series data. Here, propose adaptively determine prompts

---

based on patch tokens from the source corpus of pretrained LM (which includes V pretrained word embeddings, denoted E V)D)) Similar to [22] we project E a smaller collection of text protorypes, denoted as E' E VD by linear layer, with V < V. to avoid the potential large parameter space. We adopt modified multi-head attention layer to obtain the correlation between E' and X²,s and subsequently select M mostly related text prototypes as prompts. Concretely, for each head h E {1.-.., .H,, we have the query matrix Q = E'W; and the key matrix K²,, = S where : w E D and d = D Since we do not aim to return a weighted value matrix according to the given query but merely evaluate the correlation of text prototypes and patch tokens, omit the value matrix here. The attention score matrix is denoted as O RV'B, B, and can be calculated
S KºT 1,
O₂,, =SOFTMAX
(2
vd
We Oh E RR x' by calculating the summation of O i. per row. Each value in Ó i.h represents
the correlation degree of the corresponding text prototype in E' to all patch tokens X/.s We select M prototypes from Qi with top attention scores to form the potential prompts Z E R Mxd i.e.,
Z 2 [TPM(()),)) ).) We can obtain Z E R Rx by aggregating Z2,,, from all H heads. Finally, we employ a linear layer to project Z to the prompt tokens Z/ E Mx
Prediction Head. We input the concat of 2 and X into the LM backbone and obtain the representations R. (MM+B,) which will be flattened and projected to the final results Y, RIxP, by linear layer.
Time series across different domains could be substantially heterogeneous. Consequently, generalized global model in FL may fail to capture the disparate temporal patterns and ultimately compromises the prediction performance. Inspired by [46), which indicates that diverse data may share common feature representations, we propose to learn global encoder (i.e., modality alignment, prompt adaption and LM backbone) and coomain-custoiize prediction heads. The underlying motivation is to strike balance between generalization and personalization: (I) increasing the generalization
Personalized Strategy.
modality alignment and prompt adaption by access to cross-domain temporal patterns; (2) ensuring prediction results specific for certain domains by personalized heads. Since we keep the LM backbone intact, in each federating round, only the parameters of modality alignment and prompt adaption communicated. The server performs aggregation by averaging strategy. The training strategy differs from Federated Averaging framework, where the parameters of encoder and decoder are both aggregated at the central server after local optimization.
3.3 Training Process
We denote w/ as the global parameters of modality alignment and prompt adaption at the I-th federated round and w., prediction head parameters of at the I-th round. We clarify that (X,, Y)
here is reused to represent training batch. Xi.s, 2,, R. and Ý, denote the patch tokens, prompt tokens, representations and prediction results of such batch respectively. The training procedure of TIME-FFM is elaborated in Algorithm I (1) In the t-th federated round, the server distributes the global parameters w: (Line and 9). (2) Each domain loads the global parameters and local head parameters to perform prediction following modality alignment, prompt adaption as well as representation obtaining from LM backbone (Line 12-15) and uploads wh,, to the server after optimization. (3) Finally, the server aggregates local updated parameters by averaging mechanism to obtain the fresh global parameters w for the (L + 1)-th round (Line 6).
4 Experiments
We comprehensively compare the proposed TIME-FFM with state-of-the-art models in FL or cen- tralized settings, especially those by fine-tuning LM for time series forecasting. The numerical results demonstrate the effectiveness of TIME-FFM in time series forecasting, We employ GPT2
5

---

Algorithm 1:
Training process of TIME-FFM
Input:
Global round number T. local epoch number E, initial global encoder parameters wg. initial personalized head parameters
Function Local Execute wf): wi, f- wi
8
for 1,2, .... F. do
{woo). J. local batch number bi-
for (X;, , Y.) in b; batches do
Output: Optimized global encoder parameters w4- optimized parameters of personalized heads {w
Xis, 2, - g(w X;, E)
II
R: LM(coneat( (oircatt(xxss Xi,s 2.))
Y; + p(u : R.)
SERVEREXECUTE
13
for 1 0.,. do for i 1,2, ... NN in parallel do w1,, "LocalExecute (i, w
loss < <-| Y:Il
Update wf, and wº,,, via gradient descent.
we+1 E wí,
W W1,+1 f wf,
return wi,
// for local training
backbone of the first layers as the default LM backbone and freeze all parameters. To guarantee a fair comparison, we adhere the experimental configurations in [25].
Baselines.
Our baselines cover board collection relevant methods, which can be categorised into
3 types:
TY1 (federated fine-tuning methods): FedIT [31], FedAdapter" 47 41, and FedAdapter
48
41:
TY2 (across-datase centralized methods): UniTime [25]. GPT4TS 233, and PatchTST
43
TY3 (dataset-specif) centralized methods): TimesNet [4, DLinear [4]], FEDformer [50),
Autoformer [10], and Informer [9]
We directly cite the results from 25 if applicable.
Setups. We evaluate on 8 benchmark datasets from various domains: ETTh1, ETTh2, ETTml, ETTm2, Electricity, Weather, Exchange, and IIII, which have been widely adopted for evaluating time series forecasting performance. Each dataset corresponds to FL participant. Detailed introduction of implementation and datasets can be found Appendix A We use Mean Square Error (MSE) and Mean Absolute Error (MAE) as the evaluation metrics.
4.1 Main Results
Main forecasting results are presented in Table TIME-FFM consistently outperforms the other FL methods (in TY1) on all datasets, except ETTh2. Specifically, TIME-FFM can improve the performance gains over all datasets by 39.01% % in terms of MSE, compared with the second best- performed FL method. Furthermore, the averaged prediction results of TIME-FFi FFM are even superior to those of the centralized models. When compared with UniTime, the recently-proposed centralized unified model for cross-domain time series forecasting, TIME-FFM can provide more performance gains, which underscores the effectiveness of the proposed cross- vooss-modality adaption modules and personalized approach.
4.2 Few-Shot Forecasting
Given the remarkable few-shot learning performance of LMs, we evaluate whether TIME-FFE FFM can retain such capability for time series forecasting. In this section, we compare the prediction performance across TY1 and TY2 in few-shot settings with 10% and 5% time steps adopted as training samples, which is in line with the setups in (23 22
Main results of 10% and 5% few-shot forecasting presented in Table 2 and respectively. TIME- FFM outperforms the other FL methods and even achieves comparable performance in contrast to the centralized methods, which further underscores that TIME-FFM inherits the few-shot capability of LMs and promises proficient FM for time series forecasting. Specifically, TIME-FFM outperforms the centralized methods in the realm of 5% few-shot learning. with 20% reduction in averaged MSE w.r. UniTime. Interestingly, for all methods except UniTime, results in 10% few-shot learning worse than those 5% few-shot learning. We deduce that the pretrained LM fully-tuned in
Here modify the original GPT4TS and PatchTST per 25

---

Table 1: Forecasting performance comparisons. All results are averaged over four prediction windows, i.e., F; E {24, 36, 48, 60) for ILI and 96, 192, 336, 720} for others. Yellow the best in TY1;
Blue the second best in TY1. Underline: the best over all types; Bold: the second best over all types. Full results are presented in Table 13
Type
TY1
TY2
TY3
Method
TIME-FFM
FollT
[FedAAlspeer #f FeatAdapter"
UniTime
GPT-TS
PatchTST
TimesNet
DLimerr
FFDDoomer Autoformer]
Informer
Metric
MSE
MSE
MAE|
MSE
MAE
[MSE
MAE
MMSE
MAE|
(MSE:
MAE|MSE MSE
MAR]
MSE
MAE|
MSE
MAR]
MSE
MAE
MSE
MAE] MSE
MAE
ETThi
0.442
0.434
00:881
0,461
0.488
0:967
[0.503
0,479
J0.442
0.448
(0.502 0.463]
(o-72 0.451
[a.4ss
0.450
0.156
0,452
0.452/0.440
0..40 0.160
0.496
0.487/ [.040 0.795
E1112
|0,382
0.406
01374
0.396
10.373
0.398
0,,300
0,400
J0.378
0.408
(0.366
0.40ojo.39s
0.436
(0.44
0.422
(0.59
0.515
(0.457
0.449
0.400
0.459|4.431 1.729
ETTml
0.399
0.402
0.614
0.517
0..443
0.511
[0.640
0.516
|0.385
0.399
(0:551
0:983/0.971
0.629
0.629|0.383
(0.383
0.106
(0:103
0.407
0.,44
0.452
0.5888888888
0,517|0.961 0.734
FTTm2
0.286
0.332
0.297
0,341
0.295
0,340
0.298
0,542
[0.203
0.334
|0:321
0.356/0.440
0.373
[0.291
0.322
|0.350
0.401
[0,305
0.349
0.39
0.371 0..3111111100 [.410
Eleetiiitty
0.216
0.390
0.478
10:408
0.489
0.334
0.420
1.2216
0.305
(0.251
0.338/0.221
0.311
00.19
0.205
(0.212
0.300
0.327
0.227
0.335/0.311 0.397
Wenther
6.220
0.288
00:281
0.310
00282
0.308
0.287
0.500
J0.253
0.376
(0:203 o.
0.309/0.304
0.323
(0.299
0.287
|0.665
0.317
(0.309
0.360 333
0. 0.383/0.634 0.548
Exchange
0,338
0.391
0.389
0.423
10,322
0,419
10,300
0.437
10,364
0.404
(0.421 0.440/o.411
0.444
(0.466
0.443
(0.354
(4.11
0.300/0.13
0.539
1.300
0.998
11.1
2.107
0.924
44423
1,448
5,247
1.621
|5.251
1,600
[2.137
0.929
[3,678
1.372| (1211
0.931
22616
1.090|2.847
1.144)
3,006
1.16|
1.54
Average
0.355
0.434
0.910
0,547
1.015
0.569
|.009
0.561
|0.559
0.437
|0.300
0.521
1.996
0.553
[0.509 0.445
[0.622
0.457
1.,900 0.505
0.756
0.532]
19334 0.944
1st Count
UniTime and fewer training samples fail to support optimizing masses of parameters. While in the other methods, the pretrained LMs are frozen or fine-tuned, which can retain the original reasoning capability of LMs even with fewer training instances.
Table 2: 10% few-shot forecasting results. All results are averaged across four prediction windows, i.c., F. E {96, 192, 336, 720}. Yellow: the best in TY1; Blue : the second best in TY1. Underline: the best over both types: Bold: the second best over both types. Full results are presented in Table 14
Type
TYI
TY2
Methud
TIME-FFM
FedLoRA
FedAdaperr Ht
FedAdaater
UniTime
GPT4TS
PRIGHTST
Metrie
MSE
MAE
MSE
MAE
MSE
MAF
MSE
MAF
MSE
MAE
MSE
MAE
MSE
MAE
ETiml
0.593
0.500
0.637
0.506
0.672
0.539
0.697
0.543
0.589
0.494
0.638
0
1,071
0.662
ETTm2
0.294
0.335
0.297
0,340
0.298
0.341
0.298
0.343
0.299
0.338
0.296
0.336
0.348
0.378
Electricity
D.266
0.344
0.275
0.363
0.431
0.480
0.408
0.4%6
0.254
0.342
0.251
0.334
0.362
0.429
Weather
0.288
0.314
0.296
0.320
0.284
0.287
0.315
0.272
0.209
0.300
0.322
Exchange
0.230
0,336
0.238
0.339
0.227
0.334
0.250
0.335
0.220
0.331
0.242
0.344
0.220
0.330
Average
0.366
0.349
0,374
0.380
0.403
0.384
0.327
0.361
0.345
0.267
0.459
1 Count
2
0
7
2
2
Table 3: 5% few-shot forecasting results. All results are averaged across four prediction windows, i.e., F. E {96, 192, 336, 720}. Yellow: the best in TY1; Blue the second best in TY1. Underline: the best over both types; Bold: the second best over both types. Full results are presented in Table T6
Type
TYI
TV2
Method
TIME-FFM
FodLoRA
Ft H
FadAAdapte: P
UniTime
GPT4TS
PatchTSTT
Metric
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
ETTml
0.507
0.491
0.606
0.494
0.630
0.526
0.636
0.519
0.715
0.558
0.631
0.522
0.591
0.497
ETTm2
0.293
0.333
0.298
0.339
0.298
0.339
0.296
0L338
0.315
0.350
0.298
0.339
0.299
0.339
Electiciity
0.924
0,003
0.339
0.420
0.333
0.411
0.333
0.409
0.298
0.387
0.273
0.365
0.309
0.391
Weather
0.292
0,317
0.303
0.325
0.292
0.117
0.300
0.322
0,288
0.313
0.288
0
0.324
Exchange
0.167
0.289
0.171
0.291
0.166
0.288
0.166
1.287
0.442
0.495
0.168
0.290
0.171
0.293
Average
0.329
0.344
0.374
0.348
0.376
0.346
0.375
0.411
0.420
0.332
0.334
0.369
1s Count
2
2
7

---

4.3 Zero-Shot Forecasting
Given that language FMs are effective zero-shot forecasters, we evaluate the zero-shot learning capability of TIME-FFM, which is essential for a FM. We adhere to the pero-shot learning settings in [25], where we first train TIME-FFM on ETTh1, ETTml, and ETTm2, and then evaluate the zero-shot testing performance on ETTh2, Ellectrictyy, and Weather.
Since ETTh2 hails from the same domain of ETTh1, we directly reuse the local parameters (including both encoder and head) of ETThl for inferring ETTh2. For the other two target datasets from different domains of the source datasets, we successively reuse local parameters of the three source datasets to perform zero-shot testing. The results presented in Table 15] show that local parameters of ETThl excel on both target datasets. Hence, we adopt the model parameters of ETThl for zero-shot testing on Electricity and Weather. For other methods in TY1, we train an optimized global model on ETTh1, ETTml, and ETTm2, and then adopt the obtained global model to conduct zero-shot testing on ETTh2, Electricity, and Weather. The comparison in zero-shot forecasting is presented in Table TIME-FFM consistently ensures significant performance gains on all three datasets, with prediction MSE decreasing by 13.9% w.r.t the second best. It is remarkable that the centralized unified model UniTime exhibits inferior zero-shot testing performance compared to -FM. We attribute the performance gains of TIME-FFR to the valid knowledge transferability across domains.
Table 4: Zero-shot forecasting results, All results are averaged across four prediction windows, i.c., F3 E (96, 192, 336, 720). Yellow the best in TY1; Blue the second best in TY1. Underline: the best over both types: Bold: the second best over both types. Full results are presented in Table 17
Type
TYI
TY2
Methud
TIME-FFM
FeJIT
FedAAaapeer" !!
FedAdhpperr P
UniTime
GPT-TS
PaicTST
Metric
MSR
MAR
MSF
MAR
MSF
MAF
MSF
MAF
MSE
MAR
MSE
MAF
MSF
MAF
ETT12
0.373
0.399
0,387
0.407
0,388
0,408
0.387
0.407
0.358
0.409
0.397
0,418
0.421
Elecnicity
0.265
0.343
0.388
0.470
0.401
0.474
0.409
0,482
0.436
0.500
0.462
0.526
0.534
0.565
Weather
0.291
0.318
0.295
0.319
0.302
0.324
0,902
0.324
0,301
0.320
0.322
0.339
0.327
0,339
Average
0.310
0.353
0.369
0.399
0.364
0,402
0.366
0.404
0.375
0.410
0.394
0.428
0.427
0.444
Table 5: Ablation studies of TIME-FFM on ETTh! and ILI datasets with F3 E {336, 720} and F; E {48, 60) respectively. Bold: the best.
Forcccasting Task
ETTh1-336
ETTh1-720
ILI-48
ILI-60
Motric
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
A.I TIME-FFM
0,480
0.449
0.462
0,456
1.953
0.894
1.976
0.916
A.2 w/o Prompt Adaption
0.495
0.450
0.496
0.471
2.222
0.947
2.118
0.952
A.3 w/ Instructions
0.487
0.457
0.465
0.465
2.109
0.953
2.170
0.977
A.4 w/o Personalized Head
0.537
0.471
0.526
0.480
4.953
1.591
4.068
1.450
All
0.495
2.037
A.6 TIME FFM.D
0.499
0.450
0.503
0.472
2.453
1.022
2.427
1.026
4.4 Model Analysis
Model Ablation. We conduct ablation studies on five variants of TIME-FFM and the corresponding results are presented in Table Thereinto, TIME-FFM-D represents the distirbuted version of TIME-FFM, which ablates the aggregation process. The results demonstrate that ablating either components will compromise the forecasting performance. We have the following key observations:
(1) The prompt tokens can bootstrap the LM for target domains. The absence of prompt adaption
(A.2) will affect the forecasting performance. When employing instructions in (25) as prompts, A.3 is inferior to TIME-FFM, which underscores the efficacy of prompt adaption. (2) The ablation of personalized heads (A.4) will hurt the performance most. In A.4, global prediction head is learned for all domains, hardly ensuring the personalization for cross-domain heterogeneous data. (3) In A.6, the common temporal knowledge fails to be shared among domains, which makes poorer generalization of mrods-todality adaption modules, thus yielding inferior performance. This underscores the significance of building unified model for cross domainn traffic series forecasting.
8

---

Table 6: Ablation studies of LM on ETThl and Weather datasets with F; E {96, 192] and Pt
{336, 720) respectively. Bold: the best.
Forecasting Task
ETThl
ETTh1-192
Weather 336
Weather 720
Metric
MSE
MAE
MSB
MSE
MAE
MSE
B.1 Froeze (Default)
0.422
0.412
0.473
0.439
0.295
0.308
0.367
0.354
B.2
0,450
0.290
0.363
B.3 Full
0.394
0.403
0.448
0.431
0.287
0.305
0.360
0.351
C.1 GPT2 (6) (Default)
0,422
0.412
0,473
0.439
0.295
0.308
0.367
0.354
C.2 GPT2 (12)
0.406
0.409
0.456
0,436
0.294
0.307
0.367
0,353
Table 7: Efficiency analysis of TIME-FFM on ETTh1 dataset.
Method FedLoRa
Training Param. (M) Total Param. (M) Training Param, PCT. (%) Training Time (s/iter) Comm. Param. (M)
8,543
90-456
9:445
0.048
8,543
FedAdapter H
47.998
90.945
52.777
0.062
47.998
FeddAdapter"
47.550
90.498
52,543
0.046
47.550
TIME- FFM
90.050
0.088
GPT (12)
8.138
132.578
6.138
156
6.81
Language Model Variants. We investigate the variants of LM, in terms of optimization modes (B.1-B.3) and backbone layers (C.1 and C.2). Here we train all variants on seven datasets except Electricity. due to GPU memory limitation. In B.3, the backbones of LM are full-tuned. While in B.2, we only tune the positional embeddings and layer normalization components of the backbone 23 T shows that B.3 performs best, followed by B.2 and B.1. We argue that the performance remains comparable when we freeze all backbone parameters. This demonstrates that LMS are capable in processing time series tokens by effectively modality alignment. In C.1 and C.2, 6 and 12 backbone layers are adopted. The results shows that more backbone layers ensure better performance, which indicates the scaling laws of LMs retain in TIME-FFM for time series forecasting [5] 22
Model Efficiency. Table 7 demonstrates that TIME-FFM can reduce the training pa-
teed 0
rameter quantity and communication over- head with insignificant increase in training time. Moreover, the number of training pa- rameters and communication paramelers keeps intact, regardless of backbone layers.
tea_77
tea1.5
heal 4
teed
luad 2
Case Study. We provide case study on prompt adaption in Figure 3 (a) shows the attention scores between patch tokens and 100 text prototypes for 8 heads on ETTh1 dataset. For each head, only a small set of text prototypes (columns) have remark- able scores, which indicates that each patch token is only related to limited pretrained word embeddings and dynamically prompt adaption is promising. (b)-(d) show top M prototypes of 8 heads on ETThI, Elec- tricity, and ILI respectively. Darker colors correspond to text prototypes with higher attention scores. From these three subplots, we have the following key observations: (1) different datasets correspond to variant text prototypes;
tox__1
(a) Milti head atbartion scores on ETTal
lasd
300
(b) Top M proto
(c) Top
Moptoootyyrss prototypes Elactricity
(c) Top Mantoppesssooommmmmmmmmmmmmm on Illness
Figure 3: A showcase of prompt adaption.
(2) the distribution of text prototypes on different datasets has commonality, i.e., gathering in shadow areas. These observations indicate the global prompt adaption module has great generalization for diverse datasets and simultaneously ensures personalization across various domains.
5 Conclusion and Discussion
In this paper, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specift prompts and tackling time series heterogeneity for general-

---

purpose learning and personalized prediction. Specifically, given the differentiation of dimensionality and horizon, we introduce the modality alignment module encompassing the haannel-indeenneen and patching techniques, which may follow the track of GPT4TS and Time-ILMM. For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions. Due to cross-domain time series heterogeneity, we devise a personalized federated strategy, with global encoder and personalized prediction heads.
Rationale of TIME-FFM. Compared with the modality of text, time series is more domain-spectie and copyright-sensitive, i.e., private knowledge may be inferred from historical time series readings, especially finance and healthcare domain. Hence, it of great significance to take data privacy into account when constructing time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due data license restriction, such as Kaggle public datasets. Hence, our work uniquely bridges the gap between foundation models and federated learning, which not only enhances the privacy and applicability foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in this field.
Limitations and Future Works. We recognize some limitations of our work: the training time is increased compared with the TY1 and the performance in some case is suboptimal. In the future work, we will explore more effective and efficient modality alignment strategies. Moreover, further researches will investigate the correspondence between patch embeddings and word embeddings to explore whether time series data can be seamlessly "translated" into natural language.
Acknowledgments and Disclosure of Funding
This work is mainly supported by the National Natural Science Foundation of China (No. 62402414). This work is also supported by the Guangzhou-HKUSTCCC Joint Funding Program (No. 2024A03J0620), Guangzhou Municipal Science and Technology Project (No. 2023A03J0011 1), the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), and a grant from State Key Laboratory of Resources and Environmental Information System, and Guang- dong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B 221 1AIBB122200001 2010000.
References
[1] Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4836-4837, 2022.
[2] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.0127/11 2018
[3] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
[4] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. The eleventh international conference on learning representations, 2022.
[5] Guokun Lai, Wei-Cheng Chang. Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[6] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilis- tic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181-1199) 2020.
[7] Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting.
10

---

In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. pages 146-156, 2022.
[8] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
[9] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang. Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, pages 11106-11115, 2021.
[10] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419-22453 2021.
[11] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, 2019.
[12] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[13] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))99 2019.
[14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.139771 2023.
[15] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- cata-efficieent image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[16] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. In International Conference on Learning Representations, 2021.
[17] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via fime-frequency consistency. Advances in Neural Information Processing Systems, 35:3988-4008 2022.
[18] Shohreh Deldari, Hao Xue, Aaqib Saeed, Jiayuan He, Daniel V Smith, and Flora D Salim. Beyond just vision: review on self-supervised representation learning on multimodal and temporal data. arXiv preprint arXiv:2206.02353 2022.
[19] Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10199 2023.
[20] Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey, arXiv preprint 2rXiv:2403.14733 2024.
[21] Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Y Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, et al. Self-supervised learning for time series analysis:
Taxonomy, progress, and prospects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[22] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLMM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024.
[23] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024.
11

---

[24] Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained lims. arXiv preprint arXiv:2308. 08466, 2023.
[25] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A monguage-empowerr
unified model for cross cross- domain time series forecasting, In Proceedings of the ACM Web Conference 2024, 2024.
[26] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. caamuumiccatio-fffccieeee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[27] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[28] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre- trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9963-9977, 2023.
[29] Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. Federated learning of large language models with parameter-effiiien prompt tuning and adaptive optimization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7871 7871-7888, 2023.
[30] Ningxin Su, Chenghao Hu, Baochun Li, and Bo Li. Titanic: Towards production federated learning with large language models. In IEEE INFOCOM 2024 IEEE Conference Computer Communications, 2024.
[31] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915-6919. IEBE, 2024.
[32] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:240..02592, 2024.
[33] Chao-Han Huck Yang, Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning, pages 11808-11819. PMLR, 2021.
[34] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2336. 2023.
[35] Pin- Yu Chen. Model reprogramming: Resource-efficien cross-domain machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22584-22591 2024.
[36] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope- -Imm: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309.00366; 2023.
[37] Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du Yanfeng Wang, and Siheng Chen. Openfedlim: Training large language models on decentralized private data via federated learning. arXiv preprint 2xXiv:2402.0695.. 2024.
[38] Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:23 10049, 2023.
[39] Ronghui Xu, Hao Miao, Senzhang Wang, Philip S Yu, and Jianxin Wang. Pefad: A parameter- efficient federated framework for time series anomaly detection. In SIGKDD, pages 3621 13322,,,, 2024.
12

---

[40] Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang. Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards 11m in-emmowered time series forecasting via cross-modality alignment. arXiv preprint arXiv:2406.01638. 2024.
[41] Guangyu Sun, Matias Mendieta, Taojiannan Yang, and Chen Chen. Conquering the commu- nication constraints to enable large pre-trained models in federated learning. arXiv preprint arXiv:22100017000 2022.
[42] Pengyu Zhang, Yingbo Zhou, Ming Hu, Junxian Feng, Jiawen Weng, and Mingsong Chen. Personalized federated instruction tuning via neural architecture search. arXiv preprint arXiv:2402.1699 16919, 2024.
[43] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. The Eleventh International Conference on Learning Representations, 2022.
[44] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jacgul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[45] Diganta Misra, Agam Goyal, Bharat Runwal, and Pin Yu Chen. Reprogramming under con- straints: Revisiting efficient and reliable transferability of lottery tickets. arXiv preprint arXiv:2308. 14969 2023.
[46] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089-2099 PMLR, 2021.
[47] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pages 2790-2799 PMLR, 2019.
[48] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, 2021.
[49] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, pages 11121-11128, 2023.
[50] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International conference on machine learning, pages 27268-27677 27286. PMLR, 2022.
[51] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[52] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[53] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
[54] Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine Scaling cross- embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408. /8/2, 2024.
[55] Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. arXiv preprint arXiv:2410.1044 10469, 2024.
13

---

[56] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning, 2024.
14

---

A Experimental Details
Implementation. The Adam optimizer with the initial learning rate of 10-4 4 is adopted in the training process. The lookback window L; is set to 36 for the ILI dataset, and 96 for the others. The future prediction window F, is set to {24, 36, 48, 60) for the ILI dataset, and {96, 192, 336, 720} for other ones. We adopt the pretrained GPT2-backbone of the first 6 layers as the LM encoder. The local epoch set to I for all domains. The global round number T is set to 100. V', M, P and H are set to 100, 12, 16, and 8 respectively for all domains. S; is set to 4 for the ILI dataset, and 16 for other ones. In each round, we calculate the averaged values of validation loss. The round with lowest validation value serves as the optimal round, and then the corresponding model is used for test. All models are implemented on PyTorch with all experiments conducted on NVIDIA A100-80G GPUs.
Table 8: Detailed descriptions of datasets. The dataset size is organized in (training, validation, test).
Dataset
Dataset Size
Batch Size
OverSampling Times
Prequency
Application Domain
ETTh1
(8545, 2881, 2881)
32
hour
Electrical Asset Monitoring
ETTh2
(8545 2881)
32
hour
Electrical Asset Monitoring
ETTml
(34465, 11521, 11521)
64
15 minutes
Electrical Asset Monitoring
ETTm2
(34465, 11521, 11521)
64
15 minutes
Electrical Asset Monitoring
Electricity 321
(18317, 2633, 5261)
24
Energy Consumpttion
Weather
21
(36792, 5271, 10540)
64
10 minutes
Weather Forecasting
Exchange
(5120, 665, 1422)
24
day
International Trade
IIII
(617. 74, 170)
16
12 I week
Ilnnes Monitoring
Training Configurations. The experimental evaluations are conducted on 8 real-world benchmark datasets which include domains. We present the detailed description of these datasets in Table For fair comparison, we perform batch division and oversampling as per [25]. In each federated round, we do not train local models with all training samples, considering large quantity of training samples. Instead, we proportionately calculate the number of batches for each domain in the following steps. (I) We calculate the summation of training batches over all datsets before oversampling. (2) We count training times of each domain after oversampling, i.e., 13 for ILI and for the others, and then we perform normalization to obtain a batch ratio for each domain, i.e., 0.65 for ILI and 0.05 for the others. (3) can obtain the number of training batches for each domain (denoted as b,) by multiply the summation (in (1)) and ratios (in (2)) respectively. Actually for IIII the value is higher than the number of training batches, while the opposite is true for the others. In each round, each local model is trained with training batches sequentially until b, is reached.
We evaluate the effectiveness of oversampling strategy in TIME-FFM and present the results in Table 9 "w/o OverSampling" represents each local model is trained with all local batches in each FL round. We attribute the performance gains in TIME-FFM to it that the introduction of oversampling strategy can balance the contribution to the global knowledge. For ILI, despite data sparsity, its local knowledge can be augmented in the global encoder. We observe that such local knowledge can enhance forecasting for not only ILI itself but also the other domains.
Table 9: Effectiveness evaluation of oversampling. All results are averaged over four prediction windows, i.e., F3 E (24, 36,46 48, 60} for II.I and ( 96, 192, 336, 720) for others. Bold: Better.
Damtsets
ETTH
ETTh2
FTTml
FTTm2
Fleetricity
Weather
Exchange
II.I
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAB
MSE
MAE
MSE
MAE
TIMI-FFM
0.442
0.434
0.406
0.399
0.402
0.332
0:216
0.299
0.270
0.288
[L,388
0.391
2,107
0.924
wio OverSampling
0.456
0.445
0.296
0.415
0.405
0,410
0,300
0.341
0,212
0.295
0.272
0.289
0.345
0.393
2:364
0.989
B Hyperparameter Sensitivity
In this section, we conduct hyperparameter investigation of 3 important hyperparameters, i.e., the number of text prototypes V', the number of prompt tokens M, and the number of self-attention heads H. Figure4sshow shows prediction performance on IIII dataset with the variation of the hyperparameters respectively. We have the key observations as follows: (I) When the value of V is lower, word
15

---

20
pmdicc224
2.8
28-
pasdict366
pretict-35
proditt
26-
predics-88
prrdiit 60
poddics-6
MSE
MSE
MSE
24
2.2
2,0
20 50
100
200
12
16
32
V.
H
(a)
(b)
(c)
Figure Hyperparameter sensitivity studies on ILI dataset.
embeddings are projected into less text prototypes. Each prototype will contain both relevant and irrelevant knowledge, which will affect the accuracy prompt adaption. When text prototypes are more, stable number of prompt tokens will not cover all relevant knowledge. Hence lower higher values V will yield subpar performance. (2) Fewer prompt tokens may not fully cover the useful knowledge. Hence, the best performance is achieved when M is equal to 12. (3) Increasing the number of attention heads cannot always promise better performance because more heads may break the semantic integrity of text prototypes and patch embeddings.
C Additional Results
We compare forecasting performance with PatchTST-FL and DLinear-FL, the federated version of PatchTST and DLinear. As is presented in Table TIME- consistently outperforms the two novel federated methods on all datasets.
Table 10: Performance comparison with PatchTST-FLL and DLinear-FL. Bold: the best.
Method
TIME-FFM
PatchTST-FL.
DLinear-FL.
Metric
MSE
MAE
MSE
MAE
MSE
MAE
ETThl
0.442
0.434
0,534
0.496
0.565
0.545
ETTh2
0.382
0.406
0.399
0.415
1.040
0.738
ETTm1
0.399
0.402
0.752
0.573
0.783
0.627
ETTm2
0.286
0.332
0.318
0.357
0.987
0.730
Electricity
0.216
0.299
0.457
0.523
0.363
0.452
Weather
0.270
0.288
0.288
0.317
0.339
0.402
Exchange
0.338
0.391
0.404
0.440
0.830
0.723
Average
0.333
0.364
0.450
0.446
0.701
0.602
We further compare the forecasting performance with three baselines, i.e., iTransformer [52], N-
BEATS (53), and Crossformer (54). These three baselines be categorized into TY3. The numerical results are presented in Table We have the key observation that TIME-FFM, though trained in federated paradigm, outperform these three centralized methods.
Table 11: Performance comparison with Traasffrmmr, N-BEATS, and Crossformer. Bold: the best.
Method
TIME-FFM
iTransformer
N-BEATS
Crossformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTm2
0.286
0.332
0.288
0.332
0.294
0.345
0.757
0,610
Weather
0,270
0.288
0.258
0.278
0.263
0.282
0.259
0.315
Exchange
0.338
0.391
0.360
0.403
0.481
0.455
0.940
0.707
Average
0.298
0.337
0.302
0.338
0.346
0.361
0.652
0.544
16

---

Some researches delve into training a foundation model from scratch based on the collected time series datasets [55] 32 56] We compare our proposed federated foundation model with Moirai [2]
and MOMENT [56 in T2 Notably, TTMF-FFM achieves comparable performance with the two foundation models which are pre-trained firstly on large- time series archive.
Table 12: Performance comparison with MOIRAl and Moment. Bold: the best.
Method
TIME-FFM
Moirai
MOMENT
Metric
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
0.442
0.434
0.400
0.424
0.418
0.436
ETTh2
0.382
0,406
0.341
0.379
0.352
0.395
ETTml
0.399
0.402
0.448
0.409
0.344
0.379
ETTm2
0.286
0.332
0.300
0.341
0.259
0.318
Electricity
0.216
0.299
0.233
0.320
0.165
0.260
Weather
0.270
0.288
0.242
0.267
0.228
0.270
D Full Results
Full results of forecasting performance comparison 8 time series benchmarks presented in Table 13 TIME-FFM exhibits SOTA performance in 32 out of 42 instances, which demonstrates the effectiveness of the cross- modsl-tddlltyy adaption module, i.e., modality alignment and prompt adaption, as well as the personalized prediction heads.
Our complete results of performance comparison in 10% and 5% few-shot settings are presented in Table and 16 respectively. In both settings, TIME-FFM outperforms the other FL methods in TY1. In the setting of 10% few-shot forecasting, TIME-FFM achieves comparable performance against methods in TY2. In the setting of 5% few-shot learning. TIME-FFM attains SOTA performance on 20 out of 48 instances across five time series benchmarks. The results underscore that TIME-FFM promises effective few-shot forecaster.
E Error Bars
We conduct the experiments of TY1 for three times and report the mean values and standard deviations in Table 18 Table
The results demonstrate the superiority of our proposed TIME-FFM, which agrees with
F
Border Impacts
In this paper, we propose to build foundation model for time series forecasting hinging on the impressive capability of pretrained language models for sequence tokens reasoning. The promising advantages are two folds: (1) Data owners do not need to share the access to the private data samples which mitigates the privacy concerns and cater for data protection regulations (say GDPR). (2) The problem of "data island" can be tackled, which makes it possible to generate satisfactory performance in spite of data scarcity. To the best our knowledge, our research do not have obvious negative social impacts.
17

---

Table 13: Full results of forecasting performance comparisons. Yellow the best in TYI: Blue the second best in TY1. Underline: the best over all types; Bold: the second best over all types.
Type
TY1
TY2
TY3
Method
FIME-FM
FediT
|Fed//dagnee F8 prodAdapuer"
Unillime
GPT-TS
PacchTT
TimesNet
DLineer
|LIDfommer Autoformer
informer
Metric
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAE MSE
MAE
MAEJMSE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAE MSE MAE MSE MAB
96
0,385
0.400
0.346
0,436
0:455
0.441
0.447
0.441
[0,397
0.418
0..49 0.424 0,409 0.403 00,384 0.402
0,424/0.409
(0,386 0,400
0..37 0.419
0.449 0,459/0.865 0.713
192
0.4.9
0.430
0.480
0.454
0.486
0.059
0.481
0.461
0.434
0.439
0.503
0.4530.467
0.453 0,467 0.444
0.436
0.429
0.437 0.432
0.420 0.448 0 0,500 0.482 1.008 0.792
336
0,480
0.449
0.508
0.471
0,514
0.476
0.489
0.457
0.540
0,477 0.509 0.472
0.169
0.481
0.459
730
0.462
0.456
0.488
0.484
0.496
0.401
0.554
0.534
0.469
0.477
0.515
0.505 0.485
0.521
0.500
0.519
0,516
0.506
0.
0.514 0,512
1.181 0.865
AVG
0.442
0.434
01:481
00:661
0.488
0.467
0.503
0.439
0.442
0.448
(.502
0.46
0.442 0.451
(0,458
0.450
0.456
0.452
0.440
0.460
0.496 0,487
[1040 0.795
96
0,301
0.351
0.288
0.336
0.2899
0.340
0.298
0.316
10.296
0,345
0.303
0.3:9
0.314
0.361
00.340
0.374
0.333
0.357
0.355
0.397 0.346 3.755 1.525
192
0,378
0.507
0.373
0.387
0,375
0.390
0.379
0.504
0.374
0.394
0.391
0,390
0.407 0.431
0,400
0.414
0.477
0.476
0,420
0.439 0.456 0, 5.602 1,951
336
0:422
0.431
0.419
0.423
0.413
0.424
0/418
0:427
0.415
0.427
0.422
0:428
0.437
0.443
0.452
0.452
0.594
0.541
0.996
0:487 0:482 0, 488
4.721 1.835
730
0:427
0.418
0.436
0.416
0.443
0.425
0.444
0.447
0.440
0.434
0.448
(1446
0.468
0,831
0:657
0.466
0.511
3.647 1.625
AVGG
0,382
0.406
0.374
0.396
0.37.
0.398
0.380
0,400
00,378
0.403
0.386
0.400
0.399 0.436
0.444
0.427
00:55
0.511
0.437
0.449
0.450 0.459
4.433 1.729
96
10.36
0.609
0.095
0.610
0.616
0.198
10.322
0.363
0.509
0,463
0.927
0.601
00.338
0.375
10.345
0.372
0.379
0 0.475
0.672 0.571
192
0.378
0.389
0.639
0,512
0.641
0.507
0.639
0.508
0.366
0.387
0.537
0.476
0.964
0.620
0..74
0.387
0.380
0.389
0.426
0:441 0.553
0.4996.7995 0.669
336
0.411
0.410
0.083
0.521
0.648
0.515
0.637
0.515
0.398
0.407
0.564
00.4%
1.041
0.636
0.410
0.411
0.413
0.413
0.445
0.459
0.621 0.537
1.212 0.871
730
0.469
0.441
0.674
0.538
0.6700
0.582
0.667
0.541
0.440
0.504
0.950
0.636
0.410
0:450
0.474
0.453
0.544
0.490 0.671
0.561
0.833
AVG
0.399
0.402
0.644
0.517
00.44
0.640
0.536
0.,35
0.399
0.351
0.483
0.971 0.629
0.383
0.406
00.403
0.5077
0.448
0.452
0.517
0.961 0.734
96
00181
0.267
0.197
0.282
10.194
0.260
0.197
0.283
10.183
0.266
0.2229
0.304
0.2400
0.318
00.187
0.267
10.193
0.292
00.203 0.287 0:255 0.339
0.365 0,453
0.247
0.308
0.2660
0.25%
0.261
0.521
0.251
0.310
0.335
0.301 0.352
(1.349
0.309
(..784
0.
0.269 328 0 0. 0.340
0.533 0.563
336
0.309
0.347
0.318
0,355
0.316
0.353
0.319
0.356
01319
0.351
0.337
0.307
0.307
0.391
(1.321
0.309
0,369
0.427
0.325
0.366 0.339 0,312
1.363 0.887
720
0.406
0.404
0.415
0.808
0,414
0.507
0.416
0.409
0.420
0.410
0.130
0.451 0.432
0.408
0.403
0.554
0.522
0.421
0:415 0.433
0.432
3.379 1.338
AVG
00.286
0,332
0.297
0.341
002295
0.340
0.342 0.298
0.299
0.334
00.221
0.356
0.356/0.340
0.3440
0.373
0.291
0.322
0.255
0.403
0.327 0.371
1.410 0.810
96
1..99
0.282
0.375
0.469
0.9911
0,478
0.310
0.406
10.196
0.287
[.2232
0..21
10.198 0.290
10.168
0.272
[0.197
0.282
10.193 0 3.30 0.201 0.917/0.274
0.368
Eleetriiity
192
0.199
0.285
0.371
0.388
0.404
0.199
0.291
0.202 0.293
0.289
0.196
0.285
0.222 0.296
336
0.212
0.298
0.389
0.47%
0.408
0,489
0.333
0.421
0.214
0,305
0.333
0.225 0.318
0.198
0.300
0.209
0.301
0.214
0 0,338 0.300 0.394
720
0,253
0.330
0.424
0.497
0:447
0.511
0.384
0.450
0,254
0.335
0.289
0,366
0.259 0.341
0.229
0.320
0.245
0,333
0.266
0.355 0.254 0,361 0.373 0.439
AVG
0:216
0.299
0.300
0.478
00408
0.489
0.334
0.430
0.216
0.305
(1:251
0.335
0.338
0.221
0.311
0.193
0.205
0.212
0.300
0.214
0 0.23
0.338 0.311 0.333
0.307
96
0,191
0.230
0.198
0.250
[0.196
0.245
0.200
0.248
10.171
0.214
0.212
0.25
0.211
0.200
(0.172
0.220
10.19
0.255
(0.217 0.296 0.206 0,330 0,300 0.584
Weather
192
0.236
0.267
0.250
0,290
0,248
0.286
0.254
0.288
0.217
0.254
0.261
0.285
0.269
0.300
0.219
0.261
0.237
0.296
0.276 0.336 0.307 0.367 0.598 0.544
336
0.289
0.303
0.308
0.326
0.304
0.326
00:335
0.274
0.293
0.324
0.330
0.34|
0.306
0.788
0.335
0.339 0 0
0.578 0.523
720
0.362
0.350
0.378
0.374
0.382
0.377
0.385
0.377
0.351
0.343
0.386
0.372
0.404
0.389
0.365
0.359
0.345
0.381
0.403 0.428 0.419 0. 128 1.059 0.741
AVG
0.270
0.288
0.088
0.310
0.262
0. 103308
0.2887
0.3000
0.255
0.276
0.309 (1.203
0. 0.323
(02599
0.287
(.2665
0.317
0. 0.300 0 360 0. 338 0,783 0.634 0.548
96
0.081
0.201
0.302
0.225
10.100
0.221
0.098
0.218
10,0%6
0.209
0.142
0.26
00.137 0.200
00.107
0.234
10.088
0.218
10.148
0.397 0.325
0.325/0.847
0.752
Exchange
192
0.168
0.293
0.198
0,317
0.199
0.312
0.196
0.314
0.174
0.299
0.224
0.232
0,339
0.222
0.341
0.226
0.36d
0.176
0,315
0.271
0.380 0.300 0,369
1.204 0.895
336
0.299
0.396
0.350
0.430
0.345
0.426
1.345
0.425
0,319
0.408
0.377
0.445 0.372 0.447
0.367
0.448
0.313
0.427
0.460
500
3.5009
0.924
1.672 1,036
730
0.805
0.674
0.905
0,721
0.889
0.715
0.883
0.712
0,875
0.701
0.939
0.736
0.912 0.727
0.964
0.746
0.839
0.695
1.195
1.447
0.941
AVG
0..33
0.391
0.389
0.423
0.382
0,419
0.380
0.417
0.364
0.404
0.440 0.421
0.440)0.441
0.411
0.444
0.446
0.443
0.354
0.414
0.519
0.500
0.613 0.559
1.550 0.998
24
2.259
0.950
44.444
5.157
1.555
4.980
1,492
2,440
0.954
3.322
1.278|4.289
1.317
1,485
2.317
0.934
22.998
1,040
3.228
3,483
1,287
5.766 1.677
36
2.239
0.936
4.619
1.493
5.620
1.692
5.593
1.658
1.996
0.912
3,696
1,374
1.360
1.510
1.972
0.920
2.616
1,058
2.679
1.080 3,103
4.755
1,467
48
1.953
0.894
4.509
1.467
5.413
1.669
5.487
1.662
1.079
0.912
3.765
1.403
4.209
1.481
2.238
0.940
2.614
1.086
2.622
1. 1.078 2.669
1.085
4.763 1.469
60
1.976
0.916
4.020
1.382
4.797
1,569
1.586
2.109
0.938
3,923
1.452
3.981
2.027
0,928
2,804
2.857
1.157 2.770
1.564
AVG
2,107
0.924
4.423
1.448
5.247
1.621
5,251
1.600
2.137
0.929
3,6788
1.372
4.210
1.450
2.139
0.931
2.616
1,090
2.847
1.144 3,006
1.161
5.137
1.544
Average
0.555
0.434
0.010
0.547
1.015
0.560
1
1.009
0.561
(0.589
0.437
|0,000
0.52
|0.166
[0.500
0.445
|0.652
0.487
|0.600
505
0.756 0.533
1.934
0.944
1 Couns
32
3
19
17
3
D
0
18

---

Table 14: 10% few-shot forecasting results. Yellow the best in TY1: Blue : the second best in TY1. Underline: the best over both types: Bold: the second best over both types. '.' means 10% time series is not sufficient to constitute a training set.
Type
Method
TIMEs FPM
FedLoRA
FedAdapeer H
FodAdaater
Unitimm
GPT41S
PacciSST
Metric
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
0.571
0.481
0.638
0.496
0.651
0.518
0.708
0.535
0.582
0.485
0.611
0.486
1.136
0.672
192
0.578
0.490
0.626
0.622
0.696
0.564
0.479
0.637
0.499
1,118
0.672
ETTml
336
0.592
0.504
0.628
0.306
0.666
0.540
0.686
0.543
0.578
0.489
0.648
0.508
0.967
0.639
720
0.629
0.526
0.055
0.522
0.708
0.508
0.699
0.557
0.631
0.525
0.646
0.513
1.044
0.006
AVG
0.501
1,071
0.662
96
0.195
0.277
0.198
0.282
0.200
0.284
0.201
0.287
0.192
0.274
0.197
0.278
0.255
0.329
192
0.313
0.256
0.313
0.258
0.315
0.312
0.300
336
0.314
0.348
0.316
0.352
0.318
0.354
0.317
0.355
0.320
0.352
0.316
0.350
0.359
0.381
720
0.412
0.403
0.415
0.407
0.415
0:407
0.413
0.407
0.429
0.413
0.410
0.402
0.465
0,440
AVG
0.294
0.335
0.297
0.340
0.298
0.341
0.298
0.343
0,299
0.338
0.295
0.336
0.348
0.378
96
0.219
0,329
0.253
0.341
0.404
0.478
0.391
0.474
0.236
0.327
0.231
0.316
0.344
0.416
192
0.247
0.330
0.253
0.345
0.390
0.470
0.379
0.468
0.236
0.328
0.238
0.320
0.343
0.418
Electricity
0.349
0,361
0.129
730
0,300
0,368
0.319
0.400
0.460
0.518
0,452
0.513
0.295
0.371
0.292
0.365
0.309
0.453
AVG
0.266
0.344
0.275
0.363
0.421
0.489
0,408
0.486
0.254
0.342
0.351
0.334
0.302
0.429
96
0.207
0.258
0.210
0.258
0.201
0.252
0.203
0.255
0.191
0.242
0.215
0.262
0.215
0.259
192
0.259
0.297
0.265
0.301
0.254
0.293
0,255
0.205
0.240
0.278
0.270
0.304
0.265
0.297
Weather
0.300
0.327
0.314
0.324
0.306
0.293
0.315
0.319
0.336
0.318
0.332
730
0.381
0.373
0.386
0.308
0.386
0.388
0.375
AVG
0.2888
0.314
0.296
0.320
0.264
0.311
0.287
0.315
0.272
0.299
0.300
0.322
0.297
0.316
0.237
0.120
0.243
0.115
0.242
192
0.212
0,351
0.218
0.333
0.200
0.329
0,211
0.329
0.208
0.328
0.221
0.337
0.197
0.321
Eachange
336
0.362
0.438
0.378
0.447
0.358
0.435
0.364
0.439
0.335
0.424
0.384
0.451
0.347
0.428
AVG
0.230
0.336
0.238
0.334
0,280
0.338
0.220
0.331
0.242
0.344
0.220
0.3
Average
0.366
0.403
0.384
0,404
0.327
0.361
0.345
0.367
0.459
0.433
1 Count
25
12
Table 15: Zero-shot forecasting results of Exectricity and Weather with selecting different local parameters. Lower values correspond to better performance. Bold: the best.
Type
ETThi - Electricity
ETTml Eilecricity
ETTm2- Ellecricity
|| ETThl -a Weather
ETTml - Weather
ETTm2 Weather
Metnic
MSE
MAE
MSE
MAB
MSE
MAB
||
MSE
MAE
MSE
MAE
MSB
MAE
96
0.235
0,316
0.614
0.599
0.616
0.613
|| 0204
0.256
0.235
0,270
0.222
0.267
192
0.243
0.327
0.558
0,571
0.649
0.631
||
0:557
0.297
0.289
0.312
0,274
0.308
336
0.266
0.346
0.579
0,583
0.687
0.681
0.312
0.334
0.529
0.336
0.333
0.347
720
0.315
0.382
0.593
0.736
0.393
0.386
0.381
0.410
0.398
AVG
0.365
0.343
0.586
0.596
0,672
0.643
0.291
0.318
0.314
0,325
0,310
0.330
19

---

Table 16: 5% few-shot forecasting results. Yellow the best in TY1; Blue the second best in TY1. Underline: the best over both types; Bold: the second best over both types. means 5% time series is not sufficient to constitute training set.
Type
TYI
TY2
Method
TIME: FFM
FedLoRA
PedAdaaper R
FodAdapter"
Unillime
GPT4TS
PatcHTST
Metric
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
96
0.515
0.450
0.557
0.462
0,585
0.492
0,585
0.480
0.576
0.498
0.591
0.499
0.559
0.477
192
0.550
0.478
0.005
0.490
0.625
0.513
0,620
0.508
0.617
0.520
0.617
0.511
0.588
0.493
336
0.607
0.622
0.514
0.633
0.517
0.587
0.197
720
0.641
0.536
0.655
0.529
0.750
0.579
0.715
0.566
L.28
0.680
0.694
0.561
0.611
0.522
AVG
0.567
0.491
0.606
0.494
0.650
0.536
0,636
0.519
0.713
0,558
0.631
0.522
0
0.497
96
0.272
0.196
0.278
0.194
0.277
0.198
0.282
0.200
0.282
192
0.254
0.311
0.260
0.318
0.259
0.317
0.258
0.316
0.266
0.325
0.259
0.317
0.260
0.318
336
0.312
0.366
0
0.352
0.318
0.352
0,316
0.351
0.337
0.365
0.316
0.361
0.318
0.352
730
0.415
0.403
0:49
0.408
0.420
0,418
0.408
0.453
0.417
0.407
0.419
0.407
AVG
0.293
0.333
0.298
0.339
0.298
0.339
0.296
0,338
0.313
0.350
0.298
0.339
0.299
0.339
96
0.312
0.394
0.326
0.107
0.320
0.397
0.281
0.371
0.295
0.379
192
0.305
0.391
0.327
0.414
0.312
0.308
0.313
0.506
0.283
0.377
0.254
0.341
0.293
0.382
Electricity
336
0.321
0.401
0.340
0.422
0.338
0.417
0.385
0.412
0.294
0.365
0.271
0.354
0.308
0.392
720
0.358
0.427
0.365
0.436
0.364
0,365
0.430
0.335
0.413
0.313
0.385
0.341
0.413
AYG
0.324
0.403
0.339
0.420
0.333
0.383
0.409
0.298
0.387
0.273
0.355
0.309
0.391
96
0.214
0.265
0.222
0.269
0.212
0.262
0.219
0.267
0.209
0.260
0.207
0.259
0.221
0.271
192
0.302
0.275
0.310
0
0.270
0.305
0.25M
0.297
0.258
0.297
0.271
0.308
Weather
336
0.310
0.329
0.321
0.338
0.311
0.330
0,319
0.335
0.306
0.325
0.308
0.328
0.318
0.336
720
0.381
0.374
0.394
0.385
0.383
0.376
0.393
0.382
0.380
0.371
0.380
0.373
0.391
0.382
AYG
0.317
0.303
0.325
0.300
0.314
0.301
0.324
96
0.118
0.244
0.121
0.244
0.117
0.243
0.116
0.241
0.385
0,458
0.120
0,246
0.123
0.250
192
0.215
01211
0.337
0.315
0.333
0.215
0.333
0.498
0.528
0.216
0.334
0.230
0.337
Exchange
336
720
AVG
0.167
0.289
0.171
0.291
0.166
0.258
0.166
0.287
0.442
0.493
0.168
0.290
0.171
0.293
Average
0.329
0.367
0.344
0.374
0.348
0.376
0,346
0.375
0.411
0.420
0.332
0.364
0.334
0.369
1s Count
Table 17: Zero-shot forecasting results. Lower values correspond to better performance. Yellow the best in TY1: Blue the second best in TY1. Underline: the best over both types; Bold: the second best over both types.
Type
TY1
TY2
Method
FFM:-FFM
FeilT
FedAdigeer |f
FalAdapter
UniTime
GPTTTS
PareNTST
Metric
MSE
MAE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSB
MAE
96
0.296
0.364
0.303
0.351
0.303
0.351
0.304
0.352
0.306
0.352
0.316
0.361
0.332
0.371
192
0.373
0.391
0.391
0.401
0.391
0.402
0.300
0.401
0.389
0.401
0.400
0.410
0.423
0.421
ETT12
336
0.410
0.424
0.426
0.434
0.425
0.432
0.424
0.434
0.430
0.402
0.455
720
0.413
0.437
0.428
0.443
0.431
0.447
0.428
0.444
0.433
0.450
0.442
0.461
0.467
0.469
AVG
0.373
0.399
0.367
0.407
0.388
1.408
0.387
0.407
0.388
0,409
0.397
0.418
0.421
0.429
96
0.235
0.316
0.392
0.464
0.383
0.460
0.995
0.470
0.409
0.481
0.448
0.520
0.520
0.562
192
0.243
0.327
0.376
0.455
0.376
0.458
0.384
0.406
0.410
0.484
0.443
0.517
0.507
0.550
Electricity
336
0.266
0.346
0.404
0.477
0.412
0.484
0.462
0.536
0.566
720
0.315
0.382
0.428
0.490
0.441
0.199
0.646
0.506
0.187
0.531
0.194
0.512
0.563
0.581
AVG
0,265
0.343
0.30%
0.470
0.401
0.474
0.409
0.452
0.436
0,500
0.462
0.526
0.534
0.565
96
0.200
0.256
0.261
0:220
0.266
0.218
0.265
0.210
0.262
0:223
0.235
0.277
192
0.257
0.297
0,266
0,302
0.272
0.306
0,271
0.306
0.264
0.303
0.287
0.319
0.293
0.320
Weather
336
0.312
0.334
0.314
0.334
0.319
0.337
0.320
0.338
0.326
0.334
0.347
0.357
0.351
0.356
720
0,393
0.386
0.397
0.387
0.398
0.358
0.432
0.427
0.101
AVG
0.291
0.318
0.295
0,319
0.302
0.324
0.502
0.324
0.301
0.320
0.322
0.339
0.327
0.339
Average
0.310
0,353
0.364
0.402
0.366
0.404
0.375
0.410
0.394
0.427
0.444
20

---

Table 18: Mean values and standard deviations of TY1.
Method
TTME-FFM
FedLoRA
FedAdaapier"
FedAAapperr
ETTh]
0.44619.006
0.434a0.001
0483:0.002
0,462=0.001
0,478+0.020
011
0,496.0.008 008
0.482:0.004
ETTh2
0.383e0.001
0.407610001
0.37560.001
0.397=10002
0.375#0.002
0.399=0.001
0.377
0:401=0:002
ETTml 0.398:0.001
0.40220.001
0.66220.220
0.52340.006
0.64120.041
0.516.00013
0,673=00029
0.52940.012
ETTm2 0.286=90.01
0.33120.000
0.29820.011
0.343800001
0.29%5000003
0.54320000
0.299200001
0.34K40:002
Elecuicity
0.37720:022
0.45740:032
Weather
0.274:0.005
0.20120.004
0.284+0.002
0.310+0.001
0.283+0.002
0.3
0.285=0.002
0.311+0.002
Exchunge
0.349:0.017
0.39610.008
0.38910.002
0,42510.001
0.38410.002
0.41200002
0.380
0,41840.001
4.12+0.250
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and evaluation results in abstract and introduction elaborated Section and respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and future work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

We do not include the theoretical analysis for our proposed model, just like other relevant works.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We elaborate the our model architecture and how to perform training and inference in Section 3 and provide the experimental details in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code available at https//gghhaaacoooutyyyiiiiiailt
FFM/uree/main
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeee /etic///uuide//oeeuuuiiiiii SooodeSubiisiie ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental settings and benchmark data in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the error bars in Appendix 18
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the implementation details in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conform with all terms of NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We analyze border impacts our research in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose the risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the Section of Methodology 3 and Experiments we have provided the proper citations for the adopted technologies and results.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided detailed information of the model and the available source code at
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "g7N2SAA44z", "reviewer": "Reviewer_LqRs", "summary": "The paper introduces TIME-FFM, a federated foundation model aimed at addressing the challenges of time series forecasting due to data scarcity and privacy concerns. The approach involves transforming time series data into text tokens, leveraging pretrained language models (LMs) for analysis, and using a personalized federated training strategy.", "questions": "Include a more detailed discussion on the potential future developments and challenges for TIME-FFM, highlighting areas for further research and potential improvements.\nProvide a detailed explanation of the simulation results in Table 1, particularly regarding the discrepancies with reference 25. This will help in understanding the validity and reliability of your experimental results.\nCorrect the algorithm description to ensure that Line 6 reflects the appropriate content, as it currently appears to match what should be in Line 5.", "limitations": "Please refer to the described weakness above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tInnovative Approach: The idea of transforming time series data into text tokens and leveraging LMs is creative and could potentially address the data scarcity issue effectively.\n2.\tAddressing Privacy Concerns: The use of federated learning helps in alleviating privacy concerns and encourages data sharing without compromising sensitive information.\n3.\tPersonalized Training Strategy: The personalized federated training strategy, which combines global encoders with local prediction heads, is a thoughtful approach to handling data heterogeneity across domains.", "weaknesses": "1.\tAlgorithm Description: On page 6, the description of the algorithm seems to contain an error where Line 6 should actually be Line 5. Please carefully check the manuscript.\n2.\tExperimental Results Discrepancy: On page 6, the value of Baseline PatchTST on ETTm1 in Table 1 differs from the values reported in reference 25. You need to provide a detailed explanation of your simulation results and why these differences occur.\n3.\tDiscussion on Future Directions: The article could benefit from a more in-depth discussion of the future directions and challenges for the proposed method.\n\n[25] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. 369 Unitime: A language-empowered unified model for cross-domain time series forecasting. In 370 Proceedings of the ACM Web Conference 2024, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere thanks for the detailed and thoughtful review of our manuscript and for the encouraging appraisal of our work. We have addressed the specific concerns and respond to the constructive recommendations detailedly as follows.\n\n> **W1. Algorithm Description**: On page 6, the description of the algorithm seems to contain an error where Line 6 should actually be Line 5. Please carefully check the manuscript.\n\nThanks for your careful readings. Line 1-5 in Algorithm describes the process of global execution and the server obtains the global model parameters by Line 5. Line 6 annotates that the following lines are meant \"for local training\". We are sorry for confusing you  and will polish the description in the revised version.\n> **W2. Experimental Results Discrepancy**: On page 6, the value of Baseline PatchTST on ETTm1 in Table 1 differs from the values reported in reference 25. You need to provide a detailed explanation of your simulation results and why these differences occur.\n\nAs is described in the footnote of Page 5, we modify PatchTST into a \"unified\" version as per Reference [25]. Table 1 reports the averaged evaluation results over 4 prediction windows. The MSE and MAE values of PatchTST on ETTm1 dataset are \"0.971, 0.629\". In Reference [25], the averaged evaluation results of PatchTST in the type of \"Models Trained Across Datasets\" on ETTm1 are also \"0.971, 0.629\". Therefore, our simulation results agree with Reference [25].\n> **W3. Discussion on Future Directions**: The article could benefit from a more in-depth discussion of the future directions and challenges for the proposed method.\n\nWe sincerely show great gratitude for your valuable suggestion.  \n\n**For challenges**: We aim to propose a LM-empowered federated foundation model for time series forecasting, which is non-trivial technically, considering the following aspects.\n\n- **heterogeneous inputs**: Cross-domain time series data input into the foundation model are heterogeneous in terms of dimensions and historical readings, posing evident difficulty to modality alignment. \n- **Rigid instructions as prompts**: Prompts are adopted to bootstrap LMs for time series reasoning hinging on rigid domain-specific instructions, rather than the understanding of LMs, exhibiting poor robustness for unseen domains.\n- **Conflicts between generalization and personalization**: The ideal foundation model needs to learn the common temporal representations across domains and simultaneously enable the personalized prediction for domain-specific inputs.\n\n**For future directions**: Future directions include two aspects. \n\n- Thanks to the adopted \"personalized prediction heads\", future researches may focus on constructing a unified foundation model over different time series analysis tasks, such as classification, anomaly detection, and forecasting. \n- Further research should explore how to train a foundation model from scratch based on the existing and potential large-scale time series repositories.\n\nWe will follow your suggestions and supplement more in-depth challenge analysis and future directions in the modified version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ckhQtjTtZ3", "reviewer": "Reviewer_PDR4", "summary": "This paper introduces TIME-FFM, which is a Federated Foundation Model for Time Series Forecasting. TIME-FFM is comprised of (1) Modality Alignment which aligns time series patches with text tokens; (2) Prompt Adaption which learns the text prompts for an input time series; (3) LM backbone; (4) Prediction Head for domain specific output. Experiments are conducted on several benchmark datasets to demonstrate the effectiveness of the proposed method.", "questions": "1. How do you obtain $E'$ from $E$?\n2. Have you tried personalized Modality Alignment and Prompt Adaption for different domains?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Federated learning for time series foundation model is an interesting and promising direction. This paper presents the first attempt in this direction.\n2. Compared with SOTA federated methods (the TY1 group), the proposed method TIME-FFM could outperform the baselines. \n3. The writing is clear and easy to follow.", "weaknesses": "1. The overall novelty is limited. Modality Alignment, Prompt Adaption and different prediction head has been explored by previous methods. \n2. Compared with SOTA methods from TY2, the proposed TIME-FFM does not have a significant improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to you for providing constructive feedback on our paper. We have addressed the specific concerns as detailed below.\n\n> W1: The overall novelty is limited. Modality Alignment, Prompt Adaption and different prediction head has been explored by previous methods.\n\nThanks for your insightful concerns on our contributions. We aim to propose a language model-empowered federated foundation model for time series forecasting. \n\n- Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, Moment, etc. \n- For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to **adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions** like Time-LLM and UniTime.\n- Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with **global encoder and personalized prediction heads**. \n\nIn conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.\n\n> W2: Compared with SOTA methods from TY2, the proposed TIME-FFM does not have a significant improvement.\n\nThe methods in TY2 are centralized methods which intrinsically outperform federated methods with the same model structures. However, our proposed Time-FFM even outperforms the SOTA methods in TY2, which further indicates the effectiveness of the devised prompt adaption module and the personalized federated learning strategy.\n\n> Q1: How do you obtain $E$ from $E'$?\n\nThis is accomplished through a simple linear projection. Specifcally, given $E \\in \\mathbb{R}^{V \\times D}$, we learn a weight matrix $W \\in \\mathbb{R}^{V\\times V'}$ to identify a small set of text prototypes in $E' \\in \\mathbb{R}^{V'\\times D}$. We will revise the paper accordingly and include this technical detail in the revised version.\n\n> Q4: Have you tried personalized Modality Alignment and Prompt Adaption for different domains?\n\nThanks you for such insightful suggestions. **A.6 Time-FFM-D** in Table 5 denotes the distributed version of Time-FFM, i.e., personalized modules of Modality Alignment, Prompt Adaption, and Prediction Head for each domain. As depicted in Table 5, the performance of **A.6** is inferior to that of **A.1 Time-FFM**, which further indicates that constructing a unified model across domains outperforms training dedicated prediction models for each domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyaIFge9dV", "reviewer": "Reviewer_uvxC", "summary": "This paper proposed a federated foundation model for time series forecasting. This foundation model is trained in a distributed setting, with global shared parameters in a server, and domain / dataset specific parameters in clients. This allows the authors to personalize their predictions to local domain-specific data, while still learning general time series patterns. The global parameters belong to the prompt adaptation and modality alignment modules, whereas the local parameters include word embeddings, frozen LLMs, and client specific prediction heads. These methods are then compared with federated fine-tuning, other LLM adaptation methods for time series forecasting, and specialized time series models on standard time series forecasting datasets.", "questions": "I do not have any specific questions from the authors. But I would appreciate some clarity on the motivation and rationale behind the method. Please see my arguments above.", "limitations": "The authors have discussed some limitations of their work. I believe that the authors should mention the computational cost of their method as a limitation. Each dataset / domain / client has a large language model backbone + other parameters, which makes their method computationally expensive.\n\n----\n\nAfter rebuttal: discussions on legality of using non-permissively licensed datasets to train FMs and notions of privacy need to be discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to understand.\n2. The authors conduct multiple experiments including ablation experiments, which are well executed.", "weaknesses": "1. **Missing literature:** The authors do not cite or discuss two bodies of work which are pertinent to this discussion: (1) literature on time series foundation models, including models such as LagLLama [1], MOIRAI [2], TimesFM [3], Moment [4], Chronos [5] etc., (2) recent time series forecasting methods such as iTransformer [6], or older techniques such as N-BEATs [7] and N-HITS [8] etc. which have been shown to be strong time series forecasters, and (3) literature on federated learning for time series data.\n2. **Rationale and contributions:** I am not sure about the motivation behind the work. I feel that foundation modeling and federated learning seem at odds with each other, at least in this case. The authors argue that data owners may not be willing to share their data. But most time series foundation models, or even LLMs and LLVMs are trained on public data, and later adapted to private data, so I don't see the value proposition of such an approach.  Also I am not sure how this is a foundation model, as it doesn't solve multiple tasks, and is neither trained on multiple large-scale datasets. Secondly, why use large language models for time series forecasting, when studies have shown that there is sufficient data to pre-train foundation models from scratch [4], and that LLMs may not be better at forecasting time series data despite their significant computational cost [9]. Finally, I am not sure what the contributions of the work are. It seems like most of the work is built on architectural design choices from PatchTST, TimeLLM and Federated Averaging. Also, the personalization aspect sounds interesting, but can be addressed by fine-tuning foundation models, or custom built forecasting methods trained on specific datasets. \n3. **Baselines and comparisons:** The authors do not compare their methods with any recent time series foundation model, or recent time series forecasting models such as iTransformer, and old but performant methods such as N-BEATs / N-HITS. Recent studies such as [] have shown that older methods get the better of most recent methods in many settings.  \n\n### References\n1. Rasul, Kashif, et al. \"Lag-llama: Towards foundation models for time series forecasting.\" arXiv preprint arXiv:2310.08278 (2023).\n2. Woo, Gerald, et al. \"Unified training of universal time series forecasting transformers.\" arXiv preprint arXiv:2402.02592 (2024). \n3. Das, Abhimanyu, et al. \"A decoder-only foundation model for time-series forecasting.\" arXiv preprint arXiv:2310.10688 (2023). \n4. Goswami, Mononito, et al. \"Moment: A family of open time-series foundation models.\" arXiv preprint arXiv:2402.03885 (2024). \n5. Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024). \n6. Liu, Yong, et al. \"itransformer: Inverted transformers are effective for time series forecasting.\" arXiv preprint arXiv:2310.06625 (2023).\n7. Oreshkin, Boris N., et al. \"N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.\" arXiv preprint arXiv:1905.10437 (2019).\n8. Challu, Cristian, et al. \"Nhits: Neural hierarchical interpolation for time series forecasting.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 6. 2023.\n9. Tan, Mingtian, et al. \"Are Language Models Actually Useful for Time Series Forecasting?.\" arXiv preprint arXiv:2406.16964 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the insightful review you have provided for our manuscript. We have made the following response.\n\n> W1 & W3: Missing literature on **(1)** time series foundation models, **(2)** deep learning methods, and **(3)** federated learning methods.\n\nWe show great appreciation that the suggested literature is of great value to improve our manuscript. We will cite these works and report the corresponding performance comparison in the revision.\n\n**For (1):** Works in **(1)** aim to construct a time series \"corpus\" and then train a time series foundation model from scratch. The performance comparison is reported in **Table 1 of the attachment PDF in the \"Author Rebuttal\"**.\n\n**For (2):** We compare multiple time series forecasting methods in the manuscript and feel sorry for not covering the relevant methods you metioned. We report the performance comparison in **Table 2 of the attachment PDF in the \"Author Rebuttal\"**.\n\n**For (3):** There have not any literature focusing on building foundation models for time series forecasting. Therefore, we merely compare the federated fine-tuning methods in **TY1**.\n\n> W2.1: Rationale and contributions: **(1)** The motivation behind the work. **(2)** How this is a foundation model.\n\n**For (1):** We agree with you that LLMs or LLVMs are trained on public data. Compared with the modality of text, time series is more domain-specific and (commercial) copyright-sevsitive, i.e., **private knowledge may be inferred from historical time series readings**, especilly in finance and healthcare domain. Hence, it is of great significance to take data privacy into account when constructing time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due to data license restriction, such as Kaggle public datasets. Some foundation models like MOIRAI, have released large-scale public time series data **merely for research instead of commercial use**. \n\nHence, our work uniquely **bridges the gap between foundation models and federated learning**, which not only enhances the privacy and applicability of foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in the (commercial) field.\n\n**For (2):** We design Time-FFM as a foundation model for **time series forecasting task**, like MOIRAI and Lag-Llama. Due to the absence of public time series repositories, we merely train Time-FFM on 8 benchmark datasets. **Actually, Time-FFM can be trained on the built large-scale time series archive in MOIRAI or Lag-LIama** and then applied for downstream forecasting tasks.\n\n> W2.2: Why use LLMs.\n\nIn our manuscript, we adopt the first 6 Transformer layers of the pre-trained GPT2 in Time-FFM, while in the foundation models training from scratch (MOIRAI, Lag-Llama, MOMENT and TimesFM), stacked Transformer layers are also adopted as the encoder backbone. We simply freeze all parameters of Transformer layers, which can achieve comparable performance compared to fine-tuning or full-tuning (observed from Table 6 in the manuscript). Actually, with the available large-scale time series data, we can train our Time-FFM from scratch without being initilized by GPT2 parameters, like Chronos.\n\n> W2.3: **(1)** The contributions of the work and **(2)** necessity of personalized heads.\n\n**For (1)**: We aim to propose a LM-empowered federated foundation model for time series forecasting. \n\n- Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, Moment, etc. \n- For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to **adaptively construct prompts from how to understand patch tokens**, rather than **from rigid domain instructions like Time-LLM and UniTime**.\n- Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with **global encoder and personalized prediction heads**. \n\nIn conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.\n\n**For (2)**: Our proposed persoanlized strategy is coherent to the federated training process. Each domain participant merely uploads the updated parameters of encoders, **without additional modification of local optimization process**. Upon finishing federated training, a global encoder and multiple heads (one for each domain) are obtained. When applying to the downstream forecasting tasks, the global encoder and in-domain heads can be frozen and directly adopted, avoiding the process of fine-tuning.\n\n> L1: Limitations on computational cost.\n\n We agree with you that the computational cost is potentially high even we freeze all parameters of the LM backbone. In the revised version, we will follow your suggestion by supplementing such limitation. \n\nFurthermore, since Time-FFM works as a \"domain\" level time series forecaster, we think that federated training participants are more likely to be edge clouds with abundant computing resources, instead of resource-limited terminal devices. Upon training, Time-FFM can also be deployed at edge clouds for forecasting tasks. Hence, we think that the resource limitation might not be a bottleneck of the deployment of Time-FFM in real world.\n\nWe will carefully incorporate your comments in the revised paper. Considering the encouraging comments from Reviewers LqRs, PDR4, and 7qPS,  we believe our research findings are worth sharing with the research community. We sincerely hope that a revision is still considered.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99mgVghyMZ", "reviewer": "Reviewer_7qPS", "summary": "The authors study an important problem of time series forecasting considering the increasing concerns of privacy and copyright. The paper proposes a novel LLM-empowered federated time series forecasting model with three main components, i.e., modality alignment, prompt adaption, and personalized strategy.", "questions": "1. Are the practical deployment challenges considered?\n2. How to solve the bottleneck problem of limited FL resources? Could you provide more evidence?\n3. How to demonstrate the rationality of the prompt generated by LLM? How to justify the generated prompt is related to the corresponding domain? It would be better to provide more case studies on all datasets for a more comprehensive evaluation. In addition, it is especially encouraged to provide a corresponding theoretical analysis.", "limitations": "The main limitation is the practicality of the proposed Time-FFM. In addition, please see the weaknesses and questions for other limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The authors propose Time-FFM, a novel LLM-empowered federated foundation model for time series forecasting. Time-LLM encompasses a cross-modality module, a prompt adaption module, and a personalized federated training strategy.\n3. Extensive experiments show the effectiveness of the proposed method.", "weaknesses": "1. We often run federated learning on edge devices, which have restrictedly limited data processing capability. Time-FFM is difficult to deploy in real-world scenarios. Additionally, the high latency of LLM during inference may impede its applicability for real-time time series forecasting. This presents a practical challenge for deployment in resource-limited FL environments. Moreover, it requires at least a GPU to run LLM which is impractical and unrealistic in real-world FL scenarios.\n2. TimesFM [r1], a centralized foundation model for time series forecasting from Google, was accepted by ICML 2024. It was released on Arxiv last October. Google possesses various kinds of data from a variety of domains without any copyright and privacy concerns, which conflicts with your motivation. \n\n    [r1]. Das, et al., A decoder-only foundation model for time-series forecasting, ICML, 2024.\n\n3. More detailed explanations are needed regarding how the proposed method addresses data heterogeneity across clients.\n4. It would be better to improve and clarify the technique depth. Patching and channel-independent mechanisms are widely used in existing time series forecasting methods. In addition, the prompt adaption seems to be another version of an existing technique [22]. The projection head is also simple and straightforward. Further, the training process of Time-LLM is based on FedAvg.\n5. It is suggested to illustrate more on personalized strategy including the global encoder. In addition, it is also confusing how to deal with the conflicts between generalization and personalization.\n6. It would be better to convert TimesNet, DLiner, and PatchTST, into their federated version with FedAvg and compare their federated versions with the proposed Time-FFM. Additionally, it would be better to include a case study to intuitively show how accurately the proposed Time-FFM can predict time series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We show great gratitude to you for approving of the quality of our paper. We have make the detailed response to thoroughly release your concerns.\n\n> W1 & Q1 & Q2: Concerns on the applicability in real world.\n\nWe are sorry for not covering the deployment of Time-FFM in the manuscript. Since we make Time-FFM a \"domain\" level time series forecaster, we deem that federated training participants are more likely to be **edge clouds with abundant computing resources**, instead of resource-limited edge devices. Upon training, Time-FFM can also be deployed at edge clouds for forecasting tasks. Hence, we think that the resource limitation might not be a bottleneck of the deployment of Time-FFM and it realistic to deploy Time-FFM in real-world FL scenarios.\n\n> W2: TimesFM, as a centralized foundation model, conflicts with your motivation.\n\nCompared with the modality of text, time series is more domain-specific and (commercial) copyright-sevsitive, i.e., private knowledge may be inferred from historical time series readings, especilly in finance and healthcare domain. Hence, it is of great significance to take data privacy into consideration in the construction of time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due to data license restriction. Some foundation models like MOIRAI, have released large-scale public time series data merely for research instead of commercial use.\n\nHence, our work uniquely **bridges the gap between foundation models and federated learning**, which not only enhances the privacy and applicability of foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in the (commercial) field.\n\n> W3&W5: More details for how to tackle data heterogeneity across clients.\n\nWe aim to learn the underlying commonalities present in time sereis data, though corss-domain time series data are of great heterogeneity. In Federated Averaging framework, all domains learn a single shared model which would **perform poorly on each specific domain**. If we simply train dedicated model for each domain using their own local data (even if it is small) to train a local model, the domain-specific models may not generalize well to unseen domains. Therefore, we need to **strike a balance between the generalization and domain-specific prediction**. We devise a persoanlized federated strategy with a global encoder and personalized heads. The global shared encoder can learn the** common temporal embeddings or representations across domains**, which is in line with the success of centralized deep learning, i.e., the sharing of global feature representations among data [1][2]. The personalized heads enables **domain-specific decoders**, accounting for yielding prediction results fitting for local distributions.\n\n[1] LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 2015.\n[2] Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE TPAMI, 2013.\n\n> W4: It would be better to improve and clarify the technique depth. \n\nThanks for your insightful concerns on our contributions. We aim to propose a Language model-empowered federated foundation model for time series forecasting. \n\n- Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, etc. \n- For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to **adaptively construct prompts from how to understand patch tokens**, rather than from rigid domain instructions like Time-LLM and UniTime.\n- Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with **global encoder and personalized prediction heads**. \n\nIn conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.\n\n> W6: It would be better to convert TimesNet, DLiner, and PatchTST, into their federated version.\n\nThanks for your valuable suggeations. We report the performance comparison in **Table 3 of the PDF in the \"Author Rebuttal\"**. In the modified version, we will supplement the experiment results and the prediction case study to make the effectiveness of Time-FFM more convincing.\n\n> Q3: The rationality of the generated prompts.\n\nWe appreciate your attention on the rationality of prompts. We want to adaptively select the \"optimal\" text prototypes as prompts according to the domain-sepecific patch tokens by prompt adaption module. We think the prompts indicate how the LM backbone understand the domain temporal pattterns (conveyed by patch tokens) and thus can bootstrap domain-sepecific time series reasoning, though the prompts may have conflict with how we humans understand the patch segment [1].\n\nWe evaluate the effectiveness of prompts in Table 5. We have the key observation that the prompts can improve the forecasting performance (by comparing A.1 and A.2); the proposed adaptive prompt outperforms the domain instruction (by comparing A.1 and A.3). Furthermore, the showcase in Fig. 3 illustrates that different domain datasets correspond different text prototypes (say prompts).\n\nHence, we think ablation results and showcase can both reflect the rationality of prompt adaption. In the modified version, we will follow your suggestion to provide more case studies and theoretical analysis to make it more convincing.\n\n[1] Sun, C., Li, H., Li, Y., & Hong, S. TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series. In ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We commerce by thanking the four reviewers for their thoughtful and constructive comments. We are really encouraged to see that the reviewers appreciate some positive aspects of our paper, such as technical quality (**Reviewer LqRs, PDR4, uvxC, and 7qPS**) and presentation skills (**Reviewer LqRs, PDR4, uvxC, and 7qPS**). Your expertise significantly helps us strengthen our manuscript. We are sorry for several unclear parts and weakness mentioned by reviewers and endeavor to respond to each comment. We sincerely hope that the responses can release the reviewers' concerns. We present a brief introduction of the responses as follows.

- In response to the feedback from **Reviewer PDR4, uvxC, and 7qPS**, we have clarified the novelty and contributions of our paper. 
- In response to feedbacks from **Reviewer LqRs**, we have checked the details of the algorithm description and numerical results, and provided the in-depth analysis of the challenges and future directions.
- In response to feedbacks from **Reviewer PDR4**, we have clarified the experimental details and analysis of experimental results.
- In response to feedbacks from **Reviewer uvxC**, we have compared the performance with the suggested literature and analyzed the potential limmitations.
- In response to feedbacks from **Reviewer 7qPS**, we have detailedly analyzed the potential problems in practical application and supplemented the suggested performance comparison and analysis to validate the effectiveness of the proposed Time-FFM.

In the attachment PDF, we report the supplemented experimental results for **Reviewer uvxC and 7qPS**.

Meta-review: The paper presents a federated foundation model for time series forecasting, based on tokenizing time series data for easy ingestion by LLMs and a federated training strategy for addressing privacy concerns.  The authors and reviewers engaged in significant discussion which clarified several points, especially around whether federated learning is in fact useful in this setting, and the reviewers are (reasonably) satisfied with the rationale given.

Multiple reviewers raised questions on novelty, pointing out that the paper is largely combining previous ideas in a new way.  Personally, I think that the vast majority of computer science in general is founded on the idea that prior pieces can be combined in new ways, and feel that the field of ML is no exception.  However, my personal feeling aside, the author response discussion and response show that the reviewers have been again satisfied with the clarifications -- and I very much encourage the authors to incorporate these points into the revision.

On the question of overall impact of the work, again reviewers raised some concerns that were well addressed during the rebuttal and response phase.  Personally, my view is that the direct impact of the work is likely to be limited -- it is not clear to me that the world has a burning need for a fully federated way to adapt LLMs to time series forecasting.  However, the work is technically strong, and even more importantly I think serves as an excellent example of how components like this may be combined, and how a resulting system can and should be fully evaluated.  

As such, I believe this work is worthy of inclusion of NeurIPS, and agree with the reviewer consensus for accepting.  I very much encourage the authors to make full use of the extensive reviewer feedback to further strengthen the draft into its final form.
Decision: Accept (poster)