Synthetic Series-Symbol Data Generation for Time Series Foundation Models
Authors: Wenxuan Wang
Kai Wu
Yujian Betterest Li
Dan Wang
Xiaoyu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Time Series Analysis
Foundation Model
Data Scarcity
Primary area: deep_learning
TL;DR: Based on the complex dynamics perspective, this paper proposes a bimodal time series generation mechanism to mitigate the current data shortage problem in time series analysis.

Abstract: Foundation models for time series analysis (TSA) have attracted significant attention. However, challenges such as training data scarcity and imbalance continue to hinder their development. Inspired by complex dynamic system theories, we design a series-symbol data generation mechanism, enabling the unrestricted creation of high-quality time series data paired with corresponding symbolic expressions. To leverage series-symbol data pairs with strong correlations, we develop SymTime, a pre-trained foundation model for enhancing time series representation using symbolic information. SymTime demonstrates competitive performance across five major TSA tasks when fine-tunes with downstream tasks, rivaling foundation models pre-trained on real-world datasets. This approach underscores the potential of series-symbol data generation and pretraining mechanisms in overcoming data scarcity and enhancing task performance. The code is available at https://github.com/wwhenxuan/SymTime.

Synthetie Series-Symbol Data Generation for Time Series Foundation Models
Wenxuan Wang
Kai Wu* School of Artificial Intelligence
School of Telecommunications Engineering
Xidian University whenxuan@ieee.c
Xidian Univeristy kwu@xidian. eeweodiddinnedd......mm cn
Yujian Betterest Li School of Artificial Intelligence
Dan Wang"
School Telecommunicatoon Engineering Xidian University
Xidian University bebetterestt@uttlook com
danwang@xidian edu. cn
Xiaoyu Zhang
School of Cyber Engineering
Xidian University
xiaoyuzhang@xiddinncc edw.iaiiieneeuee.e.mm..eo.eeeeee cn
Abstract
Foundation models for time series analysis (TSA) have attracted significant atten- tion. However, challenges such as training data scarcity and imbalance continue to hinder their development. Inspired by complex dynamic system theories, we design a series-symbol data generation mechanism, enabling the unrestricted creation high-quality time series data paired with corresponding symbolic expressions. To leverage series-symbol data pairs with strong correlations, we develop SymTime, pre-trained foundation model for enhancing time series representation using symbolic information. SymTime demonstrates competitive performance across five major TSA tasks when fine-tunes with downstream tasks, rivaling founda- tion models pre-trained on real-world datasets. This approach underscores the potential of series-symbol data generation and pretraining mechanisms in over- coming data scarcity and enhancing task performance. The code is available at Symmimme
https://githhu:
Introduction
In recent years, with the rapid advancement of deep learning, foundation models for time series analysis (TSA) have garnered widespread attention due to their superior generalization capabilities, scalability and advantages in few-shot learning [ (2] Coupled with issues of data privacy 3 4 existing time series datasets are smaller compared to those in the fields of computer vision (CV) and natural language processing (NLP). Besides, current large-scale time series datasets face significant data imbalance issues, with certain types such as finance and healthcare still being relatively scarce (see Appendix [B.4). According scaling laws (5), this can lead to performance bias in the time series foundation models, reducing their generalizatio capabilities
out-of-distribution data 6 7
To mitigate the issue of training data scarcity and imbalance, this paper, starting from Takens' theorem g 9 posits that time series are representations of complex dynamical systems (10] 11 Based on
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

symbolic dynamics 13 complex systems can be expressed abstractly using mathematical symbols and formulas [14. with ordinary differential equations (ODE) and partial differential equations (PDE) being the most common methods for modeling complex systems [15 16 In an ideal scenario, continuously constructing diverse symbolic expressions allows us to cover a broader range of complex dynamical systems. As result, the time series generated from these symbolic expressions exhibit rich and varied properties. To this end, we provide a series-symbol (5²) modalityy data generation mechanism. Simulation experiments demonstrate that this approach effectively mitigates the problem of training data scarcity. To encapsulate our work, the contributions are as follows:
Addressing Data Scarcity: Our approach overcomes the challenge limited training data when building foundation models. The observation that the size of the S dataset directly correlates with model performance on downstream tasks validates this point.
Introducing Sym Time: We present SymTime, a scalable and efficient foundation model for time series analysis that leverages symbolic information to enhance representations. Pretrained on the constructed S dataset, SymTime offers broader task generality compared existing foundation models that are typically limited to zero-shot forecasting.
2 Related Work
Pre-trained foundation models (PTFMs) 18 19 20 21 have been demonstrated to adapt to variety of downstream tasks after fine-tuning on specific datasets, exhibiting excellent generalization
and scalability [22 23 Inspired by this, recent years have seen significant progress in PTFMs for TSA 24 25). with the emergence of various pre-training methods. Moirai, through masked time series modeling (MTM) and reconstruction [26] 27 has been pre-trained on large datasets (27B), yielding a universal forecasting model with zero-shot advantages (28). Timer, after generative pre-training large datasets (1B), has performed well in forecasting [29]. TimeGPT trained incoder-decoddd Transformer with 100B data [30]. COMET, using multi-level contrastive learning on large ECG dataset, has obtained medical time series PTEMs with few-shot advantages [4].
As discussed in Appendix B.4, these baseline models still face challenges related to data scarcity and data imbalance. In the next section, we introduce the proposed data generation mechanism and the corresponding mod--moylltty foundation model designed to mitigate these issues. The review of other topics can be found in Appendix D
Main Methods
Definition Time Series Foundation Model. is a deep neural network pre-trained in
supfrsed unsupervised manner on large-scale, diverse time series data. By learning general- izable time series representations, can then rapidly adapt- via few-shot or transfer learning- to efficiently solve wide range of downstream time series tasks.
Theorem 1 Takens' Theorem. This theorem demonstrates that through phase space reconstruction
3732 16 33%, a univariate time series, as a now-dimensional projection of high--dmenssionl complex system, can completely preserve the dynamic topology of the original system, thereby forming an effective external representation of the complex system
35
36
38
39/.
Theorem 2 Symbolic Dynam-
Time Series
Complex Systems
Symbolic Expression
ics. This theory encodes the evolution of continuous systems into finite symbolic expressions by discretizing the state space of complex systems, establishing an isomorphic relationship between symbolic expressions and system behaviors 1I3
Takeus Theorem Low Dimensional
Symbolli Dynamics
dr
Quantitative Mudeling
x (p -0)
dt de
Projection
Building Correspondence through Complex Systems
Figure 1: The connection between time series and symbolic ex- pressions (taking the Lorentz system example) 40
43 44).
This means that any complex sys- tem can be modeled using symbolic expressions 445 46%.

---

Mixed Distribution
ARMA Model
Masked Token
(CLS)
Nagetive [CLS]
Cutm
Cyilm
Random Sampling
Time Series Project
Symbol Linear Project
Cyss
Cea
Aoye Sampling Time Series
f(z)
hagetie
Reger
Forward
Feed Forward
tan
Multi-head Self Attention
Negative Samples
Mult-hhad Self Attention
Time Series Encoder
Momentum Model
Symbol Encoder
Positional Encoding
Positional Encodinng
tan( 0.38) Symbolic Expression
Patch Masking
Taken Embreder
// wi Map
0.58 (-0.502
xx))
2.00 (5.(5.2)
@ Generated Time Series
time series inputs
symbal inputs
Figure 2: S dataset generation mechanism (left) and SymTime network architecture (right).
As show in Figure T the two aforementioned theorems, using complex dynamical systems as conceptual bridge, fundamentally connect time series and symbolic expressions. They provide rigorous theoretical support for the semantic correspondence between temporal patterns and symbolic representations. Compared with previous time series data generation methods 47 22), the S data generation mechanism proposed in this paper more line with the nature time series generation.
3.1 Series-Symbol (S²) Dataset Generation
The pre-training of SymTime relies on a large synthetic series-symbol (S²) datase The specific generation process is shown in Figure 2 (left). Firstly, we construct multivariate input-output symbolic expression f(.) through random sampling [48). Then, we use the randomly generated sampling series X E L to forward propagate through the symbolic expression to obtain the generated series = f(X) E N where N and M represent the dimensions of the input and output series respectively, and L is the length of the series. The mathematical symbols and their explanations in this section are shown Table 9 In Appendix [B3] we present an analysis of the statistical characterization of the S data. In Appendix B..7 demonstrate that the time complexity of generating the 5 data scales lincarly with the series length L, approximating O()).
3.1.1 Sampling of Functions
Mathematical expressions can usually be represented using tree structure, where constants and variables the root nodes, binary operators are nodes with two children, and unary opera- tors are nodes with one child. Therefore, we (1) build binary tree over input variables using bi- nary operators, (2) randomly sert constants and variables into the tree, and (3) add unary opera- tor and perform affine transformation. The three key steps are illustrated in Figure
sin
(a)
(b)
Figure 3: The process of building binary tree when sampling symbolic expressions. (a) tree construction; (b) variable assign-
ment to leaf nodes; (c) unary operator insertion.
Thh code for 2 data generation is available httppp://githuu..coo/eweeeeeeeee

---

Input/Output Dimension Selection. Instead of randomly sampling input/output dimensions M ~ (1,1, maxx and N U(1.NNasx as in prior work [[2 14 49, we exhaustively traverse M e [1, Mmxxx, N E [1; Nmax] (with Mmax 6, Nmax 12) to fully cover multivariate time series representations. An input dimension M defines M variable nodes (I), (#1,...,,AM) while the output dimension yields N generated series = fi TM), 1,..., 48
Binary Operator Selection. We sample the number of binary operators b ~ [d dmax) to define the root nodes of the expression tree. Each node's operator is then drawn uniformly from U(+: -, x enhancing the diversity and complexity of the generated expressions [14 12 45 46
Tree Construction and Leaf Assignment. We randomly combine the b binary operators to con- struct a binary tree form the basic framework of the mathematical expressions (Figure Ba). Then, we randomly select m variables from the M set of variables [æ1..... TM] and insert them into the symbolic expression consisting of binary operators, where m (11 MM ) (Figure Bp). If the inserted expression does not form a full binary tree, random constant node is added to make it full.
Unary Operator Insertion. After inserting the leaf nodes to form a complete binary tree, we select the number of unary operators from U(umin Umax) and insert unary operators at random positions in the binary tree. The available unary operators include finv, abs, pow2, pow3, sqrt, sin, cos, tan, arctan, log, exp). This process is shown in Figure 3 In Ap- pendix B.8 we discussed the choice of unary operators.
To further diversify the symbolic expressions, we perform random affine transformations on each random variable za and unary operator Us in the binary tree. Specifically, we replace za and Ud with azd + and aud + b, respectively, where a and b are random constants For example, Ti - arı + and tan(-) - tan(-) b with contants a, sampled randomly.
Affine Transformation.
14
3.1.2
Generating Inputs and Outputs Series
After obtaining symbolic expressions fi, we sample X R MxL from mixed distributions
[2
48) and aundom-paramete ARMA(p. q) processes [50 [T, then compute Y E NN = (X) The ARMA(p, ?) model consists of moving average (MA) and autoregressive (AR) processes 52 which can be expressed as:
Y D
OpYi-p + -1c--1 O2e1-2 -....... -.,0--6----
(1)
where p and q represent the orders of the AR and MA models, respectively, Φy and 0, are the parameters of the AR and MA processes [51], and é, ~ NN(, 1) denotes the observed white noise sequence. Since ARMA possess both the temporal correlation of the AR process and the randomness of the MA process, series obtained from mixed distributions and ARMA sampling better reflect the characteristics of time series.
Sampling Strategy. Each input series X RM is drawn either from mixture of k (11 kmmx) distributions (with weights wy ~ U(00 1) normalized to ΣS Wj = 1, and each com- ponent chosen N ((,,, of), My ~ ~N(00, Jj U0, 1), or p;)) with probability P. < 0.5, or from an ARMA(p, 9) process (p (11 Pmax), ), q ~ (max),, parameters s, 0, 21(11111)) 1), and stationarity enforced by Σ. 0i 1, l0pl < 1) otherwise.
Series Generation and Curation. We normalize each X per channel, compute Y = S(X) via symbolic expressions, and discard any X outside I's domain or |Y] > 10a 12 53 For each random seed, we traverse all input/output channels, sampling each expression once. The resulting S dataset contains 40M series-symbo pairs (50B total length); series patched for the time series encoder (54) and expressions tokenized for the symbolic encoder [55].
3.2 Model Architecture of SymTime
As shown in Figure 2 (right), SymTime comprises a time series encoder, a symbol encoder, and momentum encoders, each trained with distinct objectives.

---

Time Series Encoder and Masked Time Series Modeling (MTM). An input time series is first divided into non-overlapping patches P=(pi. P2. Pnl using a sliding window approach [54 [7]. A 6-layer Transformer encodes non-overlapping patches P with random masking and reconstructs them via
Σ - jEMr
Cmtm
[
where My is the set of masked patch indices, and Pj represents the patch reconstructed by the time series encoder and linear projection [56] [57]. Then, we obtain the corresponding embedded sequence (= 41, 1,, In), where let, is the [CLS] token added by the time series encoder (58)
Symbolic Encoder and Masked Language Modeling (MLM). We treat symbolic expression data natural language and use the 6-layer DistilBERT [55 symbol encoder to learn the representation of symbol through natural language mask modeling [9]. The loss optimized this part is
Culmm
Σ jEMg
H(ys, Pj m
|Ms
where Ms are masked symbol positions, H is cross-entropy loss, m denote the model's predicted probability for the masked token 6, and yj one-hot vocabulary distribution with probability of for the ground-truth token. Then, we obtain the embedded sequence: S = {Sals: 81, , 8m) where Sel is the [CLS] token added by the symbol encoder.
Series-Symbol Contrastive Learning. To leverage series-symbol data pairs with strong correla- tions (The correspondence between positive and negative samples is shown in Appendix (C2). we employ contrastive learning to enhance time series representation using symbolic information. Using momentum encoders [59], we project [CLS] embeddings via linear projections It, Is and define sim(t, s) ge (4ex)) T (dax), where n' (''as) is the normalized symbol features generated by the momentum model. Similarly, sim(s, t) = (s(w))) ('((t))). We compute:
2a cxp(simm(t, sm))T)
exp(sim(s. fmT/T)
p (s)
Σm exp(sim(t, sm) 3/T)
Σm exp(sim(s, tm)/
where 7 is learnable temperature parameter 4 59 Let (2= and y (s* represent the one-hot similarity, with positive pairs having a probability
and negative pairs having [59]. We optimize
Lisc
2 E
(H((
12s
.p (2t) + H(y , )²
(5)
3.3 Momentum Distillation for Masked Data Learning
Inspired by ALBEF 60) we treat random masking noise and use momentum distillation to align the output representation of our encoder with its momentum counterpart. Let the similarity functions generated by the momentum encoders be sim' (t, s) " (s(s)) and sim' (s, t)
(o(1s)s)) ur (Cats) We compute soft pseudo targets g²2((1) and (x) by replacing sim with sim' in
Equation 4
In addition the contrastive loss Lise (Equation 5), we compute pseudo-targets 2 222,
from momentum-encoder similarities sim' and optimize
-mod = 1 E [KL (2222x (t(2(())) (t) + KL 22 (s)ll) (s)
(6)
The total pre-training objective is
L = Lmtm + Liim + aLrsc + (1 a) Liod
3.4 Fine-tuning for Downstream Tasks
We use the pre-trained time series encoder backbone. After instance normalization 61 the following two strategies:
we apply
Classification: patch the series, encode, and classify via linear head
54
62
Reconstruction (forecasting, imputation, anomaly detection) 63 64 decompose each series into trend and periodic components; regress trend directly, patch and encode the periodic part, then recombine for the final output.
5

---

Mear
fancerrttiy
FFT Veer
Draves
Eneres
Draves
ABF
Seaerrslly
Seaeoraiit
Seaerrally
Diraet
22 Dunnee
Moowe
Mineen
52 Chareel
20 Charnel
SP Charne
Miresh
(a) 100K single input channel
(b) 200K single input channel
(c) 100K single and dual channel
Figure 4: Radviz visualization of S and Monash datasets.
Experiments
We explore multiple representation measures and conduct experimental verification on variety of downstream task datasets to answer the following key questions:
RQ1: Can the unrestrictedly generated S dataset comprehensively cover diverse represen- tation types of time series data?
RQ2: Can SymTime pre-trained on the S dataset achieve competitive results across five major TSA tasks (forecasting, classification, imputation and anomaly detection)?
RQ3: Can SymTime learn fundamental representations of time series data the synthetic S2 dataset alleviate the data scarcity in TSA?
RQ4: Are the multiple pre-training objectives in SymTime effective, and can symbol expressions enhance TSA task performance?
RQ5:
How demonstrate that SymTime learns semantic information of symbols?
4.1 Statistical Characterization and Representation Coverage of 52 Dataset (RQ1)
Target. We quantify the range of representations that the S2 dataset cover through statistical metrics (ineluding stationarity (ADF Test) [65], forecastability [66], frequency domain (FFT mcan), seasonality [67]. trend (Mann-Kendall Test) (88 and prmutation entropy (69) (See Appendix B.5 for full descriptions).
Setup. We use Radviz 770 to visualize high-dimensional statistical features of 256-1ength time series segments from our synthetic S² and the Monash datasets (71). From Monash (covering weather, traffic, electricity, tourism, medicine, and energy) we sample 200 segments per domain. For S we sample 100K single-channel segments (Figure 44), then 200 single-channel segments (Figure [b)) and finally 100K mixed single- and dual-channel segments (Figure 4b).
Results. Radviz visualization confirms that 52 closely matches the Monash dataset across key statistics (stationarity, predictability, frequency, complexity, seasonality, trend), validating its use for pretraining. Expanding from 100K to 200K samples further broadens 52's coverage--suppaatt Monash in some regions. Moreover, combining single- and dual-input samples dramatically increases diversity, as multi-variable expressions f(r,, In) generate richer dynamics. These findings demonstrate that our infinitely scalable 5 dataset covers the entire time series representation space.
4.2 Validation of SymTime in Five Time Series Analysis Tasks (RQ2)
Setup. We pre-trained SymTime on the 50B-scale 52 dataset using Equation 7 as the pre-training objective, with the model architecture detailed in Table [6] Then, we evaluate SymTime on five TSA tasks: long-term forecasting, short-term forecasting, classification, imputation and anomaly detection, using the TimesNet benchmark 772). We use mean squared error (MSE) and mean absolute error (MAE) as the metrics for long-term forecasting and imputation tasks; overall weighted average (OWA) for short-term forecasting, which unique metrics for M4 benchmark [73]; accuracy for classification; precision, recall and Fl score for anomaly detection. Detailed descriptions of datasets

---

Long-torm Forocasting (MSE)
0,3148
PatchTST
Mean Square Erro (MSE)
iTransoormer
TimesNaksip.LL.
Time-LLM
Classificatinn (Accuracy)
74.8
a 1019 aorrShort-Sim F Forerasting (OWA)
0.30
68-7
0.922
0.48
Mpiral
Parl-midFormer
0.47
Timer
Memory Usage
0.48
|30MII 750MIB 1200MIB
Anomally Detection
(F1-Soara)
86.3
0.099
Impulation (MSE)
0.45
Time
9ymTime((uur)
GPT4T 9(2025)
Treenffmmen22004)
20
40
Number of Parameter (109)
60
80
100
120
160
Porl-mid(2224)
TimosNot/2023)
PatchTST/2023)
Figure 5: Model performance comparison with the state-of-the-art models in terms of five tasks (left).
Complexity analysis on long time series forecasting tasks (ETTh1 dataset, forecasting length is 720
with 96 look-back windows) (right). Note that since the original backbone of Time-LLM [7 has too many parameters, we replaced it with GPT2 194)
Long-term
Shorl-term
Forecasting
Forecasting
Classification
Imputation
Anomaly Detection
Peri-mid"
PatchTST
DLinear
Informer
PatchTST
Chronos
GPT4TS
PatchTSIT
ETSfomeer
ETStomer
5 SIPP-LLM
Time-LLM
LighTS
DLineer
Stationary
FilterNet
SPIP-LLM
ETSorreer
Light's
DLinear
Moirai
FEDformer
Transformer
Transformer
Peri-mid'
GPT4TS
FILM
Crossformer
LighTS
TSLANet
DLinear
GPT4TS
Moment
BedTADBench
Time
LighTT
TimesNet
PatchTST
PeFAD
Moment
Peri-mid
UinTS
GPT4TS
TimesNet
TimeMixer
TimesNet
TLNNet
TimesN
GPT4TS
PatchTST
TimeMixer
Pari-mia"
Pari-mid
UniTS
SymTime
SymTime
SymTime
SymTiiee
Sym Time
0.300
0.325 MSE
3.0
0.00
0.05
0.10
OWA
Acouracy (%)
MSE
F1 Score (%)
Figure 6: Validation of SymTime in 5 time series analysis tasks. We only show the average results of all datasets in this figure. See Appendix [ for full results and analysis on different tasks. ensure fair experimentation, we use the original model hyperparameters. For long-term prediction tasks, when a model is tested multiple look-back windows, we select the best result.
and metrics for each task are provided in Appendix C.I and CB3 while the pre-training and fine-tuning configurations for SymTime across downstream tasks are outlined in Appendix C.4 and C.5
Baselines. We compare with various baselines including Trransoormerbaseee models: PatchTST
[54, iTransformer (74) Autoformer (75), ETSformer (76), FEDformer [77], Non-stationary Trans-
former (78), Crossformer 79), Informer 80), Anomaly Transformer [1]. Peri-midFormer [63];
LLM-based models: GPT4TS Time-LLM [17], S2iP-LLM 82] CNN-based models: Times-
Net (72). TSLANet 83 Rocket [84. Inception
TimeiiooTiimmm Time (InTime) 85 and MICN [86]; MLP-based
models: DLinear 87 LighTTS 88) TimeMixer 89 FITS 90) and FilterNet 91.. We alse compare
with the pre-trained foundation models: Moirai (28), Timer 229 UniTS 92 and Moment [93].
Some models can be applied to all TSA tasks, while others are suitable for only one or some specific
tasks. For those pre-trained foundation models, we first load their pre-trained parameters and then fine-tune them in the same way.
Main Results. Figure (left) compares SymTime with models of the same type, while Figure 6 presents SymTime's 's performance against additional models across different tasks. These results
7

---

Table Fine-tuning results of on long-term forecasting tasks with different pre-training dataset sizes. See Appendix F.8 for full results. The look-back window length for all experiments is 96. Red: best, Blue: second best.
Datasets
ETTml
ETTm2
ETThi
ETTh2
Weather
Electircity
Traffic
Exchange
Average
Metries
MSR MAR
MSE MAF
MSE MAF]
MSF MAF
MSE MAR]
MSE MAF MSF MAF]
MSE MAF
MAF|MSE
MSE MAR
OB
0.401 0.409 00229 0.339
0.487 0.474 00376 0.412 0.257 0.289
0.1 0.289
0.284 0.471 0.310 0.415
0.358 0.366
IB
0.376 0.398 0.292 0.331
0.461 0.459 0.403 0.419 0.257 0.282 0. 199 0.285 0.473 0.303 0.370 0.410
0.354 0.361
10B
0.376
0,393 0.281 0.329
0 444 0,444 0.376 0.408 0.250 0.279 196 0.286 0.473 0.294
0.368
0.407
0.345 0,355
25B
0.378
0.393 0.278 0.325
0 434 0.438 0.371 0,405 0.253 0.282 0, 195 0,288 0.467
0.299 0.357 0.401
0.342 0.354
50B
0.371 0.390 0.274 0.321
0.430 0.436 00365 0.402 (0.247 0.276 0.187 0.276
0.457 0.291 0.359 0.401
0.336 0.349
Table 2: Fine-tuning results on short-term forecasting and imputation with different pre-training dataset sizes. See Appendix F. and F.11 for full results. Red: best, Blue: second best.
lasks
Short-term Time Series Forecasting
Time Series Imputation
Datasets
Yearly Quartly Monthly Others Avg
ETTml
ETTm2
ETThl
ETTh2
ECL
Weather
Metries
Overall Weighted Average (OWA) MSE MAE
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
0B
0.782
0.913
0.964
1,097 0.887 0.042 0.122 0.038 0,106 0.112 0.230
0.230/0.065 0.065 0.160 0.058 0. 0.155 0.036 0.053
1B
0.784
0.911
0.893
1.082 0.861 0.039 0.119
0.031 0.097 0.113 0.217
0.066 0,160 0.057 0.152 0.033 0.05U
10B
0.783
0.905
0.896
1.055
0.859 0.038 0. 19 0.030 0.095 007 0.213
0.063 0. 158 0.056 0.151 0.033 0.048
25B
0.788
0.909
0.877
1.061
0.856 0.037 0. 18
0.028 0.093 104 0.207
0.059 0. 154 0.055 0.1 0.030 0.043
50B
0.786
0.872
0.872
1.045
0.849 0.036
0.117 0.026
0.088
0.095 0.201
0.058 0.148 0.054 0.151
0.028 0.038
100
Classification Accuracy
01
1B
1081
2018
503
80
Scare
08
4.79
1,10
500
70
3.70
108
3.25
SMD
Time.
MBL
Seriks Anomaly Delection Datasels
SMAP
SWil
P6M
(a) Time Series Classification
(b) Time Series Anomaly Detection
Figure 7: Fine-tuning results of on classification and anomaly detection tasks with different pre- training dataset sizes. We use critical difference diagrams measure the performance of classification in (a) [95]. where the specific values represent the comprehensive ranking of the model on multiple datasets. See Appendix F.10 and for full results.
demonstrate that SymTime, pre-trained on the 5 dataset, successfully learns fundamental representa- tions of time series data and achieves competitive results when fine-tuned on downstream tasks. For each different task, the specific experimental settings and results are shown in: long-term forecasting (Appendix A. short-term forecasting (Appendix A.2 classification (Appendix A.3 imputation (Appendix A.4 and anomaly detection (Appendix
Complexity Analysis. We analyze the complexity of the model on the long-term forecasting E dataset, with results shown in Figure (right). We consider the parameter count, the GPU memory required for forward and backward propagation when the batch size is 1, Using MSE an evaluation metric, we find that SymTime ime achieves better performance with a smaller model parameter count and memory capacity than existing foundation models in forecasting task.
4.3 The Impact of Pre-training Dataset Size on SymTime Performance (RQ3)
Setup. Following the scaling laws of neural networks 6 96), large-scale and representationally comprehensive pre-training dataset is critical for building foundation models. Section 3.1 and 4.1 we introduce and validate the unrestricted generation capability and comprehensive representational coverage of the 52 dataset. Building on this, we proportionally divide the generated S dataset into timestamp-basee subsets (0B, 1B, 10B, 25B, 50B) and pre-train SymTime under identical
8

---

configurations. To evaluate the effectiveness of pre- pre-training, we fine-tune the models on five major TSA tasks (with 0B denoting direct fine-tumil linettuning without pre-training). The experimental results are summarized in Tables 1 Table 2 and Figure 7
Results. The results reveal a clear trend: as the scale of the pre-trained 5 dataset increases, SymTime shows progressively enhanced performance across diverse downstream tasks. Furthermore, pre-trained SymTime significantly outperforms its non-pre-trane non-pre-trained counterpart. Given the unrestricted generation capability and comprehensive representational coverage of our generation methods, the S dataset can theoretically scale to infinite size, progressively enhancing SymTime 's downstream task performance through pre-training. This demonstrates that the proposed S mechanism enables models to learn the representations of time series while effectively mitigating performance degradation caused by data scarcity and imbalanced distributions.
4.4 Ablation Study on Different Pre-training Objectives (RQ4)
Setup. We conduct ablation studies on SymTime's pre-training objectives using the ETThl and
ETTh2 long-term forecasting datasets [0). First, we establish 8 different control groups based on whether pre-training is performed, freezing the model and various pre-training losses: (1) Freeze, (2)
Real-Data, (3) w/o Pre-train, (4) w/o MTM, (5) w/o MIM, (6) w/o T2S, (7) w/o S2T, (8) w/o Symbol and (9) w/o Distill. Specific explanations for the above control groups are provided Appendix CC66 (Real-Data means use real time series data of the same scale to pre-train the time series encoder only through the MTM.). Then, We adopt MSE as the metric, load the pre-trained parameters from each ablation configuration, and conduct multiple fine-tuning trials with varied random seeds on the ETT dataset. The average results with error are shown in Figure
Results. As shown in Figure 8 pre-training with standard configurations through Equa- tion significantly enhances SymTime's long-term forecast-
Mean Square Entor (MSE)
ETT1
0.00
Eh
ing performance compared to control groups, demonstrating the validity of its pre-training ob-
0.40
jectives and the effectiveness of S2 data generation. It is worth noting that, whether on real time series data (Real-Data) or on syn- thetic 5 data (w/o Symbol), re- moving the symbol encoder part with contrastive learning and relying solely on MTM loss for temporal representation learning will degrade performance of SymTime. This suggests that semantic information provided by the symbolic encoder and series-symbol contrastive learning improves the temporal encoder's capability in long- term forecasting [97]. Similar findings are replicated in short-term forecasting tasks, detailed in Appendix C.7
0.30
SynTime
Syrmal
Figure 8: Ablation study on long-term forecasting task.
The Backbone in SymTime. There is time series encoder with a Transformer architecture and symbol encoder consisting a pre-trained LLM in SymTime. In Appendix C.8 we perform ablation experiments on the model architecture by replacing the backbone of SymTime. The results are shown in Figure [8] It is clear that replacing the backbone has no significant impact on downstream tasks. The improvement in model performance mainly comes from the dual-modal pre-training paradigm of time series and symbolic expressions (Equation 7).
4.5 Series-Symbol Representation Learning (RQ5)
We further analyze the data representations learned by SymTime through masked modeling and cross-modal contrastive loss. Given that contrastive learning aligns mutually positive samples in the representation space 998 59 we pre-train SymTime the S dataset and extract its time series encoder and symbol encoder. Using generated univariate time series and unary symbolic expressions, examine whether the representation spaces of encoders evolve during pre-training. Additionally,

---

(a) without pre-training
(b) with pre-training
(c) without pre-training
(d) with pre-training
Figure 9: The t-SNE visualization of time series encoder and symbol encoder in SymTime representa- tion space with 50 perplexity. (a)(b) time series encoder; (c)(d symbol encoder.
we specifically evaluate the time series encoder's proficiency in temporal representation learning. Results demonstrate that the time series encoder, pre-trained via large-scale masked modeling [54 27), achieves zero-shot imputation capability on both the 5 dataset and real-world time series data. Detailed experimental analyses are provided in Appendix [B66.
4.5.1 Time Series Encoder Representation
Setup. We sample 20K single-channels series-symbol pairs from S with each time series labeled by one unary operator (e-g., inv, sin, log). After patching, the time series segments are encoded by time series encoder of SymTime with and without series-symbol contrastive pretraining. The output embeddings are reduced via t-SNE 199.
Results. As shown in Figure 9 (a)(b), The untrained encoder produces entangled clusters (aside from outliers like inv, exp). whereas the pretrained encoder forms clear, operator-specinn clusters (trigonometric: sin, cos; polynomial: pow2, pow3, sqrt; etc.), confirming that contrastive learning aligns symbolic semantics with series representations. The change in representation space before and after pre-training also proves that the time series encoder has mastered the semantic information of symbolic expressions. Ablations show both contrastive losses are essential.
4.5.2 Symbol Encoder Representation
Setup. We also use 20K series-symbol pairs to verify the representation space change the symbol encoder. First, we tokenize the symbolic expression. Then, we input it into the encoder obtain its
[CLS] token 18 60 and visualize it using the I-SNE [99]
Results. As shown in Figure[(c))C) similar to the time series encoder, the pre-trained DistilBERT- based symbol encoder initially struggles to distinguish between different types of symbol (55). However, after pre-training with MLM and contrastive learning, the encoder forms distinct clusters in the representation space for symbolic expressions of the same type [12). Furthermore, the paired time- series data and their corresponding symbolic expression exhibit similar clustering characteristics, as shown in Figure 9 (b) and (d). This demonstrates that modal representation learning effectively brings semantically related data points closer together in the representation space.
5 Conclusion
To mitigate the challenges of data scarcity and distribution imbalance in time series analysis, we introduce dual-modality series-symbol (S2) data generation mechanism that enables the unre- stricted creation of high-quality time series data, along with corresponding symbolic representations. Leveraging this large-scale scale series-symbol synthetic dataset, we propose SymTime, a pre-trained foundation model that integrates both time series representations and symbolic semantic information through masked modeling and contrastive learning. Our pre-trained model demonstrates exceptional performance across five major TSA tasks, highlighting the effectiveness of both our data generation strategy and pre-training methodology. Looking ahead, we aim scale up our approach by training larger models synthetic datasets, further boosting performance on downstream tasks.
10

---

Acknowledgement
This work was supported part by Natural Science Basic Research Program of Shaanxi under Grant 2025JC-QYCX-06U in part by the National Natural Science Foundation of China under Grant 62206205, 62201419, 62472345, and 62471371, in part by the Young Talent Fund of Association for Science and Technology in Shaanxi, China under Grant 202301 29, in part by the Guangdong High-level Innovation Research Institution Project under Grant 2021 B0909050008, in part by the Guangzhou Key Research and Development Program under Grant 202206030003, and in part by the Fundamental Research Funds for the Central Universities under Grant QTZX25111. This research was supported by Huawei's Al Hundred Schools Program and carried out using the Huawei Ascend Al technology stack.
References
[1] Shiyu Wang. Jiawei LI, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Ju Shengtong. Zhixuan Chu, and Ming Jin. Timemixer++: A general time series pattern machine for universal predictive analysis. In The Thirleenth International Conference on Learning Representations, 2025.
[2] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[3] Oluwatoyin Ajoke Farayola, Oluwabukunmi Latifat Olorunfemi, and Philip Olaseni Shoetan. Data privacy and security in it: a review techniques and challenges. Computer Science & IT Research Journal, 5(3):606-615, 2024.
[4] Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang, Contrast everything: A hierarchical contrastive framework for medical time-series. In shirth-seventh Conference on Neural Information Processing Systems, 2023.
[5] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 23127):2211187812, 2024.
[6] Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, and Shirui Pan. Towards neural scaling laws for time series foundation models. In The Thirteenth International Conference Learning Representations, 2025.
[7] Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. In The Thirteenth International Conference on Learning Representations, 2025.
[8] Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young, editors, Dynamical Systems and Turbulence, Warwick 1980, pages 366-381, Berlin, Heidelberg, 1981. Springer Berlin Heidelberg.
[9] Liangyue Cao. Practical method for determining the minimum embedding dimension of a scalar time series. Physica D: Nonlinear Phenomena, 110(1):43- 50, 1997.
[10] Leo Torres, Ann S Blevins, Danielle Bassett, and Tina Rlid The why, how, and when of representations for complex systems. SIAM Review, 63(3):435-485, 2021.
[11] Yuanyuan Li, Kai Wu, Jing Liu. Discover governing differential equations from evolving systems. Physical Review Research, 5:023126, May 2023.
[12] Kazem Meidani, Parshin Shojaee, Chandan Reddy, and Amir Barati Farimani. SNIP: Bridging mathemat- ical symbolic and numeric realms with unified pre-training. In NeurIPS 2023 AI for Science Workshop. 2023.
[13] Bai lin Hao. Symbolic dynamics and characterization of complexity. Physica D: Nonlinearr Phenomena, 51(1):161-176, 1991.
[14] Pierre-Alexander Kamienny, Stéphane d' Ascoli, Guillaume Lample, and Francois Charton. End-to- end symbolic regression with transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
11

---

[15] Charles R Doering. Modeling complex systems: Stochastic processes, stochastic differential equations. and fokker-planck equations. In 1990 Lectures in Complex Systems, pages 3-52. CRC Press, 2018.
[16] Dirk Aeyels. Generic observability of differentiable systems. SIAM Journal on Control and Optimization, 19(5):595 603, 1981.
[17] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang. Yuan-Fang Li. Shirui Pan, and Qingsong Wen. Time-LMM: Time series forecasting by repro- gramming large language models. In The Twelfth International Conference on Learning Representations, 2024.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI., 2021.
[19] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Loc, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[20] Wenhui Wang. Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image a foreign language: Beit pre- training for vision and vision-languae langugge tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[21] Kangning Cui, Rongkun Zhu, Manqi Wang, Wei Tang, Gregory D. Larsen, Victor P. Pauca, Sarra Alqahtani, Fan Yang. David Segurado, David A. Lutz, Jean-Michel Morel, and Miles R. Silman. Detection and geographic localization of natural objects in the wild: A case study palms. In James Kwok, editor, Proceedings of the Thirty-Fous Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 9601 0601-9609, International Joint Conferences on Artificial Intelligence Organization, 2025. Al and Social Good.
[22] Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shehur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang. Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Schnede,, and Bernie Wang. Chronos: Learning the language time series. Transactions on Machine Learning Research, 2024. Expert Certification.
[23] Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li, Hao Wang, Aristeidis Sotiras, Yalin Wang. and Abolfazl Razi. TimeMIL: Advancing multivariate time series classification via time-aware multiple instance learning. In Forty-first International Conference Machine Learning, 2024.
[24] Sabeen Ahmed, Ian E Nielsen, Aakash Tripathi, Shamoon Siddiqui, Ravi P Ramachandran, and Ghulam Rasool. Transformers in time-series analysis: A tutorial. Circuits, Systems, and Signal Processing, 42(11):7433 74666666336666...- 2023.
[25] Yuxuan Liang, Haomin Wen, Yuqi Nic, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6555-6566 2024.
[26] Shubao Zhao, Ming Jin, Zhaoxiang Hou, Chengyi Yang. Zengxiang Li, Qingsong Wen, and Yi Wang- Himtm: Hierarchical multi-scale masked time series modeling with self-distillatio for long-term fore- casting. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 3352-3362, 2024.
[27] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long, SimMTM: A simple pre-training framework for masked time-series modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[28] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. Forty-first International Conference on Machine Learning, 2024.
[29] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In Forty-first International Conference on Machine Learning, 2024.
[30] Azul Garza, Cristian Challu, and Max Mergenthaler-Cansee Timegpt-1,2022 2024.
12

---

[31] Matthew B. Kennel, Reggie Brown, and Henry D. I. Abarbanel. Determining embedding dimension for phase-space reconstruction using geometrical construction. Phys. Rev. A, 45:3400-3441, Mar 1992.
[32] Khaled Sayed, Mahmoud Kamel, Mohammed Alhaddad, Hussein M. Malibary, and Yasser M. Kadah. Characterization of phase space trajectories for brain-computer interface. Biomedical Signal Processing and Control, 38:55 66, 2017.
[33] N.H. Packard, J. Crutchfield, J. D. Farmer, and S. Shaw. Geometry from time series. Phys. Rev. Lett. 45:712 716, Sep 1980.
[34] Andrew M. Fraser and Harry L. Swinney. Independent coordinates for strange attractors from mutual information. Phys. Rev. A, 33:1134-1140, Feb 1986.
[35] Sara P. Garcia and Jonas S. Almeida. Nearest neighbor embedding with different time delays. Phys. Rev. 71:037204, Mar 2005.
[36] Tao Wu, Xiangyun Gao, Feng An, Xiaotian Sun, Haizhong An, Zhen Su, Shraddha Gupta, Jianxi Gao, and Jürgen Kurths. Predicting multiple observations complex systems through low-dimensional embeddings. Nature Communccations 15(1):2242, 2024.
[37] Zeyu Li, Wang Han, Yue Zhang, Qingfei Fu, Jingxuan Li, Lizi Qin, Ruoyu Dong, Hao Sun, Yue Deng. and Lijun Yang. Learning spatiotemporal dynamics with a pretrained generative model. Nature Machine Intelligence, 6(12):1566-153 2024.
[38] Cosma Rohilla Shalizi. Methods and Techniques of Complex Systems Science: An Overview, pages 33 33-114, 114. Springer US, Boston, MA, 2006.
[39] Krzysztof Barański, Yonatan Gutman, and Adam Spiewak. A probabilistic takens theorem. Nonlinearity, 33(9):4940, jul 2020.
[40] N. V. Kuznetsov, T. N. Mokaev, O. A. Kuznetsova, et al. The lorenz system: hidden boundary of practical stability and the іуарunov dimension. Nonlinear Dynamics, 102:713-755 1322113--222223 2020.
[41] George Sugihara and Robert M. May. Nonlinear forecasting way of distinguishing chaos from measurement error in time series. Nature, 344:734-741, 1990.
[42] B. R. R. Boaretto, R. C. Budzinski. K. L. Rossi, et al. Discriminating chaotic and stochastic time series using permutation entropy and artificial neural networks. Scientific Reports, 11:15789, 2021.
[43] E. Tan, S. D. Algar, D. Corréa, et al. Network representations of attractors for change point detection. Communications Physics, 6:340, 2023.
[44] B.-W. Shen. Nonlinear feedback in a six sixdiimesioonal lorenz model: impact of an additional heating term. Nonlinear Processes in Geophysics, 22(6):749-764, 2015.
[45] Marco Virgolin, Tanja Alderliesten, Cees Witteveen, and Peter AN Bosman. Improving model-based genetic programming for symbolic regression of small expressions. Evolutionary computation, 29(2)2111 237, 2021.
[46] F. O. de Franca and G. S. I. Aldeia. Interaction-traasfoormaaii evolutionary algorithm for symbolic regression. Evolutionary Computation, 29(3):367 390, 2021.
[47] Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Venkat Naidu, and Colin White. ForecastPFN: Synthetically--raann zero-shot forecasting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[48] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations. 2020.
[49] Uri Alon, Frank Xu, Junxian He. Sudipta Sengupta, Dan Roth, and Graham Neubig. Neuro-symbolic language modeling with automaton-augmeen retrieval. In /CML 2022 Workshop on Knowledge Retrieval and Language Models, 2022.
[50] Robert Lund and. Time series analysis and its applications: With r examples. Journal of the American Statistical Association, 1024 2007.
[51] Pasapitch Chujai, Nittaya Kerdprasop, and Kittisak Kerdprasop. Time series analysis of household electric consumption with arima and arma models. In Proceedings of the international multiconference of engineers and computer scientists, volume 1, pages 295 300-000,22000000000000000 2013.
13

---

[52] Paul Newbold. Arima model building and the time series analysis approach to forecasting. Journal of forecasting, 2(1):23-35, 1983.
[53] Chen Xu, Hanyang Jiang, and Yao Xie. Conformal prediction for multi-dimennional time series by ellipsoidal sets, In Forty-first International Conference on Machine Learning, 2024.
[54] Yugi Nie. Nam H Nguyen, Phanwadee Sinthong. and Jayant Kalagnanam. A time series is worth 64 words: Long-term term forecasting with transformers. In The Eleventh International Conference Learning Representations, 2023.
[55] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
[56] Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. SparseTSF: Modeling long- term time series forecasting with * *1k* parameters. In Forty-first International Conference on Machine Learning, 2024.
[57] Wenzhen Yue, Yong I. Liu, Haoxuan Li, Hao Wang, Xianghua Ying, Ruohao Guo, Bowei Xing, and Ji Shi. Olinear: A lincar model for time series forecasting in orthogonally transformed domain, 2025.
[58] Alexey Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[59] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[60] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
[61] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2022.
[62] Yuxuan Wang. Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yun Zun-Zhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[63] Qiang Wu, Gechang Yao, Zhixi Feng, and Shuyuan Yang. Peri-midformer: Periodic pyramid transformer for time series analysis. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[64] Wenzhen Yue, Xianghua Ying, Ruohao Guo, DongDong Chen, Ji Shi, Bowei Xing, Yuqing Zhu, and Taiyan Chen. Sub-adjacent transformer: Improving time series anomaly detection with reconstruction error from sub-adjacent neighborhoods, 2024.
[65] Graham Elliott, Thomas Rothenberg, and James H Stock. Efficient tests for an autoregressive unit root, 1992.
[66] Georg Goerg- Forecastable component analysis. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pages 64-72, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[67] Robert B Cleveland, William S Cleveland, Jean E McRae, Irma Terpenning, et al. Stl: A seasonal-trend decomposition. J. off. Stat, 6(1):3-73, 1990.
[68] Sylvia Esterby. Review of methods for the detection and estimation of trends with emphasis on water quality applications. Hydrological processes, 10(2):127 149, 1996.
[69] Yinhe Cao, Wen-wen Tung. J. B. Gao, V. A. Protopopescu, and L. M. Hively. Detecting dynamical changes in time series using the permutation entropy. Physical Review E, 70(4):04217, 2004.
14

---

[70] Fangfang Zhou, Minghui Chen. Zeyu Wang. Feng Luo, Xiaobo Luo. Wei Huang, Yi Chen, and Ying Zhao. A radviz-based visualization for understanding fuzzy clustering results. In Proceedings of the 10th International Symposium on Visual Information Communication and Interaction, VINCI 17, page 9-15, New York, NY, USA, 2017. Association for Computing Machinery.
[71] Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecastnng archive. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarky Truck (Round 2). 2021.
[72] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
[73] Spyros Makridakis. M4 dataset, 2018.
[74] Yong Liu, Tengge Hu, Haoran Zhang. Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. In The Twelfth International Conference on Learning Representations. 2024.
[75] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[76] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for time-series forecasting, 2023.
[77] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning. pages 27268 27286. PMLR, 2022.
[78] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Non-stationary transformers: Exploring the stationarity in time series forecasting. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[79] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizzig cross- doss-dimension dependency for mul- tivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
[80] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11111 2021.
[81] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy, 2022.
[82] Zijie Pan. Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. 2SPP I.M:: Semantic space informed prompt learning with LLM for time series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[83] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, and Xiaoli Li. Tslanet: Rethinking transformers for time series representation learning. In ICML, 2024.
[84] Angus Dempster, François Petitjean, and Geoffrey Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernlss. Data Mining and Knowledge Discovery, 34(5) -4454 2020.
[85] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 346))1936-196,, 2020.
[86] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. MICN: Multi- scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
[87] Ailing Zeng. Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence. AAAI (MA233/AAAT233AATT AAAI Press, 2023.
15

---

[88] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures, 2022.
[89] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma. James Y. Zhang, and JUN ZHOU. Timemixer: Decomposable multiscale mixing for time series forecasting. The Twelfth International Conference on Learning Representations, 2024.
[90] Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: Modeling time series with $10KS parameters. In The Twelfth International Conference on Learning Representations, 2024.
[91] Kun Yi, Jingru Fei, Qi Zhang, Hui He, Shufeng Hao, Defu Lian, and Wei Fan. Filternet: Harnessing free quency filters for time series forecasting, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[92] Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. UniTS: A unified multi-task time series model. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[93] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MO- MENT: A family of open time-series foundation models. In Forty-first International Conference on Machine Learning, 2024.
[94] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[95] Hassan Ismail Fawaz. Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917-
963, 2019.
[96] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[97] Yong Liu, Guo Qin, Xiangdong Huang, Jiammin Wang. and Mingsheng Long, Autotimes: Autoregressive time series forecasters via large language models. In The Thirty-eighth Annual Conference on Neurul Information Processing Systems, 2024.
[98] Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICMI.20 2020.
[99] Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How t-sne effectively. Distill, 2016.
[100] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer-xl: Long contect transformers for unified time series forccasting. In The Thirteenth International Conference Learning Representations, 2025.
[101] Wetterstation. Weather. httpai/etter/e/m...o.eeeee...eee met.:j//amm...ddeeteeeeer.
[102] UCL.
Electricity.
http:///archiv..ccsu.... http:///......ccdddd/d/aaaattata
[103] PcMS. Traffic. http::/pems dot. ca. /ov/
[104] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxian Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, page 95-104, New York. NY, USA, 2018. Association for Computing Machinery.
[105] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018, 2018
[106] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. KDD '19, page 2828-2837, New York, NY, USA, 2019. Association for Computing Machinery.
[107] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detect- ing spacecraft anomalies using Istms and nonparametric dynamic thresholding. KDD '18 page 387-395, New York, NY, USA, 2018. Association for Computing Machinery.
16

---

[108] Aditya P. Mathur and Nils Ole Tippenhauer. Swat: a water treatment testbed for research and training on ies security. In 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater), pages 31-36, 2016.
[109] Ahmed Abdulaal, Zhuanghua I Liu, and Tomer Lancewicki. Practical approach to asynchronous mul- tivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD "21, page 2485-2494, New York, NY, USA, 2021. Association for Computing Machinery.
[110] Javier Selva, Anders S Johansen, Sergio Escalera, Kamal Nasrollahi, Thomas B Moeslund. and Albert Clapés. Video transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45(11):12922-1223 2023.
[111] Chengsen Wang, Qi Qi. Jingyu Wang, Haifeng Sun, Zirui Zhuang. Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data, 2024.
[112] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVT conference on computer vision and puttern recognition, pages 16000- 16000, 2022.
[113] Kai Wu, Yujian Betterest Li, Jian Lou, Xiaoyu Zhang, Handing Wang, and Jing Liu. Rapid plug-in defenders. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[114] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed- modal language models. In Andreas Krause. Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Searlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 265-279. PMIR, 23-29 Jul 2023.
[115] Yi Xiao, LEI BAI, Wei Xue, Hao Chen, Kun Chen, kang chen, Tao Han, and Wanli Ouyang. Towards a self-contained data-driven global weather forecasting framework. In Forty-first International Conference on Machine Learning, 2024.
[116] Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, and Rossella Arcucci. Zero-shot ECG classification with multimodal learning and test-time clinical knowledge enhancement. In Forty-first International Conference Machine Learning, 2024.
[117] Carmen Martin Turrero, Maxence Bouvier, Manuel Breitenstein, Pietro Zanuttigh, and Vincent Parret. ALERT-transfommer: Bridging asynchronous and synchronous machine learning for real-time event-based spatio-temporal data. In Forty-first International Conference on Machine Learning, 2024.
[118] Andrea Cini, Danilo Mandic, and Cesare Alippi. Graph- time series clustering for end-to-ena hierarchical forecasting. In Forty-first International Conference on Machine Learning, 2024.
[119] Jiaxiang Dong, Haixu Wu, Yuxuan Wang, Yun-Zhong Qiu, Li Zhang, Jianmin Wang, and Mingsheng Long. Timesiam: A pre-training framework for siamese time-series modeling. In Forty-first International Conference Machine Learning, 2024.
[120] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[121] Romain Ilbert, Ambroise Odonnat, Vasili Fenfanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and levgen Redko. SAMformer: Unlocking the potential of transformers in time series forecasting with sharpness-awan minimization and channel-wise attention. In Forty-first International Conference on Muchine Learning, 2024.
[122] Sumanth Varambally, Yian Ma, and Rose Yu. Discovering mixtures of structural causal models from time series data. Forty-first International Conference on Machine Learning, 2024.
[123] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transeormer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 21 214-224,, 2021.
[124] Ronghao Lin and Haifeng Hu. Multi-task momentum distillation for multimodal sentiment analysis. IEEE Transactions on Affective Computing, 15(2):549-565, 2024.
[125] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
17

---

[126] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[127] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L uuasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[128] Lu Han, Han-Jia Yc, and De-Chuan Zhan. SIN: Selective and interpretable normalization for long-term time series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[129] junxin lu and Shiliang Sun. CaudiTS: Causal disentangled domain adaptation of multivariate time series. In Forty-first International Conference on Machine Learning, 2024.
[130] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. Pyraformer: cow-complexity pyramidal attention for oangrrange time series modeling and forecasting. In International Conference on Learning Representations, 2022.
[131] Mingtao Feng, Chenbo Yan, Zijic Wu, Weisheng Dong, Yaonan Wang. and Ajmal Mian. Hyperrectangle embedding for debiased 3d scene graph prediction from rgb sequences. IEEE Transactions on Pattern Anayyss and Machine Intelligence, 47(8):641000 10--6446, 2025.
[132] Kendong Liu, Mingtao Feng. Wei Zhao, Jingtao Sun, Weisheng Dong, Yaonan Wang. and Ajmal Mian. Pixel-level noise mining for weakly supervised salient object detection, IEFE Transactions on Neural Networks and Learning Systems, 36( (1)): 1815)):181518888) 2025.
[133] Defu Cao. Wen Ye, and Yan Liu. Timedit: purppse diffusion transformers for time series foundation model. In ICML 2024 Workshop Foundation Models in the Wild, 2024.
[134] Wenzhen Yue, Yong Liu, Xianghua Ying. Bowei Xing, Ruohao Guo, and Ji Shi. Freeforme:: Frequency enhanced transformer for multivariate time series forecasting, 2025.
[135] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16) 2020.
[136] Deaglan J. Bartlett, Harry Desmond, and Pedro G. Ferreira. Exhaustive symbolic regression. IEFF Transactions on Evolutionary Computation, 28(4):950 964, 2024.
[137] Samuel Müller, Noah Hollmann. Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transform- ers can do bayesian inference. In International Conference on Learning Representations, 2022.
[138] Thomas Nagler. Statistical foundations prior-data fitted networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 266666 PMLR, 23-29 Jul 2023.
[139] Jinghao Bian, Mingtao Feng. Weisheng Dong. Zijie Wu, Jianqiao Luo. Fangfang Wu, Yaonan Wang, and Guangming Shi. Locally aware visual state space for small defect segmentation in complex component images. IEEF Transactions on Industrial Informatics, 216555965-557,, 2025.
[140] Ege Onur Taga, Muhammed Emrullah lidiz, and Samet Oymak. TimePFN: Effective multivariate time series forecasting with synthetic data. In NeurlPS Workshop on Time Series in the Age of Large Models, 2024.
[141] Zhe L., Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping, 2023.
[142] Tian Zhou, Ziqing Ma, xue wang, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. FiLM: Frequency improved legendre memory model for long-term time series forecasting. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[143] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. International Conference on Learning Representations, 2020.
[144] Donald J. Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, AAAIWS'94, page 359-370. AAAI Press, 1994.
18

---

[145] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, page 785-794, New York, NY, USA, 2016. Association for Computing Machinery.
[146] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735- 1780, 1997.
[147] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
[148] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning multivariate time series. H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[149] Fanxing Liu, Cheng Zeng, Le Zhang, Yingjie Zhou, Qing Mu, Yanru Zhang, Ling Zhang, and Ce Zhu. Fedtadbench: Federated seness anomaly detection benchmark. In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf Dependability in Sensor, Cloud & Big Data Systems & Application pages 303-310, 2022.
[150] Ronghui Xu, Miao, Senzhang Wang. Philip S. Yu, and Jianxin Wang. Pefad: parameter-efficiee federated framework for time series anomaly detection. Proceedings of 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 3621-3632, New York, NY, USA, 2024. Association for Computing Machinery.
[151] Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, KDD '21, page 3220-3230, New York, NY, USA, 2021. Association for Computing Machinery.
19

---

A Main Results and Conclusions of the Five Tasks in Time Series Analysis
A.1 Long-term Forecasting
Setup.
Time series forecasting, which analyzes historical data patterns to predict future trends, crucial for financial market analysis, inventory management, energy demand and other fields 100]. We adopt 8 real-world benchmark datasets for long-term forecasting, including ETTm1, ETTm2, ETTh1, ETTh2 [80], Weather [TOT), ECL [T02], Traffic [T03] and Exchange [T04]. The forecasting lengths are set to {96, 192, 336, 720}. To ensure the fairness of the experiment,
set up three different look-back windows (96, 336, 512] for the experiment, except Moirai and Timer are 672. For different models, we try our best to ensure that they are tested according to their original experimental configuration. For foundation models that can perform zero-shot forecasting and have been pre-trained, such as Moirai (28), Timer [29], Moment [93] and Chronos [22), we first load the pre-traine pre-trained parameters of the model, and then perform supervised fine-tuning on it in the same way.
Table 3: Long-tem term forecasting task with 96 look-back windows. The results are averaged from four different series length (96, 192, 336, 720). (8 means former.) See Appendix F. for full results, Red: best, Blue: second best. The standard deviation is within 1 1%.
SynTime (Ours)
Peri-mid
Moirai
Timer 20
Time-LLM
TSLANct
ISIP-LLM
GPT4TS
TimeMixer
Model
65
28
17
83
82
2
89
Metries
MSE MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE MAE
ETTml
0.371 0.387 0.387 0.409 0,410/0.398 0.417
0.402 0.394
00.77
0.397 0,404
0.395 00,322 0.395 0.397
FTTm2
0.274 0.321 0.290 0.328 0.29 0.29% 0.348 0.405 0.408 0.275 0.324
0.283 0.327 0.266
0.264 0.264
0.325
0.328 0.279 0.325
ETThl
0.430
0.455
0.441
0.434
0.438 0.445
0.448 0.441
0.456 0.4444
0.453 0.44|
ETTh2
0.365 0.400 0.400 0.406 0.411 0.442 0.441 0.369 0.407 0.355
0.362
0.359 0 403 0.388 0 0.408
Weather
00.247 0,2662 0.283 0.265 0.299 0.329 0.358 0.247 0.269 0.259 0.352
0.243
0.274 0,265 0.285 0,253 0,280
ECL
0.187 0. 0.267 0.167 0.252
0.177 0.267 080 0.269 0.199 0.283 0.1 0.283 0,206 0.291 0. 0.274
Traffic
0.457 0.291 0.458
0.424 0.289
0.436 0.284 0.471 0.334
0.463
0.417 0.306 0.491 0.329 0.306
Exchange
0.359 0.401 0.388 0.417 0.417 0.373
0.417
0.382 0.425 0.376 0.414 0.368
0.414 0.414 0.472 0.478 0.478 0.370
0.41 0.711 0.403 0.423
Average 00.336 0.355 0.358 0.361
|0.3 (..320..33333....44...333
0.344 0.364 0.344 0.3 0.366/0.345 0.359/0.355 0.357
Table 4: Long-term forecasting task with 336 look-back windows. The results are averaged from four different series length 196, 192, 336, 720). See Appendix F.2 for full results. Red: best, Blue: second best. The standard deviation within 1%
SymTime (Ours)
PatchTST
TimeMixer
TimesNet Autoformer
DLinear
iTransformer
TimeXer
FEDformer
Model
54
89
75
87
Metries
MSF MAF
MSE MAF
MSE
MAF
MSF
MAF
MSF MAF
MSE
MAF
MSE
MAE
MSE
MAR
MSE MAR
ETTml
0..50 0.382
0.382 ( 0.6688 0.539
0.379 0.395
00332 0.395 0.441 0.452
ETTm2
0.256 0..3160.258 0.265 0.318 0.282 0.334 0.400 0.291 0.252 0.329
0.262 0.317 0.325 0.377
ETThl
0.413
0.432 0.419 0.430 0.437 0.485 0.480 0.580
0.425 0.450 0.457 0.493 0.483 0.450 0.472
ETTh2
0.341 0.331 0.331 0.379 0. 0,396 0.425 0.409 0 0.63
0.490
0.390 0.416 0.375 0,410 0.430 0,467
Weather
0.238 0.258 0. 0.292 0.235 0.273 0.250 O. 286 0.441
0.245 0 0.298 0.238 0.272 0.287 0.290 0.313 0.363
0.298
FCL.
0.164 0.258 665 0. 0. 0.
0. 260 O. 197 O 297 0.236
0. 0.269 0.163 0.257
0.346
7. 0.267 0.214 0.327
Traffie
0.391 0.267 0.396 0. 0.268 1 0.271 0.615 0.331 0.676
0.413
0.465
0.401 0.283
0.452 0.281 00610 0.376
Exchange
0.367 0.385 0.415 0.438 0.548 0.532 1.55 0.807
0.448
0.462 0.399 0.427 0.409 0.500 0.376 0.427
Average
(0.315 0.341 |0320 00336
0. 0.352 001 0.390
0.361
0.347
0.355
0.333 0.368 0.395 0.408
Results. Tables 3 Table 4 and Table respectively show the results of SymT imme with look-back windows of length (96, 336, 512). It can be seen that SymTime demonstrates excellent performance in time series long- terg- forecasting tasks. Our model surpasses Peri-midFormer, GPT4TS and TimesNet, which are foundation models for the five major tasks, as well as Moirai and Timer, two general forecasting models. Compared with Time-LLM and 19IP-LLM, which use pre-trained large language models as backbone, SymT SymTime imme achieves better results with more lightweight Transformer encoder.
20

---

Table 5: Long-term forecasting task with 512 look-back windows, The results averaged from four different series length (96, 192, 336, 720). See Appendix F33 for full results. Red: best, Blue: second best. The standard deviation is within 1%.
SymTime Ours
PatchTST
TixeMixer
TimesNet
Autoforner
DLinear
iTransformer
TimeXer
FITS
Models
SG
89
72
251
87
74
62
9D
Metrics
MSE MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE
MAE
MSE MAE
ETTml
0.356
0.380 0.382
0.371
0.371 0.392
0.425 0.556 0.518
0.358 0.518
0.380 0.397
0.378 0.401 0.374 0,384
ETTm2
0.265 0.320 0.256 0.317 0.263 0.263 0.323 0.294 0. 0.344 0.371 0.416 0.275
0.215 0. 0.273 0.331
0.274 0.329 0.254 0.313
ETThl
0..414
0.432 0.413 0.434 0.429 0.429
0.444 0.481 0.486 0.627 0.579 0.418 0.438 0.446 0.460
0.438
0.475 0.479 0.418 0.441
ETTh2
00.365 0.405 0.357
0.373 0.4 0.410 0.397 0.432 0.687 0 609 0.6699904999 4.999 0.478 0.388 0.417
0.478
0.354 0.400 0.363 0. 0.408
weather
00.234 0.273 0.2245 0.273 0.284 0.231 0.271 0.251 08288 0 489 0 48686 0.241 0.2 0.292 0.249 0.280
0.241
0.282 0.284 0.244 0.281
ECL
0.1663 0.267 169 0 0.269 0.177 0.274 201 0.3 0.353 0.393 0.39 167 0.267 0.162 0.257 0.171 0,270 0.169 0,265
Traffic
0.395 0.268 0.399 0.272 0.410
0.410 0.270 0.624 3.334 0.705 0.435 0.433 0.305 0,383 0.273 0.466 0.287 0.420 0,287
Exchange
0..84 0.412 0.398 0.423 0.97 0.7 12 0.944 0. 0.427
0.467
0.514 0.506 0.393 0.439
Average 00.322 0.345 0.345|0.324 0.334 0. 0.104499900.0000.000444440000000
0.525
0.374
0.360 00.64 0.369 0.329 0.352
A.2 Short-term Forecasting
Setup. We adopt M4 benchmark 73 for short-term forecasting, which contains the yearly, quarterly and monthly collected univariate marketing data. Then, use symmetric mean absolute error (SMAPE), mean absolute scaled error (MASE) and overall weighted average (OWA) to measure the forecasting performance, which are calculated as detailed in Appendix C.3
Results. Table indicates that SymTime after pre-training, surpasses TimeMixer, Peri-midFormer and TimesNet on the short-term forecasting tasks in terms of SMAPE, MASE and OWA metries, achieving tfae-offthhe-a performance. Specifically, SymTime performs well on Yearly, Quarterly and Monthly datasets, demonstrating its capability capture not only the trends of annual variations but also the cyclic characteristics of seasonal and monthly encoding.
Table 6: Short-term forecasting task on M4. The prediction lenghs are {6, 48) and results weighted averaged from several datasets under different sample intervals. ( mcans former, TMixer is TimeMixer, S-LLM is SPIP-LLM, T-LLM is Time-LLM), See Appendix F.4 for full results. Red: best, Blue: second best. The standard deviation is within 1 16.
SymTime Peri-mid S-LLM T-LLMM GPT4TS TMixer PatchTST iTrans* TimesNet DLinn LighTTS FRD*
In
Models
(Ours)
6
17
2
54
74
22
87
77
X0
SMAPE
11.785
11.897
12.514
12.584
12.367
11.885
12.866
13.233
11.888
12,500
11.962
12.605
15.018
MASE
1.584
1.607
1.726
1.763
1.767
1.598
1.734
1.850
1.607
1.678
1.609
1.677
2.096
OWA
0.849
0.859
0.913
0.915
0.918
0.856
0.928
0.972
0.858
0.899
0.862
0.903
1.102
A.3
Classification
Setup. Time series classification crucial for the identification and diagnosis of patterns in complex systems and plays a significant role in various fields such as financial analysis, medical diagnosis and industrial monitoring (95). Using the experimental setup from TimesNet [72]. we test SymTime e's discriminative ability on 10 UEA multivariate time series classification datasets [105] including categories such as Industry, Face Detection, ECG, Voice and Transportation.
Results. As shown in Figure 10 SymTime achieves an average accuracy of 74.9%, surpassing all basclines, indicating that SymTime is competitive in classification tasks.
A.4 Imputation
Setup. Sensors monitoring complex systems in the real world may experience distortions or mal- functions, leading to partial missing data in the collected time series. Therefore, time series imputation is crucial for the recovery of complete datasets. We verify SymTime's 's imputation capabilities on 6
21

---

100
Average Accurapy (%)
90
RNN--Basd
Classical Methods
Transformer-Based
CNN-Based
MLP-Based Ours
80
86.0
72.5
70.9
727
70.3
73.6
74.4
71.1
70.7
71.0
71.1
71.1
68.7
72.7
74,4
67.5
70.4
74.9
48.6
40
DTW
XGBooot
Rocket
LSTM
LSTNal
LESt
InTime
TCN
TimeaNol
TSLANel
Audor
FED
ETS
iTanee
PatchTST
GPT 4TS
UnIS
Pad-mair DLinerr
LighT'
SymTime
Figure 10: Comparison of the average accuracy of SymTime and other baselines on 10 UEA datasets.
See Appendix F5 for full results.
Weather [TO1] and ECL [02]] To the model's imputation ability under varying degrees of missing data, we add random masks at proportions of (12.5%, 25%, 37.5% 50%] ] in point level on time series of length 96. Since SymTime was pre-trained by randomly masking patches level for series reconstruction and masks are added randomly in point level the imputation task. Considering the differences between these masking approaches and the potential disruption of the series's 's original trends and periodic features at higher mask rates, we adopt per-interpolation for masked series from [63] Analysis and ablation experiments regarding this method are presented in Appendix C.9 The results demonstrate that per-interpolation can be used as a codel-indeeennent feature engineering to improve the performance of downstream tasks.
datasets: ETTml, ETTm2, ETTh2 (80).
Table 7: Imputation task, where we randomly mask (12.5%, 25%, 37.5%, 50% | time points of length-96 time series. The reuslts averaged from 4 different mask ratios. (* means former.) See Appendix for full results. Red: best, Blue: second best.
SynTimm (Ours)
GPT4TS
TimesNet
Peri-mid
Moment
iTrans
PatchTST
DI Linearr
LighTTS Т
Model
2
63
93
74
64
Metric
MSE MAE
MSE MAE
MSE
MSE MAE MAE MSE MAE
MSE MAE
MSE MAE
MAE
MSE MAE
MSE MAE
ETTml
0.036 0.116000028 0.027 0.107
0.036
O. 0.16 0.168 0.1
0.185
0.049 0.143
0.090
0.204
0.068 0.182
ETTm2
0.026 0.088 0.022 0.088 0.022 0.089
0.026 0.087 0.031
1.108
0.191
0.030 0.101
0.102 0.212 0.068 0.176
ETTh1
0.095 0.201 0.093
0.289
0.199 0.091
0.139
0.234 0.269
0.126
0.231
0.169
0.283 0.159 0.278
ETTh2
0.058 0.148 0.052 0.052
0.147
0.057 0.057
0.147 0.061
0.139 0.254
0.066 0.164
0.163
0.273 0.143 0,258
ECL
0.054 0.151 0.093 0.212 0.211 0.063
1.69 0.211 0.099
0.192
0.128
0.256 0.8 0.256
0.238
Weather
0.028 0.032 0.000 0.056 0.0 0.035 0.00550.00000 0.114 0.057
0.053 0. 0.1 0.0447 0,106
Average 0.049
0.206
shows that SymTime outperforms Peri-midFormer. GPT4TS and TimesNet in over- all performance establishing SymTime as the latest state-of-the-un approach. Although SymTime's performance on the ETT series of datasets is not as strong GPT4TS, it achieves more significant effects on datasets with a higher number of channels, such as ECL and Weather.
Results.
Table
A.5
Anomaly Detection
Setup. Time series anomaly detection crucial for rapidly identifying anomalies critical areas, aiding in risk prevention and decision optimization. Due to the difficulty in annotating time series anomalies, we focus primarily on unsupervised anomaly detection. We conduct experiments on widely used anomaly detection datasets: SMD and SMAP [066,, MSL [107], SWaT [108), PSM [19], encompassing service monitoring, space & earth exploration, and water treatment applications. We adopt the same data preprocessing method as the Anomaly Transformer [8I], dividing the data into non-overlapping segments of length 100 for reconstruction. Specifically, normal data is used for model training and we employ simple reconstruction loss to help the model learn the distribution of normal data [63]. In subsequent testing phases, reconstructed outputs exceeding a specified threshold considered anomalies.
22

---

Symbols Table 8: Anomaly detection task, where calculate the F1-score (as %) for each dataset. (* means former, TNet is TimesNet, PTST is PatchTST.. A higher value of Fl-score indicates a better performance. See Appendix F.7 for full results. Red: best, Blue: second best.
immTime UniTS Peri-mid GPT4TS
TNet PTST LighrTS
DI Dlinear iTrass* Anomaly Stationary Cross
In
Auto
Model
(Ours)
92
63
2
72
54
88
87
74
81
78
79
80
75
SMD
85,66
83,69
84,08
84.49
84.37
84.62
82.53
79.76
80.19
85,49
82.97
77.22
77.88
71.17
MSL
81.77
81.16
80,68
82.03
81.14 78.70
78.95
81.87
72.47
83,31
76.68
80.59
81.07
82.22
SMAP
73.43
74.00
67.53
68.85
69.05
68.82
69.21
67.30
66.72
71.18
69.02
67.12
73.26 73.97
SWaT
93.61
92.51
91.64
92.60
92.61 85.72
93.33
92.66
92.64
83.10
92.24
90.22
80.35 79.19
PSM
97.10
97.31
96.21
97.09
97.06
96.08
97.15
96.64
94.88
79.40
97.23
92.52
90.43 88.24
Avg. F1
86,31
85.73
84.03
85.01
84.85 82.79
84.23
83.64
81,38
80.50
83.63
81,53 80.60 78.96
Results. The results in Table g show that SymTime surpasses all previous models such as TimesNet and GPT4TS in the time series anomaly detection task and becomes the latest SOTA model. Compared with UniTS [92] pre-trained on real time series data, SymTime is pre-trained on synthetic datasets and achieves better model performance.
B Analysis of Series-Symbol (S²) Dataset and Model Pre-training
Table 9: Some symbols used in data generation and their explanations.
Symbols
Explanation
Explanation
X
sampling series
Y
generated series white noise sequence the output channels number normal distrihution
f()
symbolic expression the input channels number uniform distribution
C
M
N
N
P
prohability of selecting sampling methods the order of the AR process parameters of AR process
A
total number of mixed distributions
p
O, the order of the MA process parameter MA process
Q
Op
Based on the viewpoint of complex dynamic system modeling, this paper proposes a bimodal series- symbol (52) generation mechanism alleviate the problem of shortage of training data in the field of time series analysis. Table the specific symbols we used in constructing symbolic expressions in Section 3.1
This part mainly divided into the following sections:
B. Series-Symbol Data Display: This section shows the symbolic expressions and time series data that can be generated through the 52 data generation mechanism.
B.2 Composition and Usage of the Series-Symbo Dataset: This section introduces the specific composition of the 52 dataset and how to use it when training SymTime.
B.3 Analysis: This section examines and analyzes the basic statistical representa- tion of the 52 data.
B.3
B.4 Analysis of Existing Large-scale Datasets for Time Series Pre-training: This section analyzes the data shortage and distribution imbalance faced by large-scale time series pre-training datasets in TSA.
Dataset Statistical Characterization Coverage Experiments: This section details the specific configuration of the characterization coverage experiments in the Section
B.5
Masked Time Series Modeling and Zero-shot Imputation for Representation Learning:
B.7
Time Complexity Analysis of 5 Data Generation Mechanism: In this section we
analyze in detail the time complexity of the S data generation mechanism.
B.8 Selection of the Unary Operators: In this section we analyze the unary operators
used in S data generation.
B.9 The Limitations of 2 Generation Mechanism.
23

---

B.1 Series-Symbol Data Display
In Figure 11 we show the visualization of the generated series from input channel and output channel to 4 imput channels and 4 output channels. We show two sets of cases for each input and output channel. The symbolic expressions f(-) for the generated series in (a), (c), (e) and (g) in Figure T are shown above.
The symbolic expressions with text format shown follow: Symbolic expression of [I y/1 (-0.795 add ((-0.675 mul ((0.999 add (-6.7 mul x1))))++) add ((-0.79) mul (nv((-5.9)) add (-0.751 mul x))))) sub (9.68 mul sqrt((-7.33 add (0. 756 mul x;)))))>>
Symbolic expression of Figure (c)
y/1 (-3.39 add (((0.55 56 mul mul 21) sub (31.9000 mul x2)))) sub 440.4000 mul x1)) add (0. 71 mul (((7.13 mul x²) sub (-1.68 mul (r) mul (-92 8000 add (0.054 mul (02 mul ((0.327 mul x2) sub (2.3 mul a2)))))))))) mul an)))))
add (58.2 mul x²))) mul ((-19. 7000
1/2 (1.0 add ((68.9 9 mul x2) sub (((80.9 mul (r1 mul (r) mul ((6. 1000 mul x2) sub ((34.2 2 mul sqrt (664.4 add (29. 2000 mul ar)))) add (-5. 24 mul x))))))) add (6.78 78 mul x²)) sub (((0.5730 mul x1) sub ((2.34 mul x2)) sub (-6.72 72 mul xi))) add (0. 966 mul sqrt((76 8000 add (-7.79 mul a;))))))))
Symbolic expression of Figure
V1 = (0.795 add ((0.42 mul 2z)) sub ((4.39 mul xn) add (((O. 1430 mul x2) sub ((-5.28 mul 2z) add (((-0.02 028 mul ((((1.27 mul xz) sub (((0.33) mul x22) sub
((2.99 mul rz) add (-0.932 mul (((0.606 mul 21) sub (0.967 mul xz)) mul
k))))) sub (-0.609 mul 1)))) add (-1.25 mul x))) mul 21)) sub (77.300) mul
xi)) sub (1.93 mul xz)))) sub (16.7 mul az))))))
1/2 = 2-9.2900 add ((0.398 mul ((((-49.7 mul x1) sub ((5.98 93 mul sin sin((6.54
add (-0.045 mul x))))) add ((62. mul inv 138 mul x2) add (29.0 o mul x))))) add (8.75 mul x2)))) add 9500.9500 mul x3) add (-8.1 mul x)))) mul
xz)) add ((-9.74 mul x3) add ((((-0.9 mul xz) sub (4.45 mul sqrt((-0.37) add (-0.151 mul xa))))) add (-54.6 mul xz)) sub (-0.758 mul ((85.3000 add 78 mul xz)))
ys = (-0.975 add ((-54.400 4000 mul sqrt 722 add (-9.33 mul x2))))) sub (1.45 mul ((66.4 add (-9.65 mul x2)))
Symbolic expression of Figure(ll[e]]
y/1 (-7.17 add 537 mul x)))
y/2 = (-0.843 843 add 8000 mul x1))
y/3 (57.30 3000 add (((-0. 449 mul 12) add (-1.32 mul xz)) add ((-0.9400 mul x4) add (0.51 mul )1))))
3/ 2040 add (((-6, 6000 mul inv 88 add (58.1 mul x4)))) sub ((-23.0 mul x44) add ((-91.0 mul 2z) sub 1-93.6000 mul x2)))) sub ((-6. 6000 mul x4) sub (0. 9580 mul ((rg mul 2z) mul ((-0.45 mul 12) sub ((((-9.09 09 mul x4) sub ((8.93 mul sqrt(((-26. mul 24) add (-0.907 mul z1)))) add (-6.2 2 mul r())) sub 078 mul 2))) sub (-16.5 mul +z))))))))
B.2 Composition and Usage of the Series-Symbol Dataset
We set the maximum number of input channels and the maximum number of output channels to and 12 respectively to generate symbolic expressions and series. Each symbolic expression is sampled only once. We generated a total of 40M pairs of series and symbols. The cumulative series length is 50B. The data number of each input channel and output channel in the dataset is shown in Figure
When pre-training SymTime with S dataset, we start by combining the sampled and generated series and then segmenting them into patches using a sliding window [54] I7]. The sliding window's 's kernel size and step size are both set to 16. Due to the requirement for mask time series modeling (MTM) 27 266 there is no overlap between adjacent patches. Given the varying number of input and output channels the data, the series from the maximum input and output channels can be segmented into up to 288 patches (18 x 256/10 (661) [110] For series with fewer than 288 patches, we pad them with zeros
24

---

Sandhr Sores
un
(a) input channel
output channel data example
(b) imput channel output channel data example 2
tarding
(c) input channels 2 output channels data example
(d) input channels output channels data example
Sansing Seles
Oovera Seee
(c) 3 input channels 3 output channels data example
(f) 3 input channels 3 output channels data example 2
Seesing Seea
Genere Saie
(g) input channels 4 output channels data example (h) input channels output channels data example
Figure 11: Visualization series from input channel output channel to 4 input channels output channels.
to align the length. Next, for symbolic expressions in natural language form 19 12
we
set maximum length of 512 characters and perform tokenization. Ultimately, the time series patches and natural language tokens are fed into the time series encoder and the L.MM of the Transformer architecture, respectively.
B.3 Statistics Analysis
Setup. In Section B.2 we provide a detailed introduction to the generation process, composition and usage of the 52 dataset [2
48 In this section. we first conduct random sampling analysis of
the statistical characteristics of the 52 dataset, including stationarity [65 and predictability [66] 29]
This attribute ensures that the statistical characteristics of time series data remain consistent across different time points, which is crucial for building effective predictive models and making reliable statistical inferences. To this end, we employ the Augmented Dickey-Fuller (ADF) 65 test to examine the stationarity the data, thereby determining whether the generated 52 dataset suitable for deep neural networks (DNNs) to learn representations of time series.
Stationarity. Stationarity is one the fundamental properties of time series 50 [1
25

---

Forecastability. The forecastability of time series refers to the ability and accuracy to forecast future values based historical data and statistical models [52 29]. For certain specific time series and complex systems, such stock markets, it is often challenging to predict their subsequent developments. Therefore, is necessary test whether the S dataset is non-chaotic and learnable. Forecastability is calculated by subtracting the entropy of the series' Fourier decomposition adopted from [66] and [29], where a higher forecastability value indicates better predictability. Please note that since the method provided by [66] is only applicable to multivariate time series, merge the input channels and output channels together for calculation.
Test Methods and Results. For the multiple outptt channels presented in the Table 10 randomly selected 1,000 samples to calculate their average ADF statistics, p-values, and Forecasta- bility metrics. The results indicate that the average p-value from the ADF test across all samples greater than 0.05, suggesting that the majority of the generated series the 5 dataset are non- stationary time series, posing challenge in modeling and learning [65]. However, the Forecastability metric, which is greater than 0.3 for all tested samples, indicates that the generated series Y is not produced by a chaotic system and is, overall, predictable.
652294
511887
458189
204062
800000
The number of imput channels
685406
630239
536440
470298
423522
370659
242710
210000
197533
500000
68/9674
803571
529458
467389
412157
363489
321058
285434
254082
224962
1500/113
125001
384851
895289
$19984
440377
$25873
242821
236927
253438
229783
198628
172166
151406
-00000
670634
584879
503698
452068
572847
321908
275644
2335/9
207380
177982
153827
139044
200000
673430
674300
4881 50
416483
353509
301821
254448
218685
184300
16903
134222
113004
The number of output channels
Figure 12: The number of samples in each part of the S dataset.
Table 10: Results of the stationarity and forecastability tests for the S dataset.
inputs
outputs
ADF
value
forecast
inputs
outputs
ADF
value
Forecastability
-12.77
0.0538
0.3155
-11.48
0.0619
0,3375
2
2
-1.899
0,0568
0.3199
2
-1.46
0.0733
0.3218
3
-12.40
0.0544
0.3328
3
-11.43
0.0625
0.3244
-11.66
0.0617
0.3491
-11.53
0.0640
0.3428
5
-11.38
0.0628
0.3140
-12.32
0.0597
0,3284
6
-12.43
0.0625
0.3262
6
-11.52
0.0555
0,3246
6
10
-11.65
0.0619
0.3287
6
12
-11.66
0.0520
0,3310
B.4 Analysis of Existing Large-scale Datasets for Time Series Pre-training
Large-scale datasets are crucial for building foundation models.
Almost deep learning models
today are data-driven, relying on training data [80] 85 112 [13]. Therefore, when constructing pre-traine pre-trained foundation model for time series, a large-scale and comprehensively representative pre-
training dataset is indispensable (28 29 20 18 98]. The scaling laws of neural networks indicate that the learning effectiveness deep neural networks is primarily influenced by three factors: the number of model parameters, the size of the training dataset, and the amount of computational resources [5 6 96 14 Expanding the scale of the pre-training dataset can effectively improve the model's generalization capability and performance, and the performance gains from increasing data volume
26

---

1044
Number of Point in Time-300B (Log)
279.78
The Storage Size Datasets (B)
21.58
1011
10
9.58
3.98
1010
9.28
2.68
2.28
2.38
1.40
1.30
10*
26M
20M
10"
107
414K
TA7360
weswie
daterop
CO6.8M
Autioial
LaiA-viso
Timee S00ld
Nature
Energy
Synthetic
Taaspprt
Well
Sales
Healthcane
Fosrrca
(a) data scarcity
(b) data imbalance
Figure 13:
The scarcity and imbalance time series pre-training dataset (taking the largest open- source time series dataset Time-300B compared to text datasets in natural language processing and video understanding datasets computer vision. (b) sarge-scale time series pre-training datasets face serious distribution imbalance problems.
example 77). (a) time series datasets are data-scarce
Table 11: Time-300B time series dataset from Time-MoF
Energy
Finance
Health
Nature
Sales
Synthetic
Transport
Web
Other
Total
Obs.
15.98B
413.70K
471.04K
279.72B
26.38M 0.008%
9.22B 2.98%
2.13B
1.80B
20.32M
309.09B 100%
%
5.17%
5.17%
0.0001% % 90.50%
0.69%
0.58%
0.006%
Table 12: UTSD time series dataset from Timer 22
where Mise. means Multiple Sources.
Energy
Environment
Health
IoT
Nature
Transport
Web
Cloud
Sales
Finance
Mise.
Obs.
16,86B 7.461% %
70.45M 0.031%
233.M
165M
201B
4.9B
157M
2.15B
198M
0.33M
56.52M
%
0.30%
0.073%
89%
2.17%
0.07%
0.95%
0.088%
0.00%
0.025%
Table 13: LOTSA time series dataset from Moirai 28
Energy
Transport
Climate
CloudOps
Web
Sales
Nature
Finance
Health
Total
Obs.
16.36B
4.90B
4.19B
1.52B
428M
198M
28.55M
24.92M
1.59M 0.01%
27.65B
%
59.17%
17.73%
15.15%
5.49%
1.55%
0.72%
0.09%
0.10%
100%
Table 14: Time series datasets from neural scaling laws 6
Transport
Climate
Energy
CloudOps
Health
Sales
Web
Total
# Obs.
4.82B
4.73B
2.34B 16.15%
2.15B 14.86%
240M 1.61%
140M
600M
14.46B
%
33.31%
32.71%
0.96%
0.40%
100%
are independent of the model architecture and training methods 6 115 116
117
Consequently, an
increasing number of models are adopting the approach of training larger- scale models on large-scale
pre- tre-training datasets to achieve better performance [I18]. This paper surveys the pre-traiiee pre-training datasets
used by the three current mainstream pre-trained foundation models-Time-Mor
77, Moirai (8),
and Timer [29 as well as the datasets utilized in the study of time series scaling laws 6 which
are shown Tables [T 12 [3 and [4] In Figure (3) we demonstrate that the current largest time
series datasets still smaller than those in CV and NLP.
27

---

Imbalanced domain distribution issues in large-scale time series datasets. The distribution of data across various domains indicates that the four large-scale time series pre-trainnng datasets all face issues with imbalanced domain data distribution. For instance, domains such as Nature, Energy and Transport have the most datasets [8O], while others like Sales, IoT, Web, Finance and Multiple Sources suffer from extremely low data volumes due to difficulties in data collection or data privacy concerns as shown in Figure 13 (b). According to the scaling laws of neural networks, the imbalance in the pre-training dataset distribution can lead to significant performance biases in in-domain and out-of-domain forecasting tasks for the trained foundation models [T8], meaning there is considerable performance gap between domains with less data and those with more data. To address this, this paper proposes an unrestricted method for generating high-quality time series data to alleviate the scarcity and imbalanced distribution of data in time series analysis domains.
B.5
2a Dataset Statistical Characterization Coverage Experiments
Metric. To further examine the diversity of the artificially synthesized data in the S dataset, we conduct sampling assessment from six dimensions: stationarity, predictability, frequency domain characteristics, complexity, seasonality intensity, and trend characteristics. For each dimension, we select corresponding statistical indicators for dataset evaluation and quantification, as detailed below:
Augmented Dickey-Fuller (ADF) Test: Consistent with section B.3
we employ the ADF test to assess the stationarity of time series, using its test statistic as an indicator of time series stationarity
65 29]
2. Forecastability: Based on [66] method, we determine whether time series is chaotic or be accurately predicted through machine learning models by using Fourier decomposition and entropy 29 Note that since the method provided by [666 is only applicable to multivariate time series, we invert the sampled single-channel time series to form a dual- channel series calculate the indicator.
3. FFT Mean:
We utilize the average of the Fourier transform power spectrum to evaluate the frequency domain characteristics of time series. This indicator can used to measure the overall intensity of time series and assess the energy distribution.
4. Permutation Entropy: This indicator assesses the dynamic complexity of a time series by analyzing its permutation patterns (69] We set the embedding dimension m and time delay = 1, and calculate its specific value using Shannon Entropy in Equation g See 69
for more detailed calculation.
5. Seasonality: We decompose the time series into trend, seasonal and residual components using the Seasonal-Tres Decomposition using LOESS (STL) algorithm (67]. Then, we calculate the intensity of the seasonal component in the time series according to Equation 9
6. Mann-Kendall Test: This is a con-paraettric statistical method used to detect monotonic trends in time series [68) The basic principle is compare the size relationship between each data point and other data points in the time series. Therefore, this method does not rely specific distribution of data and is not affected by outliers. We use the statistical test results of this method as the evaluation indicator, where -I indicates a downward trend, indicates an upward trend, and 0 indicates obvious trend.
Permutation =
Σ P3 ImP,
(8)
Y, - Ti 1 SS Rs
Seasonality max 0, Var(S. R.)
Var R.)
where, P3 represents the frequency of the i-th permutation model in the permutation entropy, and
K ===w =m! is the total number of permutation patterns [69] Y, represents the original time series, T,
S, and R, are the trend, seasonal and residual components decomposed by the STL algorithm (67
respectively. Var(.) means calculating the variance of series.
28

---

41-
Figure 14: Zero-shot time series imputation in out-of-dommin data.
4.
20
85-
05
0.
(a) ETTml
(b) ETTm2
1.5
63-
-25
(c) ETTh1
(d) ETTh2
1.0
25-
20-
(e) Electricity
(1 Weather
Figure 15: Zero-shot time series imputation real world time series dataset ETTml, ETTm2,
ETTh1, ETTh2 [80]. Electricity [102) and Weather [TOI]
B.6 Masked Time Series Modeling and Zero-shot Imputation for Representation Learning
Setup. Since we incorporate MTM loss the pre-training process of SymTime, this section, we assess the specific learning effects of the time series encoder in SymTime through masked modeling We test the model's performance using both pre-trained synthetic data not in the S2 As SymTime adds masks in
54 27 119
dataset and real datasets from time series imputation tasks 12
14 48
units of patches of length 16 during pre-training, we also add masks the form of 16-1ength patches. The reconstruction effect of the masked parts the time series encoder is shown in Figure and 15
29

---

S Dataset Put-of-Domain Data. In Figure 14 we generate new data using the method from the S dataset and add masks test the reconstruction ability of the time series encoder [22 14 48 The gray sections represent the masked segments, while blue and orange represent the original and reconstructed series, respectively. We input time series outside the gray parts in patches and have the model reconstruct the gray sections based on the remaining information. Since we only calculate the MTM loss on the masked parts (27 [191, the visible reconstruction does not overlap with the original input series [29] [20 [33] From the Figure 14 it can be observed that the time series encoder in SymTime ime performs well in fitting the fluctuations and trends time series, demonstrating that encoder successfully learned the fundamental representations of time series during pre-training
28
122
Real-world Time Series Data. In Figure 15 we conduct representation learning tests on 6 real datasets: ETTml, ETTm2, ETTh1, ETTh2 80], Electricity [02]. and Weather [101]. Since no real data are used for model pre-training, these datasets are also considered as out of domain data. We similarly add masks in patch units (gray sections). It can be observed that the time series encoder in SymTime also performs well in zero-shot reconstruction on real-world data [27 123
B.7 Time Complexity Analysis of S2 Data Generation Mechanism
We define the specific symbol and its explanation in Table 9 Then, use the seride-and-conques approach to anaylze the complexity of the 52 data generation mechanism.
1. Symbolic Expression Generation: We construct symbolic expressions using tree structure as a medium. When we have binary operators. we further insert (b + 1) leaf nodes (the process from (a) to (b) in Figure in our paper). Therefore, after inserting u unary operators (Figure (c)), the total number of nodes in the tree is n = 2b + u + 1. Because there are many ways construct a tree, we consider the time complexity of constructing balanced tree. Therefore, for N symbols constructed, the specific complexity of this process is ONN x nlogn).
2. Sampling series generation: When we want to generate a sampling time series with M channels, each channel has a probability of P to be sampled using mixture distribution and a probability of (1 P) to be sampled using an ARMA model. When the sampling length of the series is L, the complexity of generating k mixture distribution and ARMA (p. q) series is ((K)) and OLL(p + q)). Therefore, the time complexity of this process can be quantified as O (ML x [Pk + (1 (p + +)]).
3. Sampling through symbolic expressions and series: We simplify the specific operational details of this process and only consider the time complexity of operations with variables. For a series of length L, we have N symbolic expressions to be sampled, and each symbol has an average of M+1 variables (Each symbolic expression may contain any number of 2
variables from 1 to M, so here we take M+1 (1+2+-++)) the average probability). M
Then the process can be quantified as O(N M+1 L.).
To sum up, the symbolic expressions we construct and the parameters used in the sampling process are typically smaller than the length of the time series L. Therefore, we ignore the symbolic expression generation process and consider only the two processes of generating the sampling series and sampling using the symbolic expression. Since the number of channels, M and N, as well as k. p and q are all smaller than L, we can intuitively assume that the time complexity of the 5 data generation mechanism is linearly related the length of the series L.
B.8
The Selection of the Unary Operators
From the perspective of constructing symbolic expressions in the current S? data generation mecha- nism, binary operators primarily serve to connect multiple variables, while unary operators can increase the diversity of numerical values through specific operations. However, we choose not to all symbolic and linear operations, but only use the unary operators (inv, abs, pow2, pow3, sqrt, sin, cos, tan, arctan, log, exp) for generation.
Although ignoring certain mathematical symbols will reduce the diversity of symbolic expressions, de
we have found in numerous experiments and tests that differential dy integral f. power operations
30

---

r², and exponential operations with various bases n" will seriously affect our data generation to certain extent. The specific reasons are follows:
Value explosion: To maintain quality, we cap large magnitudes (Section 3.1
Integration,
exponential and high-order powers readily cause overflow, so they were dropped; exp alone is retained for diversity.
2. Numerical differentiation:
Symbolic differentiation introduces truncation//ruuddoolf trade- offs. Combined with reciprocal and absolute-value operators, functions such |x| or sin (+) because non-smooth or high-frequency vibration at = 0, breaking differentiation.
3. Numerical integration: Randomly built expression trees often yield integrands with sin- gularities (e.g.. So v dx) force costly oscillatory integrals (e.g.- fo sin(100z)dz);; interval selection non- non-trivial.
4. Symbolic cost: Differentiation and integration are slow and can trigger exponential memory growth. Many elementary functions lack closed-form antiderivatives (e.g.. (app-=="dda)
Considering factors like numerical stability, symbolic complexity, computational efficiency, and sampling success rate, we selectively omitted some symbolic operations. Nevertheless, the data generation framework proposed is essentially complete theory. It already incorporates the vast majority of symbolic operations, and new operators or user-defined symbolic operations can be easily added. The omission of some operations due to the above factors does not undermine the validity of this framework.
Time Complexity Analysis of
04
Actual Time Spent
Liner Regression
0.3
02
y 0.00059x
100
200
300
400
500
The Length of the Generated Time Series
Figure 16: The time complexity analysis of S Generation
To further demonstrate that the time complexity of the S data generation mechanism is linearly related (L) to the length of the generated time series, we start with a time series length and generate it every 16 lengths until 512. We switch to a different random seed for each generation and repeat the experiment 1280 times. The average time of each data generation is shown Figure [6] The linear fit line of the result shows that the time complexity of data generation is linearly related to the sequence length when sampling failure is not considered.
B.9 The Limitations of S2 Generation Mechanism
As outlined in the abstract and introduction, to address the scarcity of training data for time series foundation models, this work proposes dual-modal data generation mechanism grounded in complex dynamical systems theory (detailed in Section 3.1). This mechanism enables comprehensive coverage of time series representation spaces through unrestricted, high-quality generation. However, gener- ating symbolic expressions (complex systems) via randomized binary tree algorithms occasionally results in oversized trees, leading to overly intricate symbolic systems. This issue narrows the domain of symbolic functions [(),, hinders sampling stimulus-driven time series X, and reduces sampling linear degrade sampling speed. Consequently, these operations are excluded from the current S2 dataset generation. Future work will explore integrating ordinary and partial differential equations into the
efficiency. Notably, differential and integral operations-comppe
severely
31

---

S framework to enrich symbolic expression diversity (complex systems) and further enhance the representational capacity of generated time series data.
C Implementation Details
In this section, first provide detailed introduction the datasets and evaluation metrics used for the five TSA tasks. Subsequently, we elaborate on the training details of our experiments, including how we pre-trained SymTime on the S dataset and how we fine-tuned it on downstream task datasets. All experiments and deep neural networks training are implemented in PyTorch on 8 NVIDIA A6000 48GB GPU.
C.1 Downstream Tasks Datasets Details
We conduct experiments using the TimesNet benchmark 722.. with a detailed description of the dataset provided in Table T3] Specifically, we utilize 8 datasets including ETTh1, ETTh2, ETTml, ETTm2 80, Electricity 102., Traffic [03]. Weather [00]] and Exchange [004] to conduct long-term time series forecasting experiments. Our model, SymTime, emplloy input series of lookback lengths 96 and 512, with forecast horizons of 96, 192, 336, and 720. For short-term forecasting experiments, we employ the M4 benchmark dataset, predicting data of various frequencies [73]. In the time series imputation task, we test on 6 -Etiee,s-ETTh,, ETTh2, ETTml, ETTm2 [80], Electricity [102, and Weather [00 with mask rates of 12.5%, 25%, 37.5%, and 50%. For time series classification, we utilize ten UEA multivariate time series classification benchmark datasets [105]. For anomaly
detection in time series, we experiment with five datasets: SMD [06],, MSL [077. SMAP [107].
SWaT [08], and PSM [109].
C.2 Correspondence between Positive and Negative Samples in SymTime
In our S data generation mechanism, we randomly construct a symbolic expression f(-) and an input stimulus time series X. and then forward propagate the symbolic expression to obtain the complex system's 's response time series Y. Therefore, the generated time series Y has natural generating-hein relationship with the symbolic expression f(-). Since symbolic expressions can model complex systems, and different symbolic expressions can correspond to different symbolic systems, the generated time series also reflect the corresponding complex system's 's representation. When performing contrastive learning, we consider the time series and the symbolic expression that generated it as positive samples, and other unrelated symbols as negative samples.
C.3 Metries
We assess the five TSA tasks using various metrics. For long-term forecasting and imputation tasks, employ mean squared error (MSE) and mean absolute error (MAE). For short-term forecasting, we utilize symmetric mean absolute percentage error (SMAPE), mean absolute scaled Error (MASE), and overall weighted average (OWA), with OWA being metric unique the M4 competition. For time series classification tasks, we use classification accuracy as the metric. For anomaly detection tasks, adopt precision, recall, and F1-score as our evaluation metrics. The calculations for these metrics are as follows.
MSE =
Σ (u: 01)
(10)
1=1
MAE Σ
1y4 Q.
(11)
2=1
X Y
200
SMAPE
(12)
T
X +
Y
X, Y
100 MAPE
(13)
T
32

---

Table 15: Dataset descriptions. The dataset size is organized in (Train, Validation, Test).
Tasks
Dataset
Dim
Series Length
Dataset Size
Information (Prequency)
FTTml, ETTm2
7
(96, 192, 336, 720)
(34465, 11521, (1521)
Elecmiitty (15 mins)
ETThi, ETT12
7
336, 720)
(8545, 2881, 2881)
Electiiity mins)
Forecasting
Electricity
321
(96, 192, 336, 7203
(18317, 2633, 5261)
Electricity (Houtly)
(Long-term)
Traffic
862
(96, 192, 336, 720)
(12185, 1757, 3509)
Transporation (Hourly)
Weather
21
192, 336, 720)
(36792, 5271, 10540)
Weather (10 mins)
Exchange
(96, 192, 336, 720)
(5120, 665, 1422)
Exchange rate (Daily)
M4-Ycarly
(23000, O, 23000)
Demographic
M4-Quarterly
8
(24000, O, 24000)
Finance
Forecasting
M4-Monthly
(48000, O, 48000)
Industry
(short-term)
M4-Wcakly
13
(359, 0, 359)
Macro
M4-Daily
14
(4227, 0, 4227)
Micro
M4-Hourly
(414, 0, 414)
ETTml, ETTm2
7
96
(34465, 11521, 11521)
Electricity (15 mins)
ETTh1, ETTh2
7
96
(8545, 2881, 2881)
Electicity (15 mins)
Imputation
Electricity
321
(18317, 2633, 5261)
Electricity (15 mins)
Weather
21
96
(36792, $271, 10540)
Weather (10 mins)
FihumolCommeettaation
3
1751
(261, O, 263)
Alcohol Industry
FuceDetection
144
(5890, O, 3524)
Face (250Hz))
Handwriting
3
152
(150, O, 850)
Handwriting
Heartheat
61
405
(204, O, 205)
Heart Beat
Classification
JapeneseNowels
(270, 0, 370)
Voice
(UEA)
PEMS-SE SF
963
144
(267, 0, 173)
Transportation (Daily)
SellRegulstioSCPI
896
(268, 0, 293)
Health (256812)
SelfRegolatioSCCP
1152
(200, 180)
Health (256Hx)
SpokenArabiiiggis
13
93
(6599, 0, 2199)
Voice (1 (11025Hz)
UWaveGestueLLibarry
3
315
(120, 0, 320)
Gesture
SMD
38
100
(566724, 141681, 708420)
Server Machine
Anomaly
MSL
55
100
(44653, 11664, 73729)
Spacecraft
Detection
SMAP
25
100
(108146, 27037, 427617)
Spacecraft
SWaT
51
100
(396000, 99000, 449919)
PSM
25
100
(105984, 26497, 87841)
Server Machine
T
X; Y.
MASE = T
Σ I
(14)
[X, -X,-ql
1=1 T-q
2j=et1
1
SMAPE
MASE
OWA- 2
(15)
SMAPE ENavve2 MASEN
33

---

Table 16: The model architecture of the time series and symbolic encoders in SymTime.
Encoder
Layers
daodee
da
Heads
Params
Time Series
2048
Symbol
6
768
3072
12
67M
where, Yi is the ground true value, 9; is the model prediction, is the peridoicity of the time series data. X, Y RTxC are the ground truth and prediction results of the future with T time points and C' dimensions. X; means the i-th future time point.
C.4 Pre-training
Model Architecture The model architecture of the time series and symbolic encoders in SymTime are shown Table [6
The parameter configurations for the time series encoder and symbol encoder in SymTime are shown in Table [6] During model pre-training, we primarily set three hyperparameters: (1) the masking ratio time series patches, (2) the masking ratio for natural language symbols, and (3) the proportion factor used to balance aseudo-target in momentum distillation. Based on the masked time series modeling pre-training experimental configuration of PatchTST (54) and SimMTM (27]. we set the masking ratio for time series to 40%. Following the experimental configuration BERT masked language modeling [99 55), we set masking ratio for symbolic data to 15%. Based on the experimental configuration of momentum distillation in
Model
ALBEF 60 124 59],
we set to 0.6.
Training Configurations. During the pre-training SymTime, we employ AdamW 125 126 as the optimizer with the defult hyperparameter configuration for (81, 82) as (0.9, 0.999). Then, we utilize the OneCycle policy dynamically adjust the learning rate. We the warmup epochs 10, during which the learning rate gradually grows up to an initial value of 5 x 10-5, and then adjust dynamically using cosine annealing schedule, with the minimum learning rate conduct pre-training using data parallelism on a hardware setup consisting of 8 NVIDIA RTX A6000 GPUs with 48GB of memory each. We the batch size 128 and trained for total of 85 epochs. Unlike SNIP 122, we do not generate data on-the-fly during training for pre-training. Instead, we prepare the data in advance and then load into the device for pre-training. Due to the large size of our generat 5 dataset, we load data into the GPU in batches during each epoch for pre- pre-training.
1 10"H We
C.5 Fine-tuning
For the five major tasks in TSA, we conduct downstream task fine-tuning experiments using the configurations in Table 17 For all downstream task fine-tuning experiments, employ the Adam optimizer [125 [26] with hyperparameters (81, (22) set to (0.9, 0.999). The LR in the table represents the initial learning rate and utilize the dynamic learning rate adjustment strategy from TimesNet
72
Table 17: Experiment configuration of SymTime fine-tuning.
Model Parameter
Training Configurations
Tasks Configurations
dmodel
dæ
Layers
I.R
Loss
Batch Size
Epochs
Long-term Forecasting
3,6 6
10
10
MSE
4-64
20
Short-term Forecasting
2.3
10 2 10
SMAPE
8-32
Classification
512
2048
1-6
10-4 10-4-5x 5 10
Cross Entropy
4-64
64
Imputation
3,
10 - 10
MSE
4-64
32
Anomaly Detection
3.6
10-4 - 5 10
MSE
4-64
12
34

---

C.6 Ablation Experiments Details
Ablation study on pre-training strategies and objectives. To further verify the effectiveness of our series-symbol pre-training strategy and objectives, we establish 8 distinct ablation experiment groups and a control group. The specific configurations these ablation experiment groups are as follows.
Freeze: All parameters in the pre-trained time series encoder are frozen, with only the linear projection layer for outputring prediction results fine-tuned.
2. Read-Data: Since real time series data does not have matching symbolic expression information, we temporarily discarded the symbolic encoder and momentum model this ablation experiment and only used real time series data for pre-training using the MTM method.
3. w/o Pretrain: No series-symbol pre-training is conducted; the time series encoder with initialized parameters is used for downstream task experiments.
4. w/o MTM: The masked time series modeling (MTM) is removed from the pre-training objectives.
5. w/o MLM: The masked language modeling (MLM) is removed from the pre-training
objectives.
6. w/o T2S:
The contrastive loss from time series symbols is removed from the pre-training objectives.
7. w/o S2T: The contrastive loss from symbols to time series is removed from the pre-training objectives.
8. w/o Symbol: Only time series data from the Sa dataset used to pre-train the time series encoder via MTM, disregarding the correspondence with symbols.
9. w/o Distill: The contrastive loss in pre-training does not use the pseudo objective of momentum distillation.
C.7 Ablation Experiments Short-term Forecasting
Setup. We adopt the same experimental setup as in Section 4.4 to conduct ablation studies on short-term time series forecasting tasks. We first select the Yearly and Monthly sub-datasets from the M4 benchmark dataset [73] to perform ablation experiments on SymTime's pre-training objectives. We choose SMAPE as the evaluation metric and the average results with error bars are shown in
Figure
17
Results. Figure 17
(a) indicates
that SymTime's performance drops sharply when the backbone encoder frozen and pre-training is ducted. When some pre-training ob- jectives removed, the model's performance in short-term time se- ries forecasting also declines, but the sensitivity of performance degra- dation is as pronounced long-term forecasting experiments. Figure 17 (a) shows that as the size of the pre-training dataset increases, SymTime's 's performance on the Quarterly dataset improves significantly.
Yearly
Figure 17: Ablation study on short-term forecasting task.
C.8 The Ablation of Backbone SymTime
Setup. SymTime is composed of two encoders with Transformer architectures 127
The time series
encoder is composed of a multi-layer Transformer encoder architecture. The symbolic expression encoder is pre-trained with large language model. We conducted ablation experiment the
35

---

SymTTime model architecture by changing the encoder structure through control variables. Specifically, we adjusted the number of parameters (cmndel and dę) of the time series encoder and the type of pre-trained L.M used by the symbolic encoder to set different control groups:
SymTime: The original model architecture in Table 16 uses the DistilBERT 55
Sym' SymTimesmat" Change the time series encoder to a 3-layer Transformer model with dmodel 386 and dg 1536.
SymTimerrrge Change the time series encoder to a 6-layer Transformer model with dmodel = 768 and dg = 3072.
BERT-base,oov: Replace the pre-trained LLM with BERT-base
19
BERT-large:soar Replace the pre-trained LLM with BERT-large:.0M
19
GPT2-small 124M: Replace the pre-trained LLM with GPT2-small
94
GPT2-mediumgsss. Replace the pre-trained LLM with GPT2-medium
Mean Square Error (MSE)
0.40
SymTime
SymTimeamat
0.35
SymTimeurg
BERT-base
BERT-largestox
0.30
GPT2-smallzaa
GPT2-mediumgga
ETTm1
ETTh1 Time Series Forecasting Datasets
ETTm2
ETTh2
Figure 18: The ablation results of backbone in SymTime. We choose to conduct experimental verification on four ETT datasets 80 for long-term time series prediction.
Results. As shown in Figure 18 changing the backhone of SymTime does not significantly affect the experimental results in long-term time series forecasting tasks. Ablation experiments on pre-training objectives (Equation 7 reveal that the performance gains achieved during pre-training are primarily due to the pre-training paradigm of masked modeling of time series and symbolic expressions and contrastive learning. This pre-traiee pre-training approach is independent of the model backbone. Therefore, the basic pre- pre-trainien trainiiinn requirements are met (pre-training loss can be successfully optimized), more lightweight model can be used for fine-tuning on downstream tasks.
C.9
The Impact and Ablation of Pre-interpolation on Time Series Imputation Task
SymTime adds masks randomly at the patch level during pre-traine pre-training for time series reconstruction. While in the imputation task, masks are added randomly at the data point level. Additionally, high masking rates may disrupt the original trend and periodic features of the time series. Therefore, Peri-nidFormer's method to apply per-interpolation to the masked time series to restore the disrupted
important to note that this method is general and independent of deep learning models. The use of this method aims to further enhance the potential of deep learning models. To further verify the effectiveness and impact of the per-interpolation method, we conduct experiments on the ECL time series imputation dataset, with results shown in Table [8] Taking the ECL time series dataset 0.5 mask ratio as an example, the effect of pre-intee pre-interpolation shown in Figure 19 We perform experiments with masking rate of (0.125, 0.25, 0.375, 0.50) and compare models such ForrmeriFoomer (63], TimesNet [72), PSt [54] DLinear [87] and Pyraformer [130]. Per-interpolation represents the experimental results obtained using only linear interpolation. For missing time series 24 at time I, the method can be described as:
periodic features [63] 72 89
128 129].
+-+++
(x+-1) # None & (I++1 * None)
2
It
Et+11
if (x(-1) = None & (I++1 * None)
(16)
Ft-1:
if (x1-1) * None & (I++1
None)
36

---

Table 18: Ablation Experiments pre-interpolation in inputation task on ECL dataset. The per- interpolation results for Peri-mide mirFomrr,rrrr, TimesNet, PatchTST, DLinear and Pyraformer are copied from 63
w/o per per-interpolation
with per-interpolation
Methods
Metric
0.125
0.25
0.375
0.5
0.125
0.25
0.375
0.5
MSE
0.086
0.110
0.149
0.206
Por-interpolation
MAE
0.188
0.213
0.251
0.301
MSE
0.050
0.064
0.074
0.092
0.037
0.047
0.060
0.075
SymTime (Ours)
MAE
0.145
0.169
0.181
0.206
0.122
0.139
0.160
0.181
MSE 0.073
0.092
0.107
0.122
0.047
0.053
0.067
0.085
Peri-midFormer 63
MAE
0.187
0.214
0,231
0,248
0.140
0.162
0.179
0.195
MSE 0.088
0.092
0.096
0.102
0.081
0.083
0.086
0.091
TimesNet
MAE
0.203
0.208
0,214
0.221
0.196
0.198
0.201
0.207
MSE 0.061
0,072
0.082
0.097
0.050
0.059
0.070
0.087
PatchTST
MAE
0.170
0.185
0.198
0.216
0.148
0.164
0.181
0.202
MSE 0.084
0.113
0.141
0.173
0.050
0.062
0.789
0.105
DLinear
MAE
0.206
0.243
0,273
0.303
0.144
0.164
0.189
0,225
MSE 0.297
0.294
0.296
0.299
0.165
0.165
0.171
0.173
Pyraformer
130
MAE
0.383
0.380
0.381
0.383
0.290
0.291
0.293
0.295
where, 2z-1 and T++1 represent the values the previous and next time points, respectively, while None indicates a missing value. The results Table show that this method significantly improves the performance of all models in model-independee manner.
m
mm W
-00
(a) Original data
(b) Data with 50% missing values
(c) Per-interpolated
Figure 19: Visualization of original data, data with 50% missing values and pre-interpolated data of ECL dataset.
37

---

D Related Work
D.1 Time Series Foundation Models
In CV and NLP [8] 131 [32], PTFMs have been demonstrated to adapt to a variety of downstream tasks after fine-tuning on specific datasets, exhibiting excellent generalization and scalability. Inspired by this, recent years have seen significant progress in PTFMs for TSA 124 25 133 134), with the emergence of various pre-traiin pre-training methods. MOIRAI, through MTM and reconstruction, has been pre-trained on large datasets (27B), yielding universal forecasting model with significant zero-shot advantages 228 Timer, after generative pre-training on large datasets (1B), has performed well in forecasting 29 TimeGPT trained encoder-decoder Transformer with 100B data [30]. COMET,
using multi-level contrastive learning on large ECG dataset, has obtained a medical time series
PTFMs with few-shot advantages 4
As discussed in Appendix Section B.4 these baseline models still face challenges related data scarcity and data imbalance. In the next section, we introduce the proposed data generation mechanism and the corresponding dual-modality foundation model designed to address these issues.
D.2 Deep Learning and Symbolic Regression
The central thesis of this paper is to regard time series as representations of complex dynamical Traditionally, complex systems are modeled by observing time series utilizing ODE With the advancement of machine learning, symbolic regression (SR) [36,, as
systems [35
and PDE 15
supervised learning method, can discover hidden mathematical expressions from numerical series. Although genetic algorithms (GAs) are the mainstream approach for SR [45] 46), deep learning- based methods have also made significant progress. [14] constructed an end-to-end SR model using Transformers, while SNIP built large-scale pre-trained model through contrastive learning on symbolic expressions and numerical observations [12] Both methods treat symbolic expressions as nature language and use deep neural networks to learn their features. Therefore, this paper employs e a pre-trained LLM as symbol encoder to learn the features of symbolic expressions and jointly trains a time series foundation model imbued with semantic information through contrastive learning [12
D.3 Time Series Forecasting Models Based on Synthetic Data
Unlike the representation pre-training conducted on the large synthetic S dataset in this paper, previous TSA models trained synthetic data mainly based Prior-data Fitted Networks (PFN) 137
138, 139 This model learns prior distributions from synthetic data using Bayesian methods, enabling zero-shot inference. ForecastPFN generated a large number of synthetic time series by separately modeling the seasonal trend, global trend and noise based on given constraint expressions [7). Although PFN trained in this way offered certain zero-shot and few-shot advantages, this approach was limited to generating time series through sampling fixed expressions and performing linear combinations. In contrast, the Sa data generation mechanism proposed in this paper can sample an infinite variety of symbolic expressions [12
[29]. TimePFN constructed synthetic datasets by filtering real time series with linear and periodic convolution kernels, training PFN for zero-shot inference [14)] However, this method depends
14 48
real-world time series for filtering and linear transformations between channels. Compared to the S data generation mechanism, it can not create large-scale and fully representative synthetic datasets for model pre-training.
38

---

E
Visualization
E.1
Long-term Time Series Forecasting with 96 Prediction on ETThl (Figure 20 and ECL (Figure 21
E.2
Short-term Time Series Forecasting on M4 Weekly (Figure 22 and Monthly (Figure 23
E.3
Time Series Imputation with 50% mask rate on ETTh1 (Figure 24) and ETTml (Figure
25
Full Results
For the five downstream TSA tasks results, use (1) Peri-midFoormer for Peri-midFormer 63 (2)
uni2ts for Moirai (28), (3) Large-Time--Sriees Model for Timer [29]. (4) Time-LLM for Time-LLM
177. (5) TSLANet for TSLANet [83]. (6) S2IP- LLM for S2IP-LLM [82), (7) NeurlPS2023--One
Fits-All for GPT4TS (2). (8) UniTS for TTiTS 192]. (9) moment for Moment [93.. (10) FilterNet for
FilterNet (9T] (11) RTSF for RLinear 141. and (12) Time-Series-Library for other models, such as
TimesNet [72]. PStchTST [54]. TimeMixer 89 i'Trnnsfrmer [74], DLinear [87], Autoformer [75
and Informer 80 TimeXer [62]. Chronos-forecastoo for Chronos [22]. To ensure fair comparison,
we use the original experimental configuration in the project scripts.
E1
Time Series Long-term Forecasting with 96 look-back windows (Table 19 Table 20 and
Table 21
F.2
Time Series Long-term Forecasting with 336 look-back windows (Table 22
E33
Time Series Long-term Forecasting with 512 look-back windows (Table 23
F.4
Time Series Short-term Forecasting (Table 24 and Table 25
E55
Time Series Classification (Table 26 and Table 27
F.6
Time Series Imputation (Table 28 and Table 29
F77
Time Series Anomaly Detection (Table 30)
Pre-training and Fine-tuning Results Long-term Forecasting (Table
F9
Pre-training and Fine-tuning Results of Short-term Forecasting (Table
F.10
Pre-training and Fine-tuning Results of classification (Table
F11
Pre-training and Fine-tuning Results of Imputation (Table 34
F.12
Pre-training and Fine-tuning Results of Anomaly Detection (Table 35
39

---

G
Impact Statement
The potential value of this work lies in its ability to mitigate fundamental challenges in TSA, such as the lack of sufficient labeled data and the issue of imbalanced datasets. By generating rich, diverse, and high-quality synthetic data, our approach not only addresses these issues but also opens new avenues for improving model generalization across wide range of applications. Furthermore, the dual-modality framework, which combines time series data with symbolic semantics, introduces novel way of enriching the representation power of models, allowing them to better understand complex temporal dynamics and their underlying patterns.
We foresee that tre-training models on synthetic datasets, especially those that combine structured symbolic information with time series data, will become key development trend the TSA field. This could pave the way for more robust and scalable solutions in a variety of domains, including finance, healthcare, and climate modeling, where time series data is abundant, but labeled data is often scarce or hard to obtain.
40

---

4-2
Truth
-23
4.4
-144-
-.3
175
(a) SymTime
(b) PatchTST
(c) iTransformer
-5.
Grandd Thun
-03-
Graund Trun
Cround Trut
-5.4-
24
27-
20
175
75
175
175
(d) TimesNet
(e) DLinear
(f) Autoformer
Figure 20: Visualization of long-term forecasting with 96 prediction length of ETTh1 dataset.
Tra Prodedes
Grau Prodecdes
Prodeddes
53-
-12
-13
-1.4
(a) SymTime Time
(b) PatchTST
(c) iTransformer
Shaund Fruts
Trau Profeces
Trodicder
Prodeces
-1.
-.8
-1.
(d) TimesNet
(c) DLincar
(f) Autoformer
Figure 21: Visualization of long-term forecasting with 96 prediction length of Electricity dataset.
41

---

Srord Dvi
Grand liu
Sroud Dvn
(a) Sym Time
(b) PatchTST
(c) iTransformer
Peciaiin
Resiidion
Prodidion
M
W
(d) TimesNet
(c) DLincar
(f) Autoformer
Figure 22: Visualization of time series short-term forecasting in M4 dataset Weekly,
Ground dlln
Sroudd Paodiaiion
Ground
(a) SymTime Time
(b) PatchTST
(c) iTransformer
Prodiaionn
(d) TimesNet
(e) DLinear
(f) Autoformer
Figure 23: Visualization of time series short-term forecasting in M4 dataset Monthly.
42

---

Broae Tuu3
Creand Trait
Gread Trdhs
-00
(a) Sym'Tmee
(b) PatchTST
(c) iTransformer
2.4
Prodiciars
Prodecioo
1.-
4.4
4.2
03
-.3
-12
(d) TimesNet
(c) DI Linear
(f) Autoformer
Figure 24: Visualization of time series imputation with 50% mask rate of ETTh dataset.
Trats
Goead
-00
43-
(a) Time
(b) PatchTST
(c) iTransformer
4.5
Predicion
Predicin
Predcian
4.5
<7-
4.-
C5
-13
1.3
-15
(d) TimesNet
(c) DLinear
(I) Autoformer
Figure 25: Visualization of time series imputation with 50% mask rate of ETTml dataset.
43

---

Table 19: Full results for the long-term forecasting task compared with Peri-midFormer [63] Moirai 28 Timer 22).. Moment [93], Time-LLM [17], TSLANet [83), S'IP-LLM [82 and GPT4TS 22.
(* means former, T-LLM is Time-LLM, S-LLMM is 5²IP-LLMM) To ensure fairness the comparison, we set the look-back window length of all models to 96. Since the Timer and Moirai need to input longer series to build a token, their windows are 672. I-IP-L.MM has gradient explosion when the
window is 96, so its look-back window is 512. The standard deviation is within 0.5%. Red: best,
Blue: second best.
SymTime (Ours)
Peri-mid
Moirai
Timer
Moment
T-LLM
TSLANet
S-LLM
GPT4TS 2
Methods
63
28
29
93
17
33
82
Metrics
MSE MAE|
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
96
0.318 0.353
0.334 0.370
0.311 0.358
0.315
0.354
0.305 0.353
0.304 0.359
0.321 0.362
0.325 0.371
0.293 0.362
192
0.362 0.380
0.382 0.391 0.381 0.402
0.369 0.378 0.369 0.386
0.368 0.396 0.361 0.383
0.361 0.397
0.374 0.392
336
0.386
0.402
0,417 0.418 0,436 0,432
0.425 0,428 0.392
0.404
0.383
0.393 0.383 0.404
0.385
0,403
0.389 0,404
720
0.419
0.423
0.501 0.461 0.4466 0.476
0. 0.44 0.447
0.425 0.436
0.420
0.429
0.445 0.437
0.426 0. 0.446
0.421 0.423
Avg
00.371
0.390
0.409 0.410 0.398 0.417 0.388 0.402 0.373 0.395
0.369
0.394 0.337 0.397
0.374 0.404
0.369 0.395
96
0.174
0.257
0. 174 0.255 0.1 0.267
0. 161
0.254
0.1 0.264
0.1 0.269 0.1 0.261
0.4
0,263
0.171 0.265
192
0.238
0.299
0.249 0.305 0.244 0.311
0.429 0.425 0.285 0.294 0.239 0.305 0.243 0.303
0.232
0.306
0.226 0,304
336
0.295
0.337
0.319 0.349 0.335 0.371
0.476
0.457 0.275 0.329
0.301
0,340 0,308 0.345
0.300
0.344
0.288 0.345
720
0.390
0.392
0.418 0.405
0.425 0.444
0.545
0.497
0.383 0.397
0.382 0.381 00.403 0.401
0.359 0.386
0.372 0.398
Avg
(0.274
0.321
0.290
0.328 (0.296 0.348
0..405 0.408
(0.278 0.321
0..255
0.324 0.228 0.327
00266
0.325
0.264 0.328
96
0.376
0.400
0.382
0.403 0.369 0.408
0..74
0.404
0.385
0.402
0.386 0.395
0.387 0.405 0.380
0.403
00.888 0.399
192
0.428
0.431 0.436
0.435 0.441 0.450
0.433 0.438 0.449 0.450 0.421 0.424 00.448 0.436 0.410 0.427
0.425 0.429
336
0.463
0.456
0,492
0.455 0.469 0.469
0.458 0.453
0.455
0.472
0.438
0,450
0,451 0.437
0.426
0,442
0.444 0,455
720
0.450
0.458
0.508 0.490
00.486 0.490
0.475 0,480
0.480
0.503
0.506
0.510
0.505
0.485
0.610
0.543
0..479 0.477
Avg
00.43
0.436
0..455 0.446 (0.441
0.454
0.443 0,444
00.442
0.457|
0.438
0.445
00.448 0.441
00.456
0.454
(0.434 0,440
96
0.293
0.347 0.. 0. 0.358 0.288
0.350 0.315
0.360
0.285
0.343
0.307
0.369
0.289
0.345
0.292
0.353
0.299 0.351
192
0.364
0.397
0.388 0.403 0.390 0.426
0.411 0.423 0.368 0.403
0.349 0.384 0.362 0.391
0.355
0.388
0.351 0.394
E1
336
0.385
0.423
0.443 0.443 0.441
0.435
0.465
0.467 0.380
0.421
0.394
0.420 0.350 0.389
0.368
0.417
0.380 0.421
720
0.420
0.441 0.445 0.459
0.4887
0.435
0.521 0.515
0.423 0.466
0.426 0.454
0.418 0.439
0.434
0,460
0.422 0.446
Avg
0.365
0.402 0.400 0.416 0.400 0.411
0..428
0.441 00.364 0.408
00.369
0.407
0.355
0.391
|0.362
0.405
0..35 0.403
96
0.166
0.213 0.157 0.201 00.156 0.206
0.289 0.331 0.166 0.228
0.2 0.221 0.1177 0.216
0.162
0.213/0.188
0.213
1.14 0.224
192
0.212
0,254 0,244 0.273 0.229 0.274 0.314 0.349 0.226 0.262
0.194
0,241 00.266 0.258
0.7
0,246
0.230 0.263
Weather
336
0.267
0.294 0,283 0.303 0.282 0
0.339 0.363 0.257 0.303
0.286
0.282 0.279 0.588
0.281
0.299
0.285 0.302
720
0.342
0.344 0.364 0.355 (0.395 0.401
0.375 0.388
0..311 0.355
0.337
0.332 0.355 0.346
0.333 0.339
0.362 0.352
Avg
(0.247
0.276 0.262 0.283 (0.265 0.299
0.329 0.358 (0.245 0.287
0.247
0.269 0.255 0.352
(0.243
0.274
0.265 0.285
96
0.162
0.253
0. 151 0.245 0.7 0.221
050 0.244 0 051 0.247|
0.144 0.242 o. 176 0.261
0.9 0.251
086 0.272
192
0.173
0.264
0.166 0.259 0.158 0.243
059 0.252
0. 0.166 0.252
0.1 0.261 0.182 0.268
0.1 0.269
0.190 0.277
336
0.194
0.285
0. 184 0.268 1.166 0.255
0. 190 0.271
0. 072 0.269
0.18
0.270 0. 09199 0.285
199
0.291
0.205 0.292
720
0.220 0.304
0.207 0.29/ 00.207 0.290
0.210 0.300
00.213 0.311 0.214 0.301 00.240 0.317
0.244
0.319
0.245 0.323
Avg
|0.187 0.276 [n. 178 0.267 0.1667 0.252
177 0.267 17. 0.270
0.270/0.180
080
0.269 191 0.283
191
0.283
|0.206 0.291
96
00.432
0.280
0..426 0.277 (0.376 0.264
0.391
0.260
0..42
0.295
0.424
0.295
0.398
0.291
00.35
0.289
00.471 0.312
192
0.444
0.287 0.440 0.283 0.410
0.279
0.426
0.271 0.452
0.301
0.455
0.315 00.430
0.307
0.403
0.308
0.478 0.312
Traffic
336
0.458
0.293
0.477
0 0.442 0.287
0..45
0.297
0.467
0.309
0.494
0.335 0.494 0.3
0.425
0.299
0.499 0.319
720
0.492
0.303
0.487
0.308 0.470 0.328
0.475
0.307
0.489
0.316
0.513
0.394 00.528
0.332
0.454 0.326
0.523 0.335
Avg.
00.457
0.291
1.458
0.295 0.424 0.289
0.446
0.284 (0.463
0.305
0.477
0.334
(0.463 0.310
0.310/0.417
0.417
0.306
(0.491 0.320
96
0.084
0.201
0.083
0.199 0..899 0 00.098
0.228 00.091
0.214|0.090
0.214
0.090 0.209 0..82 0.200
0.1
0.279
0.087 0.218
Exchange
192
0.174
0.295
0.190
0.307 0.175 0.289
0.196
0.325 0.185 0.307
0.188
0.310 0.172 0.295
0.234
0.354
0.171 0.294
336
0.331
0.416
0.401
0.458 0.345 0.423
0.359
0.433 0.345 0.414
0.342
0.427 0.329 0.415
0.403
0.474
0.349 0.418
720
0.847
0.694
0.879
0.702
0.882 0.744
0.875
0.713
0.874
0.729
0.885
0.707
0.889
0.747
1.103
0.804
0.873 0.713
Avg
(0.359
0.401|
00.888
0.417 /0.73 0.417|
0..82
0.425
00.774
0.416|0.376
0.416
0.414
0.368
0.414
0.472 0.478
0..370 0.411
Average
0.336
0.349
0.355 0.358
00.446 0.361
0..32
0.378
(0.399 0.357
0.341 0.357
0.344 0.364
0.348 0.366
0..45 0.359
44

---

Table 20: Full results for the long-term forecasting task compared with FilterNet [91] TimesNet
[72). iTransformer (74), PatchTST [54]. RLinear [141], DLinear [87] and TimeMixer 89 (* means former, TNet is TimesNet, PTST is PatchTST, TMixer is TimeMixer) To ensure fairness in the comparison, we set the look-back window length of all models to 96. The standard deviation is within 0.5%. Red: best, Blue: second best.
SymTime
FilterNet
Chronos
TNet
iTrans *
PTST
RLiicar
DLincar
TMixer
Methods
(Ours)
91
74
141
87
g9
Metrics
MSE MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE MAE
96
0.318
0.353 0.321 0.361
0.324 0.371
0.331 0.372
0.343 0.377 0.324 0.365 0.355 0.376 0.345 0.372 0.323 0,361
192
0.362
0.380
0.367 0.387
0.381 0.400
0.397 0.402
0.381 0.395 0.367 0.389 0.387 0.392 0.382 0.391 0.362 0.383
ETTml
336
0.386
0.402 0.401 0.409
0.402
0.415
0.427
0.427
0.419
0.418 0.400 0.409 0.424 0.415 0.414 0.414 0,388 0.403
720
0.419
0.423 0.477 0.448
0.428
0.435
0.493
0.463
0.487
0.457 0.460 0.445
0.487 0.450 0.473 0.450 0.454 0.442
Avg
00.311 0.390
0.392 0.401
01.84
0.405
0.:42
0.416 0.407
0.412 0.388 0.402 0.413 0.408 0.403 0.407 0,382 0.397
96
0.174
0.257
0.175 0.258
0.1
0.265
085
0.265
185 0.271 0.182 0.266 0.182 0.265 0.194 0.293 0.177 0.259
192
0.238
0.299 0.240 0.301
0.268
0.320 0.256
0.310 0.254 0.314 0.250 0.311 0.246 0.304 0.283 0.360 0.245 0.306
BT1m2
336
0.295
0.337
0.311 0.347
0.289
0.341
0.314
0.345
0.315 0.352 0.313 0.350 0.307 0,342 0.376 0.423 0.298 0.338
720
0.390
0.392
0.414 0.405
0.392
0.410
0.424
0.412
0.443 0.407 0.417 0.412 0.407 0.398 0.529 0.509 0.395 0.396
Avg
0.274 0.321
0.285 0.328
0.286
0.334
0.295
0,333
0.292 0,336 0.291 0.335 0.286 0.327 0.346 0,396 0.279 0,325
96
0.376
0.400
0.382
0.402
0.408
0.402
0.409
0.425 0.394 0.409 0.381 0.400 0.386 0.395 0.396 0.411 0,385 0,400
192
0.428
0 0.430
0.429
0.459
0.450 0.469
0.460 0.447 0.440 0.429 0.433 0.437 0.424 0.446 0.441 0.441 0.431
336
0.463
0.456 0.472
0.451
0.445
0.437
0.507
0.478 0.491 I 0.464 0.475 0.460 0.479 0.446 0.490 0.468 0.482 0.450
720
0.450
0.458
0.481
0.473
0.482
0.485
0.521
0.497 0.517 0.501 0.517 0.502 0.481 0.470 0.514 0.511 0.504 0.482
Avg
00.430 0.436 0.441 0.439
0.449
0.444
0.476
0.465 0.462 0.454 0.451 0.449 0.446 0.434 0.461 0.458 0.453 0.441
96
0.293
0.347 0.293 0.343 0.299
0.354
0..31
0.372
0.300
0.350
0.301 0.351 0.318 0.363 0.348 0.401 0.293 0,343
192
0.364
0.397 0.374 0.396 0.356
0.390
0.429
0.423 0.380 0.399 0.374 0.398 0.401 0.412 0.473 0.474 0.376 0.396
ET
336
0.385
0.423 0.417 0.430
0.376
0.423
0.450
0,451 0.422 0.432 0.429 0.439 0.436 0.442 0.588 0.539 0.425 0,432
720
0.420
0.441 00.449 0.460
0.439 0.467
0.459
0 0.466
0.429 0.447 0.443 0.461 0.442 0.454 0.829 0.656 0.457 0.459
Avg
|0.365 0.402 0.383 0.407 0.368 0.408
0.447
0.428| 0.383
0.407 0.387 0,412 0.399 0.418 0.559 0,518 0,388 0,408
96
0.166
0.213
0.162
0.207
0.177
0.231
0.171
0.222 0.176 0.215 0.177 0.219 0.192 0.232 0.197 0.258 0.172 0,220
192
0.212
0.254
0.210 0.250
0.221 0.263
0.234
0.273 0.226 0.258 0.222 0.258 0.240 0.271 0.237 0.296 0.227 0.259
Weather
336
0.267
0.294 0.265 0.290
0.265
0.311
0.284
0.306 0.28 I 0.299 0.281 1 0.299 0.292 0.307 0.282 0.332 0,266 0.294
720
0.342
0.344
0.342
0.340
0.339 0.347
0.358
0.352 0.359 0.350 0.356 0.348 0.364 0.353 0.347 0.385 0.346 0.347
Avg
(0.247 0.276
0.245
0.272 0.251
0.288
0.262
0.288 0.260 0.281 00.259 0.281 0.272 0.291 0.266 0.318 0.253 0,280
96
10.162 0.253
0.147
0.245
0.1
0.249
0.1
0.271 0.8 0.240 0.180 0.272 0.201 0.281 00.210 0.302 0.157 0,249
192
0.173
0.264
0.160 0.250
0.193
0.288 0.186
0.288
0.165 0.256 0.188 0.279 0.201 0.283 0.210 0.305 0.170 0.261
336
0.194 0.285
0.3
0.267
0.213
0.304
0.203
0.304
0.179 0.271 0.204 0.296 0.215 0.298 0.223 0.319 0.186 0.276
720
0.220 0.304
0.210 0.309
0.255 0.336
0.227
0.322 0.209 0.298 0.246 0.328 0.257 0.331 0.258 0.350 0.227 0.311
Avg
10.187 0.276 0.173 0.268
0.204
0.294
0.196
0.296 0.175 0.267 0.204 0.294 0.219 0.298 0.225 0,319 0.185 0,274
96
0.442
0,280 0.430
0.294
0.420
0.294
0,589
0.316
0.393
0,268 0.461 0.298 0.649 0.389 0.696 0.429 0.479 0,299
192
0.444
0.287 0.452 0.307 0.446
0.306
0.616
0.328 0.413 0.277 0.467 0.301 0.601 0.366 0.647 0.407 0.490 0.303
Traffic
336
0.458
0.293 0.470 0.316 0.491 0.315 0.628 0.333 0.424 0,283 0.483 0.308 0.609 0.369 0.653 0.410 0.493 0,304
720
0.492
0.303 0. 0.498 0.323
0.526 0.330
0.667
0.352 0.458 0.300
0.517 0.325 0. 0.647 0.387 0.695 0,429 0.534 0.319
Avg
(0.457 0.291 (0.463 0.310 0.468 0.312
0.625 0.332 0.442
0.282 0.482 0.308 0.627 0.378 0.673 0.419 0.499 0.306
96
10.084
0.201 0.091 0.211 0.090 0.207
1. 0.246
0.094
0.216 0.088 0.205 0.093 0.217 0.093 0.226 0.091 0,210
Exchange
192
0.174
0.295 0.186 0.305 0.190 0.316 0.213 0.335 0.185 0.307 0.189 0.309 0.184 0.307 0.184 0.324 0.185 0.304
336
0.331
0.416 0.380
0.449 0.354 0.419 0.367
0.440
0.336 0.422 0,327 0.415 0.351 0.432 0.328 0.436 0.361 0.435
720
0.847
0.694 0.896 0.7 0. 0.892 0.7 16 0.978 0.753
0.893 0.716 0.886 0.706 0.886 0.714 0.880 0.705 0.974 0.741
Avg
(0.359
0.401| (0.388 0.4
1.381 0.415| 0.418
0.443
0.377
0.415 0.373 0.409
0.379 0.418 0.371 0.423 0.403 0.423
Average 0.336 0.349 0.346 0.356
0.349 0.363
0.388 0.375
0.344
0.357
0.354 0.361 0.380 0.371
0.413 0.407 0.355 0.357
45

---

Table 21: Full results for the long-ter term forecasting task compared with Autoformer 775 Crossformer
[79]. FEDformer [77]. ETSforemr [76). Stationary [78). LighTTS [88] Informer 80 (Stationary means Nonstationary Transformer. * means former) To ensure fairness in the comparison, we set the look-back window length of all models to 96. The standard deviation is within 0.5%. Red: best, Blue: second best.
SymTTmme
Autoformer
Cross
FED*
ETS
Stationary
LightTS
In
Methods
(Ours)
75
78
80
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.318 0.353
0.501
0.479
0.360
0.399
0.378
0.418
0.375 0.398
0.4188
0,415
0.390
0,411
0.619
0,549
192
0.362
0.380
0.578
0.510
0.422
0.449
0.438
0.449
0.408
0.410
0.506
0.454
0.425
0.436
0.760
0.645
336
0,386
0.402
0.668
0,552
0,589
0.557
0.456
0.462
0.435
0.428
0,530
0.482
0.463
0.464
1.093
0.812
720
0.419
0.423
0.602
0.524
0.838
0.706
0.530
0.498
0.499
0.462
0.610
0.525
0.547
0.520
1.114
0.806
Avg
0..31
0.390
0.587
0,516
0.552
0.528
0.450
0.457
0.429 0.425
00516
0.469
0.456
0.458
0.896
0.703
96
0.174
0.257
0.245
0.323
0.274
0.268
0.6
0.284 0.189 0.280
0.240
0.308
0.226
0.323
0.467
0.533
192
0.238
0.299
0.289
0.345
0.366
0.380
0.264
0.325 0.275 0.319
0.428
0.402
0.361
0.421
0.742
0.664
336
0.295
0.337
0.342
0.378
0.437
0.453
0.324
0.363
0.314
0.357
0.521
0.449
0.474
0.488
1.184
0.825
720
0.390
0.392
0:441
0.429
0.506
0.623
0.434
0.428
0.414
0.413
0.602
0.501
0.760
0.631
4.039
1.530
Avg
0.274
0.321
0.329
0.369
0.396
0.431
0,305
0.350
0,298
0,342
00.448
0.415
0.455
0.466
1.608
0.888
96
0.376
0.400
0.453
0.459
0.462
0.473
0.376
0.417
0.494
0.479
0.550
0.503
0.448
0.450
0.926
0.741
192
0.428
0.431
0.481
0.470
0.495
0.484
0.431
0.454
0.538
0.504
0.655
0.569
0.503
0.483
0.968
0.757
336
0.463
0.456
0.519
0.495
0.693
0.626
0.461
0.469
0.574
0.521
0.791
0.639
0.554
0,513
1.144 0.849
720
0.450
0.458
0.510
0.508
0.668
0.599
0.502
0.499
0.562
0.535
0.797
0.652
0.627
0.578
1.214 0.880
Avg
0..430
0.436
0..49
0.483
0.580
0.545
0.442
0.460
0,542
0.510
00.698
0.591
0.533
0.506
1.063 0.807
96
0.299
0.347
0.383
0.416
0.367
0.347
0.346
0.390 0.340
0.391
00.417
0.432
0.417
0.448
3.132
1.425
192
0.364
0.397
0.479
0.467
0.450
0.459
0.428
0.439 0.430
0.439
0.529
0.486
0.546
0.520
5.552
1.957
hh
336
0.385
0.423
0.476
0.481
0.532
0.521
0.469
0.474
0.485
0.479
0.591
0.517
0.619
0.554
4.926
1.873
720
0.420
0.441
0.503 00.494
0.614
0.633
0.473
0.486 0.500
0.497
0.601
0.531
0.972
0.704
4.201
1.741
Avg
|0.365
0.402
0.458
0.467
0.491
0.490
0.429
0.447
0.439
0.452
0.534
0.491
0.639
0.556
4.453
1.749
96
00.166
0.213
0.276
0.343
0.174
0.243
0.218
0.299 0.197
0.281
0,184
0.233
0.174
0,235
0.357
0.415
192
0.212
0.254
0.305
0.361
0.235
0.307
0.281
0.344
0.237
0.312
0.248
0.286
0.218
0.276
0.458
0.456
Weather
336
0.267
0.294
0.372
0.405
0.277
0.342
0.337
0.375
0.298
0.353
0.337
0.349
0.267
0.316
0.520
0.501
720
0.342
0.344
0.430
0.437
0.369
0.407
0.423
0.429 0.352
0.288
0.399
0.385
0.353
0.366
0.926 0.705
Avg
0.2247
0.276
0.346
0.387
0.264
0.325
0.315
0.362
0.271 0.309
0,299
0.313
0.253
0.298
|0.65 0.519
96
10.162
0.253
0.198
0.313
0.146
0.249
0.202
0.314
0.187 0.304
0.1667
0.270
0.211
0.313
0.342
0.423
192
0.173
0.264
0.218
0.329
0.163
0.262
0.211
0.323 0.199 0.315
0.183
0.284
0.223
0.326
0.360
0.442
336
0.194
0.285
0.253
0.352
0.198
0.296
0.222
0.335
0,212
0.329
0.194
0.295
0.243
0.346
0.365
0.445
720
0.220
0.304
0.2655
0.367
0.245
0.346
0.272
0.373 0.233 0.345
0.224
0.321
0.277
0.371
0.412 0.469
Avg
0..187
0.276
0.223
0.340
0.188
0.288
0.227
0.337 0,208 0.323
0.199
0.292
0.239
0.339
0.370
0.445
96
00.432
0.280
0.608
0.383
0.516
0.268
0.592
0.372 0.607
0.342
0.621
0.347
0.667
0.419
0.720
0.407
192
0.444
0.287
0.630
0.397
0.541
0.283
0.598
0.371 0.621 0.399
0.643
0.355
0.662
0.425
0.738
0.414
Trallic
336
0.458
0.293
0.622
0.387
0.566
0.351
0.636
0.397 0.622
0.396
0.650
0.360
0.683
0.436
0.833
0.470
720
0.492
0.303
0.689
0.396
0.610
0.403
0.639
0.395 0.632
0.396
0.670
0.365
0.
0.700
0.455
0.854
0.491
Avg
10.457
0.291
0.637
0.391
0.558
0.326
0.616
0.384 0.621 0.396
00.646
0.357
0.678
0.434
0.786
0.445
96
10.084
0.201
0.119
0.318
0.276
0,383
0.162
0.291 0.085
0,204
0..
0.254
0.128
0,266
0.896
0.761
192
0.174
0.295
0.315
0.407
0.540
0.552
0.276
0.382
0.182 0.303
0.251
0.361
0.292
0.402
1.146
0.861
Exchange
336
0.331
0.416
0.480
0.519
1.229
0.873
0.442
0.488
0.348
0.428
0.467
0.507
0.500
0.536
1.628
1.017
720
0.847
0.694
1.255
0.868
1.721
1.055
1.175
0.833 1.025 0.774
1.304 0.837
1.002
0.763
2.552
1.299
Avg
|0.359
0.401
0.560
0.528
0.942
0.716
0.514
0.498
0.410
0.427
|0.538
0.490
0.480
0.492
1.555 0.984
Average 0.336
0.349
0.455
0.435
0.496
0.456
0.456
0.412
0.402 0.398
(0.483 0.427
0.467 0.444
1412 0.818
46

---

Table 22: Full results for the long-term forecasting task compared with PatchTST [54
TimeMixer
89
TimesNet (22. Autoformer 775,, DLinear [871, iTransformer 744, TimeXer 62] FEDformer
( means former, PTST is PatchTST, TMixer is TimeMixer.) To ensure faimess in the
comparison, we set the look-back window length of all models to 336. The standard deviation is
within 0.5%. Red: best, Blue: second best.
SymTime (Our)
PIST
TMixer
TimesNet
Auto+
DLinear
iTrans*
TimeXer
FED
Methods
54
89
72
75
87
74
66
77
Metries
MSE MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
96
0.295 0.346
0.292 0.343
0.322 0.358
0.344 0.378
0.514 0.506 0.300
0.344
0.305 0.359
0.318 0.361
0.382 0,427
192
0.328
0.365
0.331 0.369 0.342 0.375
0.456
0.426 0.576 0.520
0.335 0,365
0.345 0.382
0.354
0.383
0.393 0.434
336
0.366
0.395
0.365
0.392 0.371 0.395
0.426
0.432 0.703 0,564
0.369
0,386
0.377 0.401
0.381
0.402
0.445 0.459
720
0.411
0.422
01.40
0.425
0.437
0.440
0.459
0.455
0.678 0.568
0.425
0.420
0.444
0.439
0.434
0.433
0.543 0.490
Avg
00.350
0.382
01352
0.382 0.368 0.392
0.421 0.423 0.618 0.539
0.357 0.379 0.366 0.395
0.372
0.395
(0.441 0.452
96
0.165
0.255
0.165
0.255 0.116 0.259
08144 0.272 0.349
0.400
0.169 0.266
0.1174 0.265
0.169
0.254
0.260 0.336
192
0.221
0.293
0,220 0.292 0.232 0.298 0.243 0.309 0.507 0.474 0.235 0.316 0.247 0.313 0.237
0.302 0,291 0.354
m2
336
0.275
0.329
0.278
0.329 0.280 0.329
0.310
0.351 0.328 0.375 0.305 0.366 0,294 0.345 0.284
0.333
0.325 0.366
720
0.365
0.387
0.368
0.385 00.359 0.387
0.393 0.405
0.418 0.433
0.457 0.463
0.374 0.394
0.360
0.381
0.423 0.451
Avg
00.566
0.316
0.258
0.315 0.262
0.318
0.282 0.334 0,400 0.420
0.420|0.291 0.2291 0.353 0.272 0.329 (.262
0.317
0..17/0.335
0.325 0.377
96
0.372
0.399
0.382
0.405 00.399 0.403
0.423
0.437 0.536
0.498
0.375
0.399 0.397 0.416 0.403
0.416/0.403
0.421 0.387 0.434
192
0.409
0.427
0.414 0.421 0,415 0.423
0.481 0.481 0.562 0.533 0.413 0.424 0.442 0.448 0.440
0.440
0.431 0.458
336
0.430
0.440
0,431 0.435 0.453 0,449
0.488
0.477 0.551 0,533 0.438
0,444 0,455 0.459 0.495 0,488 0.471 0,480
720
0.440
0.463
00 0.449 0.466
0.473
0.473
0.548
0.523 0.670 0. 0.590
0.590/0.475
0.475 0.495 0.500 0.506
0.633 0.583
0.512 0.516
Avg
00.413
0.432
0.449 0.432 0.430
0.437|
0.445
0.480
0.580 0.539
0..425 0.440 0.445 0.457
0.457|0.493
0.493
0.483
0.445 0.472
96
0.271
0.341 0.274
0.336 0.284
0.350
0.378
0.421 0.509
0.527 0.307
0.370
0.307
0.363 0.313
0.365
0.394 0.457
192
0.334
0.378
0.339
0.379 0.359 0.397
0.409
0.439 0.711 0.64] 0.402
0.431 0.393 0.413 0.375
0.404 0.426 0,456
ET
336
0.359
0.403
0.331 0.381 0.388 0.422
0.410
0.439
0.574
0.569
0.489
0.485 0.428 0.437 0.400
0.429
0.420 0.461
720
0.398
0.436
0.379
0.422
0.555 0.534
0.440 0.461
0.856
0.679
0.761 0.620
0.433 0.452
0.412
0.443
0.478 0.495
Avg
0.341
0.390
0.331
0.379 00.396
0.425|
0.409 0.440
006663
0.604/0.490
0.604
0..49
0.476 0.399 0.416
0.416/0.375
0.375
0.410
0.430 0.467
96
0.149
0. 0.227 0.273 0.5 0.224
0. 0.228
0.266
0.343
0.1 0.234
0.162 0.210
0.1
0.203
0.211 0.296
192
0.192
0.239 0,200 0.245 0.196 0.243 0.214 0.263
0.431
0.465
0.217 0.276 0,207 0.251 0.243
0,265 0.288 0.342
Wenther
336
0.245
0.282 0,259 0.298 0.2443 0.280
0.272 0.301
0.559
0.498
0.262
0.313 0.257 0.291
0.322
0.318
0.340 0,382
720
0.321
0.337
01346 0.353
0.325 0. 0.346
01343 0.353
0.506
0.495
0.328
0.370
00.227 0.337
0.414
0.373
0.405 0.430
Avg
(0.238
0.273
0.258 0.292 00.35
0.273
0.250 0.286
00.441
0.450|0.245
0.298 (0.238 0.272
0.272|0.287
0.2877
0.290
0.313 0.363
96
0.133
0.230
0. 1.1 0.361
0.1444 0.244
1.4 0.278 0..00 0.322 0.7 0.249
0..122 0.227
0.155 0.235
0.193 0.308
192
0.150
0.244
154 0.251 0.1552
0.242
0.2 0.292 0.219 0.333 66 0.261 0.1153 0.248
0.161 0.281 0.201 0.315
336
0.163
0.262
0.164 0.262
0.172
0.261
0.199 0.299 0.227 0.340
0.176 0.278 0..173 0.267
0.187
0.279
0.214 0.329
720
0.208
0.298
0.210 0.301 00.207 0.293
0.222 0.318 0.299 0.390
0.196 0.288
0.194 0.287
0. 0.183 0,273
0.246 0.355
Avg
(0.164 0.258
0.165 0.294 60.16 0.260
0.7 0.297 00.36
0.346|0.
0.1 0.269 0.163 0.257
00.
0.267
0.2114 0.327
96
0.361
0.257|
0.365
0.256 00.371 0.256
0.589
0.321 00.755
0.414
0.441
0.307
0.367 0.278
0.442
0.268
0.587 0.366
192
0.382
0.258
0.383
0.258 00.401 0.271 0.605
0.322 0.672
0.411 0.443
0.312 0.414 0.284
0.433 0.280
0.604 0.373
Traffic
336
0.395
0.271
0.398
0.271 00.408
0.267
0.621
0.338
0.667
0.408
0.456
0.319 0.399 0.280
0.454
0,278
0.621 0.383
720
0.424
0.281
0.438
0.288
0.464
0.292
0.647
0.344
0.689
0.421
0.528
0.343
0..422 0.290
0.498
0.299
0.626 0.382
Avg
(0.391
0.267
0.396
0.268 00.411 0.27
0.615
0.331 (0.666
0.413|
00.665
0.320
(0.401 0.283
00.452
0,281 0.610 0.376
96
0.085
0.204
0.093
0.214
0.092
0.217|
0.218
0.343
0.967
0.778 0.099
0.235 (0.099 0.224
0.234
0.312
0. 06168 0.285
Exchange
192
0.180
0.301
0.200
0.321 0.235 0.345
0.299
0.411 0.931
0.750 0.195 0.335 0.215 0.338
0.283
0.402
0.186 0.296
336
0.335
0.419
0.373
0.448 00.770 0.441
0.468
0.527
1.051
0.809
0.380
0.474 0.3778 0.454
0.433
0.343
0.250 0.342
720
0.869
0.700
0.875 0.695 (0.963 0.750
1.208
0.847
1.261
0.893
1.120
0.805 0,876 0.690
0.688
0,942
0.899 0.784
Avg
(0.367
0.406|
0.385
0.420
(0.415
0.438
0.548
0.532
1.053
0.807
0.448
0.462
0.399 0.427
0.409
0.500
0.376 0.427
Average
0.315
0.341
|0.20 0.348
00.336
0.352
00.401 0.390
/0.583
0.515
00.36 0.375 0.347 0.355
00.35 0.368
(0.395 0.408
47

---

Table 23: Full results for the long-term forecasting task compared with PatchTST (34] TimeMixer 19], TimesNet (72), Autoformer [75). DLinear [87. iTTrasformer 774,, TimeXer 62). FITS 190]. (* means former. TMixer is TixeMixer.) To ensure fairness in the comparison, we set the look-back window length of all models to 512. The standard deviation is within 0.5%. Red: best, Blue: second best.
SymTime
PTt
TMixer
TimesNet
Auto
DLinear
iTrans
TimeXer
FITS 90
Methods
(Ours)
54
89
72
75
87
74
66
Metries
MSE MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.313 0.348
0.290 0.344
0.315 0.354
0.358
0.388
0.495 0.499
0.304 0.347
0.310 0.364
0.332 0.374
0.306 0.349
192
0.326
0.363
0.333 0.371
0.344 0.376
0.457 0.439
0.549
0.514
0.337 0.368 0.349 0.387
0.364
0.392
0.338 0.367
336
0.376
0.390
0.369
0.392
0,385
0.397
0.412 0.427
0.536
0,509
0.366
0,385 0.376 0.401
0.387
0.407
0.421 0.384
720
0.409
0.420
01.416
0.420
0.441 0.440
0.473 0.467
0.645
0.550.04224
0.550
0.424
0.422
0.434 0.436
0.430
0.432
0.432 0.437
Avg
0.356
0.380
0.352
0.382 0.371 0.392
0.425 0.430
0.556
0.518
0.358
0.380 0.367 0.397
0.378
0.401
|0.374 0.384
96
0.169
0,258
0.166
0.256 00.173 0.261
0.192
0.279 0.278
0.353
0.166
0.262 0.112 0.272 0.177 0.263
0.165 0.254
192
0.232
0.301 0.223 0.296 0.223 0.298
0.264 0.325 0.318 0,383 0.225 0.304 0.242 0.312 0.241 0.312
0.219 0.291
m'
336
0.287
0,333
0.274
0.329 0.291 0.342
0.320
0.364 0.411
0.444
0.299 0.361 0.299 0.346
0.294 0.340
0.272 0.326
720
0.372
0.387
0.362
0.385
(03366 0.39
0.402
0.410
0.476
0.484
0.412
0.432
0.378 0.396
0.382 0.399
0.359 0.391
Avg
(0.265
0.320
0.256
0.317 0.263 0.323
0.294 0.344 0.371
0.416|0.275
0.416
0.275 0.340 0.273 0.331
00.274
0.329|
0.254 0.313
96
0.372
0.389
0.370
0.400 0..80 0.408
0.442
0.465
0.556
0.534
0.368 0.397 0.394 0.420 0.397
0.424
0.332 0.396
192
0.402
0.424
0.413
0.429 0,431 0.444
0.473
0.475 0.568 0.538 0.400 0.417 0.440 0.444 0.438 0.452 0.405 0.415
336
0.433
0.445
0,422
0.440 0.468
0.472
0.505
0.501 0.606 0.573 0.430 0,442 0,447 0.459 0.466
0,472 0.440 0,468
720
0.448
0.469
0.447 0.468
0.445
0.453
0.505 0.504
0.777 0.669
0.669)0.476 0.476 0.497 0.511 0.516
0.516 0.597 0. 0.566
00.453 0.485
Avg
00.414
0.432
0.413
0.434 0.429 0.444|
0.448 0.486
0.627 0.579|0.415
0.438 00.446 0.460
0.479
00.418 0.441
96
0.290
0.352
0.273
0.337 0.298
0.364
0.336
0.391
0..447
0.486 0.288
0.355
0..317 0.367
0.299
0.358
0.291 0.353
192
0.376
0.409
0.341 0.382 0.361 0.399
0.393
0.425
0.587 0.568 0.394 0.427 0.388 0.411 0.370
0.408
0.350 0.395
336
0.382
0.416
0.398
0.458 0.393 0.421
0.406
0.445
0.689 0. 0.608
0.501 0.491 0..422 0.437
0.372
0.410
0.375 0.424
720
0.414
0.442
0.416
0.458
0.442 0.458
0.451
0.466
1.027
0.772
0.813 0.638
0.424 0.454
0.376
0.424
0.437 0.459
Avg
0.365
0.405|
0.357
0.409 00.733 0.410
0.397
0.432
0.6887
0.609|0.499
0.609|
0..99
0.478 0.388 0.417
00.354
0.400
0.363 0.408
96
0.159
0.205
0.230
0.282
06 0.225
168
0.224
0.375
0.426
0.1 0.230
0.5 0.223
0.5 0.209
172 0.226
192
0.203
0.260
0.194 0.242 0.191 0.242
0.217 0.266 0.471
0.491 0.213 0.269 0,213 0.256 0.246
0,267
0.216 0.262
Wenther
336
0.256
0.289
0.245
0.282 0.244 0.283
0.278
0.310
0.514
0.521
0.260
0.312 0.265 0.296
0.314
0,309
0.261 0.295
720
0.319
0.339
01.12 0.332
0.316
0.333
0.342
0.352
0.596
0.506
0.320
0.358
0..342 0.347
0.391 0.353
00.36 0.342
Avg
00.234
0.273
0.245 0.284 00.231
0.271
00.251 0.288 (0.489
0,486|0.241
0.486|
0..241
0.292 00.249 0.280
0.280|0.282
0.282 0,284
0..244 0.281
96
0.128
0.256
0. 12 0.224
0.7 0.229
0. 184 0.288
0..215 0.328
0.141 0.241 00.131 0.227 0.144 0.242
0. 145 0.244
192
0.149
0.244
0. 158 0.258 0.152 0.250
187 0.291 0.224 0.333
0.154 0.254 0.155 0.250
0.157 0.256
0.153 0,250
336
0.161
0.261
0.163 0.261 0.193
0.295 0.206 0.307 0.235
0.341 0.169
0.271 00.171 0.266
0.176
0.275
0.169 0.266
720
0.216 0.308
0.225 0.331 (..288 0.324 0.226 0.323
0.738 0.570
0.2000..200.
0.204 0.304
0.111 0.285
0..21 0.306
0.208 0.298
Avg
[0.163 0.267 j.. 169 0.269 0.1 0.274 0.2001 0.302 (0.353 0.393
0.167
0.267 00.162 0.257
00.17
0.270
101169 0.265
96
0.365
0.257 0.369 0.262 (0.368 0. 0.254 0.600 0.321 (0.697 0.428
0.412 0.294
00.351 0.257
0.448
0.271
0.339 0.277
192
0.379
0.256 0.379 0.253 0.399 0.268 0.612 0.328 0. 0.700 0.429 0.422 0.299 0..733 0.268
0.448
0.282
0.409 0.280
Traffic
336
0.401
0.275
0.410
0.280 0.404
0.264 0.631 0.338 0.707 0.437
0.431
0.304
0.386 0.274
0.473
0,289
0.418 0.285
720
0.433
0.284
0.438 0.291 0.467 0.293
0.654 0.352 0.718 0.445
0.466
0.325
0.422 0.294
0.516
0.307
0.456 0.306
Avg
(0.395
0.268
0.399
0.272 00.410 0.270 0.624 0.334 (0.705 0.435|
0.443
0.305 0..83 0.273
0..466
0.287
(..420 0.287
96
0.089
0.212
b. 0.095
0.220 0.096 0.220 0.238 0.366 00.17 0.623
0.020 0.260
0.. 0.267 0.099
0.223
lo. 1.000 0.225
Exchange
192
0.176
0.298
0.215
0.336 0.7 0.318 0..49 0.497 0.810 0.739
0.241 0.376 0.322 0.417 0.207
0.331 0.201 0.326
336
0.380
0.439
0.392
0.459 0.737 0.66 0.642 0.647 0.620 0.858 0.746
0.439 0.509 00.661 0.448 0.767 0.689
0.350 0.437
720
0.893
0.698
8.890
0.680 1.038 0.808 1.550 0.948 1.491 0.965
1.199 0.831 0.8888 0.736 0.984 0.780
0.920 0.728
Avg
00.384
0.412
0.398
0.423
(0.517 0.497 0.718 0.608
(0.944 0. 0.768|
00.500
0.494
0.442 0.467 00.514 0.506
(0.393 0.439
Average
0.322
0.345 0.324 0.349
/o.466
0..44 0.403
|0.591 0.525
(0.361 0.374
(0.377 0.360 0.364 0. 0.369
(0.329 0.352
48

---

Table 24: Full results for the short-term forecasting task in the M4 dataset compared with Peri-
midFormer (63] SPIP-LLL((S-LLMM [2], Time-L.LM(T-LLLM) [77. GPT4TS (2) TimeMixer [89].
PatchTST [54), iTransformer (74). TimesNet (72). DLinear [87). Informer 80]. (* means former.)
The standard deviation within 0.5%. Red: best, Blue: second best.
Methods
SymTime (Ours)
Peri-mid*
S-LLM
T-LLM GPT4TS TimeMixer PatchTST iTrans*
TimesNet DLincar
In
Metric
663
82
17
2
89
(54)
74
72
(87
80
Yearly
SMAPE
13:355
13.483
14.931
13,450
14.847
13.369
13.677
13.724
13.463
14.340
14.698
MASE
2.997
3,080
3.345
3.184
3.628
3.009
3.049
3.157
3.058
3.112
3.293
OWA
0.786
0.800
0.878
0.819
0.911
0.787
0.802
0.817
0.797
0,830
0.864
Quarterly
SMAPE
10.060
10.037
10.655
10.671
10.389
10.131
10.922
13.473
10.069
10,510
16.172
MASE
1.183
1.170
1.249
1.276
1.228
1.186
1.326
1.722
1.175
1.241
2.136
OWA
0.872
0.882
0.939
0.950
0.919
0.893
0.979
1.240
0.886
0.930
1.513
Monthly
SMAPE
12.608
12.795
13,012
13,416
12.907
12.762
14,200
13.674
12,760
13.382
15,446
MASE
0.925
0.948
0.973
1.045
0.954
0.940
1.111
1.068
0.947
1.007
1.247
OWA
0.872
0.889
0.909
0.957
0.896
0.884
1.015
0.976
0.887
0.937
1.122
Others
SMAPE
4.941
4,912
5.540
4.973
5,266
5,085
5,658
5.598
4,995
5.122
6.839
MASE
3.327
3.260
8.426
3,412
3.595
3,403
3.626
3.957
3.346
3.608
4.536
OWA
1.045
1,031
3.792
1.059
1,121
1.072
1.167
1,213
1.053
1,108
1.435
Average
SMAPE
11.785
11.897
12.514
12.584
12.367
11.885
12.866
13.233
11.888
12.500
15.018
MASE
1,584
1.607
1.726
1.763
1.767
1,598
1.734
1.850
1,607
1.678
2.096
OWA
0.849
0.859
0.913
0.915
0,918
0.856
0.928
0.972
0.858
0.899
1.102
Table 25: Full results for the short-term forecasting task in the M4 dataset compared with LightTS 88), Autoformer 775). Crossformer (79), FEDformer [77), ETSformer (76), Nonstationary Transformer
(Stationary) 778. FiLM [142), MICN [86). Reformer [143] Pyraformer [30]. The standard deviation
is within 0.5%. (* means former.) Red: best, Blue: second best.
Methods Metric
SymTime (Ours)
LighTTS
Auto
Cross
FED*
ETS
Stationary
FiLM
MICN
Rea
Pyra
88
79
(76
78
142
86
143
130
Yearly
SMAPE
13.355
13.444
17.764
79.308
13.508
18.009
13.717
14.076
14.537
13.752
14.594
MASE
2.997
3.022
3,919
18.692
3,051
4.487
3,078
3,017
3.380
3.088
3.269
OWA
0.786
0.792
1.037
4.778
0.797
1.115
0.807
0,810
0.871
0.809
0.858
Quarterly
SMAPE
10.060
10.252
13.968
74.943
10.706
13.376
10.958
10.711
11,408
10,900
1 11.654
MASE
1.833
1,183
1.754
13.133
1.263
1.906
1.325
1,292
1,384
1.316
1.392
OWA
0.872
0.897
1,274
8,191
0.947
1.302
0.981
0.957
1.022
0.975
1.037
Monthly
SMAPE
12,608
12,798
18.200
68.892
13.925
14.588
13.917
13.362
13.803
13.949
14.963
MASE
0.925
0.957
1,574
11.199
1.062
1.368
1.097
1.016
1.078
1.096
1.165
OWA
0.872
0.894
1.371
7.654
0.982
1.149
0.998
0.941
0.985
0.999
1,066
Others
SMAPE
4.941
5,324
6.738
176.164
4,888
7.267
6,302
5.387
6.090
6.611
5.605
MASE
3.327
3.410
4.853
116.723
3.244
5.240
4.064
3.670
4.203
4.492
3.966
OWA
1.045
1.098
1.474
36.941
1.026
1.591
1.304
1.146
1.304
1.404
1.215
Average
SMAPE
11.785
11.962
16.511
78.103
12.605
14.718
12.780
12.491
13.016
12.805
13.616
MASE
1.584
1.609
2.321
18.663
1.677
2.408
1.756
1.675
1.837
1.777
1.843
OWA
0.849
0.862
1.215
7.759
0.903
1.172
0.930
0.899
0.960
0.937
0,984
49

---

Table 26: Full results for time series classification task compared with (1) classical methods: DTW
144
XGBoost [145], Rocket [4]; (2) ENN-based methods: LSTM [146), LSTNet [T04) LSSL
(3) CNN-based methods: Inception (InTime) [85) TCN [48], TimesNet 72,, TSLANet
We report the classification accuracy (%) as the result. Red: best, Blue: second best. The
standard deviation is within 1%.
Classical Methods
RNN-based
CNN-based
Dutasets Methods
DTW
XGBoost
Rocket
LSTM
LSTNet
LSSL
InTime
TCN
TimesNet
TSLANet
SymTime
144
145
84
146
[04]
147
85
148
72
83
(Ours)
EthanolConcentration
32.3
43.7
45.2
32.3
39.9
31.1
39.1
28.9
35.7
30.4
37,3
FaceDetection
52.9
63.3
64.7
57.7
65.7
66.7
65.4
52.8
68.6
66.7
69.2
Handwriting
28.6
15.8
58.8
15.2
25.8
24.6
46.9
53.3
32.1
57.9
36.7
Heartbeat
71.7
73.2
75.6
72.2
77.1
72.7
75.2
75.6
78.0
77.5
74.1
Japanese Vowels
94.9
86.5
96.2
79.7
98.1
98.4
95.1
98.9
98.4
95.1
98.1
PEMS-SF
71.1
98.3
75.1
39.9
86.7
86.1
79.6
68.8
89.6
83.8
97.1
SelfRegulationSCPII
77.7
84.6
90.8
68.9
84.0
90.8
87.2
84.6
91.8
91.8
89.8
SelfRegulationSCP2
53.9
48.9
53.3
46.6
52.8
52.2
53.6
55.6
57.2
53.3
58.9
SpokenArahicDiiitts
96.3
69.6
71.2
31.9
100.0
100.0
96.3
95.6
99.0
98.0
98.9
UWaveGestureLibrarry
90.3
75.9
94.4
41.2
87.8
85.9
92.4
88.4
85.3
89.4
89.4
Average Accuracy
67.0
66.0
72.5
48.6
71.8
70.9
73.1
70.3
73.6
74.4
74.9
Table 27: Full reuslts for time series classification task compared with (1) Transformer-based methods:
Autoformer [75). FEDformer (77), ETSformer (76), Informer [80] iTransformer (74), PatchTST
(Patch) (54). GPT4TS (GPT) 2 UniTS [92], Peri-midformer 66] and (2) MLP-based methods:
DLinear 887. LightTS [8]. We report the classification accuracy (%) as the results. (* means
former.) Red: best, Blue: second best. The standard deviation is within 1%.
Transformer-busee
MLP-based
Dutasets/ / Methods
Auto
FED*
ETS
In
iTrans*
Patch
GPT
UniTS
Peri-mid
DLincar
LightTS SynTime
77
76
80
74
54
92
63
87
88
(Ours)
EthanolConccntration
31.6
31.2
28.1
31.6
27.0
29.6
34.2
37.3
47.3
32.6
29.7
37.3
FaceDetection
68.4
66.0
66.3
67.0
67.0
67.8
69.2
67.5
68.7
68.0
67.5
69.2
Handwriting
36.7
28.0
32.5
32.8
27.2
23.2
32.7
27.0
31.5
27.0
26.1
36.7
Heartheat
74.6
73.7
71.2
80.5
75.6
75.7
77.2
80.5
86.3
75.1
75.1
74.1
JapaneseVowelk
96.2
98.4
95.9
98.9
97.6
94.0
98.6
97.8
96.8
96.2
96.2
98.1
PEMS-SF
82.7
80.9
86.0
81.5
85.5
80.9
87.9
93.1
88.2
75.1
88.4
97.1
SellRegulationSCPI
84.0
88.7
89.6
90.1
92.2
82.2
87.2
89.6
87.4
87.3
89.8
89.8
SellRegulationSCCP
50.6
54.4
55.0
53.3
54.4
53.6
59.4
61.1
55.4
50.5
51.1
58.9
SpokenArabicDDiits
100.0
100.0
100.0
100.0
98.0
98.0
95.2
98.9
98.0
81.4
100.0
98.9
WaaeeGstuurLLbbrryy
85,9
85.3
85.0
85.6
85.9
81.7
85.1
87.8
84.3
82.1
80.3
89,4
50

---

Table 28: Full results for time series imputation task, where we randomly mask {12.5%, 25%, 37.5%, 50% J time points of length-96 time series compare the model performance under different missing degrees. We compare with GPT4TS 22. TimesNet [72), Peri-midFormer [63] Moment [93]. iTTransfommer (74), PatchTST
DLinear [87 in this table. (* means former.) The standard
deviation is within 0.5%. Red: best, Blue: second best.
SymTime
GPT4TS
TimesNet
Peri-mid
Moment
iTrns
PatchTST
DLincar
Models
(Ours)
72
63
93
74
S4
87
Mask Ratio
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
12.5%
0.032
0.110
0.018
0.090
0.019
0.091
0.032
0.109
0.069 0.170
0.046
0.147
0.045
0.137
0.056
0.162
25%
0.034
0.113
0.024
0.102 0.024 0.101
0.034
0.112 0.071 0.169
0.060
0.171
0.046
0.139
0.077 0.191
37.5%
0.037
0.118
0.029
0.111 0.029 0.112
0.037
0.117 0.069 0.163
0.077
0.195
0.049
0.143
0.100
0.218
50%
0.041
0.126
0.042
0.132
0.036
0.124
0.042
0.126
0.086
0.169
0.104
0.228
0.055
0.152
0.129 0.247
Avg
0.006
06116
0.028
0.109
0.027
0.107
0.036
0.1
0.074 0.168
0.072
0.185
0.0499
0.143
0.090 0.204
12.5%
0.024
0.084
0.018
0.081
0.019
0.081
0.023
0.081
0.032
0.108
0.052
0.151
0.026
0.094
0.067 0.171
25%
0.024
0.086
0.021
0.082 0.021 0.086
0.024
0.084 0.029 0.105
0.070
0.179
0.028
0.099
0.089 0.200
Er1m2
37.5%
0.027
0.089
0.023
0.090
0.023 0.091
0.026
0.089
0.032
0.109
0.091
0.204
0.031
0.104
0.1 0.226
50%
0.030
0.093
0.027
0.098
0.026
0.098
0.030
0.095
0.031
0.110
0.117
0.232
0.034
0.109
0.140 0.253
Avg
|.0026
0.088
0.022
0.088 0.022 0.089
0.026
0.087 0.031 0.108
0.082
0.191
0.030
0.101
0.102 0.212
12.5%
0.074
0.179
0.063
0.171
0.062
0.169
0.069
0.173
0.160
0.239
0.098
0.220
0.097
0.203
0.111 0.232
25%
0.082
0.190
0.080
0.190
0.081 0.191
0.079
0.185 0.142 0.238
0.125
0.249
0.115
0.221
0.149 0.269
37.5%
0.100
0.205
0.107
0.218
0.098 0.210
0.096
0.202
0.121 0.228
0.156
0.278
0.134
0.239
0.187 0.301
50%
0.123
0.230
0.121
0.221
0.116
0.227
0.122
0.226
0.132
0.231
0.213
0.327
0.160
0.260
0.229 0.332
Avg
10.095
0.201
0.093
0,200
0.089
0.199
0.091
0.196 0.139 0.234
0.148
0.269
0.126
0.231
0.169 0.283
12.5%
0.051
0.138
0.041
0.129
0.040
0.132
0.051
0.139
0.051
0.150
0.095
0.210
0.058
0.153
0.109 0.223
25%
0.055
0.146
0.046
0.138
0.047
0.144
0.054
0.142 0.079
0.177
0.120
0.239
0.063
0.160
0.146 0.260
ETTh2
37.5%
0.059
0.152
0.060
0.160
0.054
0.154
0.058
0.148 0.056
0.155
0.149
0.266
0.068
0.167
0.180 0.290
50%
0.064
0.157
0.061
0.160
0.061
0.164
0.064
0.159
0.056
0.154
0.192
0.302
0.074
0.175
0.217 0.319
Avg
1.0088
0.148
0.052
0.147 0.050
0.148
0.057
0.147
0.061
0.159
0.139
0.254
[0.066
0.164
0.1 0.273
12.5%
0.037
0.122
0.080
0.195
0.088
0,203
0.047
0,140
0.095
0.211
0.073
0.190
0.061
0.170
0.084 0.206
25%
0.046
0.139
0.089
0.205 0.092 0.208
0.053
0.162 0.093 0.211
0.090
0.214
0.072
0.185
0.113 0.243
BCL
37.5%
0.060
10
0.094
0.217 0.096
0.214
0.067
0.179 0.094 0.211
0.107
0.235
0.082
0.198
0.141 0.273
50%
0.075
0.181
0.108
0.231
0.102
0.221
0.085
0,195
0.092 0.210
0.127
0.257
0.097
0.216
0.173 0.303
Avg
0.054
0.151
0.093
0.212 0.094 0.211
0.063
0.169 0.094 0.211
0.099
0.224
0.078
0.192
0.128 0.256
12.5%
0.005
0.035
0.026
0.047
0.026
0.049
0.025
0.037
0.033
0.073
0.038
0.087
0.028
0.049
0.039 0.091
25%
0.027
0.037
0.030
0.055 0.030 0.056
0.026
0.037 0.036 0.078
0.046
0.106
0.032
0.055
0.049 0.112
Weather
37.5%
0.029
0.039
0.033
0.061 0.032
0.058
0.029
0.041 0.034 0.075
0.055
0.122
0.035
0.059
0.057
0.125
50%
0.032
0.042
0.039
0.070
0.034
0.062
0.034
0.048
0.035
0.075
0.068
0.142
0.039
0.064
0.067 0.139
Avg
0.028
0.038
0.032
0.058
0.030
0.056
0.029
0.041
0.035
0.075
0.052
0.114
0.033
0.057
0.053 0.116
Average
0.049
0.124
0.053
0.136
0.052
0.135
0.050
0.126
0.072
0.159
0.099
0.206
0.064
0.148
0.118 0.224
51

---

Table 29: Full results for time series imputation task, where we randomly mask {12.5%, 25%, 37.5%, 50%] } time points of length-96 time series compare the model performance under different missing degrees. We compare with Stationary [78]. LighTT'S 88) ETSformer (76), FEDformer [77]. Informer 800, Reformer [143] and Pyraformer [30 this table. (Stationary means Nonstationary Transformer.) The standard deviation is within 0.5%. Red: best, Blue: second best.
SymTime
Stationary
LightTS
ETStormer
FEDformer
Informer
Reformer
Pyraformer
Methods
(Ours)
78
88
76
80
143
130
Mask Ratio
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
12.5%
0.032
0.110
0.026
0.107
0.054 0.158
0.034 0.130 0.068 0.188
0.027
0.115
0.032
0.126
0.670 0.541
25%
0.034
0.113
0.032
0.119 0.061 0.173
0.053 0.162 0.097 0.230
0.040
0.140
0.042
0.146
0.689 0.553
37.5%
0.037
0.118
0.039
0.131 0.073 0.189
0.082
0.201 0.134 0.287
0.071
0.189
0.063
0.182
0.737
0.581
50%
0.041
0.126
0.047
0.145
0.086
0.207
0.130
0.257
0.188
0.323
0.091
0.208
0.082
0.208
0.770 0.605
Avg
0.036
0.116
0.036
0.126
0.068 08182
0.075
0.187 0.121 0.257
0.057
0.163
0.055
0.166
0.717 0.570
12.5%
0.024
0.084
0.021
0.088
0.051
0.150
0.061
0.169
0.109
0.239
0.196
0.326
0.108
0.228
0.394 0.470
25%
0.024
0.086
0.024
0.096 0.069 0.176
0.093
0,214 0.166 0.295
0.295
0.414
0.136
0.262
0.421 0.482
Erlm2
37.5%
0.027
0.089
0.027
0.103 0.074
0.185
0.137
0,253 0.237 0.356
0,155
0.293
0.175
0.300
0.478 0.521
50%
0.030
0.093
0.030
0.108
0.078
0.192
0.237
0.332
0.323
0.412
0,214
0.325
0.211
0.329
0.568 0,560
Avg
0.026
0.088
0.026
0.099 0.068
0.176
0.132
0.242
0.209 0.326
0.215
0.340
0.157
0.280
0.465 0.508
12.5%
0.074
0.179
0.060
0.165
0.119
0.239
0.073
0.195
0.126
0.265
0.068
0.187
0.074
0.194
0.857 0.609
25%
0.082
0.190
0.080
0.189 0.144 0.266
0.105
0.234 0.169
0.305
0.096
0.220
0.102
0.227
0.829 0.672
37.5%
0.100
0.205
0.102
0.212
0.171 0.292
0.144
0.276 0.220
0.348
0.128
0.253
0.135
0.261
0.830 0.675
50%
0.123
0.230
0.133
0.240
0.201
0.317
0.200
0.327
0.298
0.403
0.166
0.287
0.179
0.298
0.854 0.691
Avg
00.095
0.201
0.094
0.201
0.159 0.278
0.130
0.258
0.204
0.330
0.115
0.237
0.112
0.245
0.842 0.682
12.5%
0.051
0.138
0.042
0.133
0.094
0.208
0.134
0.251
0.187
0.319
0.271
0.384
0.163
0.289
0.976 0.754
25%
0.055
0.146
0.049
0.147
0.140 0.255
0.180
0.294 0.279 0.396
0.362
0.450
0.206
0.331
1.037 0.774
ETTh2
37.5%
0.059
0.152
0.056
0.158
0.159 0.274
0.243
0,341 0.402
0.465
0.401
0.469
0.252
0.370
1.107 0.800
50%
0.064
0.157
0.065
0.170
0.180
0.293
0.353
0,408
0.604
0.504
0.437
0.487
0.316
0.419
1.193 0.838
Avg
0.0588
0.148
0.053
0.152 0.143 0.258
0.228
0.324
0.368 0.421
0.368
0.448
00.234
0.352
1.079 0.792
12.5%
0.037
0.122
0.093
0.210 0.077 0.198
0.185
0.323
0.197
0.324
0.152
0.279
0.190
0.308
0.297 0.383
25%
0.046
0.139
0.097
0.214 0.099 0.228
0.207
0.340 0.208 0.345
0.166
0.290
0.197
0.312
0.294 0.380
BCL
37.5%
0.060
101
0.
101
0.220 0.120 0.252
0.226
0.355 0.219 0.337
0.178
0.297
0.203
0.3
0.296 0.381
50%
0.075
0.181
0.108
0.228
0.138
0.272
0.251
0.372
0.235
0.357
0.189
0.305
0.210
0.319
0.299 0.383
Avg
0.049
0.151
0.100
0.218 0.108 0.238
0.217
0.347
0.215 0.341
0.171
0.293
0.200
0.313
0.297 0.382
12.5%
0.005
0.035
0.027
0.051
0.039
0.092
0.042
0.103
0.057
0.141
0.040
0.108
0.031
0.076
0.140 0.220
25%
0.027
0.037
0.029
0.056
0.045
0.105
0.056
0.131 0.066
0.155
0.045
0.130
0.035
0.082
0.147 0.229
Weather
37.5%
0.029
0.039
0.033
0.062
0.049
0.110
0.081
0.180 0.083
0.180
0.049
0.101
0.040
0.091
0.156 0.240
50%
0.032
0.042
0.037
0.068
0.054
0.117
0.102
0.207
0.103
0.207
0.054
0.114
0.046
0.099
0.164 0.249
Avg
0.028
0.038
0.032
0.059
0.047
0.106
0.071
0.155
0.077
0.171
0.047
0.113
0.038
0.087
0.152 0.235
Average
0.049
0.124
0.057
0.143
0.099
0.206
0.142
0.252
0.199 0.308
0.162
0.265
0.134
0.241
0.592 0.528
52

---

Table 30: Full reuslts for time series anomaly detection task, where P. and Fl represent the precision, recall and Fl-score (%) respectively. Fl-score is the harmonic mean of precision and recall. higher value P. and indicates a better performance. We compare with: Transformer [127], Reformer [133]. Informer [80 Autoformer
[75.
Crossformer [79], i'Transformer 774,, Anomaly
8T] Stationary 78 DI Dnee [87, LighTTS 88
ETSformer [76]. FEDformer [77], PatchTST [4
TimesNet 72,, GPT4TS 22) Peri- Formme
63
UniTS [92], where Anomaly means the Anomaly
Transformer and Stationary means the Non-stationary Transformer. The standard deviation is within
I 1% Red: best, Blue: second best.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg
Metires
FI
R
P
FI
R
(%)
LSTM
146
78.52 65.47 71.41
78.04
86.
81.93
91.06
57.
70.48
78.06
91.72
84.34
69.24
99.53
81 16.6
77.97
LogTrans 4
83.46 70.13 76.21
73.05
87.37 79.57
89.15
57.59
69.97
68.67
97.32
80.52
63.06
98.00 76.7 74
76.60
Transforner 127
78.44 65.26 71.24
89.85
73.71 80.99
90.77
61.76 73.50 96.82
66.41
79.76
99.31
83.18 90.53
79.20
TCN 148
84.06 79:07 81.49
82.44 78.60 86.90
70.45
76.59
95.71 85.09
54.59
99.77 70.57
77.24
LSSL 147
78.51 65.32 71.31 77.55 88.18 82.53
89.43
53.43 66.90 79.05 93.72 85.76 66.02
92.93 77.20
76.74
Reformer 143
72.50 84.19 77.90
73.78 81.18
90.63
73.97
66.75 80.04
73 83.03 90.62
80.74
Informer 80
72.51 84.13 77.88 90.10
73.68 81.07
90.57
61.51 73.26 99.83
67.24 80.35 99.03 83.21 90.43
80.60
Autoformer
15
78.46 65.11 71.17 75.26
90.84
73.97
65.57 79.19
99.99 78.96 88 88.24
78.96
Crossformer 79
71.89 83.41 77.22 90.32
72.74 80.59
89.68
53.63 67.12 98.00 83.59 90.22
97.49
88.02 92.52
81.53
iTransformer
74
84.73 80.19
86.15
62.54
90.68
66.
93.05 92.64
97.99 92.03
81.38
Pyraformer
30
85.61 80.61 83.04 83.81 85.93 84.86
92.54
57.71 71.09 87.92 96.00 91.78 71.67 96.02 82.08
82.57
Anomaly 8
88.91 82.23 85,49 79.61
87.37 83.31
91.85
58.11 71.18 72.51 97.32 83.10
68.35
94.72 79.40
80.50
Stationary 78
78.51 87.98 82.97 86.86 68.63 76.68
V0.62
55.74 69.02 89.26 95.42 92.24
98.17
96.30 V7 77.23
83.63
DLincar 87
75.91 84.02 79.76 89.68
75.31 81.87
89.87
67.30
93.05 92.66
98.65
94.70 96.64
83.64
LightTS 88
87.10 78.42 82.53 82.40
75.78 78.95
92.58
55.27 69.21 91.98 94.72 93.33
98.37
95.97 97.15
84.23
ETSformer 76
87.44 79.23 83.13
84.93 85.03
92.25
69.50
80.36 84.91
99.31
85.28 91.76
82.87
FEDformer
77
72.82 81.68 76.99 90.72
75.41 82.36
90.47
58.10
70.76 99.95 65.55 19.18
99.98
81.92 90.05
79.46
PatchTST 54
87.26 82.14 84.62
70.96 78.70
90.64
68.82
80.94 94 85.72
98.84
93.47
82.79
TimesNet 22
88.07 80.97 84.37 88.83 74.68 81.14
89.99
56.02 69.05 91.99 93.24 92.61
98.46
95.70 97.06
84.85
GPT4TS
87.68 81.52 84.49 82.09 81.97 82.03
90.12
68.85
93.09 92.60
98.36
95.85 97.09
85.01
FedTADBench 149
87.31 80.06 83.53 77.66
69.37 84.09
90.49 57.44 70.27 90.63 84.43 87.42
97.67 94.41 96.01
84.26
PeFAD
88.64 82.05 85.22
87.31
19.89
73.66 88.71 89.78 88.73
96.93
84.60
InterFusion 151
84.06 83.52 83.78 84.83 78.49 81.54
90.66 58. 1 70.82, 96.76 78.45 86.65
833.6
83.45 83.53
8..26
Peri-midFormer 86.97 81.37 84.08 88.66 74.02 80.68
90.02
67.53
92.55 91.64
98.46
96.21
84.03
UniTS 92
82.42 84,99 83.69 91.32 73.04
81.16
90.58
62.55 74.00 92.60 92.42
92.51
98.45 96. 97.31
85.73
SynTime (Ours)
88.08 83.37 85.66 89.46 75.31
91.06
61.51 73.43 91.39 93.61
98.90 95.36 97.10
86.31
53

---

Table 31: The full fine-tuning results of the time series long-term forecasting task under different sizes of pre-training datasets. The brief results of this experiment are shown in Table Red: best, Blue: second best.
Methods
IB
10B
25B
50B
Datasets Morizon
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
0.335
0.376
0.324
0.355
0.319
0.353
0.321
0.358
0.318
0.353
192
0.404
0.382
0.363
0.385
0.374
0.382
0.369
0.383
0.362
0.380
336
0.419
0.443
0.392
0.410
0.391
0.412
0.394
0.407
0.386
0,402
ETTml
720
0.446
0.435
0.426
0.440
0,422
0,423
0.426
0.423
0.419
0.423
Avg
0.401
0.409
0,376
0.398
0.376
0.393
0.378
0.393
0.371
0.390
96
0.181
0.271
0.195
0.257
0.183
0.265
0.177
0.264
0.174
0.257
192
0.241
0.323
0.268
0.316
0.249
0.310
0.245
0.302
0.238
0.299
336
0.334
0.361
0,310
0,350
0.295
0.346
0.297
0.338
0.295
0.337
ETTm2
720
0.417
0.401
0.394
0.400
0.398
0.396
0.391
0.396
0.390
0.392
Avg
0.293
0.339
0.292
0.331
0.281
0.329
0.278
0.325
0.274
0.321
96
0.422
0.428
0.402
0.411
0.386
0.404
0.384
0.402
0.376
0.400
192
0.457
0.459
0.447
0.462
0.430
0.432
0.434
0.436
0.428
0.431
ETThi
336
0.523
0.494
0.489
0.467
0.473
0.463
0.468
0.457
0.463
0.456
720
0.547
0.515
0.507
0.495
0.486
0.478
0.481
0.479
0.450
0.458
Avg
0.487
0.474
0.461
0.459
0.444
0.444
0.434
0.438
0.430
0.436
96
0.298
0.350
0,302
0.355
0.295
0.357
0.301
0.352
0.293
0.348
0.373
0.403
0,364
0.416
0.369
0,405
0.369
0.403
0.364
0.397
336
0.401
0.434
0.458
0.442
0.387
0.424
0.389
0.426
0.385
0.423
ETTh2
720
0.433
0.461
0.490
0.464
0.454
0.447
0.427
0.438
0.420
0.441
Avg
0.376
0.412
0.403
0.419
0.376
0.408
0.371
0.405
0.365
0.402
0.185
0.221
0.173
0.219
0.170
0.218
0.175
0.217
0.166
0,213
192
0.217
0.279
0.223
0.259
0.232
0.263
0.221
0.263
0,212
0.254
336
0.276
0.304
0.285
0.294
0.266
0.296
0.271
0.297
0.267
0.294
Weather
720
0.353
0.353
0.347
0.357
0.332
0.340
0.345
0.350
0.342
0.344
Avg
0.257
0.289
0.257
0.282
0.250
0.279
0.253
0.282
0.247
0.276
0.162
0.254
0.168
0.263
0.166
0.262
0.175
0.272
0.162
0.253
192
0.179
0.268
0.182
0.270
0.173
0.269
0.183
0.276
0.173
0.264
336
0.217
0.288
0.196
0.286
0.209
0.300
0.205
0.295
0.194
0.285
ECL
720
0.216
0.324
0.250
0.321
0.236
0.312
0.216
0.308
0.220
0.304
Avg
0.193
0.284
0.199
0.285
0.196
0.286
0.195
0.288
0.187
0.276
96
0.460
0.288
0.451
0.289
0.438
0.293
0.442
0.282
0.432
0.280
192
0.452
0.287
0.461
0.291
0.450
0.292
0.452
0.289
0.444
0.287
Traffic
336
0.461
0.330
0.469
0.296
0.472
0.294
0.467
0.296
0.458
0.293
720
0.510
0.336
0.510
0.336
0.508
0.314
0.502
0.309
0.492
0.303
Avg
0.471
0.310
0.473
0.303
0.473
0.294
0.467
0.299
0.457
0.291
96
0.120
0.235
0.087
0.203
0.098
0.208
0.087
0.205
0.084
0.201
192
0.188
0.298
0.192
0.309
0.186
0.301
0.177
0.297
0.174
0.295
Exchange
336
0.346
0.428
0.341
0.423
0.337
0.420
0.326
0.414
0.331
0.416
720
0.878
0.699
0.861
0.706
0.850
0.701
0.838
0.689
0.847
0.694
Avg
0.383
0.415
0.370
0.410
0.368
0.407
0.357
0.401
0.359
0.401
Average
0.358
0.366
0.354
0.361
0.345
0.355
0.342
0.354
0.336
0.349
54

---

Table 32: The full fine-tuning results of the time series short-term forecasting task under different sizes of pre-training datasets. The brief results of this experiment are shown in Table 2 Red: best, Blue: second best.
Methods
Metric
1B
10B
25B
50B
SMAPE
13.291
13.341
13.332
13.380
13.355
Yearly
MASE
2.981
2.986
2.985
3,012
2.997
OWA
0.782
0.784
0.783
0.788
0.786
SMAPE
10.270
10.274
10.197
10.228
10,060
Quartly
MASE
1.224
1,218
1.212
1.219
1.183
OWA
0.913
0.911
0.905
0.909
0.872
SMAPE
12.811
12.833
12.662
12.608
Monthly
MASE
1.053
0.955
0.959
0,932
0.925
OWA
0.964
0.893
0.896
0.877
0.872
SMAPE
5.186
5.070
5,003
5.034
4.941
Others
MASE
3.498
3,479
3.350
3.372
3.327
OWA
1.097
1.082
1.055
1.061
1.055
SMAPE
12.283
11.937
11.924
11.862
11.785
Avg.
MASE
1.660
1.611
1.605
1,601
1.584
OWA
0.887
0.861
0.859
0.856
0.849
Table 33: The full fine-tuning results of the time series classification task under different sizes of pre-training datasets. The brief results of this experiment are shown in Figure 7 (a). Red: best, Blue: second best.
Datascts
0B
1B
10B
25B
50B
ShhanolConcentratonn
33.08
30.04
34.22
35.74
37.30
FaceDetection
51.31
58.12
58.12
58.12
69.20
Handwriting
35.76
36.24
36.24
36.00
36.70
Heartheat
70.24
71.71
72.20
73.17
74.15
Japanese Vowels
94.86
95.95
94.86
96.76
98.11
PEMS-SF
86.71
88.44
88.44
92.49
97.11
SelfRegulationSCCP
86.69
85.67
87.71
88.05
89.76
SellRegglationSCP2
56.11
57.78
$7.78
58.89
58.89
SpokenArubicDigiss
89.77
93.91
95.91
97.04
98.86
WaaeeGestureLibrrry
78.75
85.63
85.63
87.19
89.38
Average Accuracy
68.33
70.35
71.11
71.34
55

---

Table 34: The full fine-tuning results of the time series imputation task under different sizes of pre-training datasets. The brief results of this experiment are shown in Table Red: best, Blue: second best.
Methods
1B
10B
25B
50B
Mask
Ratio
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
12.5%
0.034
0.111
0.034
0.112
0.032
0.111
0.033
0.110
0.032
0.110
25%
0.050
0.125
0.042
0.116
0.041
0.118
0.036
0.116
0.034
0.113
ETTml
37.5%
0.040
0.122
0.037
0.119
0.039
0.118
0.036
0.118
0.037
0.118
50%
0.044
0.128
0.042
0.129
0.042
0.128
0.042
0.126
0.041
0.126
Avg
0.042
0.122
0.039
0.119
0.038
0.119
0.037
0.118
0.036
0.117
0.030
0.088
0.027
0.087
0.028
0.087
0.024
0.084
25%
0.038
0.101
0.029
0.087
0.029
0.087
0.028
0.088
0.024
0.086
ETTm2
37.5%
0.040
0.120
0.032
0.107
0.096
0.027
0.027
0.045
0.114
0.036
0.109
0.034
0.110
0.032
0.106
0.030
0.093
Avg
0.038
0.106
0.031
0.097
0.030
0.095
0.028
0.093
0.026
0.088
12.5%
0.100
0.208
0.085
0.185
0.087
0.186
0.080
0.180
0.074
0.179
25%
0.120
0.238
0.116
0.222
0.106
0.221
0.103
0.207
0.082
0.190
ETThl
37.5%
0.118
0.234
0.118
0.227
0.100
0.206
0.103
0.209
0.100
0.205
50%
0.140
0.241
0.133
0.235
0.137
0.240
0.130
0.232
0.123
0.230
Avg
0.112
0.230
0.113
0.217
0.107
0.104
0.207
0.201
12.5%
0.061
0.158
0.061
0.158
0.057
0.151
0.054
0.146
0.051
0.138
25%
0.066
0.161
0.058
0.153
0.059
0.155
0.058
0.154
0.055
0.146
ETTh2
37.5%
0.065
0.158
0.070
0.164
0.067
0.160
0.059
0.155
0.059
0.152
50%
0.068
0.164
0.073
0.167
0.068
0.166
0.066
0.161
0.064
0.157
Avg
0.065
0.160
0.066
0.160
0.063
0.158
0.059
0.154
0.057
0.148
12.5%
0.046
0.123
0.039
0.123
0.039
0.123
0.038
0.123
0.037
0.122
25%
0.046
0.148
0.049
0.140
0.048
0.139
0.047
0.139
0.046
0.139
ECI.
37.5%
0.062
0.168
0.063
0.163
0.062
0.161
0.060
0.161
0.060
0.160
50%
0.076
0.181
0.076
0.182
0.075
0.182
0.076
0.183
0.075
0.181
Avg
0.058
0.155
0.057
0.152
0.056
0.151
0.055
0.152
0.055
0.151
12.5%
0.029
0.044
0.030
0.046
0.025
0.036
0.026
0.038
0.025
0.035
25%
0.038
0.054
0.030
0.044
0.036
0.052
0.029
0.042
0.027
0.037
Weather
37.5%
0.038
0.058
0.037
0.057
0.034
0.052
0.032
0.045
0.029
0.039
50%
0.040
0.057
0.036
0.054
0.036
0.052
0.034
0.046
0.032
0.042
Avg
0.036
0.053
0.033
0.050
0.033
0.048
0.030
0.043
0.028
0.038
Average
0.058
0.138
0.057
0.132
0.055
0.131
0.052
0.049
0.124
56

---

Table 35: The full fine-tuning results of the time series classification task under different sizes of pre-training datasets. The brief results of this experiment are shown Figure (b). Red: best, Blue: second best.
Datasets
OB
1OB
25B
50B
Metries
R
FI
SMD
86.70
80.72
83.60
88.08
83.37
85.66
87.
81.05
84.13
86.89
82.02
84.39
88.08 83.37 85.66
MST.
89.28 73.68
80.74
13
73.59
80.62 89.48
74.59
81.36
89.34
73.91 80.90 89.46 75.31 81.77
SMAP
89.97 54.16 67.61
90.06
53.56 67.17 90.08 54.51 67.92
90.11 11 54.63 68.02 91.06 61.51 73.43
SWaT
91.57 85,45
88.40
92.21
92.70
92.45 92.23 92.77 92.50
92.28 92.87 92.57 95.94 91.39 93.61
PSM
98.69 94.24 96.41
98.53
94.30
96.37 98.65 94.41 96.48
98.79
94. 96.40 91.39 93.61 98.90
Avg.
91.24 77.65 83.40
91.60
79.50
84.45
1,558 79.47 84.48 9 91.48
79.51 84.46 95.36 97.10 86.31
57

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made our main claims in the abstract and introduction. We have clearly pointed out the problem to be solved and the method to be used in the abstract and introduc- tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The S bimodal data generation mechanism the core method and contribution of this paper. In Appendix B.9 we further explore the current limitations in this aspect and will try improve in subsequent work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
58

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. The core ideas of this paper are further explained and proved by Takens's theorem and symbolic dynamies in the beginning Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We describe the proposed methodology in detail in Section 3 and provide further explanation in Appendix Experimental details are outlined in Appendix C Ad- ditionally, all the code related to the proposed method is included in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
59

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
guudess/CodeSumiise
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the datasets used in detail in Appendix C.I and further elaborate the model pre-training and fine-tuning configurations for downstream tasks in Appendix
C.4 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Docs the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section 4.4 we conducted multiple experiments in the ablation experiments and drew error bars in the bar graphs to further ensure the reliability of the ablation conclu- sions. However, due to the large number of experiments, detailed standard deviations are not shown. Instead, uniformly present them in the headings of each table in Appendix
60

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
deseribe the hardware environment in detail. Using the pre-trained configuration in Appendix [C.4 on our eight A6000s is enough to fill the memory of all devices. The computing resources and memory required for fine-tuning the model are shown in Figure 5 (right).
Justification: In Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and conducted it the paper conform in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
61

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We illustrate broader impacts of research in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The models compared and the datasets used this article are open source, and the corresponding papers web pages are cited.
Guidelines:
62

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code for the proposed methodology in the supplementary material and will make it publicly available at an appropriate time.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
63

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The LLM does not affect the core methodology, scientific rigor or originality the research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components. wwaperipsscccCooooee ec/Conferrences/2225/ /LLM
Please refer to our LLM policy for what should or should be described.
64


Reviews:
[{"review_id": "dkrKDiyCDn", "reviewer": "Reviewer_Atv9", "summary": "This paper introduces a synthetic data generation method for training time series foundation models. At the core of the synthetic data generation method is a composition of symbolic expressions that gets randomly generated through sampling (sampling input/output dimensions, operators, affine transformations, etc.). The input to these symbolic expressions are time series sampled from mixed distributions and ARMA processes. The transformation of the symbolic expression then forms the synthetic time series. The paper also introduces SymTime, a time series foundation model which uses patched encodings, cross-entropy loss, and additional contrastive learning and momentum distillation terms.  \n\nThe manuscript compares the generated time series dataset features with existing datasets from Monash to evaluate whether the generated time series are realistic. The authors then evaluate their models on forecasting, imputation, classification, and anomaly detection tasks. Finally, the paper conducts ablations on the pre-training objectives and analyses the obtained time series and symbol representations.", "questions": "How are positive/negative pairs selected in the contrastive learning step?", "limitations": "The authors appropriately address limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper introduces an interesting idea for synthetic time series generation based on sampling of symbolic expressions. This offers a flexible way of generating synthetic time series. As the authors state in the paper, pretraining time series models requires synthetic data due to a relative scarcity (compared the text data) and therefore novel ideas for synthetic data generation are needed. Therefore, the explored topic in the paper has significance and is original. Another strength of the paper is that the thorough evaluation of the model on several downstream tasks (forecasting, imputation, classification, anomaly detection) along with ablations and analysis of the generated synthetic time series and SymTime representations. \n\nOne weakness of the paper is the clarity of writing. I found several parts of the paper a bit confusing and even after reading several times I am still not sure how they work. While I understood the data generation process after some re-reads, I still do not understand the contrastive learning part of SymTime. For example, how are positive and negative pairs sampled? I understand that the authors covered a lot of ground and had to accommodate this in the space constraints, but I believe a clearer explanation of the core concepts would strengthen the paper. \n\nAnother major weakness is the specific setup of the forecasting results and the claims that are made on SymTime’s performance in forecasting. The paper for the long-range forecasting dataset, a fixed lookback window of 96 is used for most comparing models. This makes the conclusion that SymTime outperforms the other baseline models on this dataset invalid because it only does so under this (artificial) conditions that make the baseline methods perform worse. For example, the MSE/MAE windows for PatchTST are much lower in its original paper on these datasets, if the proposed lookback window of 336 or 512 time steps is used. Thus, SymTime does not reduce the absolute error on these datasets compared to existing methods. I have a similar criticism on the short-term datasets. Here again, while the results for SymTime are good, the OWA for the state-of-the-art method is also lower (see Gasthaus et al. (2019), \"Spline Quantile Function RNNs\") than the error of the methods presented here. \n\nAdditionally, while the long-term forecasting benchmark has been used extensively in time series forecasting, the value of this benchmark to still measure progress in this field is questionable. This has been explored in recent position papers (https://arxiv.org/abs/2502.14045) and in the NeurIPS 2024 time series workshop (https://neurips.cc/virtual/2024/workshop/84712#wse-detail-108471, see Christoph Bergmeir's talk). I think it is important for the time series forecasting filed to move towards more extensive evaluation which is proposed in several recent paper (Gifteval: https://arxiv.org/abs/2410.10393; FEV (from the Chronos paper): https://openreview.net/forum?id=gerNCVqqtR).\n\nI understand and appreciate that this work also considers other tasks (like classification and anomaly detection, where I have less insight on the SOTA). Therefore, the paper provides additional experimental evidence. However, I would ask the authors to at least run some of the baseline methods with their recommended settings (specifically PatchTST with 336/512 context length) make sure their comparison captures the error that SOTA models are capable of and therefore an accurate reflection of the SOTA.", "was_revised": true, "final_justification": "The authors addressed a major weakness in their evaluation of long-term forecasting benchmark, along with other clarifications and improvements as evident by the discussions with the other reviewers. Hence, I raise my score.", "rebuttal": "We sincerely appreciate your valuable feedback and your acknowledgement of the thoroughness of our experiments on downstream tasks. We will address your questions one by one.\n\n> **W1**: For example, how are positive and negative pairs sampled?\n\nThank you for your question. In our $S^2$ data generation mechanism, we generate a three-tuple pair of data: (sample sequence, generated sequence, symbolic expression). Specifically, we first construct the symbolic expression $f(\\cdot)$. Then, we construct the sample series $X$. Finally, by inputting the sample series into the symbolic expression, we obtain the generated series $Y=f(X)$. In this process, we believe that the generated $Y$ will have a natural correspondence with the symbol $f(\\cdot)$ that generates it.\n\nTherefore, for a certain time series $Y$, we believe that the series and the symbolic expression $f(\\cdot)$ that generates it are mutually positive samples, and that other symbolic expressions are mutually negative samples. Correspondingly, for a certain symbol $f(\\cdot)$, the time series $Y$ generated by it is mutually positive, and other irrelevant symbols are mutually negative samples.\n\nIn the abstract and **line 153**, we repeatedly mention that we generate paired data of sequences and symbols, and the pairing of the two makes them mutually positive samples. **We will make this clear in the main paper**.\n\n> **W2**: The paper for the long-range forecasting dataset, a fixed lookback window of 96 is used for most comparing models. This makes the conclusion that SymTime outperforms the other baseline models on this dataset invalid because it only does so under this (artificial) conditions that make the baseline methods perform worse.\n>\n> However, I would ask the authors to at least run some of the baseline methods with their recommended settings (specifically PatchTST with 336/512 context length) make sure their comparison captures the error that SOTA models are capable of and therefore an accurate reflection of the SOTA.\n\nThank you for your valuable feedback. As you mentioned, for long-term forecasting, the vast majority of models use a look-back window of 96. Therefore, to ensure fairness, we adjusted the look-back length of all models to 96 in our initial comparison. \n\nHowever, We are pleased to accept your suggestion. To this end, we adjust the look-back window to 336 and 512, re-run the experiments, and compared them with a series of current baseline models. Below we show the average results for four lengths {96, 192, 336, 720} on 8 datasets:\n\n\n| 336 | SymTime |        | PatchTST |        | TimeMixer |        | TimesNet |        | Autoformer |        | DLinear |        | iTransformer |        |\n| -------- | :-----: | :----: | :------: | :----: | :-------: | :----: | :------: | :----: | :--------: | :----: | :-----: | :----: | :----------: | :----: |\n| Metrics  | MSE     | MAE      | MSE       | MAE      | MSE        | MAE     | MSE          | MAE    | MSE    | MAE    | MSE    | MAE    | MSE    | MAE    |\n| ETTm1    | 0.354   | 0.381    | 0.352     | 0.382    | 0.368      | 0.392   | 0.421        | 0.423  | 0.618  | 0.539  | 0.357  | 0.379  | 0.368  | 0.395  |\n| ETTm2    | 0.260   | 0.317    | 0.258     | 0.315    | 0.262      | 0.318   | 0.282        | 0.334  | 0.400  | 0.420  | 0.291  | 0.353  | 0.272  | 0.329  |\n| ETTh1    | 0.425   | 0.436    | 0.419     | 0.432    | 0.430      | 0.437   | 0.485        | 0.480  | 0.580  | 0.539  | 0.425  | 0.440  | 0.450  | 0.457  |\n| ETTh2    | 0.352   | 0.393    | 0.331     | 0.379    | 0.396      | 0.425   | 0.409        | 0.440  | 0.663  | 0.604  | 0.490  | 0.476  | 0.390  | 0.416  |\n| weather  | 0.238   | 0.273    | 0.258     | 0.292    | 0.235      | 0.273   | 0.250        | 0.286  | 0.441  | 0.450  | 0.245  | 0.298  | 0.238  | 0.272  |\n| ECL      | 0.165   | 0.267    | 0.165     | 0.294    | 0.169      | 0.260   | 0.197        | 0.297  | 0.236  | 0.346  | 0.170  | 0.269  | 0.163  | 0.257  |\n| Traffic  | 0.393   | 0.266    | 0.396     | 0.268    | 0.411      | 0.271   | 0.615        | 0.331  | 0.676  | 0.413  | 0.465  | 0.320  | 0.385  | 0.273  |\n| Exchange | 0.370   | 0.408    | 0.385     | 0.420    | 0.415      | 0.438   | 0.548        | 0.532  | 1.053  | 0.807  | 0.448  | 0.462  | 0.392  | 0.427  |\n| Average  | **0.3198** | **0.3427** | 0.3204    | 0.3478   | 0.3357     | 0.3519  | 0.4009       | 0.3903 | 0.5832 | 0.5149 | 0.3614 | 0.3747 | 0.3323 | 0.3533 |\n\n| 512 | SymTime |        | PatchTST |        | TimeMixer |        | TimesNet |        | Autoformer |        | DLinear |        | iTransformer |        |\n| :------: | :--------: | :--------: | :-------: | :------: | :--------: | :-----: | :----------: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| Metrics  | MSE        | MAE        | MSE       | MAE      | MSE        | MAE     | MSE          | MAE    | MSE    | MAE    | MSE    | MAE    | MSE    | MAE    |\n| ETTm1    | 0.356      | 0.380      | 0.352     | 0.382    | 0.371      | 0.392   | 0.425        | 0.430  | 0.556  | 0.518  | 0.358  | 0.380  | 0.367  | 0.397  |\n| ETTm2    | 0.265      | 0.320      | 0.256     | 0.317    | 0.263      | 0.323   | 0.294        | 0.344  | 0.371  | 0.416  | 0.275  | 0.340  | 0.273  | 0.331  |\n| ETTh1    | 0.414      | 0.432      | 0.413     | 0.434    | 0.429      | 0.444   | 0.481        | 0.486  | 0.627  | 0.579  | 0.418  | 0.438  | 0.446  | 0.460  |\n| ETTh2    | 0.365      | 0.405      | 0.357     | 0.409    | 0.373      | 0.410   | 0.397        | 0.432  | 0.687  | 0.609  | 0.499  | 0.478  | 0.388  | 0.417  |\n| weather  | 0.234      | 0.273      | 0.245     | 0.284    | 0.231      | 0.271   | 0.251        | 0.288  | 0.489  | 0.486  | 0.241  | 0.292  | 0.249  | 0.280  |\n| ECL      | 0.163      | 0.267      | 0.169     | 0.269    | 0.177      | 0.274   | 0.201        | 0.302  | 0.353  | 0.393  | 0.167  | 0.267  | 0.162  | 0.257  |\n| Traffic  | 0.395      | 0.268      | 0.399     | 0.272    | 0.410      | 0.270   | 0.624        | 0.334  | 0.705  | 0.435  | 0.433  | 0.305  | 0.383  | 0.273  |\n| Exchange | 0.384      | 0.412      | 0.398     | 0.423    | 0.517      | 0.497   | 0.718        | 0.608  | 0.944  | 0.768  | 0.500  | 0.494  | 0.427  | 0.467  |\n| Average  | **0.3221** | **0.3446** | 0.3237    | 0.3487   | 0.3464     | 0.3601  | 0.4239       | 0.4030 | 0.5914 | 0.5253 | 0.3614 | 0.3744 | 0.3368 | 0.3603 |\n\nThe results show that increasing the look-back does improve forecasting performance for most models, likely because longer series contain more historical information. SymTime, PatchTST, and TimeMixer show particularly significant improvements, especially for datasets with larger numbers of channels, such as Weather, ECL, and Traffic. We summarize the possible reasons for the SymTime performance improvement:\n\n1. SymTime is Transformer encoder framework, and has also undergone large-scale pre-training.\n2. The longer look-back window allows us to construct more tokens, which is closer to the average number of tokens in our pre-training (144), thus better leveraging our pre-training performance.\n\nThe detailed test results and experimental configuration of this experiment will be added to the paper in **Section A.1** (long-term forecasting benchmark).\n\n> **W3**: Additionally, while the long-term forecasting benchmark has been used extensively in time series forecasting, the value of this benchmark to still measure progress in this field is questionable. This has been explored in recent position papers and in the NeurIPS 2024 time series workshop. I think it is important for the time series forecasting filed to move towards more extensive evaluation which is proposed in several recent paper.\n\nWe appreciate your insights on time series forecasting benchmarks. Our paper uses the benchmark from TimesNet, which evaluates models across 5 different TSA tasks to assess multi-task generalization capabilities. This benchmark is widely adopted by models like GPT4TS and Peri-midFormer. The goal of SymTime is to demonstrate strong applicability in a variety of tasks.\n\nWhile, we recognize newer zero-shot forecasting benchmarks like **GIFT-Eval** and **FEV** (used by mary models such as Sundial and Chronos), these primarily target zero-shot forecasting scenarios which differ from SymTime's objectives. Therefore, we prioritized the TimesNet benchmark.\n\nTo further validate SymTime, we follow GIFT-Eval's training splits and performed fine-tuning. Testing across all GIFT-Eval sub-datasets is ongoing due to time constraints. We commit to reporting full results upon completion, submitting them to the online leaderboard, and will include comparative analyses with other benchmarks in our paper's future work section.\n\nAnother key aspect of this paper lies in the $S^2$ paradigm, which offers the advantage of unrestricted data generation. We can construct unlimited high-quality time series datasets for model pre-training. When the data representations we generate are similar to those in real-world test scenarios, we believe that models pre-trained with $S^2$ will also achieve excellent zero-shot performance on real-world data.\n\n> **Q**: How are positive/negative pairs selected in the contrastive learning step?\n\nIn **W1**, we point out that the time series data generated by the same symbol $f(\\cdot)$ will be positive samples with the symbol expression, and negative samples with other symbols."}, {"review_id": "TOVCysbWII", "reviewer": "Reviewer_yCnr", "summary": "The paper introduces a novel approach to address data scarcity and imbalance in time series analysis. The authors propose a series-symbol data generation pipeline. Inspired by complex dynamic system theories, the authors generate high-quality univariate time series data paired with corresponding symbolic expressions, enabling the creation of diverse time series datasets. The authors present SymTime, a pre-trained encoder-only time series foundation model, leveraging contrastive learning and masked modeling. It shows notable scalability and generalization performance.", "questions": "1. Under the proposed symbolic system, what is the maximum scale of synthetic data that can be constructed?\n2. Are there any suggestions to overcome the diminishing marginal returns when pre-training on an ever-larger scale of synthetic data?\n3. What is the range of different statistical metrics in Figure 4? The generated data has relatively low diversity in some metrics (PE, FFT Mean, Seasonality). How does pre-training based on metrics of different degrees of diversity affect the performance of downstream tasks?\n4. Can the proposed method generate multivariate time series? It may also provide insights about how to define how \"multivariate\" a set of time series is.", "limitations": "Please discuss whether omitted operators (e.g., exponent and higher power) limit the generation of certain dynamics.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Quality**: The submission is technically sound, with a strong theoretical foundation and extensive experimental validation. SymTime’s architecture is well-motivated, leveraging contrastive learning and masked modeling on numerical and symbolic tokens. About this architecture option, I wonder if the author has ever tried generative modeling on a decoder-only Transformer like TimesFM and Timer. The benchmarks include five major TSA tasks (forecasting, classification, imputation, anomaly detection). Although it is comprehensive, I'd like to raise my score if the model can be evaluated more public leaderboard, such as GIFT-Eval and FEV.\n\n**Clarity**: The mathematical formulation (e.g., tree-based symbolic sampling, ARMA processes) is well-explained. The paper is generally well-written and organized, but there are some typos that could be improved (e.g., \"prmutation\" -> \"permutation\"; \"pretrain\" -> \"pre-train\").\n\n**Significance**: The paper provides new insights for (1) pre-training generalizable time series foundation models, (2) inspiring new benchmarks for time-series pre-training. As symbolically generated time-series data can effectively replace or augment real-world data for pre-training, it can be beneficial for the community if the author releases the data generation framework. \n\n**Originality**: Different from previous works, which are pre-trained solely on synthetic data or partially synthetic data. The proposed symbolic framework can provide aligned synthetic data and symbolic formulation. In addition to enabling contrastive learning, it is a promising direction is to take symbolic formulation as the aligned textual description for multimodal time series models, which can help current models better understand the underlying dynamics of time series. In terms of the model design, the contributions of SymTime mainly focus on pre-training tasks with contrastive learning. The structure is similar to previous works (such as moment, moirai).", "was_revised": true, "final_justification": "Based on the additional empirical results provided, I am confident that the framework has the potential to offer valuable insights for the development of time series foundation models. Therefore, I will maintain my positive evaluation. Additionally, I am pleased to see the release of this framework.", "rebuttal": "We sincerely thank you for your recognition of the novelty of the data generation mechanism and the effectiveness of the SymTime model. Below we will answer your valuable questions one by one.\n\n> **Quality**: Although it is comprehensive, I'd like to raise my score if the model can be evaluated more public leaderboard, such as GIFT-Eval and FEV.\n\nThank you for your recognition of the comprehensiveness of our experiments. We are happy to verify SymTime on the **GIFT-Eval benchmark**. We will split the training sets according to the requirements and fine-tune SymTime. \n\nHowever, due to the lots of sub-datasets included in this benchmark and time constraints, we have not yet completed fine-tuning. Once we have completed it, we will immediately inform you of the results and upload the online leaderboard. The results will also be added to **Section A.1** in our paper.\n\n> **Clarity**: The paper is generally well-written and organized, but there are some typos that could be improved.\n\nThank you for your recognition of the clarity of our algorithm writing. We already have corrected the above errors and further checked the full paper.\n\n> **Significance**: As symbolically generated time-series data can effectively replace or augment real-world data for pre-training, it can be beneficial for the community if the author releases the data generation framework.\n\nWe sincerely appreciate your recognition of our $S^2$ data generation mechanism. Deep learning have made significant progress in time series analysis, but we continue to strive to address existing challenges by incorporating the insights of complex dynamical systems into the essence of time series. \n\n**We promise to open source the code for our $S^2$ data generation mechanism in the future.**\n\n> **Q1:** Under the proposed symbolic system, what is the maximum scale of synthetic data that can be constructed?\n\nTheoretically, the $S^2$ data generation mechanism can generate unlimited sequence-symbol pairings. This diversity is primarily reflected in two aspects:\n\n1. **It can generate an unrestricted variety of symbolic expressions $f(\\cdot)$**: Due to the existence of random constants and a variety of symbols and operators, the symbols we can generate are infinite, and the time series generated by different symbols are also diverse.\n2. **It can generate a variety of sampled sequences $X$**: Since we switch between different random seeds and specify different initializations at each sampling, $X$ generated from a mixture of uniform and normal distributions and randomly parameterized ARMA(p, q) models is also diverse. This also makes the data we can generate infinitely.\n\n> **Q2**: Are there any suggestions to overcome the diminishing marginal returns when pre-training on an ever-larger scale of synthetic data?\n\nThis is a very valuable question for large-scale pre-training. Currently, many large-scale foundation models trained on real datasets face the problem of diminishing marginal returns. However, in the experiments in **Section 4.3** that verified the impact of pre-training dataset size on SymTime performance, we find that as we continuously increase the pre-training dataset, the model's performance continue to improve randomly. This phenomenon has little impact on SymTime. Therefore, we conduct a detailed analysis, and the specific reasons are as follows:\n\nWhen the $S^2$ dataset is generated, its representation is more \"random,\" meaning that most adjacent samples may not come from the same distribution. As our data volume increases, our $S^2$ data will appear evenly across the representation space of time series data. However, real time series datasets often consist of multiple sub-datasets, each of which comes from the same distribution and has the same generation mechanism. Therefore, as the amount of real data increases, it may only be able to fill a certain position in the representation space (a specific position and range within this distribution).\n\nTherefore, for other synthetic datasets, <u>we believe that we should make the data more random when generating it</u>. Rather than restricting the possible distribution of the data, we should make it more diverse. This approach we believe can alleviate this problem.\n\n> **Q3-1**: What is the range of different statistical metrics in Figure 4? The generated data has relatively low diversity in some metrics (PE, FFT Mean, Seasonality).\n\nThis is a very interesting question. Radviz visualizes high-D data in 2D using a **spring force balance**. After normalization, sample features act as \"pulls\" from anchor points evenly spaced around a circle. The sample's final position is determined by the equilibrium of all these pulls. Consequently, the 2D coordinates themselves lack inherent physical meaning, as they result from the combined influence of all features. This method is useful for observing clustering in reduced dimensions. The observed lack of diversity in indicators like PE, FFT Mean and Seasonality within the Radviz plot is an artifact of this balancing process across all indicators, not an indication that those specific features lack diversity in the actual data.\n\n> **Q3-2**: How does pre-training based on metrics of different degrees of diversity affect the performance of downstream tasks?\n\nTo quantitatively explore this issue, we selecte **25B** of time series data from the 50B $S^2$ dataset, which ranked high for Seasonality (S), Forecastability (F), and Permutation Entropy (PE) respectively. So the subdataset can reflect the seasonality, predictability, and permutation complexity of the data, respectively. We pre-train SymTime using the configuration in C.3 and fine-tuned it on four ETT datasets for downstream tasks. The experimental results are shown below:\n\n|Dataset|F| |S| | PE | | SymTime | |\n| ------- | :---: | :---: | :---: | :---: | :---: | :---: | :-----: | :---: |\n|Metrics|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|\n|ETTm1|0.374|0.393|0.380|0.395|0.385|0.389|0.378|0.393|\n|ETTm2|0.283|0.329|0.278|0.325|0.290|0.331|0.278|0.325|\n|ETTh1|0.433|0.438|0.445|0.443|0.443|0.440|0.434|0.438|\n|ETTh2|0.386|0.411|0.382|0.410|0.408|0.419|0.370|0.408|\n|avg|0.369|0.393|0.371|0.395|0.382|0.395|0.365|0.391|\n\nWe believe this is a very interesting experiment. The results show that metrics with varying degrees of diversity do have a certain impact on the performance of downstream tasks. We can preliminarily conclude that Forecasting and Forecasting efficiency metrics are more effective in promoting model representation learning than PE.\n\n> **Q4**: Can the proposed method generate multivariate time series? It may also provide insights about how to define how \"multivariate\" a set of time series is.\n\nOur $S^2$ data generation mechanism can generate a multivariate time series $Y = f(X)\\in \\mathbb R^{N\\times L}$ by constructing a multivariate complex system $f(\\cdot)$ and a multivariate sampling series $X\\in \\mathbb R^{M\\times L}$, where $M$ and $N$ are the input and output channels respectively and $L$ is the length of time series.\n\nFor example, we can generate a multivariate symbol and series as follows:\n$$\nf(x_1, x_2, x_3) = \\begin{bmatrix}f_1(x_1, x_2, x_3)\n \\\\ f_2(x_1, x_2, x_3)\n \\\\ f_3(x_1, x_2, x_3)\n\\end{bmatrix}\n$$\n$$\nX = \\begin{bmatrix} x_1\n \\\\ x_2\n \\\\ x_3\n \\end{bmatrix}\n$$\nWe simultaneously obtain time series data for the three input and three output channels using $Y=f(X)$. This allows us to combine sampling series from different channels through symbolic operations. Therefore, our data generation mechanism is capable of generating multivariate time series with channel correlation. Specifically, we present the generated symbolic expressions and time series data in **Section B.1 on page 21**, including multivariate time series data, from which we can observe the correlation between channels.\n\n> **Limitations**: Please discuss whether omitted operators (e.g., exponent and higher power) limit the generation of certain dynamics.\n\nThank you for this valuable comment. In the current $S^2$, omitting certain operators does restrict the creation of some dynamic data. Earlier versions included a richer set of numerical operations—differentiation, integration, exponential functions with various bases, and higher-order powers. Extensive experiments showed that several of these operators severely impeded the data-generation algorithm; the main reasons are detailed below.\n\n1. **Value explosion**: To maintain quality, we cap large magnitudes (l. 132). Integration, exponential and high-order powers readily cause overflow, so they were dropped; exp alone is retained for diversity.  \n2. **Numerical differentiation**: Symbolic differentiation introduces truncation/round-off trade-offs. Combined with reciprocal and absolute-value operators, functions such as $|x|$ or $\\mathrm{sin}(\\frac{1}{x})$ because non-smooth or high-frequency vibration at $x=0$, breaking differentiation. \n3. **Numerical integration**: Randomly built expression trees often yield integrands with singularities (e.g., $\\int_{0}^{1}\\frac{1}{\\sqrt{x}}\\mathrm{d}x $) or force costly oscillatory integrals (e.g., $\\int_{0}^{100}\\mathrm{sin}(100x)\\mathrm{d}x$); interval selection is non-trivial.  \n4. **Symbolic cost**: Differentiation and integration are slow and can trigger exponential memory growth. Many elementary functions lack closed-form antiderivatives (e.g., $\\int \\mathrm e ^{-x^2}\\mathrm d x$).\n\nConsidering factors like numerical stability, symbolic complexity, computational efficiency, and sampling success rate, we selectively omitted some symbolic operations.\n\nNevertheless, the $S^2$ data generation framework proposed is essentially a complete theory. It already incorporates the vast majority of symbolic operations, and new operators or user-defined symbolic operations can be easily added. The omission of some operations due to the above factors does not undermine the validity of this framework. We consider this a valuable point and will add it in the paper."}, {"review_id": "Rt4Kk6DWLA", "reviewer": "Reviewer_1qct", "summary": "The article focuses on improving model performance through data augmentation and the use of foundation models. Specifically, they propose a series-symbol data generation mechanism that alleviates issues of scarcity and imbalance in data. Additionally, they design a time series foundation model consisting of two encoders, where one encoder processes time series data and the other encoder processes symbol inputs. These two encoders are trained using contrastive learning to enhance the representation ability of time series. The approach has achieved good results in multiple time series tasks.\nExperiments verify the ability of data augmentation in terms of stability, predictability, frequency domain analysis, and trendiness. The comparative experiments validate the performance of the foundation model across multiple downstream tasks.", "questions": "1. Does the symbol data generation mechanism have a complexity issue when dealing with long or non-stationary time series data?\n2. Is the symbol generation mechanism required to be adjusted according to downstream tasks? Please provide detailed information on how symbols are selected.\n3. How were the 6-layer Transformer encoder for handling time series data and the 6-layer DistilBERT for handling symbol data chosen? Does layer depth have any impact on performance?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: The author proposes a data generation mechanism that addresses issues of data scarcity and imbalance, and experimental results show that it can alleviate these problems.\nS2: The pre-trained model demonstrates strong capabilities in multiple time-series tasks.\n\nW1: Lack of implementation details for the symbol data generation mechanism.\nW2: No explanation was given regarding the complexity issue of the symbol data generation mechanism when dealing with long series, as well as whether the symbol selection strategy needs to be adapted based on specific tasks.", "was_revised": true, "final_justification": "These supplementary materials and the responses from the authors to me and other reviewers basically answered my questions. I think it is sufficient to rating this work, and I will maintain my scores.", "rebuttal": "Thank you for your recognition of the data generation mechanism, model performance, and experimental results in this paper. Below I will answer your questions one by one.\n\n> **W1:** Lack of implementation details for the symbol data generation mechanism.\n\nThank you for your valuable feedback. We will open a new chapter in the appendix to further explain each step and hyperparameter settings of the $S^2$ data generation mechanism in detail, and we have responded to your questions in detail in Q1-Q3.\n\n**We will open source our $S^2$ data generation code in the future.**\n\n> **W2:** No explanation was given regarding the complexity issue of the symbol data generation mechanism when dealing with long series.\n\nWe define the specific symbol explanation and its  as follows. Then, we use the divide-and-conquer approach to analyze the complexity of our $S^2$ data generation mechanism.\n\n| symbol | explanation                                                  |\n| ------ | ------------------------------------------------------------ |\n| $L$    | The length of time series                                    |\n| $M$    | Number of input channels                                     |\n| $N$    | Number of output channels                                    |\n| $k$    | Total number of mixed distributions used                     |\n| $p$    | Autoregressive model order in ARMA model                     |\n| $q$    | Moving average model order in ARMA model                     |\n| $P$    | Probability of choosing a sampling method                    |\n| $b$    | Number of binary operators used to construct symbolic expressions |\n| $u$    | The number of unary operators used to construct symbolic expressions |\n\n1. **Symbolic Expression Generation**: We construct symbolic expressions using a tree structure as a medium. When we have $b$ binary operators, we further insert $(b + 1)$ leaf nodes (the process from (a) to (b) in **Figure 3** in our paper). Therefore, after inserting $u$ unary operators (**Figure 3** (c)), the total number of nodes in the tree is $n = 2b + u + 1$. Because there are many ways to construct a tree, we consider the time complexity of constructing a balanced tree. Therefore, for $N$ symbols constructed, the specific complexity of this process is $O(N\\times n\\mathrm{log}n)$.\n\n2. **Sampling series generation**: When we want to generate a sampling time series with $M$ channels, each channel has a probability of $P$ to be sampled using a mixture distribution and a probability of $(1-P)$ to be sampled using an ARMA model. When the sampling length of the series is $L$, the complexity of generating $k$ mixture distribution and ARMA ($p$, $q$) series is $O(kL)$ and $O(L(p+q))$. Therefore, the time complexity of this process can be quantified as $O \\left ( ML \\times [Pk + (1-P)(p+q)] \\right )$.\n\n3. **Sampling through symbolic expressions and series**: We simplify the specific operational details of this process and only consider the time complexity of operations with variables. For a series of length L, we have $N$ symbolic expressions to be sampled, and each symbol has an average of $\\frac{M+1}{2}$ variables (Each symbolic expression may contain any number of variables from 1 to M, so here we take $\\frac{M+1}{2}=\\frac{(1+2+\\cdots+M)}{M}$ as the average probability). Then the process can be quantified as $O(N \\cdot \\frac{M+1}{2}\\cdot L)$.\n\nTo sum up, since other variables that affect the $S^2$ sampling process are usually small, it can be intuitively understood that the time complexity of the entire sampling process is proportional to the length $L$. \n\nIn **Q1** below, we further answer why this paper does not sample long time series.\n\n> **W2:** whether the symbol selection strategy needs to be adapted based on specific tasks.\n\n**We do not select any symbols**. Instead, to learn the richest representation of time series, we use all symbols to generate the $S^2$ dataset for unified pre-training of SymTime. See **Q2** for details.\n\n> **Q1:** Does the symbol data generation mechanism have a complexity issue when dealing with long or non-stationary time series data?\n\nThrough **W2**, we find that the time complexity of data generation is indeed related to the length $L$. The specific explanation for avoiding sampling long and non-stationary series is as follows:\n\n1. **Out of domain**: As described in **line 132**, the symbolic expression $f(\\cdot)$ has a domain restriction. The longer the series, the more likely it is that the value will fall outside the domain, and the probability of sampling failure increases. When we fail to sample, we will resample.\n2. **Stationarity**: Non-stationary series often exhibit strong autoregressive properties, which can easily lead to numerical inflation and sampling failure (**line 132**). Furthermore, stationary series have bounded prediction errors and are better characterized; non-stationary series have errors that tend to diverge and are poorly characterized. Therefore, to ensure sampling success and forecast performance, we prioritize generating stationary series.\n3. **More channels**: Since we use mask time series modeling for representation learning, the benefits of long series on our model learning are not significant. We are more likely to generate data with more channels (with $M=6, N=12$) to help the model learn the feature between multivariate time series channels.\n\nIn summary, in order to more effectively generate data with better predictability, we adopted the strategy described in the paper in the $S^2$ data generation mechanism.\n\n> **Q2:** Is the symbol generation mechanism required to be adjusted according to downstream tasks? Please provide detailed information on how symbols are selected.\n\nAs described in **Experiment 4.1**, our goal is to generate as much data as possible through an unrestricted data generation mechanism to cover the broadest possible representation of time series, thereby training a time series foundation model (**Definition 1 in Line 54**). Using this dataset, we uniformly pre-train SymTime, enabling the model to learn the most comprehensive representation of time series data. To this end, we do not specifically tailor the symbolic expressions we use to specific downstream tasks or datasets. This approach demonstrates the generalization capabilities of SymTime as a foundation model.\n\n> **Q3-1:** How were the 6-layer Transformer encoder for handling time series data and the 6-layer DistilBERT for handling symbol data chosen? \n\n1. **Time Series Encoder**: We drew on the frameworks of time series foundation models such as **Moirai**, **Timer**, and **Moment**, as well as model architectures in the computer vision such as **ViT** and **ALBEF**. Ultimately, we chose to use a 6-layer Transformer encoder to learn the basic representation of time series data.\n2. **Symbolic Encoder**: This module aims to extract the representation of symbolic expressions during contrastive learning and enable the time series encoder to learn the pairing relationship between series  and symbols. Considering computational complexity, we decide to chose a smaller, but more linguistically robust, pre-trained LLM as the symbolic encoder. Therefore, the two most commonly used 6-layer Transformer models are **gpt2-small** and **DistilBERT**. DistilBERT was chosen due to its superior performance and the compatibility of its encoder architecture with the time series encoder.\n\n> **Q3-2:** Does layer depth have any impact on performance?\n\nBased on the ideas in **Q3-1**, we chose to determine the model architecture of SymTime. To further verify the impact of the number of model Transformer layers on downstream task performance, we set different parameters for the time series encoder and constructed the following two control groups:\n\n| name    | layers | $d_{\\mathrm{model}}$ | $d_{\\mathrm{ff}}$ |\n| :-----: | :----: | :------------------: | :---------------: |\n| SymTime | 6 | 512 | 2048 |\n| A | 6 | 768 | 3072 |\n| B | 3 | 386 | 1536 |\n\nWe keep the symbol encoder unchanged and pre-train the control group using the pre-training configuration used in **C.3** of the original paper (since the parameters of the model in A are further expanded, we reduce the batch size from 128 to 96, while keeping the other configurations unchanged). Then, we fine-tune the model on the four ETT datasets. The average forecasting results for the lengths {96, 192, 336, 720} are shown below:\n\n| Dataset | SymTime |       | A     |       | B     |       |\n| ------- | ------- | ----- | ----- | ----- | ----- | ----- |\n| Metrics | MSE     | MAE   | MSE   | MAE   | MSE   | MAE   |\n| ETTm1   | 0.371   | 0.390 | 0.370 | 0.398 | 0.378 | 0.395 |\n| ETTm2   | 0.274   | 0.321 | 0.278 | 0.323 | 0.279 | 0.324 |\n| ETTh1   | 0.430   | 0.436 | 0.435 | 0.439 | 0.434 | 0.439 |\n| ETTh2   | 0.365   | 0.402 | 0.369 | 0.405 | 0.375 | 0.410 |\n| Average | 0.360   | 0.387 | 0.363 | 0.391 | 0.367 | 0.392 |\n\nIt is clear from this that SymTime's original model parameter configuration achieves optimal results on the ETT dataset. \n\nHowever, when the number of parameters changes, the forecasting results will decrease to a certain extent, but the decrease is not significant. Therefore, as long as the model can meet the representation learning requirements during the pre-training phase, the performance improvement of the model mainly comes from our pre-training. A model that is too large will, to a certain extent, bring greater computational complexity."}, {"review_id": "oPH9mtGFZ8", "reviewer": "Reviewer_SPwj", "summary": "This paper introduces a novel mechanism for generating unlimited synthetic time series data paired with their underlying symbolic (mathematical) expressions—based on complex dynamical systems theory—to overcome real-world data scarcity and imbalance. Leveraging this dual-modality dataset, the authors pretrain SymTime, a foundation model that fuses masked time-series and symbolic-language modeling with contrastive learning, achieving state-of-the-art performance across five major time-series analysis tasks.", "questions": "- Would the synthetic data generation process be able to reproduce phenomena observed in real-world domain specific time-series datasets, e.g. volatility clustering in finance?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- I very much like the careful design of the S² generator and the theoretical background provided around Takens’ theorem.\n- The idea of synthetically generating time-series seems with their proposed approach seems to be highly effective. \n- I like the broad experimentation carried out by the authors, and the consistent gains that their method seems to achieve. \n- The ablations performed by the authors support the strength of each of the changes introduced. \n\n**Weaknesses**\n\n- If I understood correctly, there is no comparison against real-data pre-training with the same architecture. This would have been a valuable addition to the paper. \n- There are no standard deviation results in the tables presented.", "was_revised": true, "final_justification": "The authors addressed my concerns and added new experiments. My initial review was positive and I raise my confidence now.", "rebuttal": "We sincerely thank you for taking the time to review our paper and your constructive comments. We appreciate your recognition of our mechanism. Although deep learning technology has made great progress in time series analysis, we still hope to return to the essence of time series and introduce the complex dynamical system modeling perspective in machine learning to solve the current problem like data imbalance.\n\n---\n\n**We will respond to your questions one by one below:**\n\n> **W1**: If I understood correctly, there is no comparison against real-data pre-training with the same architecture. This would have been a valuable addition to the paper.\n\nThank you for the question. The paper's core is the Taken-theorem-based $S^2$ generator, which produces unlimited time-series and symbolic data. SymTime is designed to learn both modalities jointly, leveraging symbolic representations to boost time series modeling—this benefit is verified in our ablation. However, real-world data lack symbolic expressions, so we can not pre-train SymTime with only time series data through **Equation (6)**.\n\nBut in order to conduct fair and effective comparative experiments, we adopt the configuration of the ablation experiment in **Section 4.4**, randomly and uniformly selecting $1/6$ (we use 50B time series points data in $S^2$ for pre-training) of real time series points from the **Time300B** dataset (Time-MoE), pre-training the model using **Equation (2)** as the target (only using mask time series modeling, not considering symbolic expressions), and fine-tuning on the ETTh1 and ETTh2 datasets. The experimental results (mean (std)) are shown below:\n\n| Type   | SymTime      |              | Real-Data    |              | w/o Symbol   |              |\n| ------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Metric | MSE          | MAE          | MSE          | MAE          | MSE          | MAE          |\n| ETTh1  | 0.442(0.008) | 0.448(0.006) | 0.446(0.012) | 0.453(0.007) | 0.450(0.0010) | 0.454(0.006) |\n| ETTh2  | 0.382(0.007) | 0.425(0.005) | 0.389(0.009) | 0.430(0.005) | 0.401(0.009) | 0.432(0.006) |\n\nWe conduct multiple experiments by switching random seeds, with the results for each experiment being the average of forecasting lengths {96, 192, 336, 720}. The results show that pre-trained SymTime on real data outperforms models pre-trained solely on time series data from the $S^2$ dataset. However, it underperforms models pre-trained using both series and symbolic data. This result further demonstrates the relationship between symbolic expression data and time series, and that this pairing effectively enhances SymTime's performance. We did not intentionally exclude samples from the ETT dataset when sampling from Time300B, so our masked time series modeling may include our test samples. However, the performance of SymTime pre-trained on the synthetic $S^2$ dataset is not inferior to that of real data.\n\n**We will add these experimental results to the ablation experiments in the main text (Section 4.4) and provide a detailed description of our experimental setup.**\n\n> **W2**: There are no standard deviation results in the tables presented.\n\nFollowing TimesNet's benchmark across five tasks, we omitted SDs for brevity but reported their ranges in each table header (see **Tables 17–26** in Appendix). For example, **Table 17** notes SD ≤ 0.5 % for long-term forecasting. \n\n**We will highlight this note more prominently in the main text.**\n\n> **Q**: Would the synthetic data generation process be able to reproduce phenomena observed in real-world domain specific time-series datasets, e.g. volatility clustering in finance?\n\nThank you for the insightful question and examples. The diversity of our $S^2$ generator stems from (1) the unrestricted variety of symbolic systems $f(\\cdot)$ and (2) the diversity of sampled series $X$. Hence, $S^2$ can in principle generate any real-world domain. **Section 4.1** empirically confirms this.\n\nFinancial series, as highly chaotic systems, exhibit strong heteroskedasticity and volatility clustering, typically modeled by ARCH or GARCH. Searching $S^2$, we find an $f(\\cdot)$ that reproduces this behavior:\n\n$$\ny = x \\cdot \\exp\\!\\left(\\frac{\\sin^{2}(x-1)}{2}\\right),\n$$\n\nwith $x \\sim \\mathcal N(0,1)$ or any stationary input (in $S^2$ we can generate this input sampling series). The resulting $y$ shows clear volatility clustering akin to ARCH(1)/GARCH(1,1): large shocks are followed by calm periods. \n\nThe effect can be observed with the short **Python** code below.\n\n~~~python\nimport numpy as np\nfrom matplotlib import pyplot as plt\n\nnp.random.seed(42)\nx = np.random.normal(0, 1, 251)\n\nfig, ax = plt.subplots(2, 1, figsize=(12, 8))\ny = x * np.exp((np.sin((x - 1) ** 2)) / 2)\n\nax[0].plot(y, color=\"royalblue\")\nax[0].set_title(\"Financial Time Series with Volatility Clustering\")\nax[1].plot(y[:-1] ** 2, \"tomato\")\nax[1].set_title(\"Conditional Variance (Shows Clustering)\")\n~~~\n\nBy observing this visualization, we can clearly see the small fluctuations after the large fluctuations in the time domain. This is well reflected in the real-time changes in the variance of the series below, where large and small variances appear alternately. Furthermore, real financial time series may contain trend information, and the complex system generated by our $S^2$ dataset is more complex than the above formula. It may contain expected trend information (derived from the autoregressive model in our sampled series). See the visualization in **Section B.2** for specific possible representations.\n\nThe series is particularly similar to the simulated ARCH(1) model:\n$$\nr_t = \\sigma_{t|t-1} \\varepsilon_t,\n$$\n$$\n\\sigma_{t|t-1} = \\omega + \\alpha r^2_{t-1},\n$$\nwhen $\\omega=0.01$ and $\\alpha=0.9$. \n\nTo prevent explosion, we omitted autoregressive operators such as $y_t = \\phi_1y_{t-1} + \\phi_2y_{t - 2}$. (In order to prevent excessively large values from affecting the standardization during pre-training, we mentioned in **line 132** that excessively large sample values will be discarded.) But your example shows volatility clustering depends on past values, so we will reintroduce autoregression under tight constraints to yield richer series.\n\n**We believe this is a compelling and valuable case and add it to our paper to further demonstrate the validity of $S^2$.**"}]

Author rebuttal:

Meta-review: This paper studies synthetic data generation for time series. They use symbolic expressions to develop a data generator, generate a dataset, then show that models pre-trained on synthetic data can rival models pre-trained on real data when finetuned and tested on real data. The paper was received very well by reviewers, who appreciated the approach is innovative and interesting and that it seems to work well and will hopefully inspire future works. The negatives appear to largely be clarifications that can be addressed in the writing, as noted by the reviewers who were receptive to the authors' responses. One key remaining drawback identified during the discussion period was that the performance evaluations are limited to the TimesNet datasets. These datasets have been so overstudied that improvements on them mean very little, especially the forecasting datasets. So extending the evaluations to include others (e.g., GIFT-Eval) will boost the impact of this work.
Decision: Accept (poster)