TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
Authors: Daoyu Wang
Mingyue Cheng
Zhiding Liu
Qi Liu
Venue: ICML 2025 poster (2025)
Keywords: time series
self-supervised learning
diffusion model
Primary area: general_machine_learning->sequential_network_and_time_series_modeling

Abstract: Self-supervised learning has garnered increasing attention in time series analysis for benefiting various downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively capture both long-term dynamic evolution and subtle local patterns in a unified manner. In this work, we propose \textbf{TimeDART}, a novel self-supervised time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. 
Specifically, we first employ a causal Transformer encoder, accompanied by a patch-based embedding strategy, to model the evolving trends from left to right. Building on this global modeling, we further introduce a denoising diffusion process to capture fine-grained local patterns through forward diffusion and reverse denoising. Finally, we optimize the model in an autoregressive manner. As a result, TimeDART effectively accounts for both global and local sequence features in a coherent way.We conduct extensive experiments on public datasets for time series forecasting and classification. The experimental results demonstrate that TimeDART consistently outperforms previous compared methods, validating the effectiveness of our approach.Our code is available at \url{https://github.com/Melmaphother/TimeDART}.

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
Daoyu Wang Mingyue Cheng"
Zhiding Liu'
Qi Liu'
Abstract
Pre-training
Forward
Self-supervised learning has garnered increasing attention in time series analysis for benefiting var- tous downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively cap- ture both long-term dynamic evolution and subtle local patterns in unified manner. In this work, propose TimeDART, novel supervipd time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. Specifically, we first employ a causal Transformer encoder, accom- panied by patch-based embedding strategy, to model the evolving trends from left right. Build- ing on this global modeling, further intro- duce denoising diffusion process to capture fine-grained local patterns through forward dif- fusion and reverse denoising. Finally, we opti- mize the model in an autoregressive manner. As a result, TART effectively accounts for both global and local sequence features in coherent way. We conduct extensive experiments pub- lic datasets for time series forecasting and clas- sification. The experimental results demonstrate that TiRTDART consistently outperforms previ- ous compared methods, validating the effective- ness of our approach. Our code available http://ggttbb.commMMlmmhhhhhhhht
www
Reverse Process
Darclann
Fine-tuning
Caccder
Frovcial
Predicsed Window
Flatan
Pauling
Look-back Window
Two-stage training process. Pre-training: TimeDART teaches encoder to learn temporal representa- tions through noising process and autoregressive denoising. Fine-tuning: The pre-trained encoder is adapted with task- specific heads for downstream applications.
Figure 1:
ios often involve massive amounts of unlabeled real-world data. To analyze and leverage such data, self-supervised learning has emerged as a widely adopted approach (Zhang et al., 2024b). This approach extracts valuable knowledge from abundant unlabeled data, which can be subsequently transferred boost downstream task performance (Park et al., 2024; Dong et al., 2024b).
1. Introduction
Reviewing previous studies (Zhang et al., 2024b), existing methods for time series self-supervised learning primar- ily fall into three categories. The first category masked autoencoders, which focus on reconstructing masked or corrupted parts of the input and excel at learning the under- lying patterns of the time series (He al., 2022; Zerveas et al., 2021). A common challenge with these models is the inconsistency between pre-training and fine-tuning due to newly injected masked embeddings (Cheng et al., 2023b), Contrastive- based methods, as the second category, spe- cialize in sequence-level modeling and distinguish between similar and dissimilar time-series segments (Eldele et al.,
The analysis time-series data has become increasingly critical across diverse application domains, including health- care (Cheng et al., 2024), finance (Black & Scholes, 1973) and energy management (Zhou et al., 2024). These scenar-
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China. Correspondence to: Mingyue Cheng <mycheng @ .ds..edd.cn>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TimeDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
2021; Tonekaboni et al., 2021). Nonetheless, their emphasis on sequence-level modeling may limit their capacity to cap- ture fine-grained temporal variations (Dong et al., 2024a). The third is autoregressive methods, which are well-suited for modeling left-to-right sequential relationships and have promising scale-up potential (Liu et al., 2024c). However, these methods tend to overfit noise and anomalies in time series data (Lim & Zohren, 2021). Moreover, Using the common squared error loss in autoregressive time series modeling tends to make the current value's s prediction col- lapse into a Gaussian distribution centered on past values with fixed variance (Li et al., 2024b), which deviates sig- nificantly from world scenarios.
2. Related Work
2.1. Self-supervieed Learning in Time Series
Self-supervised learning has emerged a powerful pre- training approach in fields like natural language process- ing and computer vision (Brown et al., 2020; He al., 2022). Unlike supervised learning, which relies on labeled data, self-supervised methods generate supervision from the data's 's structure. Applying this to time-series data presents unique challenges due to its sequential and temporal charac- teristics. Current approaches (Zhang et al., 2024b; Cheng et al., 2025a) can be categorized into three main paradigms: masked reconstruction, contrastive discrimination and au- toregressive prediction.
We argue that an ideal self-supervised approach should ad- dress the above challenges, that is, how to simultaneously capture long-term dynamic evolution and subtle local pat- terns in unified manner. Although the autoregressive self- supervised learning shows significant potential due to its natural alignment with the left-to-right temporal dynamics, a key challenge lies in its limited capacity for modeling fine- grained patterns and its tendency to collapse into a Gaussian distribution. We attempt to address this by incorporating a diffusion denoising process within the autoregressive opti- mization framework. This design teaches the model to learn useful representations from the explicit added noise during autoregressive pre-training, and rationalizing the squared error of this regression problem using the diffusion loss.
Masked Reconstruction As fundamental approach in self-supervised representation learning, mask reconstruc- tion typically involve reconstructing masked or corrupted inputs, encouraging the model to learn meaningful repre- sentations. Masked time series modeling, introduced by TST (Zerveas et al., 2021), predicts missing time points from available data. Methods like STEP (Shao et al., 2022), PatchTST (Nie al., 2023), and CrossTimeNet (Cheng et al., 2025b) extend this approach by operating series, reducing computational costs while capturing local information. More recent work, such as TimeMAE (Cheng et al., 2023b), introduces decoupled masked autoencoders to address inconsistencies between the pre-training and fine- tuning phases. Additionally, SimMTM (Dong et al., 2024b) improves masked time-series modeling by recovering miss- ing time points through weighted aggregation of neighbors, leveraging the manifold structure of the data.
sub-
Building upon the analysis above, we propose TiReDART, a novel self-supervised time series representation frame- work that unifies two powerful generative paradigms, as illustrated in Figure 1. Specifically, we initialize vanilla Transformer encoder and partition the input time-series data into non-overlapping patches. To prevent information leak- age during autoregressive optimization, we apply a causal mask within the encoder. We independently add noise to each patch and use a cross-attentioon-asee denoising de- coder, which leverages preceding sequence information to guide the denoising process. TimeDART achieves supe- rior performance across nine publicly available datasets on downstream forecasting and classification tasks.
Contrastive Discrimination This approach aims to dis- tinguish between positive and negative instance pairs by pulling similar instances closer and pushing dissimilar ones apart. For example, TNC (Eldele et al., 2021) uses the local smoothness of time series signals to define positive neighborhoods, while TS2Vec (Yue et al., 2022) introduces hierarchical framework that operates at both the instance and patch levels. Similarly, LaST (Wang et al., 2022) aims to separate seasonal and trend components in time series data within the latent space. CoST (Woo et al., 2022) combines both time and frequency domain information to capture seasonal and trend representations, improving the discrimi- native power of the learned features.
We propose TimeDART, novel self-supervised time series representation framework that unifies autoregres- sive modeling and denoising diffusion processes to learn more transferable representations.
We introduce a causal Transformer encoder, coupled with carefully noise addition and denoising, empow- ering TimeDART to effectively model long-term dy- namic evolution and subtle local patterns.
Autoregressive Prediction In time series, autoregressive prediction originated from ARIMA (Box et al., 2015), which combines autoregressive (AR) and moving average (MA) models with differencing. Later, with the rise of RNNs, THOC (Shen et al., 2020) introduced a self-supervised pretext task for multi-resolution single-step forecasting called Temporal Self- Self-Supervision. Recently, inspired by
TimeDART consistently outperforms state-of-hee-an baselines across nine public datasets, demonstrating strong adaptability across diverse downstream tasks.
2

---

TimeDART: A Difflssin Autoregressive Transformer for Self-Supervised Time Series Representation
pre-trained large language models (Brown et al., 2020), the potential of autoregressive models in time series has gained attention, leveraging their generalization ability and task versatility (Liu et al., 2024c). These language models have even been adapted as autoregressive predictors (Liu et al., 2024b) to achieve arbitrary input-output mappings. Nevertheless, the full potential of autoregressive prediction in time series representation learning and various down- stream tasks remains to be discovered. To address this gap, TimeDART uses autoregressive optimization during pre- training, improving time series representations and enabling strong performance across downstream tasks.
(+) F Ii: where the superscript i represents the channel index for zero mean and unit variance. After reconstruction, the original mean and standard deviation are restored to main- tain input distribution consistency (Kim et al., 2021).
Patching Embedding We use patches as the basic model- ing unit, capturing richer local information for more com- prehensive representations (Cheng et al., 2023a). Applying noise and denoising to individual points could lead to exces- sive sensitivity, whereas patches offer more stable embed- dings. To preserve the autoregressive property and prevent information leakage, we set the patch length P equal to the stride S, ensuring non-overlapping patches that respect the autoregressive assumption. For simplicity, we assume L, the time-series length, is divisible by P. resulting in N = p patches, which reduces computational complexity and facilitates the processing of longer sequences.
2.2. Diffusion Models in Time Series
In recent years, Denoising Diffusion Probabilistic Mod- els (Ho et al., 2020; Li et al., 2024b) have become powerful tools for time series modeling (Lin et al., 2024; Meijer & ann-ggrined
Chen, 2024) due to their unique advantages temporal modeling. Early models, such as CSDI (Tashiro et 2021), avoid autoregressive inference while incorpo- rating additional input masking. TimeGrad (Rasul et al., 2021) introduced autoregressive denoising with Langevin sampling, enhancing multivariate prediction. More recently, conditional diffusion models like TimeDiff (Shen & Kwok, 2023) have improved time series prediction by leverag- ing external information to guide the denoising process. However, these methods primarily focus on probabilistic time series prediction tasks, aiming to generate high-quality time series data, but their direct application to downstream tasks like time series forecasting often yields suboptimal re- sults (Zhang et al., 2024a). TiReDART uniquely integrates the denoising diffusion process within a self-supervised framework, enabling inner-patch modeling that preserves the subtle local patterns time series data.
Each patch (referred to as a clean patch) then passed through a linear embedding layer, transforming it into a high-dimensional representation:
1:N = Embeddingg@1..))
(I)
where E represents the patch embeddings, and we omit the channel index (i) for simplicity.
3.1.2. CAUSAL TRANSFORMER ENCODER
We initialize vanilla Transformer encoder (Vaswani et al., 2017) as the representation network. During pre-training, prepend a learnalee start-of-sequence (SOS) embedding to the clean patch representations while exclude the final one. To incorporate positional information, we apply sinusoidal positional encoding after the embedding layer. Following this, we use a causal mask M in the processing layer, limit- ing the visibility of each patch to itself and the prior patches. Finally, the causal encoder network can be expressed as:
3. Methodology
To capture both long-term dynamics evolution and subtle local patterns, TimeDART uses causal masking for autore- gressive optimization and a diffusion denoising process to preserve continuous representations. The following sections explore the key components of our approach.
in iINN = Concat[SOS, 21:N-1] + PEI:N, f f( = Encoder M),
where s(-) processes the input sequence with the causal mask, producing the final contextualized representations.
3.1. The Proposed TimeDART
1.3. PATCH-LEVEL DIFFUSION AND DENOISING
TimeDART involves three modules: normalization and patch embedding, causal Transformer encoder and a patch- level diffusion denoising process.
Forward Process For each patch x, E LI,N the forward process ((=))=}]1)
N(z); Va(s]2; (1 a())) gradually adds noise to the patch, where ass) is the noise scheduler. Let 7(s) be the cumulative product of a over time steps, where 7ss) = II <. a(s') ), the forward process can be rewritten given the original clean patch zg:
3. NORMALIZATION AND EMBEDDING
Instance Normalization Before feeding the multivariate time series data L1.L E RC into the representation net- work, we apply instance normalization to each instance
((rt=))=N(ee) V100)) (1
(4)
3

---

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervieed Time Series Representation
Myw
Embeddings
Autoregressive Optimization
Start of Sequence Embedding
Reverse Instance Normalization
Keys & Values
Masked Denolsing Decoder
Queries
Reverse Process
Causal Transformer Encoder
TimeDART
x1
200
X2
50
X
600
Sampling independently
Myw
x1
x1000
X2
X3
x1
Instance Normalization
Cosine
Notsa Schodulor
X5
Original Multivariate Time Series
Forward Process
Figure 2: The overall architecture of TimeDART, which employs patch-level modeling, enhances the power of the autoregressive self-supervised method through a diffusion denoising process. It captures both long-term dynamics evolution and subtle local patterns to strengthen consistent and effective representation learning.
As shown in Figure 2, independently add noise to each patch at time step enabling the model to learn varying denoising scales across the sequence. This prevents task oversimplification. The noisy patches are represented as:
encoder output position j aggregates information from clean patches positions 1 to 1, enabling autoregressive optimization. The reverse process is expressed as:
2j g( f(= 1 1 N,
(7)
i = [x
(5)
where g(-) denotes the processing of the encoder output and noise-added patch embeddings by the denoising decoder. Finally, deep representations are mapped back the original space via linear projection:
We use cosine scheduling approach for the noise scheduler, where (s)) cos (T) instead of the linear decrease in DDPMs (Ho et al., 2020). This smoother transition empha- sizes the early and late stages of diffusion, improving model stability and better capturing the data distribution. Further- more, both noise-added and clean patches share the same embedding layer and weights, with sinusoidal positional encoding applied:
out Projector(2
(8)
3.2. Self-supervised Optimization Objective
Traditional time series autoregressive pre-training objective is typically a simple mean squared error (MSE), aiming to minimize the discrepancy between the regressed and true patches. This can be formalized as follows:
*NN Embedding(21.N) + PEL.N,
where 2EN denotes embeddings of the noise-added patches.
Reverse Process The reverse process is handled by the de- noising decoder which takes the encoder output as keys and values, while the noise-added patch embeddings serve as queries. self-only mask applied to the decoder to ensure that the j-th input in the noise-added sequence attends only the j-th encoder output, Informed by the causal mask, the
N
Lmse O
lag (rojector(f(21)))))) Σ
(9)
While straightforward, this MSE objective fundamentally assumes that the underlying data distribution is Gaussian.

---

TimeDART: A Diffussin Autoregressive Transformer for Selt-Supervised Time Series Representation
This assumption is evident when viewing MSE through the lens of maximum likelihood estimation:
Table I: Dataset Statistics: The top part shows forecasting datasets, bottom part shows classification datasets.
20 |0 Projector(((<$1;-1)))
DATASETS
DOMAIN
VARIABLES
LENGTH
(10)
ETTh1/ETTh2
Energy
7
17420
- -oo: (f
1TTTI/
Energy
7
69680
Electricity
Energy
321
26304
where a is constant and C is a constant determined by a. If it is valid, sampling 2) could be achieved via o = (rojector([(=1))-- toe, where is standard Gaus- sian noise. This implies time series observations are either very noisy or follow a unimodal, bell-shaped distribution. Real-world time series, however, often display complex, multimodal behaviors and intricate dependencies that sim- ple Gaussian assumption poorly captures.
Traffic
Transportation
862
17544
Weather
Weather
21
52696
Exchange
Finance
8
7588
PEMS03
Transportation
358
26208
PEMS04
Transportation
307
16992
PEMS07
Transportation
883
28224
PEMS08
Transportation
170
17856
HAR
HAR
9
14717
Epilepsy
HAR
3
413
To overcome these limitations, TimeDART replaces MSE with diffusion loss, allowing the model to express a richer, multimodal belief over time series data. This self-supervised diffusion loss, formally equivalent to the Evidence Lower Bound (Ho et al., 2020; Chen al., 2024):
EEG
EEG
2
15629
datasets for forecasting tasks (both ETT and PEMS datasets contain four subsets each) and three datasets for classifi- cation tasks. To clearly illustrate the practical scenarios and scale of these datasets, the table provides an overview of each dataset's domain, the number of samples, and the number of variables per sample. The dataset scale enables us to concretely illustrate the model size and computational requirements of the TiRT architecture. A more com- prehensive description can be found in Appendix A.
Laitt EE [|ke - o( -(= (11)
This diffusion loss, embedded in the autoregressive opti- mization objective, effectively captures both long-term dy- namic evolution and subtle local patterns. The core idea is to make the square еπrоr a reasonable loss by gradually adding and then removing noise (Su, 2024; Li et al., 2024a). Refer to Appendix C for the detailed derivation.
Embedding Layer and Backbone In TimeDART, a simple linear transformation is used for the embedding layer. Additionally, to ensure the autoregressive mecha- nism functions correctly, it is crucial that patches remain non-overlapping to prevent information leakage. Therefore, prioritize using vanilla non-overlapping patch embed- ding layer and Transformer encoder, rather than relying on existing backbones such as PatchTST (Nie et al., 2023) or iTransformer (Liu et al., 2024a). To investigate whether TimeDART maintains consistent performance across dif- ferent backbones, we conducted additional experiments by replacing the Transformer encoder with a causal TCN under specific configurations, it also satisfies the aforemen- tioned requirements. Regardless of the backbone, we use vanilla Transformer decoder as the denoising decoder, with parameters identical to those of the Transformer encoder, except for the number of layers.
3.3. Downstream Transfering
After pre-training, the denoising decoder is discarded, while the embedding layer and encoder are transferred. The en- coder is then adapted with task specifpe heads for various downstream tasks. In forecasting, fine-tuning is performed on both the look-back and predicted windows, with flat- ten head for one-step prediction, optimized using MSE loss. In classification, fine-tuning is performed on the input sequence with corresponding labels, followed by a max- pooling head that projects the latent representations onto the labels, and is optimized using cross-entropy loss.
4. Experiments
We demonstrate TimeDART's effectiveness in forecasting and classification across diverse datasets. Ablation studies confirm the power of its autoregressive mechanism and the critical role of diffusion denoising process, further showing its robustness through parameter analysis.
Baselines We compare TimeDART against four state-of- the-art self-supervised baseline methods, including a con- trastive learning method: CoST (Woo et al., 2022), and three masked modeling methods: SimMTM (Dong al., 2024b), TimeMAF (Cheng et al., 2023b) and the self-supervised ver- sion of PatchTST (Nie et al., 2023). Since we use a vanilla Transformer encoder as the backbone instead of what is proposed in existing supervised methods, we believe it is essential to compare with advanced supervised methods
4.1. Experimental Setup
Datasets We summarize the publicly available datasets used in our experiments in Table 1, which includes six
5

---

TimeDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
Table 2: Multivariate time series forecasting results, All results are averaged MSE and Mean Absolute Error (MAB) from 4 different predicted windows of {12, 24, 36, 48) for PEMS datasets and {96, 192, 336, 720} for others. The best results are in bold and the second best are underlined. Full results are detailed in Appendix D.
OURS
SELF-SUPERVISED
SUPERVISED
METHODS MOTRIC
TIMEDART
RANDOM
INIT. MAE
SIMMTM MAE
PATCHTST
TIMEMAE
CaST
PATCHTST
DLINEAR
MSE
MAE
MSE
MSE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTal
0.411
0.426
0.439
0.444
0.409
0.428
0.433
0.437
0.434
0.445
0.465
0.464
0.427
0.435
0.439
0.449
ETTH2
0.346
0.387
0.358
0.396
0.353
0.390
0.354
0.393
0.402
0.431
0.399
0.427
0.357
0.395
0.458
0.459
ETTM1
0.344
0.379
0.351
0.383
0.348
0.385
0.342
0.380
0.350
0.383
0.356
0.385
0.362
0.388
0.361
0.383
ETTM2
0.257
0.316
0.269
0.323
0.263
0.320
0.272
0.327
0.270
0.326
0.282
0.343
0.270
0.329
0.281
0.343
ELECTRICITY
0.163
0.254
0.177
0.277
0.162
0.256
0.163
0.255
0.196
0.309
0.215
0.295
0.167
0.260
0.168
0.265
TRAITIC
0.388
0.263
0.410
0.277
0.392
0.264
0,404
0.272
0.410
0.275
0.435
0.362
0.421
0.284
0.435
0.297
WEATHER
0.226
0,263
0.231
0.268
0.230
0,271
0.227
0.262
0.227
0.265
0.242
0,282
0.226
0.263
0.246
0.298
EXCHANGE
0.359
0.405
0.440
0.450
0.451
0.455
0.376
0.413
0.427
0.446
0.456
0.455
0.379
0.414
0.393
0.425
PEMS03
0.152
0.257
0.164
0.266
0.158
0.260
0.156
0.261
0.165
0.269
0.169
0.273
0,178
0.288
0.277
0.373
PEMS04
0.133
0.245
0.145
0.255
0.143
0.253
0.139
0.249
0.144
0.256
0.147
0.262
0.149
0.266
0.290
0.381
PEMS07
0.128
0.232
0.138
0.243
0.131
0.236
0.132
0.237
0.137
0.241
0.139
0.245
0.149
0.253
0.322
0.387
PEMS08
0.201
0.282
0.213
0.293
0.206
0.286
0,206
0,287
0.211
0.292
0.215
0.295
0.230
0.295
0.359
0,402
Table 3: Multivariate time series results of general pre-training and fine-tuning across different domains. All results are averaged MSE and MAE from different predicted windows of {12, 24, 36, 48} for PEMS datasets and {96, 192, 336, 720} for others. The best results are in bold and the second best are underlined. Full results are detailed in Appendix D.
METHODS
TIMEDART
RANDOM INTT.
SIMMTM
PATCHTST
TIMEMAE
COST
IN-DOMAIN
DOMAIN
METRIC
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ENERGY
ETTM2
0.255
0.313
0.268
0.328
0.262
0.321
0.265
0.325
0.269
0.329
0.278
0.337
0.257
0.316
WEATHER
WEATHER
0.227
0.263
0.238
0,272
0.233
0.268
0.229
0.263
0.229
0.263
0.254
0.284
0.226
0.263
FINANCE
EXCHANGE
0.350
0.399
0.457
0.473
0.440
0.466
0.379
0.411
0.407
0.424
0.478
0.484
0.359
0.405
TRANSPORTATION
PEMS04
0.127
0.236
0.141
0.251
0.133
0.244
0.1
0.243
0.138
0.248
0.142
0.252
0.133
0.245
that feature carefully designed architectures. For the fore- casting task, we compare with the supervised version of PatchTST (Nie et al., 2023) and the linear-based method DLinear (Zeng et al., 2023). For the classification task, we compare with FormerTime (Cheng et al., 2023a), which em- ploys a hierarchical Transformer network as the backbone. More implementation details can be found in Appendix I.
Clossificttio (average accuracy)
0.91
0.90
SIMMIN
CLireer
Pasmingr-os
Fomeetine
Taneeess
TinweekT
0.89
0.88
0.7
0.6
0.31
Fair Experiment To ensure experimental fairness, we employ a unified embedding layer and Transformer encoder with consistent model-specific parameters. For forecast- ing experiments, implement channel-independent con- figuration (Nie et al., 2023) to enhance robustness (Han et al., 2024) and facilitate cross-domain analysis. In ad- dition, we conduct forecasting experiments under two set- tings: in-domain and cross-domain. In the in-domain setup, a look-back window of L = 336 and prediction horizons H E {96, 192, 336, 720} are used for all datasets, except PEMS where L 96 and H {12, 24, 36, 48). For the cross-domain setting, all forecasting datasets are mixed for general pre-training with a consistent look-back window of =333 while downstream fine-tuning maintains. To high- light benefits of pre-training, we also include baseline Ran- dom Init., involving direct fine-tuning without pre-training. Supervised methods retain their original implementations,
Forecasting (average MSE)
Figure 3: Comparison between TimeDART and baselines for the forecasting task (MSE 4) across forecasting datasets the x--axis and the classification task (Accuracy 1) across classification datasets on the y-axis.
with only non-core training parameters adjusted for align- ment. Further details on experimental fairness are provided in Appendix B.2.
4.2. Main Result
As shown in Figure 3. we visually present the performance of TiRTDART on two mainstream time series analysis tasks: forecasting (average MSE across forecasting datasets on the x z-axis) and classification (average accuracy across classifi- cation datasets on the y-axis). In the majority of experimen-
6

---

TimcDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
Table 4: Multivariate time series classification results. Results are are reported as Accuracy (Acc.) and Macro-F1 (FI), The best results are in bold and the second best are underlined.
OURS
SELF-SUPERVISED
SUPERVISED FORMERTIME
METHODS
TIMEDART
RANDOM INIT.
SIMMTM
PATCHTST
TIMEMAE
CoST
METRIC
Acc.
F1
Acc.
F1
AcC.
F1
ACC.
F1
ACC.
F1
ACC.
F1
Acc.
F1
HAR
0.9247
0.9286
0.8738
0.8723
0.9200
0.9220
0.8789
0.8773
0.9204
0.9248
0.8997
0.8927
0,8816
0,8878
EPILEPSY
0.9712
0.9698
0.9265
0.9237
0.9565
0.9543
0.9312
0.9234
0.9459
0.9584
0.9198
0.9156
0.9315
0.9341
ERG
0.8269
0.5983
0.7752
0.5138
0.8165
0.6123
0.8076
0.5460
0.8148
0.5787
0.7918
0.5314
0.8102
0.5658
Table 5: Performance of TCN as backbone. Average MSE and MAE from 4 different predicted windows for forecast- ing while Accuracy and Macro-F1 for classification task.
Table 6: The results of ablation study, Average MSE and MAE from 4 different predicted windows for forecasting while Accuracy and Macro-F1 for classification task.
METHOD
TCN
RANDOM INIT.
TRANSFORMER
METHOD
TIMEDART
w/o AR
w//o DIFF
w/o AR-DIR
FORECASTING
MSE
MAE
MSE
MAE
MSE
MAE
FORECASTING
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
ETTH2
0.349
0.396
0.357
0.403
0.346
0.387
ETTH2
0.346
0,387
0.365
0.399
0.352
0.391
0.364
0.398
ETTM2
0,263
0.323
0.269
0.326
0.257
0.316
ETTM2
0,257
0.316
0.338
0.322
0.346
BLECTRICITY
0.165
0.254
0.177
0.278
0.163
0.254
ELECTRICITY
0.163
0.254
0.193
0,304
0.164
0.255
0.190
0.299
PEMS04
0.134
0.246
0.145
0.256
0.133
0.245
PEMS04
0.133
0.245
0.144
0,255
0.145
0.256
0.149
0.260
CLASSIFICATION
Acc.
F1
Acc.
F1
Acc.
F1
CLASSIFICATION
Acc.
F1
Acc.
F1
ACC.
Acc.
HAR
0,9252
0.9250
0,8842
0.8901
0.9247
0.9249
HAR
0,9247
0.9286
0.8966
0.8994 0. 0.9002
0.9028
0,8785
0.8756
EPILERSY
0,9723
0.9689
0.9525
0.9513
0.9712
0.9698
EPILEPSY
0,9712 0.9698
0.9505 0.9518 0.9598 0.9586 0,9486 0.9472
tal settings, TimeDART achieves substantial improvements compared to both self-supervised and supervised methods.
in the in-domain setting, except for the Weather dataset, where performance slightly lags, This may be attributed to the use of the same model-specific parameters across all domains in general pre-training. which could not be finely tuned to the specific requirements of the weather domain. Nevertheless, the results from cross-domain fine- tuning strongly indicate TimeDART's robust capabilities in such scenarios. The full results detailed in Appendix D.
Forecasting In the in-domain setting and after down- stream fine-tuning for the forecasting task, TimeDART outperforms its competing baselines in most experimen- tal scenarios. As shown in Table 2, TimeDART achieves the best results in 83.3% of the 24 evaluation metrics, with the remaining results being second best. TimeDART further achieves 6.8% reduction in MSE compared to random initialization and 3% compared to the state-of-the-ard bench- marks, despite most methods showing noticeable gains over the random initialized model. Compared to supervised mod- els like PatchTST and DLinear, TiReDART, even with ran- dom initialization, consistently performs competitively or better in most cases. Moreover, it significantly surpasses these models after pre-training, highlighting that the use of a vanilla Transformer as the backbone does not limit TimeDART's performance. In fact, this choice underscores the robustness of the proposed pre-training framework. The visualized results are provided in Appendix H to demon- strate the effectiveness of our method.
The experimental results for the classifi- cation task are shown in Table 4. To further explore the effectiveness of TimeDART's representation learning, we s also perform supervised fine-tuning on the classification task. The results in Table 4 show that after pre-training, TimeDART outperforms the baselines in terms of accuracy and F1 score in most experimental settings, even surpassing supervised methods specifically designed for classification tasks. Notably, after pre-training, TARTDART improves its average accuracy by approximately 5.7% compared to ran- dom initialization. This suggests that through high-level au- toregressive optimization and low-level noise-added patch reconstruction, TimeDART captures deep discriminative features of time series data, ensuring its effectiveness in downstream classification tasks.
Classification
In the cross-domain setting, after general pre-training, we fine-tune TimeDART on four distinct domains. As shown in Table 3, TimeDART outperforms all compet- ing self-supervised pre- pre-training methods across multiple domains, demonstrating the effectiveness of transferring knowledge from various domains. Moreover, after pre- training, TimeDART significantly outperforms random ini- tialization, highlighting the value of the pre- pre-training process. In most domains, the results after pre-training surpass those
TCN as Backbone To evaluate the effectiveness of TimeDART with different backbones, replace the Trans- former encoder with causal TCN and remove the causal mask in downstream tasks for consistency. As shown in Table 5, the causal TCN performs slightly worse than the Transformer encoder in forecasting tasks, likely due to the mismatch between the TCN and the Transformer denoising
7

---

TimeDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
Table 7: Hyperparameter sensitivity analysis of noise steps and noise schedulers. Average MSE and MAE from 4 different predicted windows for forecasting while Accuracy and Macro-FI for classification task. See Appendix F.I for full results.
FTTM2
PEMS04
EPILERSY
(4)
Nousn
STrP
NoIsE. ScHEEULIE
TOTAL
NoisE STOPS
(8) Nousn SCHEDOLLR
TaTst, NoIsE
Noisn SCHTEBLLER
VALUS
MSB
MAB
TYPS
MSE
MAB
VALUE
MSE
MAE
TYPE
MSE
MAB
VALUE
ACC.
TYPE
ACC:
750
0,258
0.317
Cos.
0.257
0.316
750
0.134
0.245
Cos.
0.133
0.245
750
0.9195
0.9156
Cos.
0,9247
0.9286
0.257
0.316
LI8.
0.266
0.320
1000
LIN.
0.343
0.254
1000
0.9247
0.9286
0.8976
0.8959
1250
0.257
0.315
RANDOM INIT.
0.269
0.325
1250
0.135
0.247
RANDOM INIT.
0.145
0.255
1250
0.9158
0.9279
RANDOM INIT.
0.9265
0.9237
Mask ratio =0.0
Mask ratio =.5
Mask ratio -10
0.10
0.20
Layers of Conoling Dacader
Patch Langhh
Causal Mask
Partial Causal Mask
Self-Only Mask
Figure 5: Hyperparameter analysis of number layers of denoising decoder and patch length in TimeDART. The triangle symbol represents the best result.
Figure 4: Masking strategies: causal mask (left) sees all prior patches, partial causal mask (middle) sees some based on the mask ratio, and self-only mask (right) sees only itself.
comparing cosine and linear schedulers. The number of diffusion steps, with higher T values increasing pre-training difficulty, and the noise scheduler, with the cosine sched- uler offering smoother transitions than the linear one, both influence the recovery of clean patches. As shown in Ta- ble 7, the total number of noise steps does not significantly impact the difficulty of pre-training However, the cosine noise scheduler performs substantially better than the linear scheduler while the latter sometimes even yields worse re- sults than random initialization. This highlights the critical importance of the noise scheduler, as insufficiently smooth noise addition can result in significantly poorer outcomes.
decoder used in pre-training. However, in the classification task, TCN performs competitively, even outperforming the randomly initialized Transformer encoder. This suggests TCN's stronger ability to capture discriminative time-series features. Crucially. TCN still surpasses its random initializa- tion, demonstrating TimeDART's adaptable sulf-supe pre-training across diverse backbones.
4.3. Ablation Study
We investigate the effectiveness of two key modules: au- toregressive mechanism and the denoising diffusion process. Four experimental settings are considered: TimeDART (the original model), w/o AR (autoregressive mechanism re- moved), w/o diff (denoising diffusion process removed), and w/o AR-diff (both modules removed). Specifically, in the autoregressive removal experiment, we eliminate both the causal mask in the Transformer encoder and the mask in the denoising decoder. The results in Table 6 demonstrate that both the autoregressive generation and the denoising diffusion model play crucial roles in the effectiveness of this approach. Notably, removing the autoregressive mechanism results in performance even worse than random initializa- tion, demonstrating its erucial role in capturing long-tem term dynamic evolution. Similarly, removing the denoising diffu- sion process also leads to significant performance degrada- tion, highlighting its effective contribution to capture subtle local pattern information.
Reverse Process We then evaluate the impact of two pa- rameters through the reverse process: the layer numbers of the denoising decoder and the patch length. The layer numbers, selected from [0, 1, 3). reflects the relative size of the denoising decoder compared to the fixed 2-layer en- coder, with layers serving as the ablation case w/o diff. As shown in Figure 5, allocating too many layers to the denois- ing decoder can result in under-training of the representation network due to an over-concentration of parameters in the denoising component. Additionally, we explore the impact of different mask strategies applied to the denoising decoder, as shown in Figure 4. We find that the self-only mask al- ready contains all the necessary information for denoising, and adding extra information actually hinders the denoising process. These results are detailed in Appendix F.2.
On the other hand, the patch length controls the amount of local segment information that each patch carries. Longer patch length is more effective for datasets like Electricity, which exhibit higher redundancy between consecutive data points, while shorter length is preferred for datasets with less
4.4. Hyperparameter Sensitivity
Forward Process We first investigate two key parameters through the forward process: the total number of diffusion steps T E (750, 1000, 1250} and the noise scheduler ((s),
8

---

TimcDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
Table 8: Fine-tuning under few-shot and linear probing settings, Average MSE and MAE from 4 different predicted windows for forecasting while Accuracy and Macro-F1 for classification task. See Appendix G for full results.
FTT12
PEMS04
HAR
Fiw-snerr MSB
LINEAR
PRODING
FIN-SHOT
(0)
PROBING
(A)
Frn-SHOT ACC.
LINEAR PROMING
PORTION
MAB
METROD
MSE
MAE
PORTION
MSE
MAE
METHOB
MSE
MAE
PORTION
METHOD
ACC.
Fi
5%%
0.365
0.392
TIMEDART
0.354
0.391
5%
0,142
0.254
TIMEDART
0.145
0.258
5%
0.5843
0.5901
TIMEDART
0.8976
0.9005
1016
0.354
SIMMIM
0.357
0.395
10%
0.149
SIMMIM
0.250
10%
0.9014
0.9053
SIMMIMM
0.8732
0.8756
FULL:
0.346
0.387
TIMEMAE
0.361
0.397
POLL
0.133
0.245
TIMEMAB
0.152
0.264
FULL
0.9247
0.9286
TIMEMAB
0.3858
0.8962
RAN. 0.358
0.396
0.368
0.401
0.145
0.255
RANDOM
0:271
0.5735
RANDOM INIT.
0.8578
redundancy, as they capture finer temporal dynamics. Fig- ure 5 underscores the importance of adaptive patch length selection based on dataset characteristics.
ET
C373
4.5. Analysis Experiment
C332
Few-shot Evaluation In real-world applications, labeled time series data often starts with a limited number of ob- servations. This can make it tough to train accurate mod- els. To see how well TimeDART performs in these situa- tions, we conduct few-shot evaluations using the ETTh2, PEMS04, and HAR datasets. Specifically, we train our mod- els using only a small fraction (either 5% or 10%) of the available training data, a method consistent with previous research (Liu et al., 2024d). We then evaluate these models on the complete test set. The results of this evaluation are presented in Table (a).
Bectrictty
PEM504
TeaDNe
6:30
18c
Figure 6: Pre-training and Fine-tuning the model to the inputs with extended length. The MSE averaged from all predicted horizons is reported.
fectively teaching the model to extract valuable representa- tions even from noisy data. This intentional design allows TiRTDART to achieve superior forecasting performance with longer look-back windows.
The results show that while fine-tuning with a small portion of samples yields weaker performance than fine-tuning with the full dataset on both forecasting and classification tasks, it still outperforms full-data supervised fine-tuning under random initialization in the vast majority of cases. This corroborates the strong capabilities of the pre-trained repre- sentation network. The results are detailed in Appendix G.
Furthermore, the look-back window increases, we ob- serve consistent improvement in forecasting in Figure 6. This demonstrates that TimeDART successfully captures meaningful information from extended input length, and crucially, robustly mitigates the impact of increasing noise that often accompanies longer inputs.
Linear Probing As an important fine-tuning setting, we also experiment with the linear probing, where we fix the pre-trained encoder and only fine-tune the newly added projector at the end of the model. Table 8 (b) illustrates the performance compared to some of the compared baselines.
5. Conclusion
In this paper, we introduced TimeDART, a novel pre-training framework for self-supervised time series representation learning that unifies two effective generative paradigms. By employing a causal Transformer encoder with a patch-based embedding strategy to model global trends and incorporat- ing a denoising diffusion process to capture fine- tne-grained local patterns, TimeDART effectively addresses both long- term dynamic evolution and subtle local features. Extensive experiments on benchmark datasets for time series forecast- ing and classification reveal that TimeDART consistently outperforms compared methods, confirming its effective- ness. A series of extra analyses confirmed TimeDART's suit- ability and robustness in numerous supersuue settings. With its unique approach, TimeDART has the potential to pave the way for future advancements in self-supervised representation learning for time series.
Our findings indicate that TimeDART's performance in the linear probing setup exceed that of several baseline mod- els, providing additional evidence for the efficacy of our pre-training framework. Concurrently, linear probing of- ten matches or surpasses full-data supervised fine-tuning from random initialization (e.g., MSE 0.354 vs. 0.358), demonstrating the generalizability of TimeDART's learned representations across various downstream tasks. The re- sults are detailed in Appendix G.
Adapt to Extended-Length Input The standard fore- casting framework can often struggle when faced with ex- tended input lengths, a challenge frequently attribute to the increased noise inherent in longer time series. However, TimeDART's pre-training explicitly incorporates noise, ef-

---

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervieed Time Series Representation
Acknowledgments
The Thirteenth International Conference on Learning Representations, 2024.
This research was supported by grants from the National Natural Science Foundation of China (62337001), the grants of Provincial Natural Science Foundation of Anhui Province (No. NN0.2408085QF193) the Key Technologies R & D Program of Anhui Province (No. 202423k09020039) and the Fun- damental Research Funds for the Central Universities (No. YD2150002501, No. 1K2150
Cheng, M., Liu, Q. Liu, Z., Li, Z., Luo, Y., and Chen, E. Formertime: Hierarchical multi-scale representations for multivariate time series classification. In Proceedings of the ACM Web Conference 2023, pp. 1437-1445, 2023a.
Cheng, M., Liu, Q., Liu, Z., Zhang, H.,, Zhang, R., and Chen, E. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv 2023b.
Impact Statement
preprint arXiv:2303.003:
This paper introduces a novel time series pre-training frame- work, where seamlessly unify autoregressive mechanism and the denoising decoder process to learn more transferable representations. Our work offers valuable insights for future research in this domain. Experimental results demonstrate the effectiveness of our method across various domains and its potential real-world applicability. We have ensured that publicly available,
Cheng, M., Zhang, J., Liu, Z., Liu, C., and Xie, Y. Hmf: A hybrid multi factor framework for dynamic intraoperative hypotension prediction. International joint conference of artificial intelligence, 2024.
Cheng, M., Liu, Z., Tao, X., Liu, Q. Zhang, J., Pan, T., Zhang, S., He, P., Zhang, X., Wang, D., et al. A com- prehensive survey of time series forecasting: Concepts, challenges, and future directions. Authorea Preprints, 2025a.
all datasets used in the experiments and we have carefully considered the ethical implications of our work. Therefore, we believe that our research does not present any ethical or moral risks.
Cheng, M., Tao, X., Liu, Q., Zhang, H.,, Chen, Y., and Lei, Learning transferable time series classifier with cross- domain pre-training from language model. International Conference on Web Search and Data Mining, 2025b.
References
Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C.,, David, P., and Elger, C. E. Indications of nonlinear deter- ministic and finitedimenniooal structures in time series of brain electrical activity: Dependence on recording re- gion and brain state. Physical Review 64(6):061907, 2001.
Dong, J., Wu, H., Wang, Y., Qiu, Y., Zhang, L.,, Wang, J., and Long, M. Timesiam: A pre-training framework for siamese time-series modeling. In ICML, 2024a.
Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., and Long, M. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36, 2024b.
Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J. L., et al. A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pp- 3, 2013.
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., and Guan, C. Time-series representation learning via temporal and contextual contrasting, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021.
Black, F. and Scholes, M. The pricing of options and cor- porate liabilities. Journal of political economy, 81():: 637-654, 1973.
Box, G., Jenkins, G., Reinsel, G., and Ljung, G. Time Series Analysis: Forecasting and Control. Wiley Se- ries in Probability and Statistics. Wiley, 2015. ISBN 9781118674925. URL books?dd-NiSCgAMQEE.
Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. F. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23): c215-c220, 2000.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Guokun Lai. Exchange Dataset. https//github..cmm/ laiguokun/multivaratt--- siim--eeree--aaa..... 2017.
Han, L., Ye, H.-J., and Zhan, D.-C The capacity and ro- bustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. JEEE Transactions on Knowledge and Data Engineering, 2024.
Chen, X., Liu, Z., Xie, S., and He, K. Deconstructing denoising diffusion models for self-supervised learning.
10

---

TimeDART: A Diffusson Autoregressive Transformer for Self-Supervieed Time Series Representation
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Liu, Z.,, Yang. J., Cheng, M., Luo, Y., and Li, Z. Gener- ative pretrained hierarchical transformer for time series forecasting. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2003-2013, 2024d.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-685 2020.
Meijer, C. and Chen, L.. Y. The rise of diffusion models in time-series forecasting. arXiv preprint arXiv:2401.03066 2024.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Nie, Y., H. Nguyen, N., Sinthong, P., and Kalagnanam, J. time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
Kingma, D. and Ba, J. Adam: A method for stochas- tic optimization. International Conference on Learning Representations, 2014.
Park, J., Gwak, D., Choo, J., and Choi, E. Self-supervised contrastive learning for long-term forecasting. In The Twelfth International Conference on Learning Represen- lations, 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424- 56445, 2024a.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. 2017.
Li, Y. Bornschein, J., and Chen, T. Denoising au- toregressive representation learning. arXiv preprint arXiv:2403.05196, 2024b.
PeMS. Traffic Dataset. http://pems.dot..a.gvvv/ 2001.
Rasul, K., Seward, C., Schuster, I., and Vollgraf, R. Au- toregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International conference on machine learning, pp. 8857-8868. PMLR, 2021.
Lim, B. and Zohren, S. Time-series forecasting with deep learming: a survey. Philosophical Transactions of the Royal Sociery A: Mathematical, Physical and Engi- neering Sciences, 379((194):2020022)) February 2021.
ISSN 1471-2962. doi: 10. 1009////t.222200000009 URL http://dx..oi.orgg// 10w.11999t/taa.00.00..0
Shao, Z., Zhang, Z.,, Wang, F., and Xu, Y. Pre-training enhanced spatial- graph neural network for mul- tivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD '22, pp. 1567-1577. ACM, August 2022. doi: 10. 10.114557534467835539966 URL http://dx.doi.org// http:/...........11//////....... 111111//5553478835999999
Lin, I.,, Li, Z., Li, R., Li, x., and Gao, J. Diffusion mod- cls for time-series applications: a survey. Frontiers of Information Technology & Electronic Engineering. 25(1): 19-41, 2024.
Liu, M., Zeng, A., Chen, M., Xu, Z., Lai, Q. Ma, L.,, and Xu, Q. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816 5828, 2022.
Shen, L. and Kwok, J. Non-autorregressive conditional diffusion models for time series prediction. In Inter- national Conference on Machine Learning, pp. 31016- 31029. PMLR, 2023.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effec- tive for time series forecasting. International Conference on Learning Representations, 2024a.
Shen, L., Li, Z., and Kwok, J. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems, 33:1 1016- 13026, 2020.
Liu, Y., Qin, G., Huang, X., Wang, J., and Long, M. Au- totimes: Autoregressive time series forecasters via large language models. Advances in Neural Information Pro- cessing Systems, 37:122154-122 122184, 2024b.
Su, J. A brief discussion on multimodal thinking in "closed- door development" (part 2): Autoregression, Jul 2024. URL http:///paees.accccoaarreee/////
Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., and Long, M. Timer: Generative pre-tran pre-trained transformers are large time series models. In Forty-first International Conference on Machine Learning, 2024c.
Tashiro, Y., Song, J., Song, Y., and Ermon, S. Csdi: Con- ditional score-based diffusion models for probabilistic time series imputation. Advances in neural information processing systems, 34:24804 24816, 2021.

---

TimeDART: A Diffusion Autoregressive Transformer for Selt-Supervised Time Series Representation
Tonekaboni, S., Eytan, D., and Goldenberg, A. Unsuper- vised representation learning for time series with tempo- ral neighborhood coding. International Conference Learning Representations, 2021.
Zhou, J., Lu, X., Xiao, Y., Tang, Su, J., Li, Y., Liu, J., Lyu, J., Ma, Y., and Dou, D. Sdwpf: A dataset for spatial dynamic wind power forecasting over large turbine array. Scientific Data. 11(1):649, 2024.
UCL.
UCI
Electricity Load Time Series https://archive.ics.uuu http://acchiveee.ss.....dd/////d uci.eddu////attases///
Dataset. Fleetrieityl LoadDiagramm200 12014, 2015.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, t., and Polosukhin, I. At- tention all you need. Advances in neural information processing systems, 30, 2017.
Wang, Z., Xu, X., Zhang, W., Trajcevski, G., Zhong, T., and Zhou, Learning latent seasonal-trend represen- tations for time series forecasting. Advances in Neural Information Processing Systems, 35:38775-387877 2022.
Wetterstation. Weather Dataset. https:/wwww.bge--ena.mpg dc/wetter/, 2006.
Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Cost: Contrastive learning of disentangled seasonal-trent. rep- resentations for time series forecasting. International Conference on Learning Representations, 2022.
Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., and Xu, B. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Zeng, A., Chen, M., Zhang, I.., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11121 1128, 2023.
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. A transformer- based framework for multi- variate time series representation learning. In Proceed- ings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 2114-2124, 2021.
Zhang, J., Cheng, M., Tao, X., Liu, Z. and Wang, D. Condi- tional denoising meets polynomial modeling: flexible decoupled framework for time series forecasting. Interna- tional Joint Conference on Artificial Intelligence, 2024a.
Zhang, K., Wen, Q. Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., al. Self- supervised learning for time series analysis: Taxonomy, progress, and prospects. IEEE transactions on pattern analysis and machine intelligence, 2024b.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-1 11106-11115, 2021.
12

---

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
A. Dataset Descriptions
To evaluate the effectiveness of our proposed TimeDART, we conducted extensive experiments on time series datasets for forecasting and 3 for classification. These datasets cover variety of application scenarios, including power systems, transportation networks, and human activity. For detailed descriptions of the datasets and their respective divisions, please refer to Table 9.
Table 9: Full dataset descriptions. Samples are organized in (Train/Validation/Tes)))
TASKS
DATASETS
LOOK-BACK
PREDICTED
CLASSES
VARIABLES
SAMPLES
DOMAIN
FREQUENCY
ETTALETTH2
336
(96,992336)
7
8209/2785/2785
Energy
Hour
ETTml,ET722
336
(96,192,336,72) 720)
14129/111455/ 1422
Energy
Mins
Electricity
336
19,,192,336,
321
17981/2537/5168
Energy
Hour
Traffic
336
196,192.336.720
862
11849/1661/3413
Transportation
Hour
Weather
336
[96, 19,,192,336,7200
21
36456/5115/10444
Weather
10 Mins
FORECASTING
Exchange
336
(96,192.338) (9.3362333)
8
4880/665/1422
Finance
Day
PEMS03
(12424,36,48)
358
15724/5241/5243
Transportation
PEMS04
96
(12,24,36,48)
307
101 10195/3398/3399
Transportation
5Mins
PEMS07
(12.24.36.48)
883
16934/5644/5646
Transportation
SMins
PEMS08
96
(12,24,36,48)
170
10713/3571/3572
Tramsportation
5Mins
8823/2947/2447
HAR
CLASSIFICATION
Epilepsy
206
4
3
137/138/138
HAR
EEG
3000
8
12787/1421/14211
EEG
ETT (4 subsets) (Zhou et al., 2021) This dataset comprises time series data of oil temperature and power load collected from electricity transformers spanning July 2016 to July 2018. is divided into four subsets, each with different recording intervals: ETThl and ETTh2 have hourly recordings, while ETTml and ETTm2 are recorded every 15 minutes.
Electricity (UCI, 2015) This dataset captures the electricity consumption of 321 clients on an hourly basis from 2012 to 2014, with measurements taken every 15 minutes (in kW). Time stamps follow Portuguese time. Each day includes 96 measurements 424x4), and during time changes in March (where one hour is skipped), the values between 1:00 am and 2:00 am are set to zero. Conversely, in October (with an extra hour), consumption between 1:00 am and 2:00 am represents the aggregated values of two hours.
Traffic (PeMS, 2001) Road occupancy rates, measured hourly, were collected from 862 sensors located along the San Francisco Bay area freeways. The data spans from January 2015 to December 2016.
Weather (Wetterstation, 2006) This dataset contains meteorological time series featuring 21 indicators. The data was collected every 10 minutes in 2020 by the Weather Station at the Max Planck Biogeochemistry Institute.
Exchange (Guokun Lai, 2017) This dataset collects the daily exchange rates of eight Canada, Switzerland, China, Japan, New Zealand, and Singapore fangapore-from 1990 to 2016.
the UK
PEMS (4 subsets) (PeMS, 2001) This dataset includes publicly available traffic network data from California, recorded in 5-minute intervals. We utilize the same four public subsets (PEMS03, PEMS04, PEMS07, PEMS08) as those used in SCINet (Liu al., 2022).
HAR (Anguita et al., 2013) The Human Activity Recognition (HAR) dataset is a multi-class classification dataset containing wearable sensor data from subjects performing activities such as walking, sitting, standing, and climbing stairs. provides rich temporal patterns for recognizing human activities.
13

---

TimeDART: A Diffusion Autoregressive Transformer for Selt-Supervieed Time Series Representation
Epilepsy (Andrzejak et al., 2001) The Epileptic Seizure Recognition dataset consists of EEG recordings from 500 subjects, where the brain activity recorded for each subject for 23.6 seconds. This dataset provides valuable resource for studying seizure patterns and developing models for reliable seizure detection.
EEG (Goldberger et al., 2000) The EEG datasets (as known as Sleep-EDF) contain polysomnographic recordings, including Eleetroenceepaalogram (EEG) signals, for multi-class sleep stage classification. They support automatic sleep stage identification and the study of sleep disorders.
B. Implementation Details
B.1. Compared Baselines
To more comprehensively evaluate the capabilities of TimeDART, we compare against the most advanced self-supervised and supervised models. Among the sepf-supe methods, we include contrastive learning and mask modeling, which help us validate the strength of TimeDART's pre-training approach. The supervised models are chosen compare against the advanced backbones proposed in existing supervised methods, ensuring that the use of the vanilla Transformer for autoregressive optimization in TimeDART does not lead to any inherent limitations in backbone selection. This selection of baselines ensures a robust and comprehensive comparison, highlighting the performance of TimeDART in different learning paradigms and validating its efficacy in wide range of settings.
SimMTM (Dong et al., 2024b) SimMTM' leverages manifold learning to recover masked time points by aggregating complementary temporal variations from multiple neighbors outside the manifold, improving the masked modeling task for time series pre-training.
PStchTST (Nie et al., 2023) PatchTST introduces a channel-independent patching mechanism for Transformer-basee time series models, enabling efficient subseries- embeddings and shared weights across univariate channels, significantly improving long-term forecasting and self-supervised pre-training performance.
TimeMAE (Cheng et al., 2023b) TimeMAE' introduces decoupled autoencoder architecture with sub-series partitioning and random masking to enhance contextual representations and encoding efficiency for time series, using bidirectional encoding and masked codeword classification for effective self-supervised learning.
CoST (Woo et al., 2022) CoST leverages contrastive learning in both time and frequency domains to disentangle seasonal ceasonal-trend representations for time series forecasting, enabling robust performance across backbone encoders and achieving state-of-the-art results.
DLinear (Zeng et al., 2023) DLinear' introduces simple one-layer linear models that bypass the temporal information loss inherent in Transformer-based self-attention, achieving superior performance in long-term time series forecasting across diverse datasets.
FormerTime (Cheng et al., 2023a) FormerTime combines hierarchical multi-scale representation learning with a novel Transformer encoder, integrating temporal reduction attention and contextual positional encoding to enhance efficiency and classification performance for multivariate time series.
B.2. Implementation Details for Fair Experiment
All experiments were implemented using Py PoTch (Paszke et al., 2017) and executed on a single NVIDIA RTX 4090 24GB GPU. To ensure the fairness of comparisons, we adopted a unified encoder as the representation network for all
httpss//gihub..oo/tuuuiiii
TimeMAE
14

---

TimeDART: A Diffussin Autoregressive Transformer for Selt-Supervieed Time Series Representation
self-supervised methods, maintaining consistency in model-specific parameters. For supervised methods, we adhered their original implementations without altering model-specific parameters, making adjustments only to non-core training parameters for alignment. For forecasting experiments, we applied the channel-independent configuration (Nie et al., 2023), ensuring consistent settings across methods.
Pre-training The representation network consists of layers for most datasets, while for the Traffic dataset, it comprises 3 layers due its higher complexity. The batch size was 16 for all datasets, except for Traffic and PEMS datasets, where it was reduced 8 due memory and time limitations. The ADAM optimizer (Kingma & Ba, 2014) was used with initial learning rates selected from (10-3,5 10-², 10-"), and L2 loss was employed for optimization. The pre-training process spanned 50 epochs. Sequence representation dimensions were chosen from {8, 16, 32, 64, 128).
Downstream Forecasting Task For the forecasting task, a consistent look-back window length of = 336 and predicted window lengths E (96, 192, 720) were applied all datasets, except for the PEMS datasets, where L 96 and H (12, 24, 36, 48). Fine-tuning was performed for 10 epochs, with settings largely consistent with pre-training. The channcl-independent configuration (Nie et al., 2023) was utilized, ensuring faimess in encoding time series data across all methods. Mean Squared Error (MSE) loss was employed as the loss function for forecasting tasks, ensuring consistency across experiments. To emphasize the contributions of pre- training, we also conducted experiments with Random Init., where the representation network was randomly initialized and directly fine-tuned without pre-training.
Downstream Classification Task For classification tasks, we followed similar settings, employing the same unified encoder and ensuring that all methods used consistent model-specific parameters. Cross-Entrops loss was utilized the loss function for classification, providing a standard measure for optimizing multi-class predictions. For fair comparison, all experiments adhered to the same training and evaluation protocols, aligning the sequence representation dimensions and training procedures.
C. Derivation Details
C.1. Task Definition
Time Series Self-Supervised Pre-training Given multivariate time series input X RCXL the goal is to train an encoder to learn feature representations without labels. The encoder learns temporal patterns through supersuu tasks, producing embeddings Z E TR xD, where L'' is the sequence length after encoding, and D) is the dimension of the hidden space. These embeddings serve as representations for downstream tasks.
Time Series Forecasting Given a multivariate time series input E RCxL, where C is the number of channels and L is the length of the back window, the goal is predict future values Y E RCCH over predicted window H. Here, X [i,..... x comprises L input vectors æ; CRC, and Y [yL-1.... |L+H] represents the predictions. Pre-training is performed on the look-back window, and both windows are used in the forecasting task.
Time Series Classification Given a multivariate time series input X E RCxL and its class label y, the goal is to map X to probability distribution over y.
C.2. Why We Add Noise Independently?
As shown in Figure 2, for patches in a sequence, we independently add noise to each patch, meaning that each patch undergoes a different noise addition process. This is done to prevent the sequence from retaining invariant statistical properties, such as mean and variance, after the noise addition. We now prove this:
Assume that patch operation is applied. After the original sequence is input, it passes through Instance Normalization layer. After normalization, the mean and variance of the original sequence along the seqllen dimension are 0 and 1, respectively. That is:
Mcan(I1.L) Var(ri:L)
(12)
(13)
15

---

TimeDART: A Diffusson Autoregressive Transformer for Selt-Supervieed Time Series Representation
The added noise follows standard gaussian distribution (0, 1). Suppose that apply the same noise addition step 8 for each point, then the noisy sequence satisfies:
1 - Vh(s))ay + V1 - (st)e, e~,1
(14)
Next, calculate the mean and variance of the noisy sequence. Since the distribution of the original sequence æ1.L is independent of the noise distribution:
(lean(f1.2) = =V (st) Mean(21:L) + /1 - (3;) Mean(c)
0
(15)
Var($1.1.) = (v>(s;)) Var(Fn:)) + (vI (VI-?(s;)) Var(e) = + (1 7(8))
(16)
Since the patch division is non-overlapping, the process of dividing patches does not affect the overall sequence distribution. Therefore, the above conclusions still apply.
Thus, if the noise addition step is the same for each patch, it will cause the mean and variance of the noisy sequence to remain the same as before, leading to an oversimplification of the pretraining task. Therefore, we adopt an independent noise addition strategy, and the corresponding experimental results can be found in Table 18.
C.3. Optimization Objective Derivation
The self-supervised optimization objective we employ follows the classical form of diffusion loss, which designed to maximize the marginal likelihood of the data p(æo). In this context, we assume that p represents the reverse denoising process, where the model learns reconstruct the original data æu from its noisy versions. This denoising process is modeled as gradual reverse transformation of the corrupted data, recovering the underlying clean distribution. The ideal loss function for this process can be formally expressed as:
Lideal
Σ H(po(af), ((0)))= = Σ E Baggri log po (2)))
(17)
Use the Evidence Lower Bound (ELBO) method and ignore the patch index subscript:
(1ucate7120) log ((a1:71200)
Lidcal S By(uo)
(18)
po (20201
log ((001i1120) po (aO0T)
(19)
Laiff
(20)
Expand and use the Bayes formula:
log g(2|-T122 11/200)
L = T
(21)
Do (x)
I1 (xx (0
= -
log p(x²²) I (0 xx*)
(22)
log ( 1xx1)
log gxx= 100 200 x²)
=Eyco
log pa (2=)
(23)
p(xT)
Po (x
Po
2ra)
(24)
16

---

TimeDART: A Diffussin Autoregressive Transformer for Self-Supervieed Time Series Representation
The first term is constant, the third term is the reconstruction loss, and only the second term is discussed:
log (x===110 120, |=",²") 220) po po(x0~1|x0) / Or g(x0.T)
log g(x² 100 xx"))
E (2207))
(25)
pe pa xx*)
Bula ,aa) [DRL(9(Z*-111r 11pe (2 (2
(26)
Applying Bayes' theorem again:
9(x0 0x00~112",""" = NNxx'1, ","""",''''"""" (x" ,,,),,,1) B21)
(27)
be assumed that the predicted distribution can also be expressed as:
po(x"~1aa (x*) = Nxx uol s), s), o
(28)
According to the KL divergence of two Gaussian distributions with the same variance, we can get:
Lai Ey(a" (x*)
20 l1 (x", (x'xx²) x0) - no(2", s)ll
(29)
Since já and xd are linearly related, we turn predict the original space, so:
Tann E o(oe)
[|<x nut [2]
(30)
Adding the cumulative sum of subscripts and the notation from the original paper, we get:
Caif
E Eagg [1a] - 2(2"", f(z f()11-1))113
(31)
D. Full Result for Forecasting Task
Due to space limitations, the complete tables for the forecasting task are provided in Appendix H. Experiments in the in-domain setting are shown in Tables 10 and 11, while experiments in the cross-domain setting are presented in Table 12. The experiments with the backbone replaced by causal TCN are shown in Table 13.
E. Ablation Study Full Result
The results in Table underscore the critical roles of both the autoregressive generation and denoising diffusion components in TimeDART. Removing the autoregressive mechanism (w/o AR) leads significant performance decline, particularly in ETTm2. At the 96-step horizon, MSE increases from 0.165 0.184, and at 336 steps, it rises from 0.279 to 0.307. This illustrates the crucial role of the autoregressive mechanism in enhancing the model's 's forecasting ability, especially across various time horizons. Similarly, climinating the denoising diffusion module (w/o Diff) results in noticeable performance degradation, as observed in ETTh2. At the 96-step horizon, MSE increases from 0.283 to 0.288, and at the 336-step horizon, rises from 0.365 to 0.372. These findings highlight the essential contribution of the denoising diffusion process to improving the model's 's learning and overall performance.
When both components are removed (w/o AR-Diff). the model's performance deteriorates significantly across all datasets. For instance, in Electricity, at the 336-step horizon, MSE jumps from 0. 166 to 0.199, clearly showing the combined importance of both modules for achieving optimal performance.
In summary, both modules are indispensable for TimeDART's success. The autoregressive mechanism is particularly important for long-term predictions, as evidenced in ETTm2, while the denoising diffusion process significantly improves accuracy and learning, especially in datasets like ETTh2.
17

---

TimeDART: A Diffusion Autoregressive Transformer for Selt-Supervised Time Series Representation
F. Hyperparameter Sensitivity
F.I In Forward Process
Noise Steps The results in Table 16 suggest that varying the total number of diffusion steps (2) has a relatively minor impact on model performance across datasets. Whether T is set to 750, 1000, or 1250, the model's effectiveness remains consistent, with minimal variation in MSE values. This indicates that once sufficient number of diffusion steps are reached, further increases offer little additional benefit.
Noise Scheduler In contrast, the noise scheduler plays more critical role shaping model performance. The cosine scheduler consistently outperforms the linear scheduler, with the gap performance widening as the prediction horizon increases. For instance, in the ETTh2 dataset, the cosine scheduler shows significantly better results at longer horizons compared to the linear scheduler, highlighting its ability to facilitate smoother noise transitions. These results emphasize the importance of selecting an appropriate noise scheduler, as it greatly influences the model's ability to effectively denoise during pre-training.
Shared Embedding Layer As shown in Figure 2, clean patches and noise-added patches share the embedding layer in the feed-forward encoder and denoising decoder. We believe that sharing the embedding layer weights allows the model to learn the representation of manually added noise, which helps better distinguish the unavoidable noise and anomalies in time series data. In addition, sharing the weights reduces the overall parameter size of the model, speeding up the computation. As shown in Table 18, the experimental results also confirm our hypothesis: sharing weights consistently outperforms using separate embedding layers.
F.2. In Reverse Process
Layers of Denoising Decoder The results in Table 17 indicate that increasing the number of layers in the denoising decoder does not consistently improve performance. While single decoder layer generally provides the best balance between model complexity and accuracy, adding more layers tends to offer diminishing returns. In fact, beyond one two layers, performance gains become negligible, and excessive layers can even hinder the training process by shifting capacity away from the representation network. This suggests that an overly complex decoder may underutilize the model's 's capacity, leading to suboptimal pre-training outcomes. Overall, the results emphasize the importance of maintaining a balanced architecture, where one decoder layer appears to be sufficient for effective performance across datasets.
As shown in Figure 4, we conduct experiments on the mask ratio of the denoising decoder. For the denoising decoder, a self-only mask is sufficient to capture all the information under the autoregressive mechanism, as the Transformer encoder output at position j already aggregates information from clean patches at positions to j - 1. We investigate whether providing additional information the denoising decoder benefits pre-training. The results in Table 15 indicate that redundant information unnecessary for the denoising process, and as the masked portion decreases, model pre-training performance worsens. This suggests that redundant information confuses the model, increasing pre-training difficulty. However, even with the inclusion of causal mask, the model outperforms random initialization, demonstrating the stability of the autoregressive mechanism.
Mask Ratio
Patch Length The results in Table 19 demonstrate that patch length significantly affects model performance, with each dataset benefiting from different levels of information density. For instance, datasets like Electricity, which exhibit high redundancy between data points, perform best with larger patches (e.g.. patch length 8), achieving the lowest average MSE of 0.1 and MAE of 0.254. In contrast, other datasets may require shorter patch lengths to capture more localized patterns. However, using smaller patches increases the computational complexity considerably, making training much more difficult and resource- resource-intensive. Thus, determining the optimal patch length depends not only on the dataset's characteristics but also on the balance between performance and computational feasibility.
G. Analysis Experiment
Few-shot Evaluation Table 20 demonstrates the full results of our few-shot fine-tuning evaluation. While fine-tuning with a reduced portion (5% or 10%) of samples generally showed weaker performance compared to using the full dataset, TimeDART consistently outperformed full-data supervised fine-tuning under random initialization across all prediction
18

---

TimeDART: A Diffusion Autoregressive Transformer for Selt-Supervieed Time Series Representation
windows for both forecasting and classification tasks. This comprehensive outperformance further solidifies the strong capabilities of pre-trained representation network.
Linear Probing Table 21 also illustrates the full performance of TimeDART in the linear probing setup, in comparison to several baseline models. Our findings indicate that TimeDARTT's linear probing performance consistently exceeded that of the compared baselines across all prediction windows. This provides further evidence for the efficacy and generalizability of our pre-training framework and the learned representations across diverse downstream tasks.
Memory and Efficiency To better illustrate the proposed TimeDART model's computation cost, we' ve provided quantita- tive results on the Traffic dataset, using lookback window of L, = 336 and a predicted window of 336, in Table 22. In summary, TimeDART is a medium-size model compared to our baselines. Thanks to our approach of training only the diffusion model without requiring lengthy sampling, both the pre-training and fine-tuning processes of TimeDART quite efficient and don't 't take up too much time, especially when compared to other pre- pre-training methods.
H. Visualization
6.15
GraundtuRh
0.35
0.75
Groandioon
6.5
0.53
0.50
0.2-
201
555
590
500
0.25
6.00
0.00
0.00
-1.1%
-325
-3.53
200
1006
204
400
403
1009
203
600
3300
ETTmZ
106
266
466
380
200
402
100
250
100
TImeDART
SimMTM
PatchTST-SSL
Figure 7: Illustration of forecasting showcases comparing TimeDART and baseline models. The look-back window is set to 336 and the predicted window is set to 192, 96, 720 for the ETTh2, Traffic, and ETTm2 dataset respectively.
In this visualization (Figure 7), TimeDART is compared against SimMTM and PatchTST-SSL, the self-supervised version of PatchTST. The ground truth, input data, and predictions plotted together. The look-back window is set to 336 for all datasets, while the predicted window varies: 192 for ETTh2, 96 for Traffic, and 720 for ETTm2. This setup ensures that different datasets are forecasted over appropriate future horizons based on their unique characteristics. TimeDART consistently shows more accurate and smoother predictions, closely matching the ground truth compared to the baselines.
I. Detailed Tables
19

---

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervieed Time Series Representation
Table 10: Multivariate time series forecasting full results with look-back window = 336. All results are MSE and MAE from 4 different predicted windows of (96, 192,336, 720). The best results are in bold and the second best are underlined. ##1 Counts" represents the number of times the method achieves the best results.
OURS
SELF-SUPERVISED
SUPERVISED
METHODS
TIMEDART
RANDOM INIT.
SIMMTM
PATCHTST
TIMEMAE
COST
PATCHTSS
DLINEAR
METRIC
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE
MAE
96
0.370
0.395
0.383
0.405
0.379
0.407 0.384 0.401 0.387
0.411
0.422
0.436
0.382
0.403 0.375
0.396
192
0.402
0.419
0.439
0.439
0.412
0.424
0.427 0.431 0.420
0.431
0.442
0.455
0.416
0.423
0.428
0.437
336
0.426
0.427
0.467
0.457
0.421
0.431
0.461 0.450
0.453
0.453
0.472
0.462
0,441
0.440
0.448
0.449
720
0.446
0.462
0.468
0.475
0.424
0.449 0.460
0.465
0.476
0.485
0.525
0.501
0.470
0.475
0.505
0.514
0.283
0.340
0.294
0.348
0.293
0.347
0.292 0.344
0.325
0.378
0.321
0.374
0.286
0.342 0.296 0.360
E1112
192
0.343
0.381
0.357
0.390
0.355
0.386
0.354 0.391 0.394
0.423
0.380
0.403
0.357
0.389 0.391 0.423
336
0.364
0.399
0.375
0.408
0.370
0.401
0.372 0.403
0,424
0.447
0.430
0.451
0.377
0.409
0.445 0,460
720
0.390
0.425
0.407
0.439
0.395
0.427
0.399 0.432 0.464
0.476
0.466
0.480
0.406
0.440
0.700 0.592
96
0.286
0.342
0.301
0.354
0.288
0.348
0.289 0.344
0.289
0.344
0.291
0.343
0.298
0.345
0.303 0.346
ET
192
0.326
0.367
0,333
0.372
0.327
0.373
0.326 0.372 0.330
0.371
0.330
0.370
0,339
0.374
0.338 0.368
336
0.357
0.388
0.360
0.389
0.363
0.395
0.353
0.387
0.366
0.393
0.382
0.401
0.381
0.401
0.373 0.393
720
0.407
0.417
0.408
0.418
0.412
0.424
0.399
0.418
0.416
0.424
0.422
0.425
0.428
0.431
0.428 0.423
96
0.165
0.256
0.174
0.263
0.172
0.261
0.171
0.257
0.174
0.263
0.183
0.298
0.174
0.261 0.170
0.264
ELECTRICITY ETTM2
192
0.221
0.294
0.240
0.307
0.223
0.300
0.236
0.304
0.233
0.303
0.247 0.315
0.238
0.307 0.233 0.311
336
0.279
0.330
0,284
0.334
0.282
0.331
0.291
0.344
0.291
0.340
0.298 0.346
0.293
0,346
0.298 0.358
720
0.364
0.385
0.377
0.389
0.374
0.388
0.388
0.404
0.380
0.396
0.401 0.412
0.373
0.401
0.423 0.437
96
0.132
0.225
0.147
0.252
0.133
0.223
0.132
0.225
0.165
0.285
0.197
0.277
0.138
0.233
0.141 0.238
192
0.150
0.241
0.163
0.265
0.147
0.237
0.148
0.241
0.181
0.297
0.197 0.279
0.153
0.247
0.154 0.251
336
0.166
0.258
0.179
0,280
0.166
0.265
0.167
0.260
0.199
0.312
0.211 0.295
0.170
0.263
0.170 0.269
720
0.203
0.290
0.218
0,312
0.203
0.297
0.205
0.292
0.238
0.341
0.255
0.330
0.206
0.295
0.205 0.302
96
0.357
0.247
0,386
0.267
0,368
0.262
0.382
0.262
0.382
0.261
0.378
0.365
0.395
0.272
0.411 0.284
TRAFFIC
192
0.376
0.256
0.398
0.269
0.373
0.251
0.385
0.261
0.399
0.267
0.371
0.352
0.411
0.278
0.423 0.289
336
0.389
0.262
0,410
0.274
0.395
0.254
0.409
0.275
0.411 0.274
0.467
0.354
0.424
0.284
0.437 0.297
720
0.429
0.286
0.446
0.299
0.432
0.290
0.438
0.291
0.446
0.298
0.525
0.378
0.453
0.300
0.467 0.316
EXCHANGE WEATHER
0. 149
0.199
0.155
0.206
0.158
0.211
0.148
0.196
054
0.203
0.162 0.235
0.147
0.197
0.1 0.236
192
093
0.240
0.198
0.246
0.199
0.249
0.193
0.240
0.191
0.241
0.212 0.255
0.191
0.240
0.217
0.275
336
0.244
0.280
0.250
0.286
0,246
0.286
0.244
0.279
0.243
0.282
0.258 0.293
0.244
0.282
0.264
0.315
720
0.317
0.331
0.319
0.335
0.317
0.337
0.321
0.334
0.318
0.334
0.334
0.346
0.320
0.334
0.325
0.364
96
0.086
0
0.102
0.229
0.100
0.226
0.088
0.207
0.098
0.226
0.102
0.229
0.094
0.213
0.087
0.217
192
0.175
0,302
0.224
0.343
0.210
0.332
0.186 0.308
0.219
0.340
0.212
0.334
0.191
0.311
0.164
0.298
336
0.344
0.431
0.384
0.453
0.389
0.460
0.374
0.446
0.400
0.466
0.384
0.452
0.343
0.427
0.333
0.437
720
0.829
0.675
1.051
0.774
1.104
0.800
0.857
0.692
0.989
0.751
1.124
0.805
0.888
0.706
0.988
0.749
#1 COUNTS
43
10
3
20

---

TimeDART: A Diffussin Autoregressive Transformer for Self-Supervised Time Series Representation
Table 11: Multivariate time series forecasting full results on PEMS datasets. All results are MSE and MAE from 4 different predicted windows of {12, 24, 36, 48). The best results are in bold and the second best are underlined. mcans doesn't t apply instance normalization on both pre-training and fine-tuning.
OURS
SUPERVUDD
SUPERVISED
METHODS
TIMEDART
RANDOM INIT.
SIMMTM
PATCHTST
TIMEMAR
COST
PATCHTST
DLINFAR
METRICS
MSB
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
12
0.082
0.188
0.089
0.198
0.082
0.189
0.081
0.191
0.093
0.207
0.102
0.213
0.099
0.212
0.120
0.240
PEMS03
24
0.127
0.236
0.138
0.248
0.133
0.239
0.131
0.239
0.138
0.249
0.142
0.251
0.139
0.251
0.198
0.312
36
0.176
0.282
0.187
0.292
0.183
0.286
0.181
0.287
0.186
0.292
0.189
0.298
0.211
0.320
0.330
0.420
48
0.223
0.321
0.241
0.326
0.233
0.326
0.231
0.325
0.242
0.327
0.244
0.331
0.262
0.370
0.460
0.520
12"
0,087
0.197
0.097
0.207
0,100
0.205
0.091
0.202
0.098
0.205
0.097
0.207
0.101
0.212
0.145
0.270
PEMS00
24"
0.121
0.235
0.128
0.243
0.125
0.241
0.126
0.240
0.126
0.242
0.133
0.251
0.135
0.265
0.220
0.335
36
0.149
0.260
0.161
0.275
0.156
0.271
0.157
0.266
0.160
0.274
0.167
0,278
0.162
0.276
0.350
0.430
48"
0.176
0.287
0.192
0.296
0.190
0.293
0.183
0.289
0.193
0.301
0.192
0.311
0.199
0.312
0.446
0.489
12"
0.066
0.174
0.078
0.183
0.072
0.180
0.071
0.178
0.074
0.181
0.076
0.186
0,076
0.185
0.109
0.238
PEMS07
0.109
0.218
0.117
0.226
0.111
0.220
0.115
0.224
0.116
0.225
0.118
0.229
0.129
0.242
0.206
0.322
36"
0.147
0.250
0.156
0,261
0.151
0.254
0.149
0.252
0.155
0.259
0.158
0.263
0,178
0.272
0.395
0.453
0,188
0.286
0,200
0.301
0.191
0.291
0.193
0.294
0.202
0.298
0.203
0.300
0.211
0.314
0.578
0.534
12
0.105
0.219
0.115
0.226
0.110
0.223
0.110
0.225
0.112
0.226
0.112
0.224
0.125
0.232
0.148
0.271
PEMS08
24
0.179
0.286
0.189
0.298
0.182
0.289
0.183
0.290
0,184
0.291
0.192
0,299
0.204
0.291
0.239
0.345
36"
0,239
0.296
0.251
0.310
0.246
0.300
0.244
0.299
0.248
0.309
0.252
0,315
0.275
0.314
0.422
0.454
48"
0.281
0.328
0.295
0.338
0.287
0.333
0.288
0.333
0.299
0.341
0.303
0.342
0.316
0.343
0.628
0.539
Table 12:
Multivariate time series full results of general pre-training and fine-tuning across different domains. All results are MSE and MAE from 4 different predicted windows of {12, 24, 36, 48) for PEMS datasets and 196, 192, 336, 720} for others. The best results are in bold and the second best are underlined.
METHODS
TIMEDART
RANDOM INTT.
SIMMTM
PATCHTST
TIMEMAE
CoST
DOMAIN
METRIC:
MSE
MAB
MSB
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAE
0.164
0.255
0.175
0.269
0.169
0.261
0.171
0.264
0.175
0.272
0.182
0.289
192
0.219
0.291
0.241
0.313
0.232
0.303
0.238
0.311
0.239
0.312
0.254
0.319
ENERGY
ETTM2
336
0.274
0.325
0.283
0.336
0,276
0.332
0.280
0.334
0,284
0.339
0.293
0.342
720
0.361
0.381
0.374
0.392
0.369
0.387
0.371
0.389
0.376
0.394
0.382
0.397
96
0.150
0.201
0.161
0.211
0.159
0.209
0.153
0.199
0.149
0.199
0.172
0.219
192
0.195
0.239
0.212
0.247
0.202
0.238
0.199
0.239
0.196
0.239
0.243
0.258
WEATHER
WEATHER
336
0.246
0.278
0.253
0.290
0,249
0.285
0.245
0.289
0.249
0.281
0.267
0.302
720
0.318
0.333
0.325
0.341
0.322
0.338
0.319
0.335
0.320
0.334
0.333
0.358
96
0.078
0.192
0.113
0.234
0,111
0.234
0.084
0.202
0.094
0.211
0.113
0.239
192
0.168
0.296
0.235
0.367
0.232
0.364
0.179
0.289
0.182
0.292
0.243
0.372
FINANCE
EXCHANGE
336
0.355
0.442
0.402
0.467
0.392
0.455
0.362
0.451
0.378
0.459
0.413
0.472
720
0.798
0.667
1,078
0.823
1,023
0.812
0.889
0.701
0.972
0.734
1.143
0.854
96
0.079
0.185
0.095
0.199
0.089
0.192
0.087
0.192
0.092
0.193
0.094
0.196
192
0.116
0.226
0.124
0.239
0.121
0.235
0.120
0.234
0.122
0.234
0.126
0.239
TRANSPORLATION
PEMS04
336
0.142
0.254
0.158
0.273
0.149
0.262
0.151
0.259
0.157
0,273
0.162
0.275
720
0.169
0.279
0.185
0.291
0.173
0.285
0.175
0.288
0.182
0.290
0.187
0.296
Table 13: Full result of TCN as backbone.
METHODS
ETTH2
ETTM2
ELECTRICITY
PEM504
BACKBONE METRICS
96
192
336
720
96
192
336
720
96
192
336
720
12
24
36
MSE
0.285
0.342
0.368
0.400
0.172
0.226
0.282
0.372
0.134
0.153
0.166
0.205
0.086
0.119
0.152
0.179
TCN
MAE
0.346
0.413
0.265
0.299
0.337
0.390
0.241
0.259
0.291
0.195
0.235
0.264
0.289
MSB
0.287
0.352
0,380
0.107
0.176
0.238
0.284
0.378
0.143
0.165
0.180
0.221
0.098
0.126
0.163
0.194
RANDOM
MAE
0.349
0.393
0.423
0.445
0.272
0.304
0.335
0.392
0.249
0.266
0.281
0.315
0.206
0.243
0.278
0.297
MSB
0.283
0.345
0.365
0.390
0.165
0.221
0.279
0.364
0.132
0.150
0.166
0.203
0.087
0.121
0.149
0.176
TRANSPORMER
MAE
0.340
0.382
0.399
0.425
0.256
0.294
0.330
0.385
0.225
0.241
0.258
0.290
0.197
0.235
0,260
0.287
21

---

TimeDART: A Diffussin Autoregressive Transformer for Selt-Supervieed Time Series Representation
Table 14: Full result of the ablation study on datasets for forecasting. The best results are in bold.
METHODS METRICS
TIMEDART
w/o AR
w/o
DIFF
w/o AR-DIFF
RANDOM INIT.
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.283
0.340
0.299
0.352
0.288
0.343
0.300
0.354
0.294
0.348
192
0.345
0.382
0.364
0.390
0.351
0.384
0.365
0.390
0.357
0.390
ETTH2
336
0.365
0.399
0.387
0.414
0.372
0.404
0.386
0.413
0.375
0.390
720
0.390
0.425
0.409
0.438
0.396
0.432
0.406
0.436
0.407
0,439
AVG.
0.346
0.387
0.365
0.399
0.352
0.391
0.364
0.398
0.358
0.396
96
0.165
0.256
0.184
0.276
0.175
0.265
0.186
0.278
0.174
0.263
192
0.221
0.294
0.245
0.317
0.228
0.300
0.246
0.318
0.240
0.307
ETTM2
336
0.279
0.330
0.307
0.355
0.281
0.331
0.311
0.367
0.284
0.334
720
0.364
0.385
0.388
0.403
0.374
0.392
0.395
0.420
0.377
0.389
AVG.
0.257
0.316
0.271
0.326
0.265
0.322
0.285
0.346
0.269
0.323
0.132
0.225
0.163
0.281
0.134
0.228
0.158
0.276
0.147
0.252
192
0.150
0.241
0.179
0.294
0.150
0.242
0.163
0.265
0.163
0.265
ELECTRICITY
336
0.166
0.258
0.195
0.306
0.167
0.259
0.199
0.312
0.179
0.280
720
0.203
0.290
0.234
0.335
0.205
0.292
0.238
0.341
0.218
0.312
AVG.
0.163
0.254
0.181
0.279
0.168
0.258
0.190
0.299
0.177
0.277
12
0.087
0.197
0.102
0.214
0.097
0.208
0.103
0.215
0.097
0.207
24
0.121
0.235
0.130
0.245
0.129
0.244
0.132
0.246
0.128
0.243
PEMS04
0.149
0.260
0.156
0.269
0.160
0.275
0.164
0.278
0.161
0.275
48
0.176
0.287
0.189
0.292
0.
0.298
0.195
0.300
0.192
0.296
AVG.
0.133
0.245
0.144
0.255
0.145
0.256
0.149
0.260
0.145
0.255
Table 15: Full result of the mask ratio on datasets for forecasting. The best results are in bold.
MASK RATIO
1.0
0.7
0.5
0.0
RANDOM INIT.
METRICS
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.283
0.340
0.285
0.342
0.289
0.346
0.294
0.347
0.294
0,348
192
0.345
0.382
0.348
0.384
0.348
0.384
0.356
0.389
0.357
0,390
ETTH2
336
0.365
0.399
0.364
0.398
0.368
0.402
0.372
0.40s
0.375
0,390
720
0.390
0.425
0.395
0.428
0.398
0.433
0.400
0.434
0.407
0.439
AVG.
0.346
0.387
0.348
0.388
0.351
0.391
0.356
0.394
0.358
0,396
96
0.165
0.256
0.167
0.258
0.165
0.257
0.172
0.259
0.174
0.263
192
0.221
0.294
0.224
0.298
0.230
0.302
0.235
0.304
0.240
0.307
ETTM2
336
0.279
0.330
0.281
0.334
0.281
0.335
0.282
0.335
0.284
0.334
720
0.364
0.385
0.364
0.385
0.368
0.390
0.370
0.392
0.377
0.389
AvG.
0.257
0.316
0.259
0.319
0.261
0.321
0.265
0.323
0.269
0.323
0.357
0.247
0.357
0.247
0.360
0.250
0.365
0.256
0.386
0.267
192
0.376
0.256
0.378
0.259
0.381
0.263
0.385
0.267
0.398
0.269
TRAFFIC
336
0.389
0.262
0.390
0.264
0.392
0.267
0.398
0.272
0.410
0.274
720
0.429
0.286
0.432
0.289
0.435
0.293
0.441
0.298
0.446
0.299
AvG.
0.388
0.263
0.389
0.265
0.392
0.268
0.397
0.273
0.410
0.277
12
0.087
0.197
0.087
0.197
0.090
0.199
0.092
0.202
0.097
0.207
24
0.121
0.235
0.122
0.236
0.125
0.240
0.128
0.242
0.128
0.243
PEMS04
36
0.149
0.260
0.151
0.262
0.153
0.264
0.156
0.266
0.161
0.275
48
0.176
0.287
0.176
0.288
0.177
0.287
0.178
0.291
0.192
0,296
AVG.
0.133
0.245
0.134
0.246
0.136
0.248
0.139
0.250
0.145
0.255
22

---

TimeDART: A Diffusion Autoregressive Transformer for Sell-Supervieed Time Series Representation
Table 16: Full result of hyperparameter analysis of noise steps and noise schedulers. The best results are in bold.
TOTAL NOISE STEPS
NOISE SCHEDULER
PARAM. METRIC
1000
1250
Cos.
LIN.
RANDOM INIT.
MSF
MAE
MSF
MAE
MSF
MAE
MSE
MAR
MSE
MAR
MSE
MAE
96
0.167
0.259
0.165
0.256
0.164
0.255
0.165
0.256
0.171
0.260
0.174
0.263
192
0.222
0.293
0.221
0.294
0.220
0.292
0.221
0.294
0.238
0.305
0.240
0.307
ETTM2
336
0.280
0.331
0.279
0.330
0.279
0.329
0.279
0.330
0.281
0.333
0.284
0.334
720
0,364
0.386
0.364
0.385
0.364
0.385
0.364
0.385
0.372
0,383
0.377
0.389
12
0.089
0.198
0.087
0.197
0.088
0.198
0.087
0.197
0.096
0.207
0.097
0.207
0.120
0.235
0.121
0.235
0.122
0.236
0.121
0.235
0.126
0.242
0.128
0.243
PEMS04
36
0.150
0.261
0.149
0.260
0.151
0.263
0.149
0.260
0.159
0.275
0.161
0.275
48
0.177
0.287
0.176
0.287
0.177
0.289
0.176
0.287
0.189
0.293
0.192
0.296
Table 17: Full result of hyperparameter analysis of the layer numbers of denoising decoder. The best results are in bold.
NUMBERS
3
RANDOM INIT.
METRIC
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.288
0.343
0.283
0.340
0.284
0.345
0.284
0.345
0.294
0.348
192
0.351
0.384
0.343
0.381
0.342
0.382
0.342
0.382
0.357
0.390
ETTH2
336
0.372
0.404
0.364
0.399
0.360
0.398
0.361
0.400
0.375
0.408
720
0.396
0.432
0.390
0.425
0.394
0.428
0.397
0.433
0.407
0.439
AVG.
0.352
0.391
0.345
0.386
0.345
0.388
0.346
0.390
0.358
0.396
96
0.175
0.265
0.165
0.256
0.166
0.257
0.167
0.257
0.174
0.263
192
0.228
0.300
0.221
0.294
0.226
0.297
0.230
0.399
0.240
0.307
ETTM2
336
0.281
0.331
0.279
0.330
0.280
0.333
0.282
0.337
0.284
0.334
720
0.374
0.392
0.364
0.385
0.372
0.388
0.379
0.394
0.377
0.389
AvG.
0.265
0.322
0.257
0.316
0.261
0.319
0.265
0.347
0.269
0.323
96
0.16
0.229
0.132
0.225
0.134
0.227
0.142
0.244
0.147
0.252
192
0.154
0.245
0.150
0.241
0.151
0.243
0.160
0.260
0.163
0.265
BLECTRICITY
336
0.170
0.263
0.166
0.258
0.169
0.258
0.175
0.274
0.179
0.280
720
0.212
0.296
0.203
0.290
0.211
0.304
0.215
0.310
0.218
0.312
AvG.
0.168
0.258
0.163
0.254
0.166
0.258
0.173
0.272
0.177
0.277
12
0.097
0.208
0.087
0.197
0.086
0.196
0.091
0.201
0.097
0.207
24
0.129
0.244
0.121
0.235
0.121
0.235
0.125
0.241
0.128
0.243
PEMS04
36
0.160
0.275
0.149
0.260
0.148
0.261
0.153
0.265
0.161
0.275
48
0.193
0.298
0.176
0.287
0.175
0.287
0.181
0.293
0.192
0.296
AvG.
0.145
0.256
0.133
0.245
0.133
0.245
0.138
0.250
0.145
0.255
METRIC
Acc.
ACC.
Acc.
F1
ACC.
AcC.
HAR
0,9002
0.9028
0.9247
0.9286
0.9186 86
0.9203
0.9048
0.9059
0.8931
0.8946
23

---

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervieed Time Series Representation
Table 18: Full result of hyperparameter sensitivity analysis of noise addition process and whether clean patches and noise-added patches share the same embedding layer. The best results are in bold.
NOISE ADDITION PROCESS
WHETHER SHARE EMBEDDING LAYER
PARAM.
INDEPENDENT STEPS
SAME
STEP
SHARED
EMBEDDING
SEPARATE EMBEDDING
RANDOM INIT:
METRIC
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.165
0.256
0.168
0.259
0.165
0.256
0.170
0.258
0.174
0.263
192
0.221
0.294
0.223
0.295
0.221
0.294
0.223
0.297
0.240
0.307
ETTM2
336
0.279
0.330
0.281
0.334
0.279
0.330
0.284
0.334
0.284
0.334
720
0.364
0.385
0.364
0.386
0.364
0.385
0.368
0.389
0.377
0.389
AVG.
0.257
0.316
0.259
0,319
0.257
0.316
0.261
0,320
0.269
0.323
0.087
0.197
0,088
0.199
0.087
0.197
0.087
0.197
0,097
0.207
24
0.121
0.235
0.123
0.237
0.121
0.235
0.122
0,237
0.128
0.243
PEMS04
0.149
0.260
0.154
0.263
0.149
0.260
0.152
0.263
0.161
0.275
48
0.176
0.287
0.180
0.290
0.176
0.287
0.178
0.289
0.192
0.296
AVG.
0.133
0.245
0.136
0.247
0.133
0.245
0.135
0.247
0.145
0.255
HAR
0.9247
0,9286
0.9168
0.9112
0.9247
0.9286
0.9104
0.8993
0.8931
0.8946
Table 19: Full result of hyperparameter sensitivity analysis of patch length. The best results are in bold.
LENGTH
16
RANDOM INIT.
METRIC
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
96
0.312
0.364
0.283
0.340
0.295
0.348
0.301
0.356
0.313
0.365
0.294
0.348
192
0.387
0.412
0.343
0.381
0.348
0.385
0.356
0.390
0.365
0.400
0.357
0.390
EТTH2
336
0.419
0.439
0.364
0.399
0.369
0.406
0.370
0.407
0.377
0.415
0.375
0.390
720
0.452
0.469
0.390
0.425
0.399
0.434
0.403
0.436
0,412
0.443
0.407
0.439
Avci
0.393
0.421
0.345
0.386
0.353
0.393
0.358
0.397
0.367
0.406
0.358
0.396
0.169
0.258
0.165
0.256
0.177
0.267
0.168
0.258
0.170
0.261
0.174
0.263
192
0.226
0.295
0.221
0.294
0.231
0.302
0.226
0.297
0.224
0.297
0.240
0.307
ETTM2
336
0.283
0,333
0.279
0.330
0.284
0.336
0.278
0.330
0.277
0.330
0,284
0.334
720
0.371
0.388
0.364
0.385
0.378
0.392
0.362
0.382
0.370
0.385
0.377
0.389
AvG.
0.262
0.3 185
0.257
0.316
0.259
0.3
0.260
0.318
0,268
0.324
0.269
0.323
96
0.165
0.285
0.149
0.254
0.135
0.234
0.132
0.225
0.146
0.250
0.147
0.252
192
0.181
0.297
0.163
0.266
0.152
0.249
0.150
0.241
0.161
0.264
0.240
0.307
ELECTRICITY
336
0.199
0,312
0.180
0.282
0.169
0.266
0.166
0.258
0.178
0.281
0.284
0.334
720
0.238
0.341
0.220
0.313
0.208
0.299
0.203
0.290
0.218
0.313
0.377
0.389
AvG.
196
0.309
0.178
0.279
0.1
0.262
0.1
0.254
1.6
0.277
0.177
0.277
12
0.088
0.198
0.087
0.197
0.089
0.198
0.096
0.207
0.099
0.209
0.097
0.207
24
0.122
0.236
0.121
0.235
0.120
0.235
0.126
0.242
0.131
0.252
0.128
0.243
PEMS04
36
0.151
0.263
0.149
0.260
0.150
0.261
0.159
0.275
0.165
0.276
0.161
0.275
48
0.177
0.289
0.176
0.287
0.177
0.287
0.189
0.293
0.197
0.303
0.192
0.296
AvG.
0.135
0.247
0.133
0.245
0.134
0.245
0.143
0.254
0.148
0.260
0.145
0.255
LENGTH
2
16
RANDOM INIT.
METRIC
Acc.
F1
Acc.
ACC.
F1
Acc.
F1
Acc.
F-1
Acc.
F1
HAR
0.8969
0.8952
0.9132
0.9145
0.9247
0.9286
0.9157
0.9168
0.9023
0.9012
0.8931
0.8946
24

---

TimeDART: A Diffusion Autoregressive Transformer for Selt-Supervieed Time Series Representation
Table 20: Full result of few-shot fine-tuning.
PORTION
5%
10%
100%
RANDOM INIT.
METRIC
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.292
0.346
0.289
0,344
0.283
0.340
0.294
0.348
192
0.356
0.389
0,354
0.387
0.345
0.382
0.357
0.390
ETTH2
336
0.375
0.403
0.365
0.399
0.375
0.408
720
0.399
0.429
0.397
0.428
0.390
0.425
0.407
0.439
AVG.
0.356
0.392
0.354
0.391
0.346
0.387
0.358
0.396
12
0.093
0,205
0.091
0.205
0.087
0.197
0.097
0,207
24
0.128
0.244
0.127
0.242
0.121
0.235
0.128
0.243
PEMS04
36
0.158
0.272
0.157
0.269
0.149
0.260
0.161
0.275
48
0.187
0.294
0,185
0.285
0.176
0.287
0.192
0.296
AVG.
0.142
0.254
0.140
0.250
0.133
0.245
0.145
0.255
PORTION
5%
10%
100%
RANDOM INIT.
METRIC
Acc.
F1 SCoRE
Acc.
FI SCORE
ACC.
F1 ScORE
Acc.
F1 ScoRE
HAR
0.8843
0.8901
0.9014
0.9053
0.9247
0.9286
0.8738
0.8723
Table 21: Full result of linear probing fine-tuning.
LINEAR PROBING
RANDOM INIT.
TIMEDART
SIMMTM
TIMEMAE
METRICS
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.297
0.345
0.281
0.338
0.281
0.339
0.289
0.342
192
0.368
0.397
0.352
0.389
0.358
0.392
0.361
0.391
ETTH2
336
0.385
0.417
0.372
0.403
0.376
0.407
0.378
0.411
720
0.421
0.446
0.409
0.434
0.413
0.442
0.416
0.442
AVG.
0.368
0.401
0.354
0.391
0.357
0.395
0.361
0.397
0.112
0.228
0.096
0.211
102
0.213
0.108
0.219
24
0.141
0.258
0.130
0.242
0. 0.134
0.245
0.135
0.247
PEMS04
36
0.187
0.286
0.165
0.279
0.165
0.279
0.172
0.283
48
0.204
0.313
0.187
0.301
0.189
0.304
0.193
0.307
AVG.
0.161
0.271
0.145
0.258
0.148
0.260
0.152
0.264
LINEAR PROBING
RANDOM INIT.
TIMEDART
SIMMTM
TIMEMAE
METRICS
Acc
F1
Acc
Acc
F1
Acc
F1
HAR
0.8542
0.8578
0.8976
0.9005
0.8732
0.8756
0.8858
0.8862
Table 22: Comparison of parameters and training times on Traffic dataset.
METHODS
PARAMS
Training TIME/PER EPOCH
TIMEDART
2.36M (PRE-TRAINING) / 2.16 (HINE-TUNING)
5 10S (PRE-TRAINNNG) 349s (HINE-TUNING)
SSL.
SIMMTM
14.5M (PRE-TRAINING) / 2.16M (FINE-TUNING)
85MIN (PRE-TRAINING) / 349s (FINE-TUNING)
TIMEMAE
1.13M (PRE-TRAINING) 2. 16M (FINE-TUNING)
91MI (PRE-TRAINING) 349s (FINE-TUNING)
COST
TRAININGG / 2. 2.16M(HNN--TUNING
24MINS(PRE- SEIINS(PRE-TRAINNNG)) 149S(HINE-TUNIN)
SL.
PATCHTST
4.27M
158s
25


Reviews:
[{"review_id": "MXT1qOQtcn", "reviewer": "Reviewer_baxN", "summary": "Authors propose a novel self-supervised time series representation pre-training framework that integrates two popular generative paradigms to enhance representation transferability. Specifically, they employ a causal Transformer encoder for autoregressive prediction while incorporating a denoising diffusion process to recover fine-grained local patterns. Extensive experiments on time series classification and forecasting tasks validate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/\n>\n> [Table1] —>para1,[Table2] —>para2,[Table3] —> para3\n\n**A1 for Q1**\n1. For efficiency, we used a lightweight denoising decoder during pre-training.  After training, similar to masked autoencoders, only the embedding layer and encoder network are transferred for downstream tasks. As a result, TimeDART introduces small additional computational overhead in terms of GPU memory usage and computation speed. Specific comparative results can be found in the [Table1] below.\n2. In forecasting tasks, we have evaluated performance on large datasets especially traffic and pems. Dataset details, including variables and lengths, are in Section 4.1. We also pre-trained on all six datasets combined, reaching approximately 43M training time points (channel independence). This data volume is significantly larger than the baseline in the article.\n3. To evaluate the scalability of TimeDART for longer sequences, we conducted additional experiments with different look-back window sizes, especially extending it to 576 and 720 in ETTh2. We observed a further reduction in prediction MSE, indicating that TimeDART scales well for longer sequences. The detailed results can be found in the [Table2] below.\n\n[Table1]\nThe data in the following table is recorded in the Traffic dataset with a look-back window=336 and a predicted window=336. \n|Methods|Params|Training Time/per epoch|\n|-|-|-|\n|TimeDART|2.36M(pt)/2.16M(ft)|510s(pt)/349s(ft)|\n|SimMTM|14.5M(pt)/2.16M(ft)|85mins(pt)/349s(ft)|\n|TimeMAE|1.13M(pt)/2.16M(ft)|91mins(pt)/349s(ft)|\n|Cost|2.66M(pt)/2.16M(ft)|24mins(pt)/349s(ft)|\n|PatchTST(supervised)|64.32M|158mins|\n\n[Table2]\nThe following table shows the average of predicted window on [96,192,336,720].\n|look-back|TimeDART|Random Init|\n|-|-|-|\n|ETTh2|MSE/MAE|MSE/MAE|\n|96|0.373/0.398|0.384/0.407|\n|192|0.364/0.391|0.374/0.401|\n|336|0.346/0.387|0.358/0.396|\n|576|0.343/0.384|0.355/0.392|\n|720|0.339/0.379|0.352/0.388|\n|PEMS04|MSE/MAE|MSE/MAE|\n|96|0.133/0.245|0.145/0.255|\n|192|0.130/0.240|0.140/0.252|\n|336|0.125/0.235|0.134/0.247|\n\n**A2 for Q2**\n1. Due to limitations in GPUs and the time constraints of the rebuttal process, and given the significant disparity in the scale of training data, it is challenging for us to directly compare with large foundation models in a short period. To address your concerns regarding representation quality and transferability, we have instead conducted some comparisons with SOTA methods based on LLMs (e.g. UniTime, GPT4TS), which we hope will help alleviate your doubts and concerns.  Specific comparative results can be found in the [table3] below.\n2. Regarding the cross-domain issue, we mixed all in-domain datasets for general pre-training in forecasting tasks and fine-tuned across domains. As noted in `a1 for q1`, this yielded approximately 43M training time points. Experiments (Section 4.2) demonstrate that TimeDART’s cross-domain representation learning, under large-scale pre-training, shows improvement compared to no pre-training.  As for going beyond the current benchmark datasets, we are considering conducting general pre-training on larger public datasets (e.g., UTSD) to further validate TimeDART’s representation ability.\n\n[Table3]\nThe look-back window is reset to 96 same to UniTime to ensure fair comparison. Model is pretrained on ETT(4subsets), Exchange, Electricity, Traffic datasets. The following table shows the average of predicted window on [96,192,336,720]. Experiments show that TimeDART can still learn better representations compared to LLM-based methods.\n\n|Models|TimeDART|Random Init|Unitime|GPT4TS|PatchTST|\n|-|-|-|-|-|-|\n|Metrics|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|\n|ETTh2|**0.376/0.398**|0.384/0.404|0.378/0.403|0.386/0.406|0.398/0.416|\n|ETTm2|**0.287/0.333**|0.301/0.350|0.293/0.334|0.321/0.356|0.340/0.373|\n|Exchange|**0.361**/0.406|0.389/0.424|0.364/**0.404**|0.421/0.446|0.411/0.444|\n|Electricity|**0.200/0.293**|0.212/0.303|0.216/0.305|0.251/0.338|0.221/0.311|\n\n**A3 for W1**\nDue to time constraints, we just provide a brief explanation. Our assumption is that traditional autoregressive pre-training excels in capturing long-term dependencies but struggles with inherent time series noise, which is hard to avoid. By introducing noise into the pre-training process, we encourage the encoder to learn both long-term dependencies and the distribution of gaussian noise $f(x_j^0|x_{1-j-1}^0, \\epsilon_1^{s_1}, ..., \\epsilon_{j-1}^{s_{j-1}})$. This improves the model's adaptability to inherent noise in downstream tasks. We are now working on a formal theoretical proof of combining autoregressive modeling with diffusion, which will be included in future work.\n\n**A4 for W2**\nPlease refer to `a1 for q1`.\n\nThank you for your encouraging score. We sincerely look forward to your reply.", "claims_and_evidence": "The claims in the paper are well-supported by empirical evidence. TimeDART’s effectiveness in capturing both global and local sequence features is validated through strong performance across multiple benchmark datasets. Extensive experiments in forecasting and classification, along with ablation studies, demonstrate its advantages over state-of-the-art methods. The integration of autoregressive modeling and denoising diffusion is shown to enhance representation learning, confirming the validity of the proposed approach.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well aligned with the problem. TimeDART effectively integrates autoregressive modeling and denoising diffusion for self-supervised time series representation learning. The use of diverse benchmark datasets across forecasting and classification tasks ensures a comprehensive evaluation. Metrics such as MSE, MAE, accuracy, and F1-score are appropriate, and ablation studies further validate the contributions of key components.", "theoretical_claims": "The paper's theoretical foundations are consistent with established principles, particularly in diffusion-based modeling and autoregressive learning. The diffusion loss formulation aligns with ELBO principles, and its application appears correct. While a deeper theoretical justification could further enhance clarity, the empirical results strongly support the proposed approach.", "experimental_designs_or_analyses": "The experimental design is well-structured and comprehensive. The authors evaluate TimeDART across diverse benchmark datasets using appropriate metrics (MSE, MAE, accuracy, F1-score). Ablation studies effectively validate the contributions of autoregressive modeling and the denoising diffusion process. The inclusion of both in-domain and cross-domain evaluations further strengthens the robustness of the findings.", "relation_to_broader_scientific_literature": "The paper builds on prior work in self-supervised time series representation learning, integrating autoregressive modeling and diffusion-based denoising. It extends masked modeling and contrastive learning approaches by introducing a hybrid generative framework. Compared to traditional autoregressive methods, TimeDART mitigates error accumulation, while diffusion models typically used for probabilistic forecasting are repurposed for representation learning, bridging gaps in existing time series pre-training strategies.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n1. The paper effectively combines autoregressive modeling with a denoising diffusion process, offering a novel perspective to self-supervised time series representation learning.\n2. The method is extensively validated across multiple benchmark datasets for both forecasting and classification tasks, demonstrating consistent performance gains over state-of-the-art baselines.\n3. The paper includes detailed ablation studies and hyperparameter sensitivity analysis, reinforcing the contributions of key components and ensuring the method's robustness.\n\nWeakness\n1. While the empirical results are strong, it would be better if the paper could provide a formal theoretical analysis explaining why the combination of autoregressive modeling and diffusion improves representation learning.\n2. It seems that the paper does not provide insights into the computational cost of TimeDART compared to other self-supervised methods, especially regarding training efficiency and scalability.", "other_comments_or_suggestions": "See above", "questions_for_authors": "Question 1: Diffusion-based models often introduce additional computational overhead due to iterative denoising steps. How does TimeDART compare in terms of training and inference time relative to standard self-supervised methods like contrastive learning or masked autoencoders? Have you evaluated the scalability of TimeDART on longer sequences or larger datasets? A comparison of computational trade-offs would clarify its practical applicability.\n\nQuestion 2: Recent advancements in time series modeling include large-scale pre-trained models (e.g., TimeGPT) that generalize across domains. How does TimeDART compare in terms of representation quality and transferability to such foundation models? Have you considered evaluating it on cross-domain tasks beyond the current benchmark datasets? Addressing this would help position TimeDART within the broader landscape of time series pre-training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xr2TQJxIz5", "reviewer": "Reviewer_TNxq", "summary": "The paper introduces TimeDART, a novel self-supervised learning framework for time series analysis that integrates autoregressive modeling with diffusion-based denoising. The framework aims to address the limitations of existing methods, such as masked autoencoders, contrastive learning, and autoregressive approaches, particularly their susceptibility to noise. TimeDART employs a causal Transformer encoder and a cross-attention-based denoising mechanism to capture both global dynamics and local patterns in time series data. The paper demonstrates the effectiveness of TimeDART through extensive experiments, showing improvements in both time series prediction and classification tasks.\n\n## update after rebuttal\nThe authors have actively responded to reviewers' comments. Though some of my concern about the utilization of diffusion model still remains, to some extent, they've empirically illustrated that it worked from the view of self-supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/\n>\n> [Table1]--->para4,para5, [Table2]--->para6, [A3 for Q3] --->para1, [A5 for Q5]--->para2\n\n\n**A1 for Q1**\n\nWe abandon autoregressive modeling and diffusion in the downstream task based on three considerations: First, we not only performed forecasting tasks but also discriminative tasks such as classification, which are difficult to directly adapt to this paradigm. Second, almost all evaluations of self-supervised methods for time series follow the approach of transferring only the encoder and attaching different heads for various downstream adaptations (e.g., TimeSiam, SimMTM). Therefore, to enable comparisons and ensure fairness, we adhered to the same transfer method. Lastly, autoregressive inference and diffusion sampling would be extremely slow.\n\nEarly of this work, we have designed a downstream denoising autoregressive paradigm for forecasting tasks and conducted experiments. However, based on the considerations above, we decided not to use this approach further. We appreciate the reviewer's question, as it allows us to revisit our initial design. The results can be found in the [Table1] below.\n\n[Table1]\nThe following table shows the average of predicted window on [96,192,336,720]. Detailed designs can be found at para5 in link above. The results show that retaining the decoder for prediction can achieve similar results.\n\n|Methods|Diff+non AR|Diff+non AR Random Init|Diff+AR|Diff+AR Random Init|TimeDART|Random Init|\n|-|-|-|-|-|-|-|\n|Metrics|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|\n|ETTh2|0.352/0.389|0.363/0.397|0.359/0.394|0.372/0.400|**0.346/0.387**|0.358/0.396|\n|Exchange|0.346/0.406|0.383/0.442|**0.339/0.385**|0.366/0.422|0.359/0.405|0.440/0.450|\n\n**A2 for Q2**\n\nEven though the denoising decoder is removed during fine-tuning, it remains **necessary**, as the ablation experiments show: removing the diffusion model during pre-training leads to a decline in performance for both downstream forecasting and classification tasks.  \n\nWe can use a simpler MLP as a decoder. For simplicity, we directly use the concat strategy to concatenate the output of the causal encoder and the embedding of the noise added patches in the dim axis. In order to keep the parameters consistent with the original 1-layer Transformer (about $12d_{model}^2$), we use a Squencial(Linear($2d_{model}$, $4d_{model}$), ReLU, Linear($4d_{model}$, $d_{model}$) ) MLP as the denoising decoder. The results can be found in the [Table2] below.\n\n[Table2]\nThe following table shows the average of predicted window on [96,192,336,720]. The results show that simpler's MLP is slightly weaker than TRM, but almost the same.\n\n|Decoder|MLP|TRM|Random Init|\n|-|-|-|-|\n|Metrics|MSE/MAE|MSE/MAE|MSE/MAE|\n|ETTh2|0.347/0.387|0.346/0.387|0.358/0.396|\n|PEMS04|0.134/0.245|0.133/0.245|0.145/0.255|\n||Acc/F1|Acc/F1|Acc/F1|\n|HAR|0.9197/0.9186|0.9247/0.9286|0.8738/0.8723|\n\n**A3 for Q3**\n\nPlease refer to `A1 for Q1, paragraph 1` in our rebuttal to the first Reviewer baxN, where the table of gpu memory and calculation is given.\n\n**A4 for Q4**\n\nYou have raised an important point, however, we must clarify the entire data processing and experimental evaluation section:  \n\nYou mentioned that in `/data_provider/data_factory.py#L41`, setting `drop_last` to `True` might lead to discarding some samples, potentially biasing the results. However, at #L44 and #L45, when the downstream task is forecasting, the `batch_size` is set to 1. With this, no forecasting test samples will be dropped even `drop_last` is set to `True`. Furthermore, at #L42 and #L43, when the downstream task is classification, although `batch_size` is set to `args.batch_size`, at #L69 and #L70 `drop_last` is explicitly reset to `False`. Thus, the classification task is also unaffected.  For the sake of rigor: we double-checked the code and printed the key parameters. The results confirm that the critical parameters are as described above.\n\n**A5 for Q5**\n\nPlease refer to `A1 for Q1, paragraph 3` in our rebuttal to the first Reviewer baxN, where the table of different look-back windows is given.\n\n**A6 for W1**\n\nAs mentioned in `a3 for w1` in our rebuttal to first Reviewer baxN, we did not mechanically combine these two techniques in pre-training. Instead, we approached the uniqueness of time series data with careful consideration. Simply using autoregressive optimization during pre-training does not yield satisfactory results when transferred to downstream tasks. However, through our novel approach of explicitly introducing noise, we found that the pre-training performance indeed improved, which inspired the development of our current complete work.\n\n**A6 for W2**\n\nPlease refer to `A1 for Q1`.\n\n**A7 for W3**\n\nPlease refer to `A3 for Q3` above or `A1 for Q1, paragraph 1` in our rebuttal to the first Reviewer baxN.\n\nYour score is very important to us. We hope we can solve your questions. We sincerely look forward to your reply.", "claims_and_evidence": "The claims in the paper are clear and well-supported by evidence. The authors effectively argue that combining autoregressive modeling with diffusion denoising helps capture both global and local patterns in time series data. The use of diffusion denoising is justified to mitigate the overfitting problem of autoregressive models to noise and anomalies, and the cross-attention decoder is introduced to address the local dependence issue of diffusion models.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for the problem.", "theoretical_claims": "The theoretical claims are supported by a statistical invariant property analysis and a rough derivation of the optimization objective provided in the appendix. There are no significant issues with the theoretical foundations of the paper.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound. However, there is a concern about the consistency of hyperparameters across different methods, as the results of some baseline methods (e.g., PatchTST) appear weaker than those reported in their original papers.", "relation_to_broader_scientific_literature": "The main contribution of the paper lies in the combination of autoregressive and diffusion models in self-supervised learning for time series analysis. This approach addresses the overfitting problem in autoregressive methods and enhances the ability to capture local information through diffusion models.", "essential_references_not_discussed": "There are no essential references missing from the discussion.", "other_strengths_and_weaknesses": "Strengths:\n* The writing is clear, and the code is provided, which enhances the reproducibility of the results.\n* The experiments are well-designed and demonstrate the superiority of the proposed method.\n\nWeaknesses:\n* The combination of autoregressive and diffusion models is straightforward. Both techniques are commonly used in many other fields.\n* The combination of autoregressive modeling and diffusion denoising is only used during the pretraining stage, and a different paradigm is adopted for downstream tasks, which weakens the innovation of the approach.\n* The runtime efficiency of the algorithm is not reported, raising concerns about the computational cost of combining autoregressive modeling with multiple denoising steps.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Why do the authors abandon autoregressive modeling and diffusion in the downstream task, given that these methods are central to their claims about capturing global and local patterns? What’s the performance of keeping autoregressive modeling and diffusion model in forecasting task?\n2. If the denoising decoder is removed during fine-tuning, is the diffusion model still necessary? Could other simpler backbone networks achieve similar results?\n3. What is the runtime efficiency of the algorithm, especially given the potential computational cost of combining autoregressive modeling with multiple denoising steps?\n4. I noticed that in the data factory script, the setting of drop_last is True (/data_provider/data_factory.py#L41). This will lead to wrong results of the whole experiment [1].\n5. What are the effects of different lookback windows on the results?\n\n[1] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7D2Pano6h2", "reviewer": "Reviewer_6U5m", "summary": "This paper presents a self-supervised time series representation learning method. It combines autoregressive modeling with the denoising diffusion process. Key ideas involve normalizing and patch-embedding data, using a causal Transformer encoder for long-term evolution and a patch-level diffusion/denoising mechanism for local patterns. Results show TimeDART outperforms baselines in forecasting and classification tasks, both in-domain and cross-domain.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/\n>\n> [Table1]--->para7, [Table2]--->para8\n\n**A1 for Q1**\n\nWe conduct detailed few-shot experiments on the performance of the model with 5% or 10% fine-tuning data, including forecasting and classification tasks. The results can be found in the [Table1] below.\n\n[Table1]\n\n|Portion|5%|10%|100%|Random Init|\n|-|-|-|-|-|\n|Metrics|MSE/MAE|MSE/MAE|MSE/MAE|MSE/MAE|\n|ETTh2|0.356/0.392|0.354/0.391|0.346/0.387|0.358/0.396|\n|PEMS04|0.142/0.254|0.140/0.250|0.133/0.245|0.145/0.255|\n||Acc/F1|Acc/F1|Acc/F1|Acc/F1|\n|HAR|0.8843/0.8901|0.9014/0.9053|0.9247/0.9286|0.8738/0.8723|\n\n**A2 for Q2**\n\nThe following is our detailed derivation from formula [7] to formula [8], which we will integrate into the paper in the future to highlight the comprehensiveness and rigor of the paper.\n\nThe original theoretical loss should be in the form of cross entropy:\n\n$$\nL_{ideal}=H(p_\\\\theta(x^0\\_{1:j}),q(x^0_{1:j}))=\\\\mathbb{E}_{q(x^0\\_{1:j})}(-\\\\log p\\_\\\\theta(x^0\\_{1:j}))\n$$\n\nUse the Evidence Lower Bound (ELBO) method and ignore the patch index subscript:\n\n$$\n\\begin{align}\nL\\_{ideal} &\\\\leq \\\\mathbb{E}\\\\_{q(x^0)} \\\\left(\\mathbb{E}\\_{q(x^{1:T}|x^0)}\\\\left[\\\\log \\\\frac{q(x^{1:T}|x^0)}{p\\_\\\\theta(x^{0:T})}\\\\right]\\\\right) \\\\\\\\\n&=\\\\mathbb{E}\\_{q(x^{0:T})} \\\\left[\\\\log \\\\frac{q(x^{1:T}|x^0)}{p\\_\\\\theta(x^{0:T})}\\\\right] \\\\\\\\\n&:= L\\_{diff}\n\\end{align}\n$$\n\nExpand and use the Bayes formula:\n\n$$\n\\\\begin{align}\nL &= \\\\mathbb{E}\\_{q(x^{0:T})} \\\\left[ \\\\log \\\\frac{q(x^{1:T}|x^0)}{p\\_\\\\theta(x^{0:T})} \\\\right] \\\\\\\\\n&= \\\\mathbb{E}\\_{q(x^{0:T})} \\\\left[ \\\\log \\\\frac{\\\\prod\\_{s=1}^T q(x^s|x^{s-1})}{p(x^T) \\\\prod\\_{s=1}^T p\\_\\\\theta(x^{s-1}|x^s)} \\\\right] \\\\\\\\\n&= \\\\mathbb{E}\\_{q(x^{0:T})} \\\\left[ \\\\log \\\\frac{ q(x^T|x^0)}{p(x^T)} + \\\\sum\\_{s=2}^T \\\\log \\\\frac{q(x^{s-1}|x^s, x^0)}{p\\_\\\\theta(x^{s-1}|x^s)} - \\\\log p\\_\\\\theta(x^0|x^1) \\\\right] \\\\\\\\\n\\\\end{align}\n$$\n\nThe first term is a constant, the third term is the reconstruction loss, and only the second term is discussed:\n\n$$\n\\\\begin{align}\n    \\\\mathbb{E}\\_{q(x^{0:T})} \\\\sum\\_{s=2}^T \\\\log \\\\frac{q(x^{s-1}|x^s, x^0)}{p\\_\\\\theta(x^{s-1}|x^s)} &= \\\\int\\\\text{d}x^{0:T}~ q(x^{0:T})\\\\cdot\\\\sum\\_{s=2}^T \\\\log \\\\frac{q(x^{s-1}|x^s, x^0)}{p\\_\\\\theta(x^{s-1}|x^s)} \\\\\\\\\n    &=\\\\sum\\_{s=2}^T\\\\mathbb{E}\\_{q(x^0,x^s)}\\\\left[D_{KL}(q(x^{s-1}|x^s, x^0)\\\\|p\\_\\\\theta(x^{s-1}|x^s))\\\\right]\n\\\\end{align}\n$$\n\nApplying Bayes' theorem again:\n\n$$\nq(x^{s-1} | x^s, x^0) = \\\\mathcal{N}(x^{s-1},\\\\tilde{\\\\mu}_t(x^s,x^0),\\\\tilde{\\\\beta}_s I)\n$$\n\nIt can be assumed that the predicted distribution can also be expressed as:\n\n$$\np_\\\\theta(x^{s-1}|x^s) = \\\\mathcal{N}(x^{s-1};\\\\mu_{\\\\theta}(x^s,s),\\\\sigma_s^2 I)\n$$\n\nAccording to the KL divergence of two Gaussian distributions with the same variance, we can get: \n\n$$\nL\\_{diff}=\\\\mathbb{E}\\_{q(x^0,x^s)}\\\\left[\\\\frac{1}{2\\\\sigma\\_s^2} ||\\\\tilde{\\\\mu\\_s} (x^s, x^0)-\\\\mu\\_\\\\theta(x^s,s) ||^2\\\\right]\n$$\n\nSince $\\\\tilde{\\\\mu}_s$ and $x^0$ are linearly related, we turn to predict the original space, so:\n\n$$\nL_{diff}\\\\sim \\\\mathbb{E}_{\\\\epsilon, q(x^0)} \\\\left[ || x^0 - x^{out}  ||^2 \\\\right].\n$$\n\nAdding the cumulative sum of subscripts and the notation from the original paper, we get:\n\n$$\nL_{ours} =  \\sum_{j=1}^{N} \\\\mathbb{E}_{\\\\epsilon, q(x_j^0)} \\\\left[ || x_j^0  - g(\\hat{z_j}^{in} - f(z\\_{1:j-1}^{in}) ||^2 \\\\right]\n$$\n\n**A3 for Q3**\n\nBelow is a table of the key parameters for the complete model, pre-training, and fine-tuning process, where `p1/p2/…` means we searched for these parameters (such as d_model, learning_rate), or we dynamically modified these parameters based on the size of the dataset (such as batch_size):\n\n\n[Table2]\n\n|Tasks|Encoder||Decoder||Pre-training|||Fine-tuning||||\n|-|-|-|-|-|-|-|-|-|-|-|-|\n||e_layers|d_model|d_layers|d_model|learning_rate|batch_size|epoches|learning_rate|lr_scheduler|batch_size|epoches|\n|Forecasting|2|8/32/128/512|1|encoder_d_model|0.001,0.0005,0.0001|8,16|30,50|0.001,0.0005,0.0001|cosine/exponential decay|8,16,32,64|10|\n|Classification|2|64/128/256|1|encoder_d_model|0.001,0.0005,0.0001|16,64,128|30,50|0.001,0.0005,0.0001|cosine/exponential decay|16,64,128|10|\n\nThank you for your encouraging score. We sincerely look forward to your reply.", "claims_and_evidence": "most of the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are sensible.", "theoretical_claims": "yes, equations follow basic mathematical operations and seem logically sound.", "experimental_designs_or_analyses": "the forecasting experiments are well-configured and analysed", "relation_to_broader_scientific_literature": "The model addresses the challenges of traditional self-supervised methods, offering an approach with improved performance in time series forecasting", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "strengths:\n\n(1) Demonstrating that autoregressive methods can be applied to self-supervised tasks and that diffusion models can explicitly introduce noise and model local patterns, which offers new insights to the forecasting\n\n(2) The paper is well-designed and the presentation is clear.\n\n(3) The technics of combination of the autoregressive  and the denoising diffusion process seems sound.\n\nWeaknesses:\n\n(1) For self-supervised learning, one of the significant advantages lies in its performance after few-shot fine-tuning in downstream tasks. I'm curious about how TimeDART would perform when only 5% or 10% of the samples are used for downstream fine-tuning\n\n(2) The transition from the description of the reverse process in Equation (7) to the elaboration of the optimization objective in Equation (8) seems rather large. The authors should provide a more detailed derivation of the optimization objective\n\n(3) add a complete table to illustrate the specific model and training parameters for each dataset.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1nNZ5qePf", "reviewer": "Reviewer_S1wT", "summary": "This paper introduces TimeDART, a self-supervised time series representation learning framework that integrates autoregressive modeling with a denoising diffusion process. The method consists of a causal Transformer encoder with a patch-based embedding strategy to capture global trends, while a denoising diffusion process refines fine-grained local patterns. The authors claim that this combination improves transferability and representation quality for downstream tasks. Empirical evaluations on nine publicly available datasets demonstrate the effectiveness of TimeDART, outperforming various state-of-the-art baselines in time series forecasting and classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/\n>\n> [A1 for Q1]--->para4,para5, [A3 for W1] —> para3\n\n**A1 for Q1**\n\nPlease refer to `A1 for Q1` in our rebuttal to the second Reviewer TNxq.\n\n**A2 for Q2**\n\nOf course, we are willing to elaborate on our understanding of these two generative paradigms in the context of time series representation learning, and we welcome your corrections if there are any inaccuracies:\n\nFrom the perspective of the characteristics of the data itself, we believe that time series data possesses traits of both language data and image data. First, language data generally has high information density and exhibits strong contextual dependencies. Autoregressive modeling methods share a natural alignment with human language [1], and similarly, time series data emphasizes dependencies from past to current states. Secondly, image data typically has lower information density [2] and places greater emphasis on jointly modeling global spatial relationships and locality. Time series data also shares similar characteristics.\n\nRecently, there have been excellent works applying diffusion to autoregressive language models, such as LlaDA [3], as well as works applying autoregressive approaches to image generation, such as VAR [4]. These are inspiring contributions. As contemporaneous work, we carefully examined the distinct roles that diffusion modeling and autoregressive modeling play in time series representation learning. \n\nOn one hand, autoregressive modeling can capture relationships from left to right. However, we recognize that time series data also shares characteristics with image data, including locality features. Due to its relatively low information density, it is challenging to effectively fit time series data using autoregressive methods without employing discretization techniques like VQVAE. \n\nOn the other hand, diffusion models can effectively model locality features in time series, such as abrupt weather changes within a single day or week. However, modeling time series requires a significant focus on trend modeling [5],[6]. otherwise, the model risks overfitting to drift.\n\nThis is the source of our motivation: by combining these two approaches, we can capture both long-term dynamic evolution and subtle local patterns in a unified manner .\n\n[1] GPT-1: Improving Language Understanding by Generative Pre-Training \n\n[2] MAE: Masked Autoencoders Are Scalable Vision Learners \n\n[3] LlaDA: Large Language Diffusion Models\n\n[4] VAR: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction\n\n[5] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting\n\n[6] FDF: Flexible Decoupled Framework for Time Series Forecasting with Conditional Denoising and Polynomial Modeling\n\n**A3 for W1**\n\nFor non-diffusion self-supervised methods, comparisons are provided in Tables 2, 3, and 4 of the paper, including methods based on mask modeling and contrastive learning. Specific baseline details can be reviewed in Section 4.1 (Baselines) of the article. The experiments demonstrate that, compared to the baseline methods we selected, TimeDART achieves superior downstream fine-tuning performance after pre-training.\n\nFor LLM-based methods, please refer to `A2 for Q2` in our rebuttal to the first Reviewer baxN where we compare TimeDART with LLM-based methods like UniTime.\n\n**A4 for W2**\n\nThank you for your sincere suggestions. We will continue to refine our paper based on the feedback from all reviewers, especially the analysis regarding the differences between autoregressive and diffusion modeling that you mentioned in question 2.\n\nThank you for your encouraging score. We sincerely look forward to your reply.", "claims_and_evidence": "In this work, the authors hold that combination of auto-regressive and diffusion optimization schemes can be used to obtain transferrable time series representation so as to benefit the target downstream tasks. The specific experimental results are provided in the experimental part.", "methods_and_evaluation_criteria": "Yes, the chosen benchmarks (ETT, Electricity, Traffic, Weather, PEMS, EEG, Epilepsy, HAR) are widely used in time series forecasting and classification. The evaluation settings are appropriate:", "theoretical_claims": "The paper primarily relies on empirical validation rather than formal theoretical proofs. The diffusion loss formulation aligns with standard ELBO principles and appears correct. While the integration of autoregressive modeling and diffusion is well-motivated, a more detailed theoretical justification could further strengthen the claims.", "experimental_designs_or_analyses": "The experimental design is mostly sound. The main results of the proposed methods and compared baselines, the ablation of this model et al are discussed in the experimental parts.", "relation_to_broader_scientific_literature": "The paper advances self-supervised time series representation learning by integrating autoregressive modeling with denoising diffusion, addressing limitations in masked modeling (e.g., TimeMAE) and contrastive learning (e.g., TS2Vec). It extends diffusion models beyond probabilistic forecasting (e.g., TimeGrad, CSDI) to representation learning, improving both global trend and local pattern capture. A comparison with large-scale time series foundation models (e.g., Chronos, TimesFM et al) would further clarify its positioning in the field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. Novel integration of diffusion models and autoregressive modeling for time series representation learning.\n2. Extensive empirical validation across nine datasets for forecasting and classification.\n3. Strong performance against self-supervised and supervised baselines, demonstrating robustness.\n4. Cross-domain evaluation, showing adaptability across different time series applications.\n5. Comprehensive ablation studies, highlighting the contributions of different components.\n\nWeakness:\n1. Baseline selection could be improved—how does TimeDART compare to LLM-based time series models or non-diffusion self-supervised methods?\n2. I Suggest the author further improve the writing details to improve the impact of this paper.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Q1: Whether the pre-trained time series model can be fine-tuned in a new manner? For example, it obeys the same training procedure as the pre-training stage. \n\nQ2: The authors combine two types of prevalent generative optimization paradigm together for self-supervised time series representation. Can you describe the great difference between this two training schemes in terms of time series representation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed a self-supervised time series representation learning method. This research direction is significant and still in an early stages. The reviewers have generally provided positive feedback, and I also believe that this paper demonstrates good performance in both theoretical analysis and experimental validation. I agree with the comments of reviewer TNxq and suggest authors address the comments in the final revision. Overall, this paper presents a solid work and is potentially useful to the community. I am confident that this paper can be accepted.
Decision: Accept (poster)