Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting
Authors: Yuxuan Yang
Dalin Zhang
Yuxuan Liang
Hua Lu
Gang Chen
Huan Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Time Series Forecasting
Self-supervised Learning
pseudo labels
adaptive mask
Primary area: applications

Abstract: Time Series Forecasting (TSF) is a crucial task in various domains, yet existing TSF models rely heavily on high-quality data and insufficiently exploit all available data. This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets. During the optimization of a simple reconstruction network, intermediates are used as pseudo labels in a self-supervised paradigm, improving generalization for any predictor. We introduce the Self-Correction with Adaptive Mask (SCAM), which discards overfitted components and selectively replaces them with pseudo labels generated from reconstructions. Additionally, we incorporate Spectral Norm Regularization (SNR) to further suppress overfitting from a loss landscape perspective. Our experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models. This work offers a new perspective on constructing datasets and enhancing the generalization of TSF models through self-supervised learning. The code is available at https://github.com/SuDIS-ZJU/SCAM.

Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting
Yuxuan Yang!,, Dalin Zhang', Yuxuan Liang', Hua LuL, Gang Chen Huan Li
'he State Key Laboratory of Blockchain and Data Security, Zhejiang University
"aanzhouu High-Teck Tech Zone (Binjiang) Institute of Blockchain and Data Security
Space Information Research Institute, Hangzhou Dianzi University 'The Hong Kong University of Science and Technology (Guangzhou)
$ DDpartmentt of Computer Science, Aalborg University, Denmark (yangyuxuan, cg, lihuan. cs} edu. zhangdalin@hdu. edu. en, i iuxlouugott com, luhua@cs, aau. dk
Abstract
Time Series Forecasting (TSF) is crucial task in various domains, yet existing TSF models rely heavily on high-quality data and insufficiently exploit all available data. This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets. During the optimization of a simple reconstruction network, intermediates used pseudo labels in self-supervised paradigm, improving generalization for any predictor. We introduce the Self- Correction with Adaptive Mask (SCAM), which discards overfitted components and selectively replaces them with pscudo labels generated from reconstructions. Additionally, incorporate Spectral Norm Regularization (SNR) to further sup- press overfitting from a loss landscape perspective. Our experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models. This work offers new perspective on constructing datasets and enhancing the generalization of TSF models through self-supervised learning. The code is available at hutpss//ggttubbcoom//b/////////.
Introduction
Time Series Forecasting (TSF) is crucial task with extensive applications in energy, finance, engineering, and many other domains. Recent advances in deep learning have resulted in TSF methods that outperform traditional methods in precision, robustness, and scalability [53 36).
Nevertheless, deep learning-basse TSF methods still face significant challenges such as overfitting, dependence high-quality datasets, and inconsistent performance across datasets issues acerbated by flawed evaluation practices 34 30 Central to these challenges the problem of low-quality label associated with inherent noise and anomalies in raw data. Existing strategies, such as multimodal data integration [3. and dataset scaling [35), focus on augmenting or refining datasets but fail address the core limitation: the reliance raw labels as ground truth. To address this, we pose two critical questions:
1. Can the reliance on high-quality labeled time series datasets be alleviated, given their scarcity?
'In TSF, given two adjacent windows time series, æ (input window) and y (output window), "labels" refer to fitted by predictions g = f(x; 0) in a supervised learning setting.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2. Can the potential of existing time series datasets be better exploited to improve model perfor- mance?
We posit that both answers are positive by redefining how labels are generated. Instead of treating raw labels as immutable targets, we selectively replace them with "pseudo labels" generated self- supervisedly.
The key idea is that the pseudo labels can be created from inherent search through candidate datasets created by an auxiliary reconstruction task. In this process, raw labels are partially replaced with reconstructions, guided by an adaptive mask that identifies overfitted raw components and selectively replaces them with pseudo labels for predictions in the supervised setting. This self- supervised learning paradigm significantly enhances the generalization of TSF models compared to traditional supervised learning, which rigidly adheres to raw labels.
eppy:
Self-Correction with Adaptive Mask (SCAM)
mask
lb
yll
Imasked rupcator
Reconstruction Network g(;4))
apply mask
911
Mmaskec updatol
Predictor f(:
predittion
reconstruction)
yabelsy
Non Seleectd Region
Hask for E--28
lapayy
mask
ll
yll
Figure 1: The illustration of the proposed method SCAM.
From an intuitive perspective, a straightforward reconstruction network g(: Ø) can be employed to generate pseudo-labels in a self-supervised manner. To assess the feasibility of training such reconstruction network, we conduct an initial evaluation using grid search algorithm (Section 3.1 Appendix B). The network g is optimized to learn an identity mapping (i.e., reconstruction) of the raw labels over multiple training epochs. During each epoch, the intermediate parameters of can be frozen to produce an approximate reconstruction of the dataset. Empirical observations reveal that when certain predictors f(: 0) are trained from scratch on these intermediate datasets, they exhibit superior performance compared to their counterparts trained exclusively on the raw datasets.
While the reconstruction-bast based approach demonstrates potential for pseudo-label generation, deter- mining optimal parameters presents significant challenge. Specifically, the reconstruction loss (i.e.. 1 ū|) when combined with the original supervised loss (i.e., || y|I), tends to drive g(: )) toward reproducing the raw labels. This phenomenon induces an undesirable "overfitting" behavior during training.
To better analyze overfitting, derive mask form of the co-objective loss, which partitions original time series into distinct components (see Figure]] D. We employ criterion, namely the sharpness metric Amaz 10). which detects overfitting by evaluating the sharpness of loss landscape. Using Amaz and practical evaluations of the decomposed loss, we identify the overfitted components. This process, termed Self-Correction with Adaptive Mask (SCAM), discards raw data labels (y) based on reconstruction results (j) and current predictions (g), smoothing the loss landscape and enhancing the generalization of the predictor model f(:
We apply masked updates on a reconstruction network g(;; () and a predictor model ((: during training. During inference, the prediction will be generated directly by the updated f(; f) with no additional cost. To further generalize across models of varying complexities, we also propose to apply Spectral Norm Regularization 46] 26 to parameters of the first or last linear layers without altering optimizers like previous works 10
We summarize our contributions follows:
Novel Perspective: We explore a novel approach of self-enhancing TSF datasets by incorporating auxiliary reconstruction task into TSF model training.

---

Self-supervised Paradigm:
We propose self-supervised paradigm that generates pseudo labels from reconstructions and adaptively replaces overfitted raw labels to improve models' generaliz- ability.
Detailed Analysis: We confirm the effectiveness of the proposed self-supervised mask formulation with extensive analyses over various backbones and real-world datasets.
2 Preliminary
2.1 Problem Formulation
Many previous TSF studies adopt a paradigm that learns a direct mapping between two adjacent windows: the history series (imputs) and the future series (labels). Let the history series (resp. future series) be [x1, X2,..,,.... XN = X E RLXN (resp. Y2.. .yx]........... series length L (resp. H) and dataset size (number of segmented windows) N. For simplicity, we formulate the problem in the univariate scenario, as it naturally extends to the multivariate case by treating each variable as an additional dimension.
Y E HR with time
Definition 1. A typical TSF process is formulated as supervised-leaming problem, iee, to find 0" arg min (If(X;) Y,,, where a specified metric ||. is used to measure ervors, typically the 1- f--norm.
When splitting the data into training and test sets, the training set Dern = {Xern, Yern] J and the model f(::0) can determine minimal target error Lier = (X 0*) Ytest on the set Drest = {Xtests Ytesth
Usually, TSF models struggle small or noisy datasets. Now, suppose we can obtain additional candidate datasets beyond the observed raw dataset; ideally, this would address the issue. In this sense, we assume a family of such candidate datasets D [D:], where the optimal candidate dataset D; enables better generalization for predictor when evaluated by Ltar on the raw test set.
Without constraints, defining such a dataset family can be overly arbitrary. To handle this, we parameterize the family using neural network g(X; Φ) X, which is trained on reconstruction loss Lree ||X- - X|- During an iterative optimization process (e.g., typical full-batch gradient descent with learning rate 17). candidate dataset D; is generated at each iteration step as follows:
D. X }.
(1g (x) X1) ig (v: 04) 1y YJ),
(1)
Φ, = Ø0 Σ E Σ m Vou Φx) - xx]-
This approach shifts the focus from designing models with different inductive biases to identifying candidate datasets that improve a model's generalization. For TSF scenarios where data are often noisy and challenging to clean, replacing raw datasets with such parameterized candidates can lead to more robust performance. To conclude, we formalize the idea as follows.
Hypothesis 1. Given time series data split into Dern. (rtny Yern and Drest = (Xtest Ytesth and a predictor model f(;;0, 0). In a family of training sets D3 = (Dimn) parameterized by g(; Φ) as in Eq. 1 there exist optimal D = (xx,"')
(1g(1 x²)} [g(15; (d"}}} such that IX 0 (d)))) Ytessll S (IIf(Xiesti 0 (ot)) Yusr |l. Vo.
-
where 8 (dx) indicates that 6* is optimized Dir (X, Y.) (1((1; o.)), folus; o.)1).
2.2 Proposed g(::0) for Reconstruction
We proceed introduce a simple reconstruction network used in subsequent exploration. Similar to a predictor model, the reconstruction network operates in a sequence-to-sen tonnnce--t-seeee so-quuence fashion, learning function g(:: σ) that maps raw series Y to reconstructed series Y. Note that reconstruction is applied only to Y. time series datasets are typically generated from a single series using a moving window approach, where X and Y are almost fully overlapped. By skipping reconstruction for X, the predictor model can use raw series as inputs, avoiding extra inference costs or potential distribution shifts.
3

---

As depicted in Figure 2 the proposed g(: o) comprises four convolutional layers, a cross-layer concatenation for multi- resolution integration, and lightweight feedforward network (FFN) to decode the reconstructed results. The convolutional lay- ers primarily act as a parameterized smoothing mechanism, sim- ilar to techniques for seasonal-trend decomposition [48] [6] [7].
Y
Conv
Conv
Concat
+
+ y
Conv
The FFN then mixes information from different positions in a
time series to reconstruct each data point using features extracted by convolutional layers (further details in Appendix Without introducing extra noise or patch-wise/pp wise/point-wwiiie wise masks [27] we directly learn an identity mapping.
Conv
Figure 2: Convolution-FFFy recon-
struction network.
In essence, g(: Φ)) is designed traverse diverse alternatives with varying levels of fidelity, which differs in purpose from the usual reconstruction task [22 24 I8]. Due to this difference, the architecture of g(:; might benefit from novel designs; however, this not examined in this study. The proposed g(:: <) is merely a simple prototype to validate our ideas and is not claimed superior to other unexplored options for this novel task.
Method
3.1 Initial Case
We begin with a simple case where g(:; evolves from a randomly initialized state toward an approximation of the raw target series Y. This process can be viewed as a grid search along the axis of the following reconstruction loss:
Eree = [9(;: o) yll.
(2)
In this setup, optimize g(: o) for reconstruction loss using full-batch gradient descent, while the predictor f(: 0)) is optimized with mini-bated batch SGD in a vanilla training setting. At each optimization epoch i, freeze the current parameters of 9(:: and generate a candidate dataset D,. The predictor (s; 0) is re-initialized before the epoch and trained on D, until convergence. The prediction results the original test set are recorded for each g(55 (b) and corresponding trained ((:07) A case experiment is conducted on the ETTh dataset using a vanilla 2-layer MI.P model (for faster and more guaranteed convergence). The pseudo-code for this grid search is provided Appendix B
So far, we can construct a series candidate datasets {D;} through two-step optimization process, each easily distinguished by its reconstruction loss, which intuitively measures its similarity the raw dataset. Evaluating the performance of the same predictor trained on these varying candidate datasets leads to three major observations:
(a) Mean 1Il
(b) Loss scurch
Observation (Improved Labels Enhance Perfor- mance). Figure fa) demonstrates the existence of better-labeled candidates which bring better perfor- mance for given predictor, indicated by the seat- ter points below the red dotted line - which rep- resents the baseline performance of the predictor trained on the raw dataset.
Figure 3: (a) shows the distribution of candidate datasets during the grid search. (b) shows the un- smooth loss curve in grid search. Each scatter in both figures denotes converged predictor {-; 9)
individual dataset parameterized by g d.).
(Variable Performance w.r.t. Reconstruction Metric Erec). A feasible method should evaluate rank candidate datasets during training without relying test set performance. However, this remains challenging as datasets with comparable free values frequently exhibit substantial differences in actual performance (see Figure 3a)).
Observation 2
Observation 3 (Difficult Training). Training involves an unstable loss curve (Figure 3b)), meaning many potentially superior candidate datasets (or equivalently g(:;)) could missed. Moreover, training is costly. To ensure the predictor converges, & is only updated after O is fully trained. This renders grid search impractical for more complex models (e.g.. PATCHTST and ITR ANSAANSRRRMAA MER) or larger datasets.

---

In the next section, we propose replacing the brute-force grid search algorithm with a co-objective training approach that improves training stability and overall performance.
3.2
Co-objective Training
The grid search (Section 3.1 is framed as a two- step optimization process with two distinct objec- tives involved in finding optimal candidate datasets. The reconstruction optimízation primarily provides a trajectory of parameters Φi. without emphasizing optimality. In contrast, the prediction optimization evaluates the predictor's 's actual performance.
5.0
2.3m
430
10
15
Our analysis of grid search results suggests that sim- plifying the training process into co-objective op- timization would be beneficial. Since the solution of two-step optimization (though not optimal on the test set) essentially lies on the Pareto Front of the corre- sponding co-objective optimization (see Figure 4b)), a natural approach is to search along this front. Again, the trajectory of the optimization, rather than its strict optimality, contributes to improved test set performance, the co-objective training can still facilitate the construction of effective candidate
(a) to-objective
(b) Simplified Pareto
Figure 4: (a) shows the overfitting tendency sup- ported by the increasing Garget which measures emror on the lest set. (b) shows the Pareto Front of objectives, tilting top-left indicating the over- fitting in (a).
datasets.
single-step optimization using mini-batch SGD would be sufficient, enabling a more smooth trajectory of & during updates (Figure 5
b) vs aaa) and Figure 4a Figure (()).. Moreover, enabling gradient computation pred 1 y w.r.t. Φ introduces regularization effect, making y updated more cautiously towards y. This allows us to constrain the update of candidate datasets (now specifically represented by the reconstructed 77) by jointly constraining gradients w.r.t. both and db
minimize 0,0
C= 11 yl1
|| ell
subject to
ÿ g(v; Ø),
i fz;
Vo.ošill S 8, Vye Y
where ei is trained using both loss terms whereas trained solely on Epred = 11 ~1l- A gradient constraint is added to ensure a smooth trajectory of y. enabling the co-objective to traverse potential candidate datasets more carefully.
The gradient constraint has surrogate || Vell and practically implemented by Spectral Norm Regularization (SNR), as discussed in Sec- tion and Appendix For simplicity, we tem- porarily omit this constraint, as a 2-layer MLP converges quickly with small Voll Using the same setting as the grid search, we evalu- ate the revised loss function for co-optimizing the predictor and the reconstruction network. Figure 4 shows that this co- co-training improves Ltarget loss while simplifying the two-step train- ing process, leading to more stable optimization and reduced training costs.
Free dor of & Space gicc
Ceniral Trgge Drer
ahan ter whel
ARC SONA serrted
(a) Grid search
(b) Co-optimization
Figure 5: Illustrations of grid search and optimization.
However, as the co-training process progresses, it becomes increasingly prone to overfitting (see Figure 4 a)). Overfitting is a fundamental issue in machine learning, tied to the generalizability of models. In this specific case, this issue arises as the reconstructed dataset gradually approaches the raw dataset, causing the target loss to converge to those of the raw dataset. Similar to the two-step grid search, determining reasonable threshold to identify optimal parameters remains challenge.
5

---

To address this specific overfitting issue, we propose solutions summarized in two main components of our method: Self-Correction with Adaptive Mask (SCAM) in Section 3.3 and Spectral Norm Regularization (SNR) in Section 3.4
3.3
Ceff-Correction with Adaptive Mask (SCAM)
0,00
0235
0.05
64375
2,000
3:20
n293
64223
oces
R
100
6004 66653
0250
103
0.56
0480
best
train
Muxcr
frai)
Figure 6: Loss curves of 4 parts divided by the adaptive mask. From left to right, 2 3ol-yloM200
2 5 @ Me @ M, ツ- @ M and ly - - @ M. The four parts combined are the loss L in Eq. M refers to the mask where m, > 0 and M refers to the mask where m; < 0. The right y-axis (in green) indicates the value for (Etest - ltrain), the differences losses measured on the test set and the train set. Values (on right y-axis) above 0 (the black dotted line) indicate overfitting.
Mask Form of Self-supervised Loss. In traditional supervised-leameee learning paradigm, the target loss ltarget = |0 yll is used only to train a model, implying that all data points are equally treated as valid labels. However, in our approach, where predictors
trained alongside search for candidate datasets guided by the reconstruction loss, the labels perceived by the predictors are adaptively shifted. Specifically, ú f(z: 0) is trained to fit ÿ. meaning only the second term is optimized for the predictor [(;:0) (Eq. 3 The reconstruction loss term, on the other hand, is optimized to provide self-supervised labels for the predictor. We frame this
seffs-upervise--eammeee that adaptively adjusts labels in TSF problems. By comparing the revised loss with the traditional
paradigm
supervised loss, we explicitly separate the auxiliary loss from the supervised loss: When revisiting the objective C in co-training, the additional loss term Laux does not directly contribute to the target objective. Instead, this term depends on the relative positions of ÿ. ÿ and y. When the reconstructed ÿ is viewed as correction of labels, Laur indicates where the correction should be placed. Time series are naturally sparse in real scenarios, often containing spiking signals due irregular events or anomalies. Louz encourages the reconstructed ý to lie between the prediction ý and the actual labels y. which can undermine sparsity when used as labels.
C = ly
91 +
(1V
1 15 yl
1 91
Loup
Laus
Let = ÿ Ú, B = - y,
C Lsup + (IAI + |B3 AIA- BI)
12 12min [|A),
if AB > 0,
Lsup +
(4)
0,
if AB 5
Let m = (ÿ
(5 - y),
(2min (lū - 01.15 1- vl}.
if mf 0,
C Lsup +
(0,
if mfm <
Loup + 2 (lø - 01 0 M + + yl O M2) 3 M.
Here, M is binary mask defined by m (ÿ- ( v) 0, Mr ensures V/ أl < lg y/. and Mc represents its complement.
Decoupling Overfitted Components by Adaptive Masks. From the above derivation, we obtain mask-based co-training loss, allowing to analyze the causes of overfitting via the mask. As
described for Lave Eq. the mask M defines the relative positions of y, T. and ÿ. Specifically, M
masks Epred and free to zero when m, (is (i) y1) < 0. Similarly, Ltorget 1/ can

---

also be divided using this mask. By comparing train/test differences in the divided losses (particularly ltargets since l ppred and free are zero when m, < 0), we identify that overfitting primarily stems from ltarget @ M, as shown in Figure
Further evidence is provided by analyzing the loss landscape. A sharpness metric for optimization, proposed by [10), measures the generalization capability of a model. Specifically, the sharpness metric defined as Amaz- V the largest eigenvalue of the Hessian matrix. A higher Amaz indicates a sharper loss landscape, which correlates with a worse generalization or more severe overfitting. By computing this metric on the converged parameters, we observe that ltarget M does exhibit a sharper landscape compared to ltrrget @ M. To address this, we keep the less sharp part of the sup term, i.e., Suu @ M, resulting in the revised loss:
L
M + 2 (l0 0ll o M 10 vll M
M.
(5)
masked Caup
Cavx
Training with this revised loss reduces the sharpness of the loss landscape. As shown in Figure 7 this effectively mitigates overfitting in the co-training scenario. This represents the final loss form in proposed method, termed Self-Corres Ceff-Correction with Adaptive Mask (SCAM), where the mask M is constructed based on an auxiliary reconstruction task. The adaptive M effectively identifies and removes overfitted components of time series labels, enabling the search for more validated parametric candidate datasets g(y: Φ1).
3.4 Spectral Norm Regularization (SNR)
Note that we have omitted the gradient constraints in Eq. S This term is, in fact, positively correlated with V (r; f) (discussed in Appendix E). This further supports our previous analysis, we have only tested MLP models, which converge easily with lower Vaf. However, when the predictor is replaced by Transformer-bused model, the dominant source of overfitting shifts from target to fpred- This phenomenon is not unique to our self-supervised learning paradigm. For example, [TO] proposed sharpness-awaid optimization to address overfitting in supervised settings. While gradient penalties are theoretically effective, they may not be practical for complex models like Transformers, computing second-order derivatives can significantly hinder optimization.
more direct approach is to regularize parame- ters using the sharpness metric, known as Spec- tral Norm Regularization [26 46]:
W
W Wnormalized = 7 ||
(6)
seech
where || ll2 is the spectral norm (the largest eigenvalue of parameter matrix) and y is learnable scale factor. When applying to self- attention (SA) in Transformer-basee architec- ture, SNR significantly undermines the expres- siveness of attention score matrices (entropy collapse 449 Hence, Ilbert et al. [T0] conclude that SNR is inapplicable to SA parameters. However, we observe that linear layers- - typically the embedding layer before SA and the projection layer after SA also contribute to the overall sharpness of the loss landscape. Consequently, we propose applying SNR selectively to the pre- and post-SA linear layers. In this way, SNR can work with MLP-based models as they are also composed of multiple linear layers. Further empirical studies on SNR are discussed in Section E,
Orggrnl
Redied 11ah
(a) Sharpness comparison Loss curve comparison
Figure 7: Effectiveness of the revised loss form
4 Experiment and Analysis
We address three major questions for experiments:
Q1: Is SCAM effective across different backbone models and datasets with varying features?
Q2: How do SCAM and SNR contribute to the potential improvement in model performance?
Q3: How does the self-supervised reconstruction task benefit the predictor models?
7

---

Table 1: Performance boost by adding SCAM and SNR different backbones. Better results are in
bold.
Models
MLP
Ours
CYCLENET
Ours
PATCHTST
Ours
ITRANS
Ours
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
0.464
01.488
0.455
0.441
0,429
01.66
0.443
0.454
0.431
ETTh2
0.382
0.405
0.366
0.394
0.388
0.409
0.362
0.393
0.387
0.407
0.370
0.398
0.383
0.407
0.377
0.402
ETTml
0.391
0.402
0.388
0.398
0.379
0.395
0.368
0.388
0.387
0.400
0.381
0.394
0.407
0,410
0.387
0.399
0.3255
0.266
0.341
0.309
0.281
0.326
0.288
0.283
0.327
Electricity
0.204
0.285
0.203
0.283
0.168
0.259
0.166
0.258
0.205
0.290
0.191
0.275
0.178
0.270
0.173
0.267
Traflic
0.522
0.335
0.494
0.308
0.472
0.301
0.448
0.290
0,481
0.304
0.455
0.288
0.428
0.282
0.411
0.266
0.262
0.281
0.243
0.271
0.268
0.259
0.275
0.257
Our main evaluation involves seven datasets: Electricity, Weather, Traffic, and four ETT datasets (ETThI, ETTh2, ETTmI, ETTm2), all of which well-established TSF benchmarks and publicly available [40). We also test the proposals using four PeMM datasets of a larger scale, as reported
in Appendix H The predictor (i.e., the backbone model integrated with SCAM) covers representa- tive TSF models, including both MLP-based and Transformer- based architectures. MLP [15] is a vanilla 2-layer baseline equipped with RevIN [2] while CYCLENET [17] is SOTA MLP-based model explicitly capturing cyclic trend in time series. PATCHTST [27] and (TRANSFORMER [2T Transformer-based models, representing channel-independee
methods, and channel-dependeer
respectively. Following previous settings [53 41 for direct and fair comparison, we set prediction length H E (96, 192, 336, 720} and look-back length to 96 for all datasets. We provide dataset descriptions, implementation details, and reproduction instructions Appendix
4.1 Main Experiment (Q1)
Table demonstrates consistent performance improvements in all backbones across all datasets when SCAM and SNR are incorporated in the self-supervised-leamee
paradigm. The full results with detailed breakdowns by prediction lengths are provided in Appendix H These gains are particularly notable on ETT datasets, which are known for their noisy nature and relatively small size. Notably, Transformer- based models like PATCHTST and ITRANN,, which typically underperform compared to lightweight models MLP and CYCLENET on these datasets, show significant enhancements in generalization with SCAM. On the Weather dataset, the boost is more modest, likely due to the intrinsic chaotic nature of atmospheric data.
Regarding Q1, our method demonstrates general effectiveness across various backbones and datasets. A well-known discrepancy between MLP-based and Transformer-basee models is their dataset preferences: Transformer-ben perform better on noisy datasets [30 34). SCAM helps bridge this gap by enabling Transformer-base models to perform competitively on traditionally challenging datasets and enhancing the robustness of MLL--ased models.
methods excel on large, regular datasets, while MLP-based methods
4.2 Ablation Study (Q2)
Table 2: Ablation study for CYCLENET.
Table 3: Ablation study for ITTANNS.
+S0MM
"NE
EFild
MAE
XSS
ETTR
MSE
STTml
XIX
MSE
MAE
MIE
Windha
MAE
SCAM
1988
ETTh
M0E
EITH
KYTal
JISE
Efited
MIKE
M55
Wisthy
0444
0.656
0.817
6.50
0.357
11356
13333
9.275
0446
0.440
0.800
0417
6.415
0.414
0.99
0.341
6.385
0.366
9438
095
0.33
0.995
100
0.255
0311
0.266
0.22
0.645
0.392
04477
0.415
11300
11347
01253
8431
0.69
0.362
0.313
0.38
0.388
1L363
0.399
0,34
0.366
6,51
0.10
0,37
0,812
0.09
1L383
0337
6.27%
To answer 02, we present the performance gains from SCAM and SNR through an ablation study (Ta- bles 2 and 3 using ITRANS (Transformer--assee and CYCLENET (MLP-based) as representatives (the SOTA one in their category).
SNR, a practical alternative to the gradient penalty in Eq. 8 consistently enhances performance across backbones. However, ITRANS, being more prone overfitting on small datasets (discussed in [00).
"The results in both tables may have discrepancy in baseline results from Table This is because the Table has baseline results from the original papers, and Table 2 and are from our runs for module modification. See
|
See G.3 for further explanations.
8

---

benefits less from SNR compared to CYCLENET. likely due to insufficient data for generalization in higher-complixity architectures. With SCAM, both models exhibit significant improvements. leveraging the expanded effective training data.
In summary, SCAM is the primary driver of performance gains, offering consistent and clear improve- ments. While SNR enhances results as standalone method, it serves best as complement to SCAM, acting as an effective surrogate for the gradient penalty in the self-supervised objective.
4.3 SCAM: Multiple Instance Learning View (Q3)
Multiple Instance Learning (MIL. [25] 3) is a classical
weakly-supervised binary classification problem where
typically bag of instances is labeled positive if at least one instance is positíve. [8 [S] extend MIL to Time Series
Ground Reconstruction Prediction }
bag of in- stances (time points), enabling a model predict based on instance-level classifications that are more interpretable.
Classification by treating an input window
To answer Q3, hypothesize that the effectiveness of SCAM shares similarities with MIL. by leveraging instance-level signals, though it does not strictly follow a MIL framework. For quick illustration, we synthe- Figure 8: Case study visualized mask. size toy dataset where the ground truth is defined A (in(w;rr r) + B sin (w22z) with added noise sampled alternately from (1, a1) and (0, in different windows. As shown in Figure when the noise deviation is large (left part), SCAM tends optimize free- M @ M | yll. prioritizing robust reconstructions: when is small, SCAM shifts focus to optimizing fpred = M @ Mzllū yll and (zorget = || - y||. emphasizing accurate predictions.
64 10 for 1
-1
M.M. for F 99
This study only reveals a part of SCAM's self-supervision effectiveness, which further explored in Appendix B
5 Related Work
We discuss related techniques from meta-learning and self-supervised learning perspectives, with an inventory of TSF models in Appendix [F]
Meta-Learning for Time Series. Meta-Learning, by definition, seeks to perform in a learn-to-leam paradigm. Generally speaking, meta-learning for TSF includes optimization- and metric-based methods. Optimization-basee methods target optimal initial parameters [9) often involving two-
loop optimization process [28] 38). Metric-based methods 7 38 learn metric function that provides
more expressive measurements of distances between data samples, commonly comparing training and validation samples.
Our method SCAM aligns with the broader scope of meta-learning. Specifically, the grid search (Section 3.1 Appendix [B) follows the two-loop structure similar aptimizatiionbasee methods. However, it diverges by focusing on dataset space rather than parameter space. Additionally, the final mask form of SCAM in essence provides a more accurate metric tailored for a supervised-learning setting. Previous works
(38 7 learn metrics on the sample level (a window of time series), whereas ours focuses on the instance level (individual data points of time series).
Self-supervised Learning for Time Series. Self-supervieed learning trains models without relying on manually labeled data by using auxiliary tasks like generation, contrast, and reconstruction to learn expressive representation or create pseudo labels. In the realm of time series, this approach is discussed more in Time Series Classification (TSC) [1] 22 22). Recent works g 5 present a novel perspective that instances in time series/segments can have multiple labels. They propose methods that significantly improve both performance and
corresponding saadll--upprrise--leaaame interpretability.
In this work, we integrate the multiple-lubel perspective by employing an auxiliary reconstruction task, commonly used in TSC, to enhance the performance of TSE. The pseudo labels, often discussed in TSC, are derived from existing, manual ones, while ours are created in self-supervised paradigm.

---

Conclusion and Future Work
This paper presents self-supervised approach SCAM that enhances TSF models by selectively replacing overfitted components with pseudo labels derived from intermediate reconstructions. When combined with Spectral Norm Regularization, SCAM improves generalization and robustness across TSF models. Future work will explore extending SCAM to tasks such as time series outlier detection and error correction.
Acknowledgment
This work supported by the National Natural Science Foundation of China (Nos. 62402420, 62402414), Fundamental Research Funds for Central Universities (Grant No. 226202400109) the Zhejiang Provincial Natural Science Foundation (No. LD24F020015), and the National Regional Innovation and Development Joint Fund (No. U24A20254). Huan is the corresponding author of this paper.
10

---

References
[1] Md Atik Ahamed and Qiang Cheng. Timemachine: A time series is worth 4 mambas for long-term forecasting. In ECAI 2024: 27th European Conference Artificial Intelligence. 19-24 October 2024, Santiago de Compostela, Spain-Including 13th Conference on Prestigious Applications of Intelligent Systems. European Conference on Artificial Intelli, volume 392, page 1688, 2024.
[2] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, al. Chronos: Learning the language of time series, arXiv preprint arXiv:2403.08815 2024.
[3] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Puttern recognition, 77: 329-353, 2018.
[4] Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, and Chenghao Liu. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters. arXiv preprint arXiv:2408. 17253, 2024.
[5] Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li, Hao Wang, Aristeidis Sotiras, Yalin Wang, and Abolfazl Razi. Timemil: Advancing multivariate time series classification via time-aware multiple instance learning. In Forty-first International Conference on Machine Learning, 2024.
[6] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM international conference information & knowledge management, pages 402-411, 2021.
[8] Joseph Early, Gavin Cheung, Kurt Cutajar, Hanting Xie, Jas Kandola, and Niall Twomey. Inherently interpretable time series classification via multiple instance learning. In The Twelfth International Conference on Learning Representations, 2024.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. aggosttg meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-170 1135. PMLR, 2017.
[10] Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and levgen Redko. Samformer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention. In Forty-first International Conference on Machine Learning. 2024.
[11] Shayan Jawed, Josif Grabocka, and Lars Schmidt- Siannidt-Thheme. Self-supervised learning for semi- supervised time series classification. In Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore, May 11-14, 2020, Proceedings, Part I 24, pages 499-511. Springer, 2020.
[12] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[13] Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
[14] Seunghan Lee, Taeyoung Park, and Kibok Lee. Learning to embed time series patches indepen- dently. arXiv preprint 23 16427, 2023.
[15] Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: investigation on linear mapping. arXiv preprint 275. 2023.
11

---

[16] Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. Sparsetsf: Modeling long-term time series forecasting with Ik parameters. In Forty-first International Conference on Machine Learning, 2024.
[17] Shengsheng Lin, Weiwei Lin, HU Xinyi, Wentai Wu, Ruichao Mo, and Haocheng Zhong. Cyclenet: Enhancing time series forecasting through modeling periodic patterns. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[18] Jiexi Liu and Songean Chen. Timesurl: Supf-super contrastive learning for universal time series representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13918- 13926, 2024.
[19] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A inguage--mpoweree unified model for cross-domas time series forecasting. In Proceedings of the ACM on Web Conference 2024, 2024.
[20] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:988 1-993,, 2022.
[21] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[22] Ziyu Liu, Azadeh Alavi, Minyi Li, and Xiang Zhang. Self-supervised learning for time series: Contrastive or generative? arXiv preprint urXiv:2403.09809 2024.
[23] Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations, 2024.
[24] Yanqing Ma, Carmine Ventre, and Maria Polukarov. Denoised labels for financial time series data via self-supervised learning. In Proceedings of the Third ACM International Conference on Al Finance, pages 471-479 2022.
[25] Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning, Advances in neural information processing systems, 10, 1997.
[26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[27] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arX11:2211.14770,, 2022.
[28] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAA! conference artificial intelligence, volume 35, pages 9242 9250, 2021.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[30] Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S Jensen, Zhenli Sheng, et al. Tíb: Towards comprehensive and fair benchmarking of time series forecasting methods. arXiv preprint arXiv:2403. 2024.
[31] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3): 1181-1191, 2020.
[32] John Schulman. Trust region policy optimization. arXiv preprint arXiv:1002.0547,, 2015.
12

---

[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[34] Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang, Di Yao, Tao Sun, Guangyin Jin, Xin Cao, et al. Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis. IEEE Transactions on Knowledge and Data Engineering, 2024.
[35] Jingzhe Shi, Qinwei Ma, Huan Ma, and Lei Li. Scaling law for time series forecasting. arXiv preprint arXiv:2405.15124, 2024.
[36] Sean Taylor and Benjamin Letham. Forecasting scale. The American Statistician, 72(1): 37-45, 2018.
[37] Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. In The Twelfth International Conference Learning Representations, 2024.
[38] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Deeptime: Deep time-index meta-learning for non-stationary time-series forecasting. 2022.
[39] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. 2024.
[40] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 2021.
[41] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
[42] Liang Xi, Zichao Yun, Han Liu, Ruidong Wang, Xunhua Huang, and Haoyi Fan. Semi- supervised time series classification model with self-supervised learning. Engineering Applica- tions of Artificial Intelligence, 16:105331, 2022.
[43] Zhijian Xu, Yuxuan Bian, Jianyuan Zhong, Xiangyu Wen, and Qiang Xu. Beyond trend and periodicity: Guiding time series forecasting with textual cues. arXiv preprint arXiv:2405.13522. 2024.
[44] Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters, In The Twelfth International Conference on Learming Representations, 2024.
[45] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He. Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mips are more effective learners in time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
[46] Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv preprint 1xxiv:1705. 10941, 2017.
[47] Chengqing Yu, Fei Wang. Zezhi Shao, Tao Sun, Lin Wu, and Yongjun Xu. Dsformer: A double sampling transformer for multivariate time series long-term prediction. In Proceedings of the 32nd ACM international conference on information and knowledge management, pages 2023.
[48] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[49] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang. Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. International Conference on Machine Learning, pages 40770- 40803. PMLR, 2023.
13

---

[50] Yitian Zhang, Liheng Ma, Soumyasundar Pal, Yingxue Zhang, and Mark Coates. Multi- resolution time-series transformer for long-tent term forecasting. In International Conference on Artificial Intelligence and Statistics. PMLR, 2024.
[51] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
[52] Lifan Zhao and Yanyan Shen. Rethinking channel dependence for multivariate time series forecasting: Learning from leading indicators. In The Twelfth International Conference on Learning Representations, 2024.
[53] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, 2021.
[54] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268-27286. PMLR, 2022.
14

---

A Limitations
Our work constrains the discussion for TSF models in MLP- and Transformer-bases ones, excluding other potentially feasible solutions [23 M. We omit these newly explored architectures since they have not outperformed the baseline methods we have adopted under the same budget. Also, we have not verified our method on LLM-based models (or models of an LLM-level scale) 39 due to the excessive costs required.
B
Grid Search Algorithm in Initial Case
In Section 3.1 we have introduced the grid search process to illustrate how the reconstruction network o( evolves approximate the raw target series Y by minimizing the reconstruction loss free = g Φ) y/l- The following Algorithm provides a detailed implementation of this process, where the reconstruction network ((53) and the predictor model [(: 0) are jointly optimized to minimize both reconstruction and prediction losses. Specifically, for each candidate reconstruction parameter Ø; (line 1), the predictor's parameters and its optimizer are initialized (line 2). In the inner loop (line 3), the predictor is optimized by first reconstructing Vs using g(: o,) (line 6), then calculating the prediction loss pred (line 7) and reconstruction loss Eyee (line 8). The prediction loss ppred is backpropagated to update 0 (line 8), and this process repeats until the gradient Vo falls below a threshold a or the maximum steps are reached (line 3). After optimizing 0, the reconstruction loss Fee is backpropagated to update (line 11), and this process is repeated for all N candidates to find the best & (line 1).
Algorithm 1 Grid Search along Erec
Parameters: ø w.r.t reconstruction network g(: o);
i w.r.t predictor model f(:;0)
Optimizers Opta w.r.t Φ (outer loop);
Opla w.r.t 0 (inner loop)
Initialize : Opt
for i +- to N do
>. N for number of candidates Initialize 0, Opts: j +- while V > a and j do
J for maximum steps optimize predictor free +0
foreach (z,y) E (X do
6
Ji x g(y: (i)
P ppred [I] 0) Øi Eree +- free + 1| yl
Backpropagate fored and update using Opts
10
j+i+1
11
Backpropagate free and update & using Opts
C Further Discussion on Gradient Constraint
Recall that Section 3.2
we have proposed the co-training objective as:
minimize 0,6
= |10 yll + || Øll
subject
D g(v: Φ), 1 = f(;
Ve.oh.ll <
Vye E Y.
With constraints on the gradient ÿ. the co-training update allows for a more stable optimization compared to the grid search using two-step optimization. When viewing each intermediate Φ; as an individual candidate dataset, applying cautious updates to ūJ = g Φi) introduces additional candidate datascts along the optimization trajectory, enriching the search process.
15

---

However, since Ve.eū is not practically computable for each yi within single step of optimization, we turn to look for a surrogate term to replace this constraint.
Note that
V.00: = [V
og
Vagl
V
Vag
VeL V.L
Vo vol(0-v) + 09 Vo (T- + 1)
Vog
(7)
Vag
V vル
V
(6-iiVf (1 (5-1)+(ū-))
og
Vofl, because 10 - yl
< | g).
Reconstruction is inherently a simpler task compared to forecasting, which allows the last approx- imation Eq. 7 to hold after just few steps of initial optimization. When ÿ is far distinct from original labels the reconstructed series ÿ becomes nearly unpredictable, leading to instability in the optimization process. Therefore, adding a constraint on g can interfere with the convergence of the predictor model. To address this, Eq. / uses //lll S 8 as a surrogate for maintaining stability during optimization.
offers an optional assurance of gradient constraint, which
The constrained form of the optimization is equivalent to the penalized form using Lagrangian Duality. Eq. 3 can be rewritten as:
minimize 0.0
L 19 Φ) yll lo(u 0(=;0)1
+ +B Vo]
For readers who are familiar with Reinforcement Learning, this derivation resembles the transfer from a constrained optimization penalized one (e.g., from TRPO [32] PPO [33]). In brief, while the penalized form is theoretically equivalent to the constrained form, it is challenging to choose fixed B that works universally across all datasets or even within a single dataset (because intrinsic characteristics can vary over time). Thus, a more general form of constraint is required to better serve the penalty, similar to the concept of gradient clipping in PPO.
Note that Vef S 5 implies the Lipchitz condition for an arbitrary function f. This means (z) 0) - fx=;; 0)1l < C(0)] - -22l-
(9)
where C(0) is a constant with respect to the parameter 0. When considering a typical Fully Connected Layer defined (a;; W. = aWW: x b). the condition becomes
Ilo (Wx1) - a (Wx2)
[I(a(T)) + o' (20) (Wan - x0)) (x + a d((20) (Waz so))ll, by Taylor expansion (10( (W (x1 - x2))
S'noz [|]] x 2|
S C(W)] x2||-
(10)
When the derivative of activation function a has an upper bound o'moz (as is often the case for common activation functions like ReLU, Sigmoid, etc.), the Lipchitz condition holds long
[[ =2) C(W, b)
x2||-
(11)
We expect the constant C to be relatively small so that the penalty works. In fact, C' here corresponds to the spectral norm of the matrix W. which is defined as
Wa x
|| = max
(12)
|
16

---

By applying the Spectral Norm Regularization (SNR) in Eq. 6 we can ensure the constant C' equals exactly 1.
However, in practice, SNR have limitations when applied to the parameter matrix in self-attention mechanisms. This phenomenon is termed entroyy catastrophe discussed by [10] In this paper, by analyzing the sharpness of different components in the predictor model, we propose to use pre-SNR and post-SNR combined, which specifically normalizes the first and last linear layer in the TSF models (see Section 3.4).
D Implementation Details of g(:; ()
As introduced in Section 2.2 we propose simple enough reconstruction network g(:: that serves our objective. Despite its simplicity, the architecture incorporates some special designs that enhance its performance. Specifically, these include the conv-concul layer and the point-wise FFN, which are detailed in Appendix D.I and Appendix D.2 respectively.
D.1 Conv-concat Layer
Transpose and Unfold. The implementation details of the conv-concat layer involve two key operations that are designed for the following two benefits:
1. The convolution outputs be concatenated into embeddings of the same length, enabling features from different frequencies to be ideally fused into one embedding.
2. The features evenly arranged along the temporal dimension, ensuring that each embedding in the sequence has the same large Receptive Field.
To achieve these benefits, introduce two-step operation: Transpose and Unfold, which work together to ensure both uniform embedding structure and large Receptive Fields.
Specifically, we set kernel size 3, stride = 2, padding 1, and the number of kernels doubled for each subsequent layer. Using this setup, as illustrated in Figure we ensure that the number of features remains invariant across different layers, with only the shape of the features changing. Now we can fuse the outputs from different convolution layers together by flattening/unfoldingg the features to the original shape of (T. 1). Again, considering the effectiveness of -ise FFN presented in Appendix D.2, we expect the concatenated features to be near-equally arranged along the temporal dimension preserve the sequential relationships in the embedding. To achieve this, we first transpose the features and then unfold them. This practice can ensure a Receptive Field of (2!+1- wide for each embedding, where I is the total number of convolution layers.
Convolution
Transpose Unfold
Convolution
Transpase Unfold
Convolut
Enbeddding
Figure 9: The illustration of transpose and unfold operation in the Convolution Encoder.
Effective Receptive Field. As what was proposed by 23 the Effective Receptive Field (ERF) is reasonable consideration for designing cntvolutioo--baeee architectures. To evaluate the ERF of our conv-coneat layer, we input an impulse function and visualize the resulting ERF, as shown in Figure T0 The visualization demonstrates that, without requiring an extra-large convolution kernel, our proposed method achieves near-global ERF. This is made possible by combining the outputs from different layers, each capturing distinct frequency patterns.
17

---

Layer
(b) Layer 2
(c) Layer 3
(f) Layer
Figure 10: Effective Receptive Field (ERF) the proposed conv-concat layer. By the transpose and unfold operation, the ERF of each convolution layer covers the entire input series with different frequencies.
D.2 Point-wise FFN
We employ wisn FFN as parameter-sharing module decode the outputs from the convolu- tion layer. The FFN is essentially a two-layer MLP that resembles the common design linear projector in predictor models, mentioned in Section E.
To further illustrate, we present the three different parameterizations of the linear projector commonly used in TSF models in Figure
Patch-dependent Design: PATCHTST (27 adopts a patch-dependent linear projector (shown in Figure aa) that first flattens all features within patches and utilizes an extra-large weight matrix of shape l.d d, where 7. the sequence length and dl the dimension of the latent embedding.
Patch-independent Design: [4] proposes that the necessity for patch-dependent designs de- pends on the specific task. For instance, tasks like forecasting may require patch-dependent projectors, while tasks like contrastive learning might favor patch-independent design (shown in Figure b)).
Our reconstruction process does not require exploiting the patch correlations necessity and can even extend this independence to point-wiig wise scope (shown in Figure (T)).). This approach is feasible only when each point-wise embedding is sufficiently rich in information, a property achieved through convolution layer, which provides near-global ERF for each point.
Point-wise Design:
W.Lax
W.NN N
W: N
W.
W:Ad X
Widxa
30
(a) patch-dependent
(b) patch-independent
(c) point-wise
Figure 11: Three types of linear projectors.
The sharing of parameters in our linear projector allows for an increase in parallel modules. In practice, we generate multiple reconstructed samples for the same raw input sample and feed them to the predictor simultaneously. This approach inherently expands the scale of the dataset, further enhancing training efficiency.
E
Further Analysis
E.1
Distribution of Reconstructed Datasets in Grid Search
The distribution of sampled predictions is shown in Figure 12 The predictions are evaluated using
three loss metrics: (1) Prediction loss: Epred j Ill (2) Reconstruction loss: Erec = Il yll- and (3) Target loss: ftarget | y|l. In Figure 12, scatter points illustrate the relationships among these losses across various candidate datasets generated by g(:;)) on the ETThl dataset. Each scatter point is colored according to the mean reconstruction loss (free) of its corresponding dataset. Lighter colors (e.g., yellow) represent datasets with higher Eree, while darker colors (e-g., purple) correspond to datascts with lower frec.
18

---

Raw
0.8
Mlean Recconstrue Iattuuttionn uuttooo Loss
0.6
0.5
3.0
1.0
10.0
0.4
0.5
0.5
0,00
0.05
044
0.2
0.15
0.2
0.00
0.55
0.10
0.35
0.00
0.05
0.10
0.15
0.2
0.4
0.8
Lece
free
Figure 12: Distribution of losses of candidate datasets generated by (+:)) ETThI.
The distribution is visualized across three projections:
1. lreed Plane illustrates the relationship between the reconstruction loss Erec and the prediction loss Epred. both of which are actively optimized during training. As Eree decreases (darker colors), the points in the distribution become more condensed, indicating reduced flexibility the candidate datasets. This trend suggests that datasets with very low reconstruction loss may lack the diversity needed for optimal predictor performance.
2. lree-laarget Plane highlights the relationship between the reconstruction loss free and the target loss learget, where ltarget serves as the primary evaluation metric for predictor performance. Interestingly, datasets closer to the raw data (darker colors, with lower fre) do not consistently lead to better target values. This observation, which indicates that overly strict reconstruction constraints may hinder prediction quality, is further supported by Figure 3
is involved in calculating both (pred and target. Notably, candidate datasets with intermediate distances from the raw data (green points) demonstrate better generalization. These datasets are characterized by relatively higher fpred and lower target and are distributed more toward the bottom-right region of the plane, reflecting improved prediction quality.
lpred-ltarger Plane directly examines prediction performance,
These results suggest that datasets with moderate reconstruction loss neither too high nor too low strike a better balance between flexibility and generalization. This balance ultimately leads to improved predictor performance, as overly strict reconstruction constraints may limit model adaptability, while overly high reconstruction loss may fail to capture meaningful patterns.s
E.2 Demystifying the Self-supervision in SCAM
Conv-concat Layers Feature Amplifier. In theory, the reconstruction network can be extended to larger scales as it does not interfere with inference efficiency. However, in our implementation, it is kept as simple as possible to prioritize training efficiency. Despite its simplicity, our observations reveal that the two different components of the reconstruction network function in distinct ways, which offers insights into designing more effective reconstruction networks.
As mentioned in Appendix D.2] the FFN in g(: () is designed point-wise, decoding the concatenated outputs of the convolution layers for each point independently. To further investigate the utility of conv-concat layers, we skip the initial linear layers and activation functions in g(; o). directly applying the final linear transformation to the latent outputs of the conv-concat layers. This process generates an intermediate series, which we term as undecided j//
Figure 13 shows that the i/ (red plots) effectively amplifies the sparse spiking signals present in the raw data. This behavior can be interpreted as the conv- layers acting as a feature amplifier. emphasizing and enlarging important patterns in the input data.
Channel-wise Distribution Alignment. Distribution shift, a core challenge in long-term Time Series Forecasting (TSF). is decisive for a TSF model to generalize future data after training. The most widely adopted approach to address this issue is Reversible Instance Normalization (RevIn) [12 20], which aligns the distributions of historical and future data.
While RevIn significantly improves the performance of TSF models, it falls short in aligning distri- butions across channels in the multivariate forecccatttteoooeattttt setting. Figure [4 highlights this limitation:
19

---

60
23
Label y
Reconstruction Undecoded 5 from Conv layers
(a) Visualization of g. j' and y during training on Channel of ETTh1. Epochs displayed here are 2, 3, 4, 10.
23
46
20
Label y
Reconstruction Undecoded from Conv layers
(b) Visualization of ÿ and j' during training and y on Channel 4 of ETThl. Epochs displayed here are 1, 2,
10.
Figure 13: Visualization our difference components in g(::
even when the raw data y is normalized by RevIn, the distribution distances (measured by the KL divergence metric) remain significant, as shown in the leftmost plots of all subfigures.
In this view, the proposed reconstruction network (-: effec- tively serves as a general channel-wise distribution alignment mechanism. Interestingly, when examining the intermediate decided i' across channels, we observe that, although the sparse features are amplified, the distribution distances are reduced compared to those in the raw data. Overall, the final recon- structed series exhibits better alignment across channels. How- ever, exceptions such as Channels 3 and 6 (see Figure 14 c)) demonstrate that the alignment nels.
1243
Ohans
(a) Distributions of Channels and
uniform across all chan-
For models designed with channel-independenc (CI), such as PATCHTST (27) and CYCLENET [17], aligning distribu- tions across channels is of critical importance, especially when trained on datasets with a large number of variates. The channel- wise alignment introduced by our self-supervised reconstruc- tion task provides an effective solution to distribution shifts, thereby enhancing the performance of CI predictor models.
(b) Distributions of Channels and
(c) Distributions of Channels 3 and 6.
Evolution of Adaptive Mask SCAM. Section 4.3 we
MLdic 482
Cosle
Kethe MAM
have explored the benefits of the adaptive mask in SCAM from a Multiple Instance Learning (MIL) view, highlighting its sen- sitivity to instances with varying deviations and its ability to apply different masking strategies accordingly.
(d) Distributions of Channels 4 and 7.
In this part, further visualize the evolution process of the adaptive mask during training on real datasets, using ETTh I as an example, as shown in Figure 15
Figure 14: Distribution alignment for channels in ETThl with data all normalized by RevIN.
At the very first epoch, the model heavily relies on the predic-
tion ý to reconstruct the series ÿ. This benefits the convergence of both the predictor model f(; 8) and the reconstruction network g( Φ). As previously analyzed, the reconstructed series ā aligns the data distribution, enabling [((:; () to learn effectively from ÿ. Correspondingly. the prediction in the first epoch basically draft with smooth curvature, preventing ÿ from quickly overfitting the noisy raw labels y. In subsequent epochs, the proportion of gray masks applied to the loss term l yll
20

---

-0
20
60
40
60
20
40
80
20
60
23 40
60
80
Label y Reconstruction
Prediction
17 for 1
M.O.M. for 1P -1
M MoM., for 1 -1
(a) Visualization masks during training on Channel of ETThl. Epochs displayed here are 1, 2, 3, 4, 10.
2.
25
1.5
Labely Reconstruction
Prediction
18 for 1--11
M. for 19 M for
(b) Visualization masks during training on Channel of ETTh1. Epochs displayed here are 1, 2, 3, 4, 10.
Figure 15: Evolution of the adaptive masks in SCAM during training.
increases. This indicates that the model progressively emphasizes labels, resulting in improved reconstruction and prediction quality.
precise fitting the true
In comparison with the case study in Section 4.3 the adaptive mask tends to collapse more rapidly from intermediate states where all three masking strategies are simultaneously active. This is possibly due to the noisy and unpredictable nature of real-world datasets. We plan to explore less aggressive strategies compared to binary masking, which hopefully will increase the robustness of the current method.
E.3 In-depth Examination SNR
We further verify the effectiveness of SNR by analyzing the loss landscape. Transformer-basee models, as discussed, are more prone to overfitting, particularly noisy datasets. This weakens our adaptive mask's ' ability to discard overfitted components, as fpred - the predictor's training loss also contributes overfitting. Gradient penalty offers solution this issue. For TSF models specifically, where both inputs and outputs are time series with abrupt distribution shifts, parameter robustness is critical for mitigating overfitting. Previous works 46 26 have shown that regulating parameters via their spectral norm enhances stability against input perturbations, which is particularly beneficial in TSF.
0.64
0 0.5922
projector
0:179
w/o SNR pre-SNR post-SNR
Sharpness Amax
0-4647
0.4
0496
Chonnel Wise Scli-Altcstion
0.2
Lincoder
5.024
Embeediing
0.0
Embedding
Encoder
Projector
Figure 16: Sharpness of difference components in ITRANS.
Figure 17: ITRANS as a TST example
In Figure 16 the bars represent the sharpness of different components in ITRANS, divided into three parts: embedding, encoder, and projector (see Figure 17). The embedding and projector are linear layers, while the encoder comprises channel-win wise attention blocks. This architectural pattern is
21

---

common among Time Series Transformers (TSTs), with the encoder being the primary focus and less emphasis placed on pre- or post-encoder linear layers.
Without SNR, the input linear layer (embedding) exhibits the highest sharpness, indicating is the most overfitted component (see red bars). This suggests that overfitting in Transformers originates not only from self-attention mechanisms but also from the linear layers. By applying SNR to the pre- and post-encoder linear layers (Section 3.4. we observe smoother loss landscapes, validating SNR's 's ability reduce sharpness effectively.
0.386
0.382
wi SHR
w pm SKR
pott
we buth sHR
To determine the best practices for SNR, conduct multiple runs with different random seeds to evaluate various design choices. As shown in Figure [8] applying SNR to the post-encoder línear layer Figure 18: Empirical results both linear layers proves most effective: 'post-SNR' achieves lower on variants of SNR.
mean metrics and reduced variance, while 'both SNR' yields better mean performance but with higher variance. We recommend both practices, depending on the use case, and adopt 'both SNR' for all tests in this paper.
F Related Work Revisit
In this part, we will introduce some popularly adopted methods for TSF, in which our baseline models are included.
Channe-inndependen Transformers. Time Series Transformers (TST) (33 40 54
have recently
led to significant progress in the TSF problem, demonstrating convincing superiority over traditional methods and convolution-based models. Inspired by [48], 227 incorporates the Channel Independent (CI) design (sharing weights across all channels) to introduce PATCHTST, a new state-of-the-ae (SOTA) model that significantly benefits from CI and the patching operation. Multiple works follow this practice and achieve excellent performance TSF 66 19 50].
Channel-wise Transformers. Building on TST, more recent research has focused on designing Transformers capable of capturing channel dependencies inherent in multivariate time series data. Notable examples include CROSSFORMER (51), (TRANSFOOMEER (ITRANS) [21], DSFORMER (47), and CARD 37 These models shed light on exploiting inter-series relationships to improve forecasting accuracy. Furthermore, having observed that Time Series Transformers inherently unstable due to sharp loss scape. [10] propose sharpness-awe aware optimizer to mitigate such issues. Their work focuses on an optimization-leww approach involving a two-step backward. Nonetheless, channel-wise Transformers still suffer from overfitting on small datasets. By sharpness analysis, our proposed SCAM locates the overfitting issue and provides solution from a data-level perspective.
Linear or MLP-based Models. In contrast to the quadratic complexity of Transformers, lightweight linear MLP-based models have emerged as competitive alternatives offering simplicity and efficiency. RLINEAR and RMLP [15] verify that vanilla linear model or a 2-layer MLP, when combined with widely-adopted normalization method [12] can achieve near SOTA performance in TSF. Further research (52) channel dependencies within linear and MLP-based models has yielded performance improvements over previous CI approaches. Moreover, the models KKI 45 that directly learn linear regression or MLP-based models complex frequency features have achieved remarkable performances. Most recently, SPARSETSF (16), a highly lightweight model, incorporates ID convolutions as down-sampling modules and learns linear parameters the nown-sampled values of the original series. CYCLENET 177. SOTA model that explicitly captures periodic trend features to enhance vanilla linear or MLP-based models to be on par with Transformer-basee models.
G Experiment Details
G.1 Datasets
We conduct experiments on real-world datasets evaluate the performance of the proposed SCAM. The datasets are detailed below.
22

---

ETT [53]: This dataset contains 7 factors of electricity transformers, recorded between July 2016 and July 2018. The subsets ETThl and ETTh2 are recorded hourly, while ETTml and ETTm2 are recorded every 15 minutes.
Electricity 40 This dataset records the hourly electricity consumption of 321 clients.
Traffic 40): This dataset collects hourly road occupancy rates measured by 862 sensors across the San Francisco Bay Area freeways, spanning from January 2015 to December 2016.
Weather 40
This dataset includes meteorological factors, recorded every 10 minutes the Weather Station of the Max Planck Biogeochemistry Institute in 2020.
PeMS: This dataset contains public traffic network data from California, collected at 5-minute intervals. We use the same four subsets (PeMSO3, PeMS04, PeMS07, PeMS08) as adopted in (TRANSFORMER (ITRANS)) 21
For the ETT datasets, we divide them by ratio {0.6, 0.2) into train set, validation set, and
test set. For Electricity, Traffic, and Weather, follow the same split ratio of (0.7. 0.1, as For the PeMS datasets, we split them using the ratio {0.6, 0.2, 0.2)
in TIMESNET [41 53 40
following the same setting as ITRANS 21] All datasets are scaled using the mean and variance of their respective training sets, a standard practice in TSF 41. The statistics of all used datasets are listed in Table
Table 4: Statistics of evaluation datasets.
Datasets
ETTh1
ETTh2
ETTml
ETTm2
Electricity
Traffic
Weather
PeMS03
PeMS04
PeMS07
PeMS08
of TS Variates
321
862
21
358
307
883
170
TS Length
17420
17420
69680
69680
26304
17544
52696
26209
16992
28224
17856
G.2 Backbones
evaluate the proposed SCAM against the following baseline backbone models:
1. MLP 15
2-layer MLP model combined with Reversible Instance Normalization 12
CYCLENET [77. a 2-layer MLP equipped with efficient cycle modeling that belongs general scasonal-trent decomposition method (Cycle/MLL in the original paper).
3. ITRANS 21,, Transformer-bascl model that computes attention scores on the inverted series along the channel dimension.
4. PATCHTST 27 Channel-Independee Transformer-base model that uses patching to tokenize the input.
It should be pointed out that the scale of time series dataset is determined by combination of the number of variates and the length the dataset. Therefore, fair comparison should take both factors into account. In our experiments, we observe that some baseline models, such as ITRANS, benefit greatly from datasets with a larger number of variates, while others, like PATCHTST, tend to perform better on longer datasets.
G.3 Reproductivity
Hyperparameters and Settings. All experiments and methods are implemented in Python and PyTorch 29 and conducted on two Nvidia RTX A5000 Ada generation GPUs (32GB VRAMs) and two Nvidia RTX A6000 GPUs (48GB VRAMs). We use the ADAM optimizer [13 with a learning rate initialized 77 0.001 for all settings. Unlike prior works (1 16 that set a fixed, small number of training epochs, we adopt an early stopping strategy based on the MSE metric of the validation set, with patience of 20 epochs.
For the reconstruction network g(: Φ), the hyperparameters are detailed in Table 5
We use 4 convolution layers in total, with consistent settings for kernel size, stride, and padding size as explained in Appendix [ Bim_multiplic represents the expansion ratio convolution channels. The channels for the four convolution layers are set to 1, 2, 4, and 8, respectively, and are further multiplied by 4 to increase the model's capacity. Hidden_ding _dm is the dimension size the FFN, and of series indicates that use 8 point-wise linear projectors in parallel as explained in Appendix D.2
23

---

Table Hyperparameters of the reconstruction network ():
# of convolution layers
4
dim multiplier
4
hidden dim
128
# of series
8
and Figure 11 The hyperparameter settings provided in Table 5 are applied consistently across all datasets.
Rerun Baselines as Additional Comparisons. ensure fair comparisons with baseline re- sults, we include both the results reported in the original papers and the results from our own re-implementations. Specifically: The MLP and CYCLENET results are taken from CYCLENET paper [17]. The PATCHTST and ITRANS results are from the ITRANN paper [21] as the PATCHTST paper 27 does not provide results with a look-back length of 96.
While the results from the original papers and our reruns show no major discrepancies, minor differences do exist. To provide complete transparency, we present both in Table and Table 9 in Appendix H. Results from the original papers are more reliable as they reflect the authors' intended implementations, while our reruns ensure consistency training settings for direct comparison.
The Result Discrepancy on PeMS Datasets. We also evaluate the larger traffic dataset, PeMS, which has been previously examined in both ITRANS and CYCLENET. However, observe major discrepancies between our results and those reported in the original papers [7].
While the original papers state that prediction lengths of {12, 24, 48, 96) were used, their reported results closely align with what we obtain using prediction lengths of {12, 24, 36, 48). The issue of reproduction inconsistencies is also widely discussed in the TRANSFORMMER GitHub issues. For reference, we provide the results obtained using our settings, which we hope will aid clarifying these discrepancies.
G.4 Efficiency
Table 6: The efficiency analysis of SCAM. All metrics are measured on the Electricity dataset with
batch size of 32.
of parameters
Max Memory Allocated (GB)
Training Time (ms/iter)
MI.P
24.8K
0.102
6.126
+SCAM
52.4K
1.095
30.451
CYCLENET
78.7K
0.099
6.879
+SCAM
106K
1.092
30.832
PATCHTST
+SCAM
365K
2.072
53.044
ITRANS
3.3M
0.791
20.395
+SCAM
3.3M
1.729
44.357
The SCAM method introduces an additional lightweight reconstruction network to aid the training. The additional computation cost is only introduced in training, because the additional module is discarded like a scaffold.
For the training efficiency, the baselines we adopted are acknowledged for their low cost. We only introduce a constant increase in memory and time consumption, enabling our method to integrated into backbones that scale in parameters.
24

---

H
More Experiment Results
In this section, we present detailed results from the previously mentioned experiments:
Tables 8 and
Summarize the main experiments on the ETT, Electricity, Traffic, and Weather datasets, with detailed breakdowns of different prediction lengths provided.
Baseline results in Table S are taken from the original papers.
Baseline results in Table 9 are reproduced by us.
Table
Reports experiments on the PeMS datasets, including baseline results from our runs.
Tables 10
and 11
Provide the complete results of the ablation studies.
In Table performance.
and
color RED indicates better performance and color BLUE indicates worse
Table 7: Full results of experiments PeMS datasets, comparing backbone models and their
integration with our proposal. All baseline results are reproduced by us.
Modils
Mip
-Ours
EYCLuNNer
-Ours
PATCHEST
+0hrr
ITRANS
+Oues
MAA
MSE
M.A.
MSE
MAL
MSE
MSE
12
134833
0.191
0.882
0.189
0,0773
0.179
0.872
0.178
134778
0.186
0.072
08.176
0.075
0.386
0.074
11.100
PAMSOn
03.265
0.132
0.240
0.106
00254
0.089
36
0.196
0.297
0.189
0.291
0.147
0.256
0.144
0.255
0.172
0,276
0.136
0.244
0.127
0.237
0.130
0.242
0.344
0.304
0.517
0.174
Avg.
0,168
0.269
0.150
0.256
0.127
0.235
0.125
0.235
0,148
0,253
0.118
0.234
0.116
0.226
0.117
0.226
00211
0
02200
0.080
24
0.168
0,273
0.167
0.273
0.137
0.244
0.137
0.244
0.162
0,268
0.116
0.228
0.121
0.228
0.105
0.213
0.335
0.333
0.187
0.187
00521
0.139
48
0,326
0.390
0.320
0.387
0,235
0,329
0.234
0.328
0.297
0.367
0.168
0.279
0.186
0.288
0.191
0.295
Avg.
0,312
0.208
0.301
0.163
0.162
0.197
10.99
0.129
0.130
12
01079
0.185
0,680
0.185
0,989
0,171
0,009
0.171
0,076
0,180
0.068
0.160
0.065
0.139
0,060
0.154
PAMROTT
0.248
0.246
0.130
0.109
10.44
0.106
0.085
36
0.210
0.306
0.209
0.304
0.153
0.260
0.152
0.260
0.184
0,296
0.148
0.248
0.135
0.422
0.132
0.237
0,360
0.357
00332
0.277
0.183
Avg.
0.179
0.275
0.177
0.273
0.132
0,257
0.131
0.236
0.159
0,260
0.124
0.228
0.115
0.218
0,115
0.217
0.198
0.094
0.199
0,082
0
0.007
0.19)
0.130
0.071
0.167
24
0.153
0.257
0.152
0.255
0.124
0,228
0.125
0.228
0.137
0,240
0.163
0.222
0.107
0.207
0,100
0.198
36
11,22
0.312
0.310
0.170
0,268
0.370
1.194
01291
0.163
0.142
0.239
0.130
48
0.297
0.362
0.293
0.360
0.220
0,307
0.218
0.305
0.255
0.334
0:225
0.276
0.204
0.293
0,200
0.299
Aug.
0.192
0.282
0.190
0.281
0.149
0.149
0.108
102664
0.170
0.333
0.125
1mpl Arg.
3.24%
1.45%
0.37%
0.25%
19.61%
14.00%
2.31%
1.55%
25

---

Table 8: Full results for performance comparisons between backbone models and their integration with our proposals on ETT, Electricity, Traffic, and Weather datasets. The results for MLP and CYCLENET are taken from the CYCLENET paper 27 while the results for PATCHTST and ITRANS are from the ITRANSFORMER paper (21).
Modell
MLP
.Ours
CYCLUNET
40hrs
PATCHTST
4Ours
ITRANS
4Ours
Meene
MSE
MAE
MSE
MAE
MAL
MSE
MSE
MAE
MSE
MAE
96
0.383
(1401
0.373
0.39%
01.775
0.395
0.368
0.390
0.414
0.419
0.373
01.398
0.386
0.405
0.373
0.401
E7122 ETThi
0.437
0.432
0.428
0.424
0.445
0.424
0.441
0.436
0.432
0.436
336
0.994
0.461
0.474
0.441
0.196
0:455
0.470
0.440
0
0.466
0.465
0.447
0.487
0.458
0.466
0.455
0.499
0.464
0.484
0.462
0.458
0.444
0.491
0.455
Avg.
0:461
0.448
0.437
0.433
0.457
0.481
0.431
0.429
0.069
0.455
0.427
0.433
0.454
0.418
0.431
0.440
0.345
0.283
0.344
0.280
0.348
0.285
0.297
0.349
0.293
0.342
192
0.371
0.394
0.362
0.385
0.372
0.3%6
0.357
0.384
0.388
0,400
0.367
0.389
0,580
0.400
0.373
0.393
0,420
0.470
0.404
0.411
0.479
0.400
0.420
0.426
0.433
0.409
0.428
0.32
0.417
0.429
720
0:438
0.450
0.413
0.435
0.450
0.458
0.409
0.436
0.431
0,446
0.419
0.442
0.427
0.445
0.424
0.442
Avg:
0.362
0.405
0.366
0.388
0.409
0.362
0.393
0.387
0.407
0.370
0,398
0.383
0.407
0.377
0,402
96
0.327
0.366
0.325
0.361
0.319
0,360
0,306
0.349
0.329
0.367
0.316
0,354
0.354
0.368
0.315
0.353
ET1ml
192
0.370
0.386
0.367
0.383
0.360
0
0.349
0.375
0,367
0,385
0.360
0,382
0.377
0.391
0.369
0.387
0,404
0.400
0.405
0.379
0.410
0.393
0.420
0.403
730
0.462
0.445
0.462
0.443
0.447
0.441
0.438
0.435
0.454
0.439
01.544
11.437
01.491
0.459
0.460
0.445
0,400
0.381
0.410
0.387
96
0.178
0.259
0.175
0.259
0.363
0.246
0.161
0.244
0.175
0.250
0.176
11.261
0.180
0.264
0.179
0.264
ETTed
0.302
0.290
0.225
0.301
0.241
0.309
336
0.299
0.340
0.295
0.336
0.281
0.437
0.282
0.323
0,305
0.343
0.303
0,340
0,311
0.348
0.305
0.343
720
0.398
0.294
0.391
0.380
0,384
0,402
0.400
0.404
0.412
0.407
0.406
0,400
Avg.
0,280
0.325
0.276
0.323
0.266
0.341
0,262
0,309
0,281
0.326
0.281
0.326
0,258
0.332
0.283
0.327
96
0.265
0,181
0,364
0:229
0.134
0,226
0,181
0.270
0.163
0.248
0.148
0.240
0.145
0.237
Edecasccy
192
0.187
0.270
0.186
0.368
0.152
0.244
0.152
0.244
0.188
0.274
0.172
10257
0.162
0.253
0.158
0.252
336
0.203
0.287
0.202
0.385
0.170
0.264
0.170
0,263
0,204
0.291
0.191
0,278
0.178
0.369
0.176
0.271
720
0,244
0.319
0.243
0.317
0.212
0.299
0.210
0.296
0,216
0.324
0.239
0,319
0.225
0.317
0.212
0.366
Avg.
0.204
0.285
0.203
0.283
0.168
0.259
0.166
0.258
0.205
0.290
0.191
0,275
0.178
0.270
0.173
0.267
96
0,510
0.331
0.477
0.301
0.458
0.296
0.420
0.275
0,162
0.295
0.433
0,280
0.395
0.268
0.374
0.347
Iraffic
192
0.505
0.327
0.478
0.300
0.457
0.295
0.437
0,283
0,466
0.246
0.447
0.287
0.417
0.276
0.399
0.259
0.299
0.453
0.304
0.455
0.283
0.419
0.269
730
0.553
0.350
0.531
0.328
0.502
0.314
0.482
0.310
0.514
0.323
0.486
0.302
0.467
0.302
0.451
0.291
0.448
0.455
0.282
96
0.161
0.219
0.176
0.214
0.158
0.205
0.158
0.203
0.177
0.218
0.169
01.111
0.174
0.214
0.173
0.213
Weather
0.259
0.223
0.207
0.247
0.206
0.225
0.259
0.215
0.251
0.221
0.254
0.223
336
0.282
0.299
0.179
0.295
0.262
0.289
0.260
0.285
0278
0.291
0.274
0.293
0.278
0.296
0.278
0.296
730
0.357
0.347
0.365
0.345
0.344
0.344
0.343
0,342
0354
0.348
0.353
0.345
0.258
0.347
0.353
0.344
Avg.
0.202
0.281
0.258
0.278
0.443
0.271
0.242
0.268
0.259
0.281
0.253
0.275
0.258
0.278
0.257
0.278
Ingri
Avg.
3.24%
2,60%
3.92%
3.40%
4.48%
2.77%
3.24%
1.94%
26

---

Table 9: Full results for performance comparisons between backhone models and them integrated with our proposals on ETT, electricity, traffic and weather datasets. All baseline results are run ourselves
Modells
MiP
-Ours
CYCLUNDr
+Ohrr
PATCHEST
+Ours
ITRANS
+Ours
Metric
MSF
MAE
MSF
MAR
MIS
MAF
MSF
MAE
MSE
MAF
MSF.
MAF
MSF
MAE
MSF
MAE
96
0.390
0.399
0.373
0.396
0.379
0.400
0.368
0.390
0.390
0.407
0.373
0.398
0.383
0.405
0.373
0.401
ETTJ
192
0.444
0.478
0.435
0.434
0.437
0.432
0.424
0.424
0.461
0.446
0.424
0.427
0.434
0.433
0.432
0.43
336
0.478
0.441
0.474
0.442
0.477
0.466
0.470
0.440
0.086
0.457
0.465
0.447
0.570
0-452
0.466
0.455
720
0,478
0.475
0.464
0.459
0.489
0.466
0.462
0.461
0.484
0.409
0.444
0.458
0,464
0.420
0.455
0.466
Avg.
0.445
0.437
0.437
0.433
0.441
0.436
0.451
0.429
0.455
0.445
0.427
0.433
0.435
0.440
0.431
0.44)
0.291
0.343
0.347
0.280
0.208
0.545
0.285
0.321
0.562
0.293
E71h2
192
0.365
0.391
0.362
0.385
0.374
0.3%6
0.357
0.384
0.394
0.401
0.367
0.389
0.394
0.408
0.373
0.393
0.419
0.477
0.400
0.429
0.409
0.447
0.417
720
0.427
0.443
0.413
0.435
0.430
0.467
0.409
0.436
0.437
0.454
0.419
0.442
0,435
0.449
0.424
0.442
Avg.
0.377
0,401
0.366
0.390
0,406
0.362
0,393
0,387
0.407
0.370
0,396
0,400
0.417
0.377
0.402
96
0.345
0.371
0.325
0.361
0.315
0.358
0.306
0.349
0,339
0,570
0.316
10.354
0.351
0.378
0.315
0L353
ET7ml
192
0.385
0.391
0.367
0.353
0.359
0.382
0.349
0,375
0
0.393
0.360
0,362
0.393
0.399
0.369
0.387
336
0.422
0.412
0.400
0,405
01389
0.407
0.379
0.394
0.411
0.441
0.393
11400
01422
0.421
0.403
0.412
730
0.493
0.451
0,462
0,443
0.454
0.441
0.438
0.435
0.474
0.449
0.454
0.437
0.487
0.460
0.460
0.445
Avg.
0.413
0.406
0.388
0.398
0.379
0.397
0,368
0,388
0.401
0,406
0.351
0.394
0.413
0.414
0.387
0.399
96
0.186
0.270
0.175
0.259
0.164
0.248
0.101
0.244
0,180
0.263
0.176
0.261
0.189
0.275
0.179
0.264
ETTml
192
0.249
0.309
0.240
0.300
0.228
0,289
0.225
0,286
0,248
0,310
0.241
0,300
0,260
0.318
0.241
0.302
336
0.308
0.345
0.295
0.336
0.285
0.328
0.282
0.323
0.307
0.345
0.303
0,340
0.326
0.359
0.305
0.343
720
0.398
0.394
0.384
0,387
0.380
0.404
0.404
0,403
0.412
0.406
0.400
Avg.
0.287
0.331
0.276
0.322
0.266
0.318
0.262
0.309
0.287
0.351
0.281
0.336
0.299
0.341
0,283
0.327
Elestriitt
0.067
0.230
0.134
0,181
0.268
0.163
0:211
0.145
192
0.191
0.272
0.186
0,268
0.154
0.246
0.152
0.244
0.193
0.277
0.172
0.257
0.178
0.267
0.15M
0.252
0.088
0.202
0,264
0.170
0.286
0.191
0.176
0.271
720
0.251
0.325
0.243
0.317
0.212
0.299
0,210
0.296
0,240
0.319
0.239
0.319
0.213
0.306
0.212
0.306
Avg.
0.209
0.288
0.203
0.168
0,260
0.166
0,258
0,205
0.257
0.191
0,275
0.185
0.216
0.173
0.367
96
0.504
0.314
0.477
0.361
01432
0.292
0.420
0.275
0.468
0,301
0.433
10.200
11,013
0.276
0,374
0.247
Tralic
0.522
0.330
0.478
0.300
0.443
0.294
0.437
0,283
0,474
0.305
0.447
0,287
0,423
0.283
0.399
0.259
336
0.534
0.334
0.492
0.305
0.460
0.304
0.453
0.291
0.498
0.334
0.455
0.2255
0.441
0.292
0.419
0.269
720
0.540
0.337
0.531
0.338
0.489
0.321
0.482
0.310
0.547
0.340
0.486
0.302
0.465
0.303
0.451
0.291
Avg.
0.525
0.329
0.494
0.308
0.456
0,303
0.448
0.290
0.197
0.317
0.455
0.288
0.433
0.288
0.411
0.266
96
0.192
0.232
0.176
0.214
0.165
0.209
0.158
0.203
0.185
0.225
0.169
0.211
0.211
0:257
0.173
0.213
Weather
192
0,237
0.266
0.223
0.256
0.209
0.248
0.206
0.244
0.229
0.262
0.215
0,251
0.255
0.285
0.223
0.257
336
0.290
0.304
0.279
0.295
0.269
0.292
0.260
0.285
0.283
0.301
0.274
0.293
0.303
0.319
0.278
0.296
720
0.366
0.352
0.355
0.344
0.345
0.351
0.353
0.362
0.353
0.344
Avg:
0.271
0.788
0.258
0.278
0.348
0.275
0.242
0.268
0.264
0.285
0,253
0.275
0.285
0.306
0.257
0.278
Impr
Avg.
4.11%
2.52%
2.18%
5.48%
3.61%
5.51%
4.17%
Table 10: Full results of ablation study with backbone predictor as CYCLENET.
Models Metric
CYCI ENET
4SNR
+SCAM
+hrth
MSB
MAI
MSE
MAE
MSE
MSE
MAE
965
0.381
0.399
0.374
0.392
0.372
0.393
0,366
0,390
0.438
0.432
0.423
0.424
0.424
0.424
336
0.478
0.446
0.475
0.444
0.470
0.440
0.470
0.440
720
0.478
0.472
0.470
0.469
0.480
0.473
0.462
0.461
Avg.
0.444
0.436
0.438
0.432
0.436
0,432
0.431
0.429
0.297
0.347
0.288
0.339
0,283
0,336
0,280
0.333
ETTh2
192
0.379
0.399
0.371
0.392
0.357
0,384
0.357
0.384
336
0.419
0.433
0.412
0.425
0.400
0.420
0.400
0.420
720
0.430
0.447
0.415
0.437
0.421
0,440
0.409
0,436
Avg.
0.381
0.407
0.372
0.398
0.365
0.395
0.362
0.393
0.315
0.312
0.353
0,349
0.3416
0.349
111m2 ET1ml
192
0.359
0.382
0.356
0.379
0.352
0,372
0,349
0.375
336
0.389
0.407
0.384
0.400
0.385
0.402
0.379
0.394
720
0.454
0.441
0.449
0.435
0.441
0.436
0.438
0.435
Avg.
0.379
0.397
0.375
0.392
0.371
0,390
0.368
0.388
96
0.164
0.24%
0.162
0.245
0.161
0.244
0.161
0,344
192
0.228
0.289
0.227
0.287
0.226
0.267
0.225
0.286
336
0.285
0.328
0.285
0.326
0.283
0.326
0.282
0.323
720
0.387
0.387
0.385
0.385
0.381
0,386
0.380
0.384
Avg.
0.266
0.313
0.265
0.311
0.263
0.311
0,262
0.309
96
0.165
0.209
0.159
0.203
0.159
0,202
0.158
0,201
Weather
0.209
0.248
0.210
0.249
0,244
0.206
0.244
336
0.269
0.292
0.267
0.291
0.260
0.285
0.260
0.285
720
0.348
0.344
0.347
0.343
0.343
0,338
0.343
0.342
Avg.
0.248
0.273
0.246
0.272
0.242
0.267
0.242
0,268
27

---

Table 11: Full results of ablation study with backbone predictor as ITRANS.
Models Metric
ITRANS
+5NR
+SCAM
+buth
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.383
0.405
0.388
0.407
0.376
0.397
0,373
0,401
136
0.470
0.452
0.472
0.457
0.470
0.451
0.466
0.455
Avg.
0.438
0.440
0.443
0.445
0.436
0.438
0.431
0.440
0.321
0.331
0.370
0.296
0,293
0.342
ETTh2
192
0.394
0.408
0.398
0.410
0.385
0.399
0.373
0.393
336
0.449
0.447
0.431
0.437
0.418
0.429
0.417
0.429
720
0,435
0,449
0.437
0.453
0.426
0,443
0.424
0.442
0,400
0.417
0.399
0.417
0.381
0.404
0.377
0.402
0.376
ETTm2 ETTml
192
0.393
0.399
0.385
0.395
0.376
0,393
0.369
0.387
0.403
720
0.487
0.460
0.494
0.461
0.461
0.447
0.460
0.445
96
0.189
0.275
0.204
0,289
0.184
0.279
0.179
0,264
192
0.260
0.318
0.275
0.331
0.252
0.321
0.241
0.302
336
0.326
0.359
0.324
0.358
0.323
0.357
0.305
0.343
720
0.423
0.412
0.421
0.410
0.412
0.411
0.406
0.400
Avg.
0.299
0.341
0.306
0.347
0.293
0.342
0,283
0.327
96
0.211
0.257
0.212
0.247
0.190
0.226
0.173
0.213
Wealher
0.252
336
0.303
0.319
0.301
0.319
0.295
0.307
0.278
0,296
0.367
0.357
0.353
0.344
Avg.
0.285
0.306
0.283
0.302
0.274
0.289
0.257
0,278
28

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and the Introduction include this paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
29

---

Justification:
The limitations of our work are discussed in Sec A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We ensure the correctness of loss derivation in Sec 3
The equivalence of the
gradient constraint in Eq 3 and the Lipchitz condition that SNR ensures is provided in See in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
30

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided guidelines for reproducing our experiment Sec G.3 in Ap-
pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access our code in the anonymous link We will make the code publicly accessible upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce)
for more details.
31

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
soi.:oiiiisioooooociccocoo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the experiment details in Sec G in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We have reported error bars like previous works. We believe this is acceptable because the relative improvement of our method is significantly beyond the statistical error. For experiments requiring statistical verification, provided the box plot in Fig 18 where deviation is considered.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The hardware information is listed in Sec G We also include See G.4 on
efficiency in the Appendix. Only the training costs are discussed since our method does not introduce any additional cost in inference (actual deployment).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt.ghgrgg....oooo..../////////a
Question:
Answer: [Yes]
Justification: Yes, we follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While work may have various societal implications, believe significant enough to warrant specific mention here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The code and datasets used in the paper are publicly available and properly credited.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
34

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will make the code publicly available upon acceptance of the paper and provide detailed documentation
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/codeemoodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB)
35

---

approvals (or an equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development of this work does not involve LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy htteer///eerrpppcccc///offrrrers for what should or should not be described.
36


Reviews:
[{"review_id": "MOWzrQ6Rj0", "reviewer": "Reviewer_b9qu", "summary": "This paper proposes a self-supervised method called SCAM, which improves time series forecasting by replacing overfitted raw labels with pseudo-labels from a reconstruction network, guided by an adaptive mask. It also uses spectral norm regularization to stabilize training. Experiments show consistent performance improvements.", "questions": "No", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n**1**: Introducing self-supervised reconstruction into time series label generation is a promising idea, and the proposed SCAM method shows practical application potential.\n\n**2**: The method is evaluated on various mainstream TSF models and multiple datasets, demonstrating good adaptability and performance improvement.\n\nWeaknesses:\n\n**1**: Although the method outperforms standard supervised training, it lacks in-depth comparison with existing self-supervised reconstruction or label denoising approaches, such as Denoising Label Learning and Mixup for time series.\n\n**2**: The CNN+FFN architecture used for reconstruction is relatively simple, and the paper does not explore whether more advanced designs could lead to further performance gains.\n\n**3**: Although the paper mentions that SNR helps mitigate attention collapse in Transformers, it lacks systematic empirical analysis on why SNR is effective only on linear layers and how the application points of SNR are selected.\n\n**4**: Although the method is tested on multiple datasets, it remains limited to conventional TSF tasks and has not been validated in more complex scenarios, such as industrial sensor anomaly detection.", "was_revised": true, "final_justification": "The authors address my concerns during the rebuttal period.", "rebuttal": "**Q1: In-depth comparison with existing reconstruction or label denoising approaches**\n\n> **Summary of Below:** We address the lack of direct self-supervised baselines for continuous-label time-series forecasting (TSF) by adapting MixUp and LatentMixUp, demonstrating their potential while showing that our raw method SCAM is competitive, supported by empirical results.\n\n**(1) Baseline Selection.** \nWe extensively explored SSL methods, including _Denoising Label Learning_ and _MixUp_. However, these are either:\n- Operate on vastly different data modalities (making them unsuitable for time series), or\n- Work with discrete labels, which are essentially different from continuous labels in forecasting, requiring additional adaptation.\n\nOur work is the **first** to construct **continuous pseudo-labels** for TSF by integrating an auxiliary self-supervised reconstruction task. **Thus, no off-the-shelf baselines are directly applicable to our scenario.**\nTo address this, we adapted _MixUp_ and _LatentMixUp_ [1,2] as self-supervised baselines for TSF—demonstrating SSL without a reconstruction network.\n\n[1] _mixup: Beyond Empirical Risk Minimization_ \n\n[2] _Embarrassingly Simple MixUp for Time-series_ \n\n**(2) Adaptations of SSL Baselines.** \nThe core idea of _MixUp_ is linear interpolation:\n$$\\begin{aligned} \\tilde{x} &= \\lambda x_{1} +(1-\\lambda)x_{2} \\\\\\\\ \\tilde{y} &= \\lambda y_{1} +(1-\\lambda)y_{2}, \\end{aligned}$$\nwhere a predictor $f$ is optimized on $||f(\\tilde{x})-\\tilde{y}||$. _LatentMixUp_ extends _MixUp_ to latent space, which replaces $x$ with $h$.\n\n***Adaptation 1:*** For **discrete labels**, linearly combining one-hot vectors is reasonable due to sparsity. However, randomly interpolating time series can be meaningless. We find that selecting $x_1$​ and​ $x_2$ based on **implicit periodicity** (using priors from [3]) works effectively within _MixUp_.\n\n[3] _CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns_ (NeurIPS 2024 spotlight)\n\n***Adaptation 2:*** Following standard classification models, we treat the TSF model’s hidden state as the input to the final linear projector (a common design in TSF baselines). Other alternatives were tested but underperformed.\n\nFollowing _MixUp++_ [2], we use multiple $\\lambda$ values ([0.3, 0.5, 0.7]) instead of a single one.\n\n**(3) Empirical Results and Insights.**\nExperiments follow Section 4 and Appendix G (rerun with minor adjustments; results may slightly differ from the paper).\n\n**Table 1:** Performance of _MixUp_ and _LatentMixUp_ with MLP and iTransformer.\n\n| | ETTh1 | ETTh2| ETTm1| ETTm2 |\n| ------------------ | --------- | --------- | --------- | --------- |\n| method  | MSE | MSE| MSE | MSE |\n| MLP+MixUp | 0.376| 0.285 | 0.331 | 0.177 |\n| MLP+LatentMixUp | 0.381 | 0.289 | 0.339| 0.183 |\n| **MLP****+scam**   | **0.373** | **0.280** | 0.**319** | **0.175** |\n| iTrans+MixUp| **0.371** | 0.296 | 0.315 | 0.187  |\n| iTrans+LatentMixup | 0.412 | 0.295  | 0.396 | 0.197|\n| **iTrans+scam**  | 0.373 | **0.293** | **0.315** | **0.179** |\n\n\nInsights from Table 1:\n- **SSL is promising for TSF.** To our knowledge, _MixUp_ (with intra-period mixing) has not been applied to TSF before. Our adaptation shows its potential.\n- **Non-latent SSL outperforms latent SSL.** Both _MixUp_ and our method (_SCAM_) manipulate raw time series, which outperforms _LatentMixUp_. \n\nAdapting SSL baselines, although potentially performant, requires **additional effort** without on-the-shelf implementations. We will expand on this in the paper and explore it further in future work. We hope these efforts address your concerns.\n\n**Q2: Exploration of other advanced designs of reconstruction networks**\n\n> **Summary of Below:** We focus on the SSL framework rather than reconstruction architecture design, and the empirical results show minimal performance differences between feasible alternatives.\n\n **(1) Reasons for Limited Exploration of Reconstruction Network Designs**.\n1. **Focus on SSL Paradigm**: Our work emphasizes the novel self-supervised learning (SSL) framework rather than architectural innovation or feature engineering.\n2. **Task-Specific Performance**: Predictor model performance does not directly translate to reconstruction tasks.\n3. **Minimal performance gaps**: Early experiments showed minimal performance gaps across architectures.\n\nIn summary, the reconstruction network’s role is solely to enable our SSL paradigm. Extensive architectural exploration would deviate from our core contribution and lack prior work for reference.\n\n**(2) Explored Reconstruction Network Designs**.\nWe evaluated multiple architectures, among which feasible ones are listed below, categorized by **embedding** and **decoder** components:\n\n**Table 3**: Design Choices for Reconstruction Networks\n\n| **Embedding** | **Decoder**       | **Abbr.** |\n| ------------- | ----------------- | --------- |\n| Patch-emb     | Patch-dependent   | `patch_d` |\n| Patch-emb | Patch-independent | `patch_i` |\n| Linear  | Linear-dependent  | `linear`  |\n| Conv-sparse   | Point-wise        | `sparse`  |\n| Conv-ffn| Point-wise        | `ours` |\n\n1. **Patch_d/patch_i**: Patch embeddings with dependent/independent linear decoders (Appendix Figure 11).\n2. **Linear**: Vanilla MLP (linear embedding + decoder).\n3. **Conv-sparse**: Inspired by SparseTSF [4], uses multi-stride convolutions for sparse embeddings.\n4. **Ours (Conv-FFN)**: Pyramidal convolution (Figure 9), our final choice.\n\n[4] *SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters* (ICML 2024 Oral)\n\n**Table 4**: Performance Comparison\n\n|Method | ETTh1 | ETTh2| ETTm1| ETTm2|\n| ------- | --------- | --------- | --------- | --------- |\n| | MSE  | MSE | MSE | MSE  | MSE |\n|patch_d| 0.374| **0.280** | 0.320| 0.176 |\n|patch_i| **0.373** | **0.280** | 0.322 | **0.175** | \n|linear| 0.380 | 0.287 | 0.324 | 0.177| \n|sparse| 0.375| 0.281| **0.318** | 0.176 | \n|ours| **0.373** | **0.280** | 0.319  | **0.175** | \n\nThe performance gaps between these options are not very significant, which explains why we do not focus much on the topic. \n\n**(3) Conclusion**\n\nReconstruction network design is secondary to our SSL framework. While feasible alternatives exist, their performance differences are negligible, and optimizing them falls outside our paper’s scope. We appreciate the reviewer’s feedback and hope this clarification addresses the concerns.\n\n**Q3: systematic empirical analysis of SNR**\n\n> **Summary of Below:** The ineffectiveness of SNR on self-attention is a conclusion drawn from [5]. We focus more on the empirical analysis of the detailed effective field for SNR on common components of TSF models. \n\n**(1) The existing empirical study regarding SNR**\n\nThe application of SNR or SNR-like (spectral normalization operation) to alleviate overfitting is a classical approach. However, very few studies have discussed its application specifically in Time Series Forecasting. [5] first studies this topic, and has concluded the ineffectiveness of pure SNR on attention matrices with empirical studies already.\n\nHowever, their conclusions are limited to the two constraints:\n\n1. They study channel-wise transformers particularly. The overfitting phenomenon is attributed to channel-wise attention.\n2. They verify the ineffectiveness of Spectral Normalization on self-attention only following previous works [6].\n\n**(2) The empirical study in our paper**\n\nAs pointed out in our paper (in Appendix E.3), we observe that the overfitting problem, measured by sharpness metric, does not solely exist in attention-relevant component (Encoder in Figures 16,17), but also exists in linear layers. We therefore add SNR to linear layers in TSF models for empirical verifications. As described in Appendix E.3, by adding SNR, the overfitting issue is alleviated for all three components.\n\nWe also conduct empirical analysis on where the SNR should be applied (Figure 18) supported by actual performances. These results may be model-dependent and therefore only serve as a reference.\n\nThe ablation study regarding SNR is also included in Section 4.2. We conclude that the alleviation of overfitting is especially crucial for SCAM, and using SNR alone may cause sub-optimal performance sometimes.\n\n**(3) Conclusion**\n\nWe do not intend to lay much emphasis on the root theory of SNR and extend the previous analysis in [5], and supplement their conclusion about the feasible field of the SNR method. Our empirical results show that SNR is not a silver bullet but works well with SCAM.\n\n_[5] SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention_\n\n_[6] Stabilizing Transformer Training by Preventing Attention Entropy Collapse_\n\n**Q4: Application in more complex scenarios like industrial sensor anomaly detection**\n\nOur work restricts the topic to Time Series Forecasting only, due to the great gap between the self-supervision in classification and forecasting. The fundamental essence of our method, which is distinguishing noise, however, does relate to anomaly detection. The mask value, under reasonable adjustments can be utilized as noise indicators, or augment training data for external anomaly detectors. We will seriously consider such an opportunity to extend our work to other tasks, including anomaly detection and will add such discussion in the section about future works in the camera-ready version.\n\n> *Looking forward to further discussions.* All experiment results displayed in the rebuttal are selected due to the total character constraint; we are pleased to provide comprehensive statistics in the following phase if requested. Additionally, further concerns about the content of our response or other topics are welcome in the discussion phase. We appreciate your valuable advice and look forward to your reply."}, {"review_id": "rAUz1VzOlI", "reviewer": "Reviewer_mY2H", "summary": "This paper focuses on the important issue of performance degradation caused by low-quality labels. The authors innovatively propose to re-label time series datasets by replacing original labels with pseudo-labels generated by reconstruction networks. Specifically, the authors put a prediction network and a reconstruction network into their proposed structure with three training objectives among prediction, raw label, and reconstructed label. Additionally, the authors introduce Self-Correction with Adaptive Mask (SCAM), using loss landscape sharpness to identify overfitting components and dynamically replace raw labels with pseudo-labels. Moreover, the authors present Spectral Norm Regularization (SNR) as a training normalization strategy to constrain the gradient of the prediction network and smooth the loss landscape.", "questions": "1. In equation 4, the SCAM mask strategy relies on the empirical decomposition of loss functions. Is it possible to use theories such as PAC-Bayes to clarify the correction boundary of pseudo-labels?\n2. The reconstruction network is computationally heavy. Can knowledge distillation or neural architecture search be used to lower the computational resource consumption of the reconstruction network? For example, how much performance degradation will occur if we replace dense convolution layers with one-dimensional sparse convolution layers?\n3. The binary mask M may lose transition state information. Can a Soft Mask or uncertainty weighting be designed to achieve continuous label correction?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The proposed framework in this paper makes use of the self-supervising philosophy to reduce noisy labels dependence. Also, the adaptive character of SCAM is based on solid mathematical optimization heuristics (loss landscapes and overfitting), leading to high-quality toxic label filtering. Meanwhile, the authors conduct detailed analysis and ablation studies, demonstrating that their method is effective and convincing.\n\nThe authors concatenate representations from multiple stacked Convolution layers and put the concatenated representation into an FFN layer for multi-resolution reconstruction. Such a Convolution design usually causes high computational resources and relies heavily on hyperparameter choosing. In this paper, the authors use a four-layered convolutional network with kernel sizes 1,2,4,8, and the experiments on benchmark datasets confirm the feasibility of such parameter choosing. However, the authors do not include parameter analysis of the reconstruction network in their analysis. If the model’s performance is easily affected by the reconstruction network parameters, the proposed method is less attractive in realistic applications, as it heavily relies on the developer’s experience. Meanwhile, the analysis in the supplementary materials shows that adding SCAM to current backbones costs additional computational resources and time in a large proportion with adding a small amount of parameters (around 30k). This might indicate that the reconstruction network is comparably heavy and time-consuming. It would be very meaningful to look into how necessary the design of the current reconstruction network is in future works.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Correction Boundary of Pseudo-Labels**\n\n> We acknowledge the unfeasibility of static statistical methods and provide posterior correlation proof that is also theoretically backed.\n\n**(1) Definition of the Boundary/Mask in Current Methods**\n\nAs briefly explained in Section 3.3, we observed a clear tendency toward overfitting when adding a new reconstruction loss ($\\ell_{rec}$) to the supervised loss ($\\ell_{tar}$). The addition has two key effects on SSL:\n\n1. The loss reaches a lower optimum.\n2. Overfitting occurs, preventing convergence to the lowest optimum.\n\nThis observation suggests the existence of two opposing components, which motivates the derivation of **Equation 4**. We reformulate the problem as a binary classification problem. The loss components are the objects with the loss statistics of training samples as attributes. The \"label\" refers to whether a loss component contributes to overfitting.\n\n**(2) Complexity in Label Acquisition**\n\nWhile Bayesian or PCA analysis could help decide the boundary for the binary classification. The ground-truth labels are not determined by observable attributes. The labels—quantified by the overfitting metric ($\\ell_{test} - \\ell_{train}$)—are accessible after backward-propagation on the combined loss. Thus, we cannot decide whether **a single component** causes overfitting.\n\n**(3) Empirical Analysis: Proof of the Current Boundary from a Correlation Perspective**\n\nA more practical approach is to verify classification results (mask $M$) in an end-to-end manner. Below, we provide statistical evidence:\n\nBy applying mask $M$ to $\\ell_{rec}$ and $\\ell_{tar}$, we obtain:\n\n- $\\ell_{rec} \\odot M$, $\\ell_{rec} \\odot \\overline{M}$, $\\ell_{tar} \\odot M$, $\\ell_{tar} \\odot \\overline{M}$\n    (denoted as **rec_pos, rec_neg, tar_pos, tar_neg**)\n\nUsing Pearson Correlation, we evaluate how training loss correlates with test performance. A negative score indicates that the training loss negatively impacts test performance (i.e., overfitting). We focus on correlations with two target test losses, i.e. the overall performance metric.\n\n**Table 1**: Train-Test Loss Correlation (Epoch 1, 50, 100)\n\n||tar_pos_test (1,50,100)|tar_neg_test (1, 50, 100)|\n|---|---|---|\n|**tar_pos_train**|0.000221, -0.10482, -0.101608|0.009486, -0.05257, -0.08058|\n|**tar_neg_train**|-0.02193, 0.110645, 0.033152|-0.02061, 0.060247, 0.13079|\n|**rec_pos_train**|-0.05054, -0.13014, 0.333769|0.052278, 0.014121, 0.037747|\n|**rec_neg_train**|-0.04216, 0.012024, 0.266016|0.029819, 0.088595, 0.130759|\n\nA clear trend emerges: **tar_pos_train** exhibits decreasing correlation scores over time, suggesting tar_pos_train as the contributing factor of overfitting.\n\n**Table 2: Trend of Train-Test Loss Correlation**  \n_(Trend = (end correlation - begin correlation) / total epochs)_\n\n|correlations|Trend (corr/epoch) |\n|---|---|\n|tar_pos_train / tar_pos_test|-4.58e-5|\n|tar_pos_train / tar_neg_test|-13.4e-5|\n|tar_neg_train / tar_pos_test|9.57e-5|\n|tar_neg_train / tar_neg_test|7.50e-5|\n|rec_pos_train / tar_pos_test|39.2e-5|\n|rec_pos_train / tar_neg_test|14.5e-5|\n|rec_neg_train / tar_pos_test|50.9e-5|\n|rec_neg_train / tar_neg_test|37.2e-5|\n\n**(4) Conclusion**\n\nWe adopt some **posterior statistical analysis** to support our conclusion. We acknowledge the value of determining the boundary on **a priori** and will discuss this further in the **Limitations/Future Work** section of the camera-ready version.\n\n**Q2: Computational Efficiency Optimization of the Reconstruction Network**\n\n> We provide updated efficiency results, which corrected a bug causing overestimation of costs. Further, we explore different architectures that trade performance for better efficiency, along with detailed component analysis. \n\n**(1) Additional Computation Costs Are Only Incurred During Training**\n\nThe reconstruction network acts as a **scaffold** during training, meaning the predictor operates independently at inference time without requiring the reconstruction network.  No additional costs will be incurred in actual deployment.\n\n**(2) Correction of Efficiency Analysis for the Reconstruction Network**\n\n**Table 6 in Appendix G.4** initially overestimated the additional computational cost due to a bug in our codebase, which inadvertently doubled both the cost and parameter counts.\n\n**Root Cause:**  \nThe issue stemmed from the use of `tensordict.nn.EnsembleModule`. We patched the official PyTorch implementation in `src/models/modules/ensemble.py` to fix a bug in parameter registration for `TensorDictModule`. The original implementation created duplicate parameter copies instead of using the correct ones for computation.\n\n**Table 3:** Corrected efficiency benchmark. \n\n|Benchmark|Total Memory (MB)|Raw Memory (MB)|Additional Memory (MB)|Additional Params (K)|\n|---|---|---|---|---|\n|ETT (7 channels) MLP|30.31|18.15|12.16|2.7|\n|ETT iTrans|620.8|71.9|548.9|2.7|\n|Electricity (321 channels) MLP|36.72|22.23|14.49|2.7|\n|Electricity iTrans|739.43|181.47|557.96|2.7|\n\n **(3) Optional designs for different performance-efficiency trade-offs**\n\nTo study the trade-offs in reconstruction network design, we propose an **embedding-decoder abstraction**, where:\n- **Embeddings** convert time series to vectors.\n- **Decoders** reconstruct vectors back to time series.\n\nThis framework allows us to analyze the cost of the following alternatives:\n\n1. **Patch_d / patch_i**: Patch embeddings with dependent/independent linear decoders (Appendix Figure 11).\n2. **Linear**: Vanilla MLP (linear embedding + decoder).\n3. **Conv-sparse**: Inspired by SparseTSF, uses multi-stride convolutions for sparse embeddings.\n4. **Ours (Conv-FFN)**: Pyramidal convolution (Figure 9), our final choice.\n\n **(4) Detailed Component Analysis**\n\n- The memory for different components refers to peak memory usage in backpropagation of the corresponding module.\n- The **predictor memory** reflects peak usage since its loss computation depends on the reconstruction loss.\n\n**Table 4:** Component analysis on ETT. \n\n| method  | embedding mem (MB) | decoder mem (MB) | predictor/peak mem (MB) | additional memory (MB) | Backbone memory (MB) | additional params (K) |\n|---|---|---|---|---|---|---|\n|patch_d|78.13|97.51|111.33|39.43|71.9|149.4|\n|patch_i|76.77|96.01|109.82|37.92|71.9|6.3|\n|linear|65.92|72.21|85.44|13.54|71.9|37.2|\n|sparse|2683.1|2927.61|2940.84|2868.94|71.9|86.2|\n|ours|363.06|607.57|620.8|548.9|71.9|2.7|\n\n**Table 5:** Component analysis on Electricity. \n\n| method | embedding mem (MB) | decoder mem (MB) | predictor/peak mem (MB) | additional memory (MB) | raw memory (MB) | additional params (K) |\n|---|---|---|---|---|---|---|\n|patch_d|81.38|100.32|228.47|47|181.47|149.4|\n|patch_i|80.01|98.82|227.59|46.12|181.47|6.3|\n|linear|69.6|75.89|204.08|22.61|181.47|37.2|\n|sparse|724.03| 2968.54|3097.31|2915.84|181.47|86.2|\n|ours|366.21|610.72|739.43|557.96|181.47|2.7|\n\n**Key Observations**\n- **Channel count is the primary factor** influencing memory overhead. which can be alleviated by channel sampling.\n- **Efficiency-performance trade-offs**: Some alternatives (e.g., `patch_i`, `linear`) offer marginal efficiency gains at slight performance costs (see **Response to Reviewer b9qu Q2** for performance details).\n\n**(5) Conclusions**\n- We corrected the initial memory overestimation due to a bug.\n- Multiple architectural variants were explored to optimize efficiency.\n- As concluded in **Appendix G.4**, **channel sampling [1]** remains the most effective way to control reconstruction costs—aligning with our claim that reconstruction networks do **not** bottleneck TSF model scalability.\n\n_[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting_\n\n**Q3: Soft Mask for Continuous Label Correction**\n> We verify the performance of a simple Tanh-based softmask that upderperforms. There is room for more delicate softmasks left for future studies. \n\n**(1) Types of soft mask**\nThe suggestion to explore soft mask designs is indeed insightful. In machine learning, \"soft\" masks typically refer to differentiable gating mechanisms, such as:\n- **Softmax** (for multi-category routing)\n- **Tanh** (for binary gating)\nIn our method, softening the mask involves reformulating the original binary mask definitions:\n1. Original hard mask:  \n    $M=\\mathbb{I}[(\\tilde{y} - \\hat{y})(\\tilde{y} -y) >0]$\n\n    $M_{<}=\\mathbb{I}[|\\tilde{y}-\\hat{y}| - |\\tilde{y} - y|<0]$\n    where $\\mathbb{I}[\\cdot]$ is the indicator function.\n2. Proposed soft mask (using Tanh):  \n    $M=\\text{Tanh}[(\\tilde{y} - \\hat{y})(\\tilde{y} -y)]$\n\n    $M_{<} = \\text{Tanh} [|\\tilde{y}- y| - |\\tilde{y} - \\hat{y}|]$\nThis remaps mask values from $\\{0, 1\\}$ to $(0, 1)$, enabling continuous gradient flow.\n\n**(2) Experimental Results**\n\nWe evaluated soft vs. hard masks across multiple datasets and architectures:\n\n**Table 6:** Performance comparisons of soft and hard masks. \n\n|Method| ETTh1| ETTm1|Weather|\n|---|---|---|---|\n|MLP+scam+Soft |0.376|0.344|0.207|\n|MLP+scam+Hard|0.373|0.319|0.176|\n|iTrans+scam+Soft|0.390|0.337|0.200|\n|iTrans+scam+Hard|0.373|0.315|0.173|\n\n**Key Findings**\n1. **Hard masks consistently outperform soft masks** across all benchmarks.\n    - We hypothesize this is because hard masks enforce **focused optimization** on effective loss components, whereas soft masks dilute gradients by assigning partial weights to suboptimal terms.\n2. **Tanh as a baseline soft mask**:\n    - While our experiments used Tanh (a standard choice for binary gating), other soft masking functions (e.g., Sigmoid, Learned thresholds) could be explored.\n3. **Extension to multi-category masks**:\n    - For scenarios with **multiple reconstructions**, the mask could be generalized to a multi-category version (e.g., using Softmax to route losses from different reconstructions).\n\nWe will add these discussions to the Limitations or Future Work section in the camera-ready version. We appreciate your suggestions very much and hope our responses can address some of your concerns."}, {"review_id": "66uxJvsuyK", "reviewer": "Reviewer_wXrM", "summary": "This study devised a new data augmentation technique, using a custom adaptive mask and a CNN network, for time series data which create pseudo-labels over the time-series, smoothing the loss landscape and enhancing the generalization of the predictor model.The technique created, the Self-Correction with Adaptive Mask (SCAM) and the use of Spectral Norm Regularization (SNR) were tested on real-world datasets by additioning their procedures to known techniques such as MLP, CycleNet, PatchTST and ITransformer. Of note, their procedure discards overfitted components, and is applicable when one wants to work on the prediction of each element in a time-series, while others work on a set of elements.", "questions": "1. How many replicates have been performed for the different experiments?\n2. Do the method(s) apply to both short-term and long-term forecasting?\n3. How does the N (number of candidates) and J (steps) determined - Algorithm 1?\n4. Are there any cases when the reconstructed networks diverge too much from the true labels and cause more error than leaving the original data? In figure 8, it seems like for low noise, it actually makes the prediction less accurate?", "limitations": "No societal impact was discussed in the paper.\n\nOne possible avenue would be to discuss the implication for wather forecasting, traffic or electricity distribution since the authors used datasets in those specific domains. For example, better forecasting in agriculture?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths\n- Innovative technique to mask some part of the time-series so the deep learning network can leverage the “best points” when training, facilitating the training process.\n- Results seems to indicate some improvement over the existing SOTA TSF methods.\n- Good definitions for the different loss functions and the objectives are well defined.\n\n*Weaknesses\n\n- The authors introduced two questions in the introduction which are never answered in the study (line 26 and 27).\n- New concepts, important for the method (e.g. Conv-concat Layer - Figure 2?), are only discussed in the Annexe, but are central to the reconstruction networks.\n- Related-work could have been longer with more contextual information on concurrent methods and the tested deep learning network (e.g. ITransformer, CycleNet). The information is available in the paper, e.g. lines 276-280, but it's out of place and limited. For example, annexe is far more informative and section D.2 should be put in the main section. \n- Figures and tables are too small to be informative in the main paper. No formal statistics for the results and no information on the number of replicates. Furthermore, the difference between with/without addition of SCAM on the results is small.\n- Source code on the git could have been simplified so that only the relevant code (or some sample notebook) be available. \n- Experimental procedure for the results provided (Table 1) are unclear and sould be provided in the main text.", "was_revised": true, "final_justification": "Dear editors, \n\nThe authors answered all my questions. The answers to the other reviewers also greatly improved the comprehension of the paper and the results. \n\nLooking forward to a revised version of the paper, including the better description of Algo. 1. to be included in the paper. Still, I think some of the text should be revised in  the final version to add more context to this work, and rely less on the Annexe section.\n\nI adjusted the status of the paper from 'Borderline reject' to 'Accepted' following the authors answers.", "rebuttal": "**About weaknesses**\n\n**w1 two questions in the introduction:** We answer the questions right next to them (line 29, \"We posit that both answers are positive...\"). Admittedly, the answers are not directly given with concise reasons. The whole idea of our paper is to construct pseudo labels natively from raw datasets that increase the quality of labels. We have backed this claim throughout the paper.\n\n**w2 introduction of the reconstruction network:** We have a detailed introduction of our implementation in Appendix D. The idea to adaptively provide pseudo labels during the training of a TSF model is novel and therefore leaves room for more extensive discussion on the designs of the reconstruction network. We simply provide one choice and do not claim its optimality or SOTA performance on this novel task.\n\n**w3 about related work:** We will make adjustments accordingly, and do our best to satisfy NeurIPS's formatting requirements.\n\n**w4 presentation of figures and tables/main experiments results:**\n\nRegarding the figures and tables, we are happy to provide a more reader-friendly version after releasing our code repository (Sorry that we cannot provide them right now due to NeurIPS's rebuttal policy).\n\nFor statistical results of the main experiments, we conduct 1 replicate only for all experiments (under a fixed random seed). The main experiments (Table 7, Table 8 in the paper) contain 352 individual experiments, each of which approximately costs 0.6 to 8 GPU hours, adding up to around 1000 GPU hours.\n\nThe statistical significance is guaranteed by extensive experiments already. Our method shows improved performance against 4 different baselines on 11 datasets, 8 different forecasting horizons.\n\nRegarding reproductivity, we have provided detailed guides in Appendix G. Additionally, we ensure our code base can produce exactly the same results when fixing the random seed (we use 1234 as the seed in all experiments).\n\n**w5 about** **source code** **and simple demos:** We are happy to provide a more illustrative notebook and detailed documents upon acceptance (we cannot revise the code base due to NeurIPS policy). For the current code repository, we provide the full framework, which enables free combination of different model components by simple revisions of the config files. Additionally, the pytorch lightning framework that we have adopted works cleanly with logging tools like wandb and aim, which enable easy management of experiment results. The complete training and evaluation framework is built on our own which we believe will benefit the community.\n\n**w6 experiment procedure in the main text:** We are pleased to consider the arrangement of the current main text content. Please understand that NeurIPS has formatting requirements that only allow for a very concise main text.\n\n**Q1 number of replicates for different experiments:**\nWe use 1 replicate, which is explained in w4.\n\n**Q2 application to both short-term and long-term forecasting:**\nYes. Table 7 is the experiment results on short-term forecasting on 4 PeMS datasets, and Table 8 is the experiment results on long-term forecasting on 7 datasets (which is also reported in the main text). \n\n**Q3 Regarding Algorithm 1 in the Initial Case**\n\n **(1) Background of Algorithm 1**\n\nIn Section 3.1 (_Initial Case_), we implement a grid-search-like algorithm that constructs multiple candidate datasets to train a predictor from scratch. \nLet $A = \\{a_i\\} $ and $B = \\{b_i\\}$ be two datasets with the same number of samples. \nThe distance between them is defined as $d(A, B) = \\sum |a_i - b_i|$.\n\nA candidate dataset can be parameterized based on a raw dataset $A$ as $A' = g(A, \\phi) = \\{g(a_i, \\phi)\\}$. This allows us to map datasets to an interval $(d_{\\text{inf}}, d_{\\text{sup}})$, where:\n- $d_{\\text{inf}}$ is the lower bound for $d(A, A')$,\n- $d_{\\text{sup}}$ is the upper bound.\n\nFor any $d_i \\in (d_{\\text{inf}}, d_{\\text{sup}})$, the corresponding $A'_i = g(A, \\phi_i)$ is not uniquely determined—there may be infinitely many $\\phi_i$ that map to the same $d_i$. To resolve this, we introduce an optimization process as an external constraint.\n\nIn each step (Line 11 of Algorithm 1: $\\phi_{i+1} \\leftarrow \\phi_i - \\eta \\nabla_{\\phi}$), we optimize $d(A, A_i')$ (i.e., $\\ell_{\\text{rec}}$ in Line 11),\nThis procedure effectively generates a new dataset $ A_{i+1}' = g(A, \\phi_{i+1}) $ that maps to $d_{i+1} \\in (d_{\\text{inf}}, d_{\\text{sup}})$. \nWe can then sample as many points in $(d_{\\text{inf}}, d_{\\text{sup}})$ as desired, with each point corresponding to a unique $\\phi_i$ and dataset $A'_i$.\n\n\nWe refer to this as a \"grid search\" because we explore the loss interval via gradient descent, where each point in the interval maps to a distinct dataset. Predictors are then independently trained on these datasets to identify the best-performing one.\n\n**(2) Answers to the Question**\n\n- **$N$ (Number of Optimization Steps):** Combined with the learning rate $\\eta$, this hyperparameter determines the lower bound $d_{\\text{inf}}$ and the number of samples explored within $(d_{\\text{inf}}, d_{\\text{sup}})$.\n\n- **$J$ (Termination Condition):** This ensures algorithmic completeness by guaranteeing predictor convergence. We set a gradient threshold $\\alpha$ and a maximum optimization step for predictors $f$ as termination criteria.\n\n**(3) Conclusion**\n\nTo summarize, Algorithm 1 serves as an illustrative example of the concept, demonstrating that \"better\" datasets can exist compared to the raw dataset. Here, \"better\" means a predictor trained on $A'$ achieves superior performance (on $A$'s test set) than one trained on $A$.\n\nWe are happy to include these clarifications in the camera-ready version. If further questions arise, we welcome discussion in the following phase.\n\n**Q4: The Fidelity of Reconstructions**\n\n**(1) Explanation of Figure 8**\n\nFigure 8 is provided solely for illustrative purposes, as it depicts results on synthesized data during the early stages of training—when neither the predictor nor the reconstruction network has fully converged. For real-world scenarios, we present actual reconstruction fidelity in Figure 15, which reflects performance during later training phases.\n\n**(2) Empirical Analysis**\n\nTo quantitatively assess reconstruction fidelity, we report the following statistics:\n\n**Table 1:** Reconstruction loss ($\\ell_{rec}$) after full training (MLP + SCAM). _Horizon_ denotes the forecasting window length (i.e., the time series segment to reconstruct).\n\n||ETTh1|ETTh2|ETTm1|ETTm2|Weather|Electricity|\n|---|---|---|---|---|---|---|\n|Horizon = 96|2.250E-04|2.435E-04|2.023E-04|2.254E-04|2.012E-04|3.158E-04|\n|Horizon = 336|1.996E-04|2.134E-04|2.055E-04|2.162E-04|1.866E-04|3.281E-04|\n\nThe consistently low $\\ell_{rec}$ values at convergence confirm that reconstruction fidelity is well-preserved in later training stages. Intuitively:\n\n- **Early stages:** Our method generates easy-to-learn pseudo-labels (which may initially diverge from raw data).\n    \n- **Late stages:** Continuous optimization ensures reconstructions converge to recover the original series, enabling predictors to better capture unprocessed temporal patterns.\n    \n\n**(3) Architectural Choices to Ensure Fidelity**\n\nOur reconstruction network architecture (see Response to Q2, Reviewer b9qu, for alternatives) provides the strongest theoretical guarantees for fidelity. This is empirically validated by:\n\n1. **Low $\\ell_{rec}$ values**, which correlate with a reduced mask ratio for $M_{<}$ (i.e., greater emphasis on $|\\tilde{y} - y|$ optimization).\n2. **Comparative metrics** (Table 2), where our method achieves superior fidelity (lowest $\\ell_{rec}$ and $M_{<}$ ratios):\n\n**Table 2:** Reconstruction metrics. _mask_y_: $M$ ratio; _mask_l_: $M_{<}$ ratio; _loss_rec_: $\\ell_{rec}$.\n\n|Method|mask_y ratio|mask_l ratio|loss_rec|\n|---|---|---|---|\n|patch_d|0.3558|0.0807|0.0087|\n|patch_i|0.4378|0.0780|0.0125|\n|linear|0.4198|0.3198|0.0318|\n|sparse|0.4708|0.0170|0.0008|\n|**ours**|**0.3837**|**0.0141**|**0.0002**|"}, {"review_id": "s4uWnK9NzI", "reviewer": "Reviewer_GL6M", "summary": "This paper focuses on data quality for enhancing TSF by proposing a self-supervised paradigm. The core insight is that raw time-series data, inherently as labels, often contain noise or anomalies, which degrade model performance. To mitigate this, the authors introduce ​​Self-Correction with Adaptive Mask (SCAM)​​, which generates pseudo-labels via an auxiliary reconstruction network. SCAM identifies and replaces overfitted label components with reconstructed pseudo-labels, while ​​Spectral Norm Regularization (SNR)​​ suppresses overfitting in this procedure. Experiments across representative datasets and diverse backbones (e.g., MLP, Transformer-based models) show consistent improvements in generalization. The method decouples data quality assessment from prediction, offering a novel angle for TSF robustness.", "questions": "Q1. How are SNR-normalized parameters initialized? Are original weights retained and spectrally normalized during training, or is SNR applied as a reparameterization?\n\nQ2. Could you explicitly define the dimensions of \"quality\" used in this work? For example: 1) Is \"label quality\" solely measured by noise magnitude (e.g., SNR), or does it include temporal consistency/plausibility? 2) For reconstructed pseudo-labels (Sec 3.2), is quality assessed via ℓ_rec (reconstruction error) or downstream ℓ_target (forecasting accuracy)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nS1. Telling a story with intuitive visualizations and the theory is clearly derived, e.g., Eq 4 for loss decomposition and Fig 6 for adaptive masked parts.\n\nS2. Addressing understudied label quality issues in TSF is of interest. The self-supervised paradigm reduces dependency on curated data, benefiting real-world applications --- important for practical scenarios where data are noisy without known ground truth.\n\nS3. Novel formulation of label noise via reconstruction-based candidate datasets. SCAM's adaptive masking and SNR integration are inventive with insights into time series data and current strong models.\n\nS4. Experimentally validated across diverse settings; Appendix details aid replication; particularly, the reproductivity report in G.3 and efficiency study in G.4 are helpful.\n\nWeaknesses:\n\nW1. SNR implementation details (Sec 3.4) can be clearer; a brief pseudocode would help.\n\nW2. The \"quality\" discussed in the paper should be made explicit.\n\nW3. Some presentation issues, like Fig 14 on Page 18 and widow in line 310 (should be minor).", "was_revised": true, "final_justification": "The author's rebuttal addressed all of my concerns. I will maintain my current positive score and support its acceptance.", "rebuttal": "**Q1: implementation details of SNR**\n\n**(1) Power iteration to compute spectral norm**\nSNR is implemented in a reparameterization manner, meaning that the initialization of SNR-applied parameters follows the conventional Xavier initialization.\n\nThe SNR ensures a parameter matrix $W$ has a spectral norm $\\Vert W\\Vert_{2}$ exactly equals to 1. Essentially what SNR substitutes $W$ with a normalized matrix $\\frac{W}{||W||_{2}}$.\n\nTo normalize matrix $W$, we first calculate its the spectral norm $\\Vert W\\Vert_{2}$, which is  defined as $$\\Vert W\\Vert_{2} =\\max \\frac{\\Vert Wx\\Vert_{2}}{\\Vert x\\Vert_{2}}=\\max \\frac{x^{\\top}W^{\\top}Wx}{x^\\top x}.$$This corresponds to the largest singular value of $W^{\\top}W$.\n\nWe compute this value numerically using **power iteration**, applying the following update rule iteratively:\n\n$$u_{i+1}=\\frac{W^{\\top}Wu_i}{||W^{\\top}Wu_i||}$$\n\nAfter several iterations, we obtain $||W||^2_2 \\approx u_t^{\\top}W^{\\top}Wu_t$. The initial value of $u$ can be set as all-ones vector.\n\n**(2) Brief Proof of Convergence**\n\nLet the singular values of $W^{\\top}W$ be $\\lambda_1 > \\lambda_2 > \\dots >\\lambda_k$, with corresponding singular vectors $\\eta_1, \\eta_2, \\dots, \\eta_k$ forming a basis of a k-dimension vector space. \nExpressing $u_i$ in this basis, we have:\n$$u_i = c_1 \\eta_1 + c_2\\eta_2 + ... + c_k\\eta_k$$The iteration becomes:\n$$u_{i+1} = c_1\\lambda_1\\eta_1 + c_2\\lambda_2\\eta_2 + ... + c_k\\lambda_k\\eta_k$$\nLet $l_{i}$ be $||W^{\\top}Wu_i||$. After $t$ iterations, we have: $$u_t / \\lambda_1^t = (\\prod c_1 l_i) \\eta_1 + (\\prod c_2 l_i) (\\frac{\\lambda_2}{\\lambda_1})^t + ... + (\\prod c_k l_i) (\\frac{\\lambda_k}{\\lambda_1})^t.$$As $t \\to \\infty$, the terms $(\\frac{\\lambda i}{\\lambda_1})^t\\to 0$, leaving $$u_t \\approx \\lambda_1^t (\\prod c_1l_i)\\eta_1.$$Due to  normalization in each iteration, $u_t$ converges to a unit vector aligned with $\\eta_1$, yielding:   $$||W||_2^2=\\eta_1^{\\top}W^{\\top}W\\eta_1 \\approx u_t^{\\top}W^{\\top}Wu_t.$$\n\n**(3) Implementation details**\nIn practice, we stop gradients (via `tensor.detach()`) for both $W$ and $u_i$​ during power iteration. This means the original parameter $W$ in a linear layer is scaled only by a constant. The full codes regarding SNR are available in our code base (in `src/models/modules/snrlinear.py`)\n\nWe are happy to provide pseudocode and supplementary illustrations in the camera-ready version to further clarify the SNR method.\n\n**Q2: Definition of Label Quality**\n\n**(1) Current Metrics for Label Quality**\n\nLabel quality is evaluated using end-to-end test metrics. Given a raw dataset $A$, we split it into training set $A_{train}$ and test set $A_{test}$. Our reconstruction method generates a corresponding dataset $B_{train}$ that is exclusively used for training. We then train two separate predictors: $f_A$ on the original dataset $A$ and $f_B$ on the reconstructed dataset $B$. When comparing their performance on $A_{test}$, if $f_{B}$ demonstrates lower prediction error (i.e., better performance), we conclude that the labels in $B_{train}$ are of higher quality. Thus, the quality of our reconstructed pseudo-labels is ultimately assessed through the downstream target metric $\\ell_{target}$.\n\n**(2) More Information-Rich Metrics**\n\nWe propose that sharpness ($\\lambda_{max}$) [1], as shown in Figure 7, serves as an effective metric for evaluating a model's robustness to data noise. This metric, combined with the SNR score of parameter matrices, focuses more on model characteristics rather than direct data properties. We fully agree that temporal consistency and plausibility should be considered when assessing data noise. However, current techniques still lack methods that can directly leverage prior knowledge about noise patterns in the data. We greatly appreciate your suggestion and look forward to expanding the self-supervised learning framework toward more model-free and data-centric approaches.\n\n_[1] Samformer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention._"}]

Author rebuttal:

Meta-review: This paper proposes a self-supervised method called SCAM, which improves time series forecasting by replacing overfitted raw labels with pseudo-labels from a reconstruction network, guided by an adaptive mask, and employs spectral norm regularization (SNR) to stabilize training. The idea is considered novel and well-motivated (mY2H, GL6M), practically effective across multiple datasets and backbone models (b9qu, wXrM), and theoretically grounded with clear visualizations (GL6M, mY2H). Although some weaknesses remain, including limited comparisons with existing self-supervised/denoising methods (b9qu), computational overhead of the reconstruction network (mY2H, wXrM), the authors have addressed the issues in their responses.
Decision: Accept (spotlight)