Title: Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time Series Forecasting

Self-supervised Learning

pseudo labels

adaptive mask
Abstract: Time Series Forecasting (TSF) is a crucial task in various domains, yet existing TSF models rely heavily on high-quality data and insufficiently exploit all available data. This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets. During the optimization of a simple reconstruction network, intermediates are used as pseudo labels in a self-supervised paradigm, improving generalization for any predictor. We introduce the Self-Correction with Adaptive Mask (SCAM), which discards overfitted components and selectively replaces them with pseudo labels generated from reconstructions. Additionally, we incorporate Spectral Norm Regularization (SNR) to further suppress overfitting from a loss landscape perspective. Our experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models. This work offers a new perspective on constructing datasets and enhancing the generalization of TSF models through self-supervised learning. The code is available at https://github.com/SuDIS-ZJU/SCAM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on data quality for enhancing TSF by proposing a self-supervised paradigm. The core insight is that raw time-series data, inherently as labels, often contain noise or anomalies, which degrade model performance. To mitigate this, the authors introduce ​​Self-Correction with Adaptive Mask (SCAM)​​, which generates pseudo-labels via an auxiliary reconstruction network. SCAM identifies and replaces overfitted label components with reconstructed pseudo-labels, while ​​Spectral Norm Regularization (SNR)​​ suppresses overfitting in this procedure. Experiments across representative datasets and diverse backbones (e.g., MLP, Transformer-based models) show consistent improvements in generalization. The method decouples data quality assessment from prediction, offering a novel angle for TSF robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. Telling a story with intuitive visualizations and the theory is clearly derived, e.g., Eq 4 for loss decomposition and Fig 6 for adaptive masked parts.\n\nS2. Addressing understudied label quality issues in TSF is of interest. The self-supervised paradigm reduces dependency on curated data, benefiting real-world applications --- important for practical scenarios where data are noisy without known ground truth.\n\nS3. Novel formulation of label noise via reconstruction-based candidate datasets. SCAM's adaptive masking and SNR integration are inventive with insights into time series data and current strong models.\n\nS4. Experimentally validated across diverse settings; Appendix details aid replication; particularly, the reproductivity report in G.3 and efficiency study in G.4 are helpful.\n\nWeaknesses:\n\nW1. SNR implementation details (Sec 3.4) can be clearer; a brief pseudocode would help.\n\nW2. The \"quality\" discussed in the paper should be made explicit.\n\nW3. Some presentation issues, like Fig 14 on Page 18 and widow in line 310 (should be minor).\n\nQUESTIONS:\nQ1. How are SNR-normalized parameters initialized? Are original weights retained and spectrally normalized during training, or is SNR applied as a reparameterization?\n\nQ2. Could you explicitly define the dimensions of \"quality\" used in this work? For example: 1) Is \"label quality\" solely measured by noise magnitude (e.g., SNR), or does it include temporal consistency/plausibility? 2) For reconstructed pseudo-labels (Sec 3.2), is quality assessed via ℓ_rec (reconstruction error) or downstream ℓ_target (forecasting accuracy)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study devised a new data augmentation technique, using a custom adaptive mask and a CNN network, for time series data which create pseudo-labels over the time-series, smoothing the loss landscape and enhancing the generalization of the predictor model.The technique created, the Self-Correction with Adaptive Mask (SCAM) and the use of Spectral Norm Regularization (SNR) were tested on real-world datasets by additioning their procedures to known techniques such as MLP, CycleNet, PatchTST and ITransformer. Of note, their procedure discards overfitted components, and is applicable when one wants to work on the prediction of each element in a time-series, while others work on a set of elements.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths\n- Innovative technique to mask some part of the time-series so the deep learning network can leverage the “best points” when training, facilitating the training process.\n- Results seems to indicate some improvement over the existing SOTA TSF methods.\n- Good definitions for the different loss functions and the objectives are well defined.\n\n*Weaknesses\n\n- The authors introduced two questions in the introduction which are never answered in the study (line 26 and 27).\n- New concepts, important for the method (e.g. Conv-concat Layer - Figure 2?), are only discussed in the Annexe, but are central to the reconstruction networks.\n- Related-work could have been longer with more contextual information on concurrent methods and the tested deep learning network (e.g. ITransformer, CycleNet). The information is available in the paper, e.g. lines 276-280, but it's out of place and limited. For example, annexe is far more informative and section D.2 should be put in the main section. \n- Figures and tables are too small to be informative in the main paper. No formal statistics for the results and no information on the number of replicates. Furthermore, the difference between with/without addition of SCAM on the results is small.\n- Source code on the git could have been simplified so that only the relevant code (or some sample notebook) be available. \n- Experimental procedure for the results provided (Table 1) are unclear and sould be provided in the main text.\n\nQUESTIONS:\n1. How many replicates have been performed for the different experiments?\n2. Do the method(s) apply to both short-term and long-term forecasting?\n3. How does the N (number of candidates) and J (steps) determined - Algorithm 1?\n4. Are there any cases when the reconstructed networks diverge too much from the true labels and cause more error than leaving the original data? In figure 8, it seems like for low noise, it actually makes the prediction less accurate?\n\nLIMITATIONS:\nNo societal impact was discussed in the paper.\n\nOne possible avenue would be to discuss the implication for wather forecasting, traffic or electricity distribution since the authors used datasets in those specific domains. For example, better forecasting in agriculture?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the important issue of performance degradation caused by low-quality labels. The authors innovatively propose to re-label time series datasets by replacing original labels with pseudo-labels generated by reconstruction networks. Specifically, the authors put a prediction network and a reconstruction network into their proposed structure with three training objectives among prediction, raw label, and reconstructed label. Additionally, the authors introduce Self-Correction with Adaptive Mask (SCAM), using loss landscape sharpness to identify overfitting components and dynamically replace raw labels with pseudo-labels. Moreover, the authors present Spectral Norm Regularization (SNR) as a training normalization strategy to constrain the gradient of the prediction network and smooth the loss landscape.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed framework in this paper makes use of the self-supervising philosophy to reduce noisy labels dependence. Also, the adaptive character of SCAM is based on solid mathematical optimization heuristics (loss landscapes and overfitting), leading to high-quality toxic label filtering. Meanwhile, the authors conduct detailed analysis and ablation studies, demonstrating that their method is effective and convincing.\n\nThe authors concatenate representations from multiple stacked Convolution layers and put the concatenated representation into an FFN layer for multi-resolution reconstruction. Such a Convolution design usually causes high computational resources and relies heavily on hyperparameter choosing. In this paper, the authors use a four-layered convolutional network with kernel sizes 1,2,4,8, and the experiments on benchmark datasets confirm the feasibility of such parameter choosing. However, the authors do not include parameter analysis of the reconstruction network in their analysis. If the model’s performance is easily affected by the reconstruction network parameters, the proposed method is less attractive in realistic applications, as it heavily relies on the developer’s experience. Meanwhile, the analysis in the supplementary materials shows that adding SCAM to current backbones costs additional computational resources and time in a large proportion with adding a small amount of parameters (around 30k). This might indicate that the reconstruction network is comparably heavy and time-consuming. It would be very meaningful to look into how necessary the design of the current reconstruction network is in future works.\n\nQUESTIONS:\n1. In equation 4, the SCAM mask strategy relies on the empirical decomposition of loss functions. Is it possible to use theories such as PAC-Bayes to clarify the correction boundary of pseudo-labels?\n2. The reconstruction network is computationally heavy. Can knowledge distillation or neural architecture search be used to lower the computational resource consumption of the reconstruction network? For example, how much performance degradation will occur if we replace dense convolution layers with one-dimensional sparse convolution layers?\n3. The binary mask M may lose transition state information. Can a Soft Mask or uncertainty weighting be designed to achieve continuous label correction?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a self-supervised method called SCAM, which improves time series forecasting by replacing overfitted raw labels with pseudo-labels from a reconstruction network, guided by an adaptive mask. It also uses spectral norm regularization to stabilize training. Experiments show consistent performance improvements.

STRENGTHS AND WEAKNESSES:
Strengths:

**1**: Introducing self-supervised reconstruction into time series label generation is a promising idea, and the proposed SCAM method shows practical application potential.

**2**: The method is evaluated on various mainstream TSF models and multiple datasets, demonstrating good adaptability and performance improvement.

Weaknesses:

**1**: Although the method outperforms standard supervised training, it lacks in-depth comparison with existing self-supervised reconstruction or label denoising approaches, such as Denoising Label Learning and Mixup for time series.

**2**: The CNN+FFN architecture used for reconstruction is relatively simple, and the paper does not explore whether more advanced designs could lead to further performance gains.

**3**: Although the paper mentions that SNR helps mitigate attention collapse in Transformers, it lacks systematic empirical analysis on why SNR is effective only on linear layers and how the application points of SNR are selected.

**4**: Although the method is tested on multiple datasets, it remains limited to conventional TSF tasks and has not been validated in more complex scenarios, such as industrial sensor anomaly detection.

QUESTIONS:
No

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03