Selective Learning for Deep Time Series Forecasting
Authors: Yisong Fu
Zezhi Shao
Chengqing Yu
Yujie Li
Zhulin An
Cheems Wang
Yongjun Xu
Fei Wang
Venue: NeurIPS 2025 poster ()
Keywords: selective learning; time series forecasting; deep learning
Primary area: deep_learning

Abstract: Benefiting from high capacity for capturing complex temporal patterns, deep learning (DL) has significantly advanced time series forecasting (TSF). However, deep models tend to suffer from severe overfitting due to the inherent vulnerability of time series to noise and anomalies. The prevailing DL paradigm uniformly optimizes all timesteps through the MSE loss and learns those uncertain and anomalous timesteps without difference, ultimately resulting in overfitting. To address this, we propose a novel selective learning strategy for deep TSF. Specifically, selective learning screens a subset of the whole timesteps to calculate the MSE loss in optimization, guiding the model to focus on generalizable timesteps while disregarding non-generalizable ones. Our framework introduces a dual-mask mechanism to target timesteps: (1) an uncertainty mask leveraging residual entropy to filter uncertain timesteps, and (2) an anomaly mask employing residual lower bound estimation to exclude anomalous timesteps. Extensive experiments across eight real-world datasets demonstrate that selective learning can significantly improve the predictive performance for typical state-of-the-art deep models, including 37.4% MSE reduction for Informer, 8.4% for TimesNet, and 6.5% for iTransformer.

Selective Learning for Deep Time Series Forecasting
Yisong Ful Zezhi Shao', Chengqing Yu Yu!.?, 1.2 Yujie Li Zhulin An An!.?,
Qi Wang', Yongjun Xu!.2; Fei Wang 1.2+
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences ?University of Chinese Academy of Sciences
Department of Automation, Tsinghua University (fuyisong24s, shaozezhi, yuchengging22b, liyujie23s, anzhulin] Sicpullinnnnitiieeeee ac. cheemswang@mail. tsinghua edu. en, {xyj, wyyj,wangeeieeict.coom ac.cn
Abstract
Benefiting from high capacity for capturing complex temporal patterns, deep learn- ing (DL) has significantly advanced time series forecasting (TSF). However, deep models tend to suffer from severe overfitting due to the inherent vulnerability of time series noise and anomalies. The prevailing DL paradigm uniformly opti- mizes all timesteps through the MSE loss and learns those uncertain and anomalous timesteps without difference, ultimately resulting in overfitting. To address this, we propose a novel selective learning strategy for deep TSF. Specifically, selective learning screens subset of the whole timesteps to calculate the MSE loss in opti- mization, guiding the model to focus generalizable timesteps while disregarding non-generalizable ones. Our framework introduces a dual-mask mechanism target timesteps: (1) uncertainty mask leveraging residual entropy to filter uncertain timesteps, and (2) an anomaly mask employing residual lower bound estimation to exclude anomalous timesteps. Extensive experiments across eight real-world datasets demonstrate that selective learning can significantly improve the predictive performance for typical state-of-the art deep models, including 37.4% MSE reduction for Informer, 8.4% for TimesNet, and 6.5% for iTransformer.
Code: httpahrwrreegaaaaccccooeeeeaaaaa https: wwwpehuw....oeeeteeeeeeeeeeeeeee http://tww...o/eeeeaaooo/////aaa
Introduction
Time series forecasting (TSF) plays a crucial role in many real-world applications, such as traffic flow prediction [26, 45, 43], weather forecasting [38, 65, 80, 12], and energy consumption planning [33, 57]. The rapid advancement of deep learning (DL) has spurred breakthroughs in TSF, with numerous deep models pushing the boundaries of predictive performance and becoming pivotal in the field [81, 64, 82, 76, 35, 29].
Despite the strong capacity for capturing complex temporal patterns, deep TSF models are prone suffer from severe overfitting issues under certain scenarios due to the characteristics of real-world time series data [42, 58, 7]. Unlike other data modalities such natural language and images, time series is inherently susceptible to noise and anomalies introduced by random exogenous factors [28, 56, 31]. For example, industrial sensors are easily affected by noise from mechanical vibrations and electromagnetic disturbances, and stock prices exhibit non-stationary fluctuations given the policy interventions. These interference factors are challenging model and typically change over time, exhibiting uncertain and anomalous patterns at specific range of timesteps. However, the current
"Corresooding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Non-generalizable
w/ SL
0.52
w/ SL
Uncertain
Anomalous
0.50
Overficting
Selective learning
Mask out
Text
0.48
0.46
8.1%
Well-ficcing
0.44
Uncertain
Anomalous
10
Epoch
Prediction
Ground trudh
Masked values
Gradient direction
Figure 1: Left: When optimizing the model through MSE loss, our proposed selective learning calculates MSE only on a subset of timesteps, while masking out uncertain and anomalous ones that non-generaliiable. Right: Test MSE curves of iTransformer during training on the ETThl dataset (prediction length = 336). The model exhibits severe overfitting, but this is effectively mitigated through selective learning, yielding 8.1% reduction in test MSE with stable convergence.
DL paradigm treats each time step equally when training the models with regression loss functions (e.g., MSE/MAE loss). This causes overfitting issue when forcing models to learn uncertain and anomalous timesteps unavailable to generalize, deteriorating models' performance. For example, iTransformer [29] encounters significant overfitting when trained on the ETThl dataset. As shown in Figure (right), its test MSE during training gradually increases after the third epoch.
To address these challenges, we propose selective learning, a novel learning strategy for deep TSF. As illustrated in Figure (left), the main idea of selective learning to involve only generalizable timesteps, a subset of the time series, in optimization and discard identified uncertain or anomalous ones. In implementation, we propose a dual-mask mechanism to filter out non-gene non-generalizable timesteps dynamically. (1) For uncertain timesteps, we introduce the entropy of the prediction residual distribution as the uncertainty measure. With the sliding window sampling in time series, we obtain multiple samples of predicted timestep under different historical windows, thereby quantifying the entropy of the residual to serve as an indicator to filter out high-entropy ones. (2) For anomalous timesteps, train estimation model to obtain the residual lower bound of each timestep. By masking timesteps where current residuals are closest to the lower bound, removes non-generalizable anomalies dynamically while keeping to-be-learned timesteps.
Extensive experiments across eight real-world datasets show that selective learning achieves consistent performance gains on six well-acknowledged deep models. It proves particularly effective for models that are susceptible to overfitting, where achieves a 37.4% reduction MSE and for Informer [81]. and 15.6% in MSE for Crossformer [79]. Notably, selective learning maintains its benefits even for state-of-ther arr baselines, such as TimesNet [63] (8.4%) MSE reduction) and iTransformer [29] (6.5% % MSE reduction). Furthermore, we conducted comparative analyses with alternative training objectives in $5.4. The consistent leading performance of selective learning further demonstrates the advantage of optimizing generalizable subsets over global sequence optimization.
In summary, our contribution is three-fold:
We propose selective learning, novel learning strategy for deep TSF to identify and expel reeliiaerll timesteps in optimization. is the first trial to address the overfitting issue from the timestep granularity in the field.
Technically, we devise a tractable strategy by introducing a dual-mask mechanism to filter out uncertain and anomalous timesteps dynamically during training.
Our method is agnostic to deep learning backbones and examined across several real- world datasets. The results demonstrate effectiveness and show consistent performance improvement over all basclines.

---

2 Related Work
2.1 Deep Models for Time Series Forecasting
In recent years, numerous deep models have been proposed to capture complex dependencies in TSF. Transformer-basee models have gained significant attention for their ability capture long-term temporal dependencies through attention mechanisms [81, 64, 82, 74, 79, 35, 29]. For example, Informer [81] introduces a ProbSparse attention to reduce the quadratic complexity. PatchTST [35] splits time series into patches and employs channel-independee independent strategy. i'Transfommer [29] embeds each series independently to the variate token and applies self-attention to capture multivariate correlations. In contrast to Transformers, CNN-based models [63, 32, 25] exhibit strong proficiency in extracting local patterns. Typically, TimesNet [63] transforms time series into 2D tensors and employs CNN to capture inter- and intra-period dependencies. Additionally, MLP--aaed models offer efficient alternatives with lightweight architectures [44, 13, 55, 70, 76]. DLinear [76] leverages simple linear layer with decomposition, and TimeMixer [55] captures multi-scale information through MLP layers. Our proposed selective learning can be easily applied to these deep models.
2.2 Training Strategies for Time Series Forecasting
The prevailing DL paradigm computes the regression loss (e.g., MSE/MAE) uniformly across all timesteps, and some works have explored alternative training strategies. For example, i'Transformer [29] proposes training strategy that randomly selects subsets of variables for large-scale multivariate time series. Merlin [75] employs a knowledge distillation [15] strategy to enhance the model's robustness against data missing. These approaches optimized for specific scenarios tightly coupled with model architectures, limiting their broader applicability. The most relevant work is MTGNN [67], which applies the idea of curriculum learning [4] to TSF by progressively increasing the prediction length during training. However, overlooks that difficulty is not solely determined by prediction length but is also influenced by intrinsic data characteristics. Selective learning addresses this issue by masking non non- generalizable timesteps while maintaining broad applicability.
Another line of work proposes alternative training objectives to replace the regression loss. For example, Soft-DTW [8]. DILATE [23], and TILDE-Q [24] align the shape between predictions and target sequences under temporal distortions. FreDF [51] combines the MSE loss with a frequency loss, mitigating the label correlation. PS loss [21] enhances the alignment by incorporating patch-wise distribution information. These works focus on matching the shape distribution in temporal or frequency domain between sequences, but none have recognized that global alignment over the whole sequence is not optimal, certain timesteps in the target sequence are inherently non-gener.
3 Preliminaries
Notations For a multivariate time series with N variables, let X, E R represent the t-th timestep. Given historical time series X,- t.t (Xt-1. Xt-t+1, . R RLXN where is the look- back window size, the TSF task is to predict future values X = (X,, X.- .x+r-1) E FXN with forecasting window size F. Considering a historical time series X E TTNN for training, the training dataset Dtrain {(X,LLa, xx T-F is constructed by a sliding window approach with stride 1.
The current DL paradigm is to find the best mapping from the samples in Dirain, i.c., X f(Xe-La) 0). where f(: 0): R + FxN is deep neural network parameterized by f. Mean squared error (MSE) measures the discrepancy between the prediction X and the ground truth X and is one of the commonly used loss functions to optimize 0:
Problem Statement
F-1
CMSE ( =
Σ
|
(1)
F
2-
0 = 0, - - VoLMEE.
(2)
where vj is the learning rate. We use T to denote the number of iterations during training, distinguishing it from the timestep index t.
3

---

(a) Selective Learning
Output Xer+F
Dual-Mask Mechanism
Input X
Deep Model
Residual ErioF
Uncercainty
Masking
Mmmn Pip
Mask
Ground truth Necss
Anomaly
Mask
Cst.
Me
Update
|M
(b) Uncertainty Mask
Uncercain
(c) Anomaly Mask
Unlearmed
Anomalies
Abs. Res.: [fel
Residual Entropy
Sliding windows
Time
Distance lower bound: [ec]
1PE21
Figure 2: (a) Overall framework of selective learning. (b) Uncertainty mask. (c) Anomaly mask.
Selective Learning
4.1 Overview of Selective Learning
We propose selective learning, model-agnostic learning strategy for deep TSF to address the overfitting issue. The main idea of selective learning is to calculate MSE loss only on subset of timesteps. This enables models to focus selectively generalizable timesteps while disregarding non-generalizubbe ones. Figure (a) illustrates the overall framework of selective learning. The implementation details and workflow of selective learning are provided in Appendix C.3.
As illustrated in Figure (left), we identify two critical categories of timesteps that degrade model generalizability. (1) Uncertain timesteps. Primarily originating from inherent noise in time series (e.g., signal disturbances [23]), they are characterized by high predictive uncertainty. Consequently, the gradients at these timesteps update towards random directions, resulting in undesirable fitting to the noise. (2) Anomalous timesteps. They are mainly caused by exogenous exceptional events (e.g.. sensor malfunctions [73, 72]). The model's predictions, though possibly confident, exhibit significant errors. This forces the model learn instance-specifie features through biased gradient updates, ultimately harming generalization.
To this end, propose a dual-mask mechanism to dynamically filter out these non-generalizable timesteps, as shown in Figure 2. Formally, we define selective learning as follows: Given a deep TSF model f(: 0) at 7-th iteration, we find a mask M(T) E 1) that constrains optimization only subset of timesteps:
(sl(O)= N N. (M
Σ (I (X++i - (x) 13
(3)
1=0
0 0++1000 -VVLLL.
(4)
where M M! v M! and the element-wise OR operator. M .M
10. are
the uncertainty mask and anomaly mask, respectively. We will describe them in detail in the following sections. Since uncertain and anomalous patterns do not necessarily appear synchronously across all variables, we adopt the channel-independerr strategy [35], generating masks for each variable independently. For theoretical tractability, we will focus on the univariate case in subsequent analysis, with natural extensibility multivariate scenarios due to channel independence.

---

4.2 Uncertainty Mask
We propose an entropy-based uncertainty masking approach for those timesteps that exhibit high predictive uncertainty. Let et = Xt X, denote the residual of t-th timesteps. The differential entropy of is
(Ex)) = f plee))lm(ee))) (np(e)dt.
Since training samples constructed via sliding windows over 1 each timestep will be predicted ne times in one epoch, where = min(t L + F"}. Therefore, we can estimate the residual distribution of e, using the most recent ny predictions. Let ((yble)) be the likelihood model of the residual, and we have
S arg (1) .e. .....,tt)))) clmi)) 10 (c.)) - f ((bblc) (n ((bble,)
(6)
(7)
where g is the i-th most recent prediction residual. In practice, we assume that the residual e NN o,), therefore we have
1
(c:) In In(2Ted?),
(8)
07 = n4 Σ
- =,)
(9)
Since these residuals et are computed at different training time they originate from distinct [(;:; 0z). Under Assumptions 2-4 (Appendix A.1), Theorem provides an upper bound for the error introduced by different 0.
Theorem 1 (Upper Bound for Variance Estimation Error ). The error bound between variance estimation under distinct parameters oū and that under identical parameters (0()) sutisfies:
bo? (0,)) S 41, (oyg)KGG(KK
(10)
where is the number of iterations per epoch, and L,,, R, G are constants.
The proof is provided in Appendix A. According to Theorem 1, we can always control the estimation error by choosing sufficiently small learning rate 7 and large batch size.
We employ hard thresholding You on the top-r. % residual entropy and obtain the uncertainty mask ( E 10 satisfying
0,
A(e)) Yuo
(M
(11)
1, otherwise.
4.3 Anomaly Mask
Predictions for anomalous timesteps typically exhibit significantly larger residuals due to deviations in ground truth values. The most intuitive solution mask timesteps with high |c/| filter anomalies. However, this naive approach suffers from critical limitation: it indiscriminately excludes both genuine anomalies and currently unlearned yet (but potentially generalizable) patterns, particularly during the early stage of training when the learning process remains incomplete.
To overcome this limitation, draw inspiration from practices in other fields [34, 48, 27, 50), and define S(X,) as the deviation between the residual and its theoretical lower bound c.
S(X,) = XX f(x; - L.B.
(12)
This formulation enables key separation: Anomalous timesteps exhibit elevated residual lower bounds, resulting comparatively small S(X) values. In contrast, unlearned timesteps demonstrate larger S(X) values due to significant gaps between current residuals and their theoretical minima.

---

In practice, we train a lightweight model g(:: Φ) on Dirain to estimate the residual lower bound, thereby estimating S(,, by:
S(X,) = X1 /(X;0):)
Xx (X; Φ)
(13)
residual
estimated LB E/-"
The details of the estimation model are discussed in Appendix C.3, Analogous the uncertainty mask, we employ hard thresholding Na to filter out the top-r,% of timesteps with the smallest
((x)) and obtain the anomaly mask M (0,
[0, S(X.) Ta
(M
(14)
1, otherwise.
Notably, instead of using the estimated residual lower bound as a static masking criterion, S(X,) can dynamically adjust the masking based on the current predictions. This approach offers two key advantages: (1) Static masking significantly alters the distribution of Dtrain thereby introducing bias, whereas dynamic masking adapts the mask during training to mitigate this in expectation. (2) For rare but critical extreme events (e.g., extreme weather) [10, 78] that are less generalizable, dynamic masking first learns the most generalizable timesteps and gradually attempts to learn timesteps previously considered anomalies. See Appendix E.2 for detailed discussions.
Experiments
5.1 Experimental Setup
Datasets We thoroughly evaluate the effectiveness of the proposed selective learning on 8 real- world datasets, including Electricity, Exchange, Weather, ILI, and 4 ETT datasets (ETThI, ETTh2, ETTml, ETTm2), which have been extensively used for benchmarking [42, 64, 37]. detailed description of the datasets is provided in Appendix C.I.
Baselines Selective learning is a model- model-agnostic training strategy, and it is compatible with any
deep TSF models. We carefully select six well-acknowledged deep models as the baselines, including Transformer-basee models (Informer [81]. Crossformer [79]. PatchTST [35]. iTransformer [29]). CNN-based models (TimesNet [63]), and MLP-based models (TimeMixer [55]). See Appendix C.2 for the introduction to the baselines. We select DLinear [76] as the estimation model for all baselines, and we further discuss the effects of different estimation models in $5.5.
Experimental Settings All baselines follow the same experimental setup with prediction lengths F E {24, 36, 48, 60) for ILI and F E (96, 192, 336, 720] for others [63]. We search for the look- back window L and report the best results. For fair evaluation, when training baselines with selective learning to enhance their performance, we follow their original hyperparameter settings and only tune the masking ratios ra and rn- We utilize Adam [20] for the model optimization. We evaluate the performance of all baselines using two commonly used metrics, MSE and MAE. All experiments are implemented with PyTorch and conducted on NVIDIA GeForce RTX 4090 24GB GPUs.
5.2
Main Results
Table shows the forecasting results with and without selective learning. The results are averaged over three runs. The lower MSE/MAE indicates a more accurate prediction. Our comprehensive evaluations demonstrate that selective learning consistently enhances model performance in all 192 cases (see full results in Appendix G.I). Selective learning proves particularly impactful for early- generation architectures that are susceptible to overfitting, where it achieves an average reduction of 37.4% in MSE and 25.4% % in MAE for Informer [81] (66.8% MSE and 42.6 6% MAE reduction the ETTm2 dataset), and 15.6% % in MSE and 10.5 % in MAE for Crossformer [79]. Notably, it maintains its benefits even for seate-of-the-an baselines, such as iTransformer [29] (6.5% MSE and 4.4% MAE reduction) and TimeMixer [55] (4.3% MSE and 3.3% MAE reduction), where the improvements persist in models already equipped with RevIN [19]. This confirms that selective learning provides additional performance gains over existing distribution shift mitigation techniques.

---

Table 1: Comparison of forecasting results without/with selective learning (SL). We use prediction lengths F E {24, 36, 48, 60) for ILI and E {96, 192, 336, 720} for other datasets. Results are averaged from all prediction lengths. Better results are in bold, and A denotes the improvements caused by selective learning. Full results of ETTh2 and ETTml are provided in Appendix G.I.
ETTh1
ETTm2
Electricity
Exchange
Weather
ILI
Method
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Informer
1.289
0.917
1.485
0,919
0.342
0.420
1.520
0.985
0.337
0.374
4.953
1.544
+SL
0.538
0.534
0.494
0.527
0.292
0.386
0.814
0.712
0.273
0.299
3,997
1.360
A
-58.3%
-118%
-66.8%
-42.6%
-14.4%
-8.21 %
-46.4%
-27.8%
-19.1 1
-20.1% %
-19.3%
-11.9%
Crossformer
0.455
0.465
0,588
0.528
0.182
0.277
0.755
0.649
0.226
0,284
3.982
1.342
+SL
0.431
0.441
0.370
0.408
0.168
0.264
0.527
0,525
0.213
0.265
3.681
1.285
A
-5.33%
-5.22%
-37.1%
-22.7%
-7.71%
-4.87% %
-30.1%
-19.2%
-5.97%
-6.61%
-7.56%
-4.26%
PatchTST
0,427
0.433
0.271
0.329
0.167
0.262
0,342
0,396
0.228
0.262
2.076
0.921
+SL
0.410
0.417
0.252
0.312
0.165
0,258
0.337
0,384
0.225
0.250
1.905
0.895
A
4.10%
+3.70%
-6.83%
-5.32%
-1.05%
-1.53%
-1.46%
-3.10%
-1.32%
-4.77%
-8.26%
-2.74%
TimesNet
0.499
0,486
0.289
0.343
0.198
0.301
0.382
0.429
0.248
0.284
2.493
1,028
SL
0.429
0,439
0.258
0,316
0.191
0.294
0.363
0.413
0.239
0.271
2.154
0.931
A
-14.0%
-9.67%
-10.7% %
-7.74%
-3.29%
-2.33%
-4.9%%
-3.73%
-3.54%
-4.58% %
-13.6%
-9.36% %
Transformer
0,458
0.457
0.273
0.332
0.164
0.257
0.364
0.413
0.235
0.269
1.909
0.914
+SL
0.415
0.425
0.256
0.313
0.157
0.249
0.343
0.399
0.229
0.257
1.710
0,857
99.29%
-6.90%
-6.40 %
-5.51%
.4.27%
-2.92%
-5.78%
-3.45%
-2.87
-4.28%
-10.4%
-6.24%
TimeMixer
0.443
0.445
0.265
0.323
0.163
0.259
0.348
0.400
0.230
0,276
2.163
0.932
+SL
0.411
0.421
0.251
0.309
0.160
0.254
0.335
0.394
0.226
0.268
2.026
0.895
A
-7.11%
-5.40%
-5.01%
-4.26%
-2.15%
-2.12%
-3.60%
-1.38%
-1.74%
-2.81%
-6.37%
-4.06%
5.3 Zero-shot Forecasting
We conducted zero-shot forecasting experiments to evaluate the generalization benefits of selective learning across different datasets. Following prior works [83, 17, we trained the models dataset DA and assessed on unseen dataset Dn without further training. As shown in Table 2, selective learning consistently enhance the performance of the baselines across diverse datasets in zero-shot forecasting, demonstrating its generalization advantage. Notably, in challenging generalization scenarios (ETTh2-ETTh and -ETTm2,EETTI,, selective learning achieves significant improve- ments, with MSE reduced by 22.6% % and MAE by 14.5% on average. Furthermore, in cases like ETThI-ETTh and ETTmi-EETT2,, the results outperforms training from scratch on the target dataset, underscoring the benefits of selective learning.
Table 2: Zero-shot forecasting results ETT datasets without/with selective learning. DA - DB denotes that the model was trained on DA and tested on DA The results are averaged from all prediction lengths. Better results are in bold, and red indicates a better result than training from scratch on DB without selective learning.
TimesNet
iTransformer
TimeMixer
Method
w/o
+SL
w/o
+SL
w/o
+SL
Metric
MSR
MAE
MSR
MAR
MSE
MAE
MSE
MAE
MSR
MAE
MSE
MAE
ETTh1 >ETT12
0,469
0.465
0.419
0.439
0.421
0.434
0.394
0.420
0.424
0.438
0.389
0.416
ETThI-EETm2
0.359
0.397
0.351
0.389
0.322
0.371
0,310
0.359
0.313
0.360
0.300
0.352
ETTh2 >ETThl
0.828
0.642
0.595
0.517
0.622
0.555
0.501
0.487
0.679
0.568
0.560
0.510
ETTmI-ETTh2
0.483
0.485
0.465
0.469
0.447
0.456
0.428
0.444
0.441
0.453
0.434
0.445
ETTmI-FETTm
0.321
0.363
0.288
0.339
0.276
0.334
0.271
0.326
0.275
0.326
0.267
0.319
ETTm2->ETTHR
0.727
0.578
0.459
0.448
0.554
0.498
0.443
0.439
0.451
0.466
0.428
0.425
5.4 Comparison with Other Training Objectives
Our proposed selective learning exhibits strong compatibility. It is completely model-agnostic and can applied to any deep learning architecture with various normalization methods [19, 11, 14]. Moreover, it maintains compatibility with learning strategies such as curriculum learning for TSF
7

---

Table 3: Comparison between selective learning (SL) and other training objectives with iTransformer as backbone. The results are averaged from all prediction lengths. The best results are in bold, and the second-best are underlined. Full results are provided in Appendix G.2.
Training objective
SL
PS
FreDF
TILDE-Q Q
MSE
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
0.415
0.425
0.427
0.440
0.450
0.455
0.432
0.439
0.458
0.457
ETTm2
0.257
0.315
0.264
0.320
0,262
0.319
0,263
0.319
0.273
0.332
Exchange
0.343
0.399
0.366
0.409
0.376
0.413
0.369
0.414
0.364
0,413
Weather 0.229
0.257
0.233
0.265
0.239
0.274
0.232
0.262
0.235
0.269
[67, 42]. However, selective learning operates on point-wise training objectives, In this section, we compare our selective learning with alternative non-point-wise training objectives, including shape-base based (TILDE-Q [24]), frequency-basee (FreDF [51]), and distribution-ben based (PS loss [21]) objectives. As shown in Table 3, selective learning consistently achieves superior performance. These results demonstrate that training the model by global alignment over whole sequences, whether in shape or distribution, the temporal or frequency domain, proves suboptimal, validating the effectiveness of selective learning.
5.5 Ablation Study and Hyperparameter Analysis
Ablation Study To study the effectiveness of the components selective learning, we conduct an ablation study covering: (1) removing either mask from the dual-mask mechanism, and (2) replacing the dual-mask mechanism with random masking with the same masking ratio. The results in Table 4 demonstrate that the model with full selective learning consistently achieves the best performance. Removing either mask leads to significant performance degradation across all four datasets, indicating that both masks contribute essential and distinct functionalities in filtering out non-generalizable patterns. Additionally, replacing the dual-mask mechanism with random masking reduces model performance to levels comparable to worse than the unmasked counterparts. This suggests that randomly attending to a subset of timesteps fails to enhance the model's performance and generalizability. The effectiveness of selective learning fundamentally stems from our dual- mask mechanism, which directs model attention to generalizable timesteps while filtering out non-generalizable ones.
Table 4: Ablation results for selective learning with iTransformer as backbone. The results are averaged from all predicted lengths. Full ablation results are provided in Appendix G.3.
Dataset
ETTh1
ETTm2
Electricity
Weather
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Selective Learning
0.415
0.425
0.257
0.315
0.157
0.249
0.229
0.257
Uncertainty mask
0.436
0.443
0.265
0.322
0.162
0.256
0.232
0.266
w/o
Anomaly mask
0.431
0.438
0.266
0.323
0.159
0.252
0.234
0.267
Replace
Random mask
0.457
0.460
0.274
0.332
0.165
0.261
0.237
0.269
Effects of Masking Ratio After validating the effectiveness of the dual-mask mechanism through ablation studies, we further investigate the effects of the masking ratios. When investigating a particular mask, we vary its masking ratio while fixing the other masking ratio at O. The results are shown in Figure 3. We can observe that larger masking ratios demonstrate superior performance on highly non- -ta-stationary datasets (ETThl and Exchange). This indicates severe overfitting in such datasets, where models benefit from focusing selectively on the most generalizable patterns. In contrast, datasets exhibiting periodic patterns (Weather) show improved performance with smaller masking ratios. Besides, it can be observed that on the Exchange dataset, the 90% anomaly masking ratio yields peak performance. This occurs because market-induced non-generalizable anomalies in this dataset exert significantly greater influence than noise. However, in most scenarios, the best

---

ETTh1
Exchange
Weather
0.26
0.46
MSE
MSF
MSE
0.25
0.31
0.24
0.6
0.4
0.8
0.6
Masking ratio
Masking ratio
Masking ratio
Uncertainty mask
Anomaly mask
Optimal results
Figure 3: Forecasting results under different masking ratios. The prediction length is 336.
results are achieved by combination of two masking strategies, the ratios of which constitute critical hyperparameters of selective learning. Detailed guidelines for selecting optimal ratios are provided in the appendix C.4.
Effects of Estimation Model In the ex- periments above, we employed DLinear as the lightweight estimation model. In this section, we investigate the effects of differ- ent estimation models. We the i'Trnss- former as the backbone model and addition- ally compare three estimation models: MLP, TimeMixer, and iTransformer. The results are shown Figure 4. We can observe that: For highly statiinti datasets (ETTh1 and Exchange), simpler models demonstrate superior performance; Conversely, datasets exhibiting periodic patterns (Weather) ben- efit from more complex estimation models. Despite this, the choice of the estimation model has limited overall impact per- formance, underscoring the robustness of the selective learning.
0.45
D.427
DUineer
TimeMizer
mansformr
0.40
0.35
0.343
3,349
0.228 0,229 0.229
0.20
0.15
ETTh1
Exchenge
Weather
Datasets
Figure 4: Forecasting performance with iTransformer backbone and various estimation models. The re- sults are averaged from all prediction lengths.
Learning Curve Analysis
In Figure 1, we initially illustrated iTransformer's training dynamics, highlighting selective learning's capacity to mitigate overfitting. To further validate this, Figure presents additional learning curves on the ETTh dataset. While all three models exhibit varying degrees of overfitting, their counter- parts trained with selective learning achieve stable convergence and superior performance. This demonstrates the efficacy of selective learning in mitigating overfitting and enhancing generalizability.
Informer
Crossformer
TimeMixer
0.7
wio SL
0.55
w/ SL
0.6
SL
0.50
0.5
10
20
30
Epoch
# Epoch
EEpach
Figure 5: Test MSE curve the ETTh1 dataset. The prediction length is 336.

---

Conclusion
In this work, we introduced selective learning, novel strategy to mitigate overfitting in deep TSF by selectively computing regression loss generalizable timesteps. Our dual-mask mechanism, comprising an uncertainty mask (based on residual entropy) and an anomaly mask (leveraging residual twwer-bound estimation), dynamically filters non-generalizable timesteps. allowing models to focus on robust patterns. Extensive experiments across eight real-world datasets validate that selective learning improves predictive accuracy and model generalizability. The scope of this work is currently constrained to in-domain time series forecasting. Future work can investigate the generalization to diverse time series analysis tasks (e.g., classification, imputation) and explore pretraining strategies for time series foundation models. See Appendix for limitations discussion and future directions.
Acknowledgement
This work is supported by the NSFC underGrant Nos.62372457 and 62502505, the Youth Innovation Promotion Association CAS 2023112 the Postdoctoral Fellowship Program of CPSF under Grant Number 57Z22021177,, the China Postdoctoral Science Foundation No.2025M77154 and HUA Innovation fundings. We sincerely thank all the anonymous reviewers who gerenously contributed their time and efforts.
References
[1] Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.
Sercan O Arik, Nathanael C Yoder, and Tomas Pfister. Self-adantive forecasting for improved deep learning on non-stationary time-series. arXiv preprint arXiv:220.002000 2022.
[3] Tianyi Bai, Ling Yang, Zhen Hao Wong, Fupeng Sun, Xinlin Zhuang, Jiahui Peng, Chi Zhang. Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan, and Conghui He. Efficient pretraining data selection for language models via multi-actor collaboration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguistics, July 2025.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41 18-48,22 2009.
[5] Jialin Chen, Jan Eric Lenssen, Aosong Feng. Weihua Hu, Matthias Fey, Leandros Tassiulas, Jure Leskovec. and Rex Ying. From similarity to superiority: Channel clustering for time series forecasting. Advances in Neural Information Processing Systems, 37:1 37:130635-130663 2024.
[6] Mouxiang Chen, Lefei Shen. Han Fu, Zhuo Li, Jianling Sun, Chenghao Liu. Calibration of time-series forecasting: Detecting and adapting context-driven distribution shift. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 341-353 2024.
Si-An Chen, Chun-Liang Li, Serean Arik, Nathanael Christian Yoder, and Tomas Pfister. Tsmixer: An all-mip architecture for time series forecasting. Transactions on Machine Learning Research, 2023.
[8] Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In International conference machine learning, pages 894-903. PMLR, 2017.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. decoder-only foundation model for time-scrics forecasting. In Proceedings of the 41st International Conference on Machine Learning. 2024.
[10] Daizong Ding, Mi Zhang, Xudong Pan, Min Yang, and Xiangnan He. Modeling extreme events in time series prediction. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pages 1114 1122, 2019.
[11] Wei Fan, Pengyang Wang. Dongkun Wang. Dongjie Wang. Yuanchun Zhou, and Yanjie Fu. Dish-ts: a general paradigm for alleviating distribution shift in time series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages
2023.
10

---

[12] Yisong Fu, Fei Wang. Zezhi Shao, Boyu Diao, Lin Wu, Zhulin An, Chengqing Yu, Yujie Li, and Yongjun Xu. On the integration of spatial-temporal knowledge: lightweight approach atmospheric time series forecasting, 2025.
[13] Han, Xu- Yang Chen, Han-Jia Ye, and De-Chuan Zhan. Softs: Efficient multivariate time series forecasting with series-core fusion. In The eiihth Anmual Conference on Neural Information Processing Systems, 2024.
[14] Lu Han, Han-Jia Ye, and Dhuan Zhan. Sin: Selective and interpretable normalization for long-term time series forecasting. In Forty-first International Conference Machine Learning, 2024.
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint /03xv:/50.00253/ 2015.
[16] Jincai Huang, Yongjun Xu, Qi Wang, Qi Cheems Wang, Xingxing Liang, Fei Wang. Zhao Zhang, Wei Wei, Boxuan Zhang, Libo Huang, et al. Foundation models and intelligent decision-mmkinng Progress, challenges, and perspectives. The Innovation, 2025.
[17] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Zhang. Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-fang Li. Shirui Pan, et al. Time Ilm: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations, 2024.
[18] HyunGi Kim, Siwon Kim, Jisoo Mok, and Sungroh Yoon. Battling the non-stationarity in time series forecasting via test-time adaptation. arXiv preprini arXiv:2501. 250992501004970,, 2025.
[19] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park. Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International conference on learning representations, 2021.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
[21] Dilfira Kudrat, Zongxia Xie, Yanru Sun. Tianyu Jia, and Qinghua Hu. Patch-wise structural loss for time series forecasting. arXiv preprint 2rXiv:2503.00877, 2025.
[22] Ying-yee Ava Lau, Zhiwen Shao, and Dit- Yan Yeung. Fast and slow streams for online time series forecast- ing without information leakage. In The Thirteenth International Conference Learning Representations, 2025.
[23] Vincent Le Guen and Nicolas Thome Shape and time distortion loss for training deep time series forecasting models. Advances in neural information processing systems, 32, 2019.
[24] Hyunwook Lee. Chunggi Lee, Hongkyu Lim, and Sungahn Ko. Tilde-q: transformation invariant loss function for time-series forecasting, arXiv preprint 22X.v2210.110500 2022.
[25] Chenghan Li, Mingchen Li, and Ruisheng Diao. Tvnet: A novel time series analysis method based on dynamic convolution and 3d-variation. In The Thirteenth International Conference on Learning Representations, 2025.
[26] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
[27] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen, al. Not all tokens are what you need for pretraining. Advances in Neural Information Processing Systems, 37:29029- 2024.
[28] Haoxin Liu, Harshavardhan Kamarthi, Lingkai Kong, Zhiyuan Zhao, Chao Zhang, and B Aditya Prakash. Time-series forecasting for out-of-distribution generalization using invariant learning. In Procedingss of the 41st International Conference on Machine Learning, pages 31312-31325, 2024.
[29] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itrans- forner: Inverted transformers are effective for time series forccasting. The Twelfth International Conference on Learning Representations, 2024.
[30] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: generative pre- trained transformers are large time series models. In Proceedings of the 41st International Conference on Machine Learning, pages 22369-32399 2024.

---

[31] Jiecheng Lu, Xu Han, Yan Sun, and Shihao Yang. Cats: Enhancing multivariate time series forecasting by constructing auxiliary time series as exogenous variables. In International Conference on Machine Learning, pages 32990-33006. PMLR, 2024.
[32] Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. In The rwelfth international conference on learning representations, pages 1-43, 2024.
[33] Ziqing Ma, Wenwei Wang. Tian Zhou, Chao Chen, Bingqing Peng, Liang Sun, and Rong Jin. Fusionsf: Fuse heterogeneous modalities in vector quantized framework for robust solar power forecasting. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5532-5543, 2024.
[34] Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630-15649. PMLR, 2022.
[35] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Leurning Representations, 2023.
[36] Quang Pham, Chenghao Liu, Doyen Sahon, and Steven Hoi. Learning fast and slow for online time series forecasting. In The Fleventh International Conference on Learning Representations, 2023.
[37] Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aaying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. Proc. VLDB Endow. 17:2363-223 2377, 2024.
[38] Suman Ravuri, Karel Lene, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):622-677,222 2021.
[39] CLEVELAND RB. Stl: A seasonal-trend decomposition procedure based on loess. J Off Stat, 6:3-73, 1990.
[40] Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly detection in time series: a comprehensive evaluation. Proceedings of the VLDB Endowment, 15(9):177--117 2022.
[41] Zezhi Shao, Yujie Li, Fei Wang, Chengging Yu, Yisong Fu, Tangwen Qian, Bin Xu, Boyu Diao, Yongjun Xu, and Xueqi Cheng. Blast: Balanced sampling time series corpus for universal forecasting models. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Data Mining V. 2. pages 2502 2513.2222222 2025.
[42] Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang, Di Yao, Tao Sun, Guangyin Jin, Xin Cao, et al. Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogencity analysis. IEEE Transactions on Knowledge and Data Engineering, 2024.
[43] Zezhi Shao, Fei Wang, Zhao Zhang, Yuchen Fang, Guangyin Jin, and Yongjun Xu. Hutformer: Hicrarchical u-net transformer for long-term traffic forecasting. arXiv preprint arXiv:2307. 145966 2023.
[44] Zezhi Shao, Zhao Zhang, Fei Wang. Wei Wei, and Yongjun Xu. Spatial temiiall identity: simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM international conference on information & knowledge management, pages 4454-4458 2022.
[45] Zezhi Shao, Zhao Zhang, Wei Wei, Fei Wang, Yongjun Xu, Xin Cao, and Christian S Jensen. Decou- pled dynamic spatial-temporal graph neural network for traffic forecasting. Proceedings of the VLDB Endowment, 15(11 15(11)22333-7746, 2022.
[46] Xiaoming Shi. Shiyu Wang, Yuqi Nie, Dianqi Li. Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. arXiv preprint arXiv:2409. 16040, 2024.
[47] Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-hased hard example mining for deep metric learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[48] Shivakanth Sujit, Somjit Nath, Pedro Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 36:23237- 23258, 2023.
12

---

[49] Tristan Thrush, Christopher Potis. and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. In The Thirteemth International Conference on Learning Representations, 2025.
[50] Ruilin Tong, Yuhang Liu, Javen Qinfeng Shi, and Dong Gong. Coreset selection via reducible loss in continual learning. In The Thirteenth International Conference on Learning Representations, 2025.
[51] Hao Wang, Lichen Pan, Yuan Shen, Zhichao Chen, Degui Yang, Yifei Yang. Sen Zhang, Xinggao Liu, Haoxuan Li, and Dacheng Tao. Fredf: Learning to forecast in the frequency domain. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Qi Wang, Yiqin Lx, Yixiu Mao, Yun Qu, Yi Xu, and Xiangyang Ji. Robust fast adaptation from adversarially explicit task distribution generation. In Proceedings the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. I. pages 148 1481-1491, 2025.
[53] Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning, pages 10018-1002) PMLR, 2020.
[54] Qi Cheems Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predictive task sampling for efficient and robust adaptation. arXiv preprint arv::25001 1009, 2025.
[55] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU. Timemixer: Decomposable multiscale mixing for time series forccasting. In The Twelfth International Conference on Learning Representations, 2024].
[56] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[57] Zhiyuan Wang, Xovee Xu, Goee Trajcevski, Kunpeng Zhang, Ting Zhong, and Fan Zhou, Pref: Probabilis- tie electricity forecasting via auguee-e state space model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12200-12207. 2022.
[58] Andreas S Weigend, Morgan Mangcas, and Ashok N Srivastava. Nonlinear gated experts for time series: Discovering regimes and avoiding overfitting. International journal of neural systems, 6(04):373-39) 1995.
[59] Qingsong Wen, Weiqi Chen, Liang Sun, Zhang Zhang, Liang Wang, Rong Jin, Tieniu Tan, et al. Onenet: Enhancing time series forecasting models under concept drift by online ensembling, Advances in Neurul Information Processing Systems, 36:69949-69980, 2023.
[60] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary. Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data, 2019.
[61] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting ligh-quality data for training language models. In Forty-first International Conference on Machine Learning. PMLR, 2024.
[62] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Proceedings of the 41st International Conference on Machine Learning, pages 53140-53164, 2024.
[63] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. The Eleventh International Conference on Learning Representations, 2023.
[64] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 14222419-22400 2021.
[65] Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence, 56))6002- 2023.
[66] Xingjian Wu, Xiangfei Qiu, Zhengyu Li, Yihang Wang, Jilin Hu, Chenjuan Guo, Hui Xiong, and Bin Yang. CATCH: Channel-aware multivariate time series anomaly detection via frequency patching. In The Thirleenth International Conference on Learning Representations, 2025.
[67] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 753-763, 763, 2020.
13

---

[68] Sang Michael Xie, Hieu Pham. Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V. Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798-6995 1881886688888818881 2023.
[69] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201-34233 2023.
[70] Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters. In The Twelfth International Conference on Learning Representations. 2024.
[71] Yiyuan Yang. Chaoli Zhang. Tian Zhou, Qingsong Wen, and Liang Sun. Dedetector: Dual attention contrastive representation learning for time series anomaly detection. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3033-3045, 2023.
[72] Chengging Yu, Fei Wang. Zezhi Shao, Tangwen Qian. Zhao Zhang, Wei Wei, Zhulin An, Qi Wang, and Yongjun Xu. Ginar+: A robust end-to-end framework for multivariate time series forecasting with missing values. IEEE Transactions on Knowledge and Data Engineering. 37(8))4635 4648, 2025.
[73] Chengging Yu, Fei Wang, Zezhi Shao, Tangwen Qian, Zhao Zhang, Wei Wei, and Yongjun Xu. Ginar: An and multivariate time series forecasting model suitable for variable missing. In Proceedings of the 30th ACM SIGKDD conference knowledge discovery and data mining, pages 3989 4000, 2024.
[74] Chengging Yu, Fei Wang, Zezhi Shao, Tao Sun, Lin Wu, and Yongjun Xu. Dsformer: A double sampling transformer for multivariate time series long-term prediction. In Proceedings of the 32nd ACM international conference on information and knowledge management, pages 3062-300000 207222 2023.
[75] Chengqing Yu, Fei Wang. Chuanguang Yang. Zezhi Shao, Tao Sun, Tangwen Qian, Wei Wei, Zhulin An, and Yongjun Xu. Merlin: Multi-view representation learning robust multivariate time series forecasting with unfixed missing rates. Proceedings of the st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 3633-3644, 2025.
[76] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu, Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11 11121111228 1128, 2023.
[77] Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Qiu Jiantao, Lei Cao, Ju Fan, Ye Yuan, Guoren Wang, and Conghui Hc. Hamessing diversity for important data selection in pretraining large language models. In The Thirteenth International Conference Learning Representations, 2025.
[78] Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael I Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 526-537) 2023.
[79] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
[80] Tianjie Zhao, Sheng Wang, Chaojun Ouyang, Min Chen, Chenying Liu. Jin Zhang, Long Yu, Fei Wang, Yong Xie, Jun Li, et al. Artificial intelligence for geoscience: Progress, challenges, and perspectives. The Innovation, 5(5), 2024.
[81] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 1 11106-11115, 2021.
[82] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term term series forecasting, In International conference on machine learning, pages 27268 27286. PMLR, 2022.
[83] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 011433222433553 2023.
14

---

A Proofs
A.1 Assumptions
Assumption 2 (Lipschitz Continuity). We assume that f reserves the Lipschitz continuity w.r.f. i.e. v01, O2 E O satisfying
[| 01) -(x: S Ly]10 02l1:
(15)
where L is the Lipschitz constant.
Justification can be ensured by the Lipschitz-continuuss activations the neural network and the continuously differentiable MSE loss.
Assumption (Bounded Prediction Residual), We assume that there exists a constant R. 0 such that
Eel S R VI (I....,T)
(16)
Justification Empirically, residuals often follow light-tailed distribution, where extreme deviations are rare. Therefore, there exists finite gigh-probbbiltyy bound.
Assumption (Bounded Gradient). We assume that there exists a constant > 0 such that
| fll G. E+
(17)
The boundedness of gradients ensured by Lipschitz-continuuus activations and weight constraints of the neural network, preventing explosive updates and ensuring stable optimiza- tion. Gradient clipping can further enforce it.
Justification
A.2 Proof of Theorem
(6)
Proof. Let T2 denote the training time corresponding to the i-th most recent prediction residual e (3)
such that ex O0.,,) = 6
1
|0? - (0,(1
(0,,) - ((0)) -(
74
1
(0,,), - ((0, )1 + + (0,)
71
< 2 max (0,,) - (0,)
(18)
< 2 max (0,)) c.())) la (0,.) + c(
2 max [(X, 0,,); f(X, 0, )²l (e: + (0,)
< 2 max L 0 2R (Assumption 2 and 3)
< RLG max |7 7a/- (Assumption 4)
Given that adjacent epochs are separated by no more than 2K iterations, we obtain:
|0, - -,,(0)))) < 4L, R2G((2 - 1).
(19)
15

---

B Related Work
B.1 Data Selection Strategies in Deep Learning
Data selection strategies aim to improve the efficiency, generalizability, and robustness of deep learning models by carefully selecting subsets data for training. Instead of using the entire dataset indiscriminately. these methods prioritize samples that contribute more significantly to model learning dynamics or downstream performance. Some studies such as curriculum learning [4] and hard sample mining [47], select or filter samples based their difficulties. In LM pretraining, samples or tokens are typically selected for high quality [68, 61, 27, 3], great importance [69], or strong diversity [77] to enhance both training efficiency and generalization. Data filtering pipelines often rely on heuristic metrics, such as perplexity [60, 49] and toxicity, or learned metrics [34, 27, 48], to remove duplicated, or low-quality data. Unlike the above studies, approach is closely tied to the characteristics of time series data, focusing on filtering noise or anomalies. By identifying and removing such irregular patterns, the model can learn from representative and generalizable timesteps and achieve more stable and reliable predictions.
C Implementation Details
C.1 Dataset Descriptions
We conduct experiments on 8 real-world datasets to evaluate the effectiveness of the proposed selective learning, including:
ETT (Electricity Transformer Temperature) [81] contains features of electricity trans- former data collected from two separate counties from July 2016 to July 2018. It contains four datasets: ETTh1, ETTh2, ETTm1, ETTm2, where ETTh1 and ETTh2 are recorded every hour, and ETTml and ETTm2 are recorded every 15 minutes.
Electricity [64] records the hourly electricity consumption data of 321 clients from 2012 to 2014. Each variable represents client's 'ss electricity consumption.
Exchange [64] collects the panel data of daily exchange rates from 1990 to 2016 from 8 countries, including Australia, Britain, Canada, Switzerland, China, Japan, New Zealand, and Singapore.
Weather includes 21 meteorological factors collected every 10 minutes from the weather station of the Max Planck Biogeachemistry Institute in 2020.
ILI (Influenza-Like Illness) [64] includes the weekly recorded patient data from the Centers for Disease Control and Prevention the United States between 2002 and 2021.
We follow the same data processing and maan--alidation--e set split protocol used in (6)),) where the train, validation, and test datasets are strictly divided according to chronological order to ensure no data leakage issues. The statistics of the datasets are provided in Table 5.
Table 5: Statistics of the datasets.
Dataset
Dim
Prediction Length
Dataset Size
Split
Frequency
Domain
ETThl, ETTh2
7
[96, 192, 336, 720)
14,400
6:2:2
Hourly
Electricity
ETTml, ETTm2
7
196, 192, 336, 720)
57,600
6:2:2
15min
Electricity
Exchange
8
(96, 192, 336, 720)
7,588
7:1:2
Daily
Economy
Weather
21
(96, 192, 336, 720)
52,696
7:1:2
10min
Weather
Electricity
321
(96. 192, 336, 720)
26,304
7:1:2
Hourly
Electricity
ILI
7
[24, 36. 60)
966
7:1:2
Weekly
Health
16

---

C.2 Baselines
Informer [81] is Transformer for time series forecasting (TSF) with ProbSparse self-
attention mechanism.
Crossformer [79] utilizes attention to capture both temporal and multivariate correlations.
PatchTST [35] splits the input time series into patches, which serve as input tokens of the Transformer. It proposes a channel-independent strategy.
TimesNet [63] transforms time series into 2D tensors and employs CNN to capture inter- and intra-period dependencies.
iTransformer [29] embeds each series independently to the variate token and applies self-attention to capture multivariate correlations.
TixeMixer [55] is MLP-based model that captures multi-scale patterss by decomposing time series into different scales and mixing them through MLP layers.
Notably, PatchTST, iTransformer, and TimeMixer are equipped with RevIN [19] to handle the distribution shift issue. Selective learning can provide additional performance gains while maintaining full compatibility with existing normalization techniques.
Algorithm 1 The workflow of selective learning.
1: INPUT:
The model f(: 0). the training set Ptrain {(x,-ii Lat x ) tt= the estimation model g(:, Φ) trained on Dtrain and the number of iterations Ng..
T-F
2: OUTPUT: Optimized model f(,, 0,).
3: Initialize (-,00u) and a historical residual archive $
4: for in 10,1, -1 do
5: X = f( 0,)
Forward
6: // Calculate the residual and update $
7:
6 6 = X
8:
+ EEi+F
9:
// Uncertainty mask
if 2 then
11:
// Update the residual entropy once per epoch
12:
if r then
13:
for & in (0...... T) do
Calculate Me()) by (8) end for You = Top-r,% % H(ee) fort (..... , 1}
15:
17:
end if
Calculate M by (11
19:
end if
// Anomaly Mask
Calculate 5(X,) by Eq.(13)
22:
Ta Top-r.% S(X)) fort (0.... F0....FF-111. F)
Calculate (T by (14(14)
23:
24:
M(r) _M(") VM
25:
LSL. N-IM ErD [IM - Lat 0
26:
0++1 = =, VoCLL // Optimization through selective learning loss
27:
end for
C.3 Implementation Details
this section, we provide the implementation details of selective learning. The overall workflow of selective learning is provided in Algorithm 1.
Uncertainty Mask We employ a global threshold 7x computed across the entire training sequence X to determine uncertainty masks. This choice prevents performance degradation caused by
17

---

masking normal (but relatively uncertain within specific samples) timesteps when masking ratios increase. For computational efficiency, the threshold updated only once per epoch. This design additionally mitigates the cold-start issue of uncertainty masking in the first epoch, where inadequate residual entropy estimates would otherwise lead to suboptimal masking decisions.
When processing extremely large datasets that exceed memory capacity for saving full-sequence residuals, we recommend two alternative approaches: (1) adopting per-sample thresholds with con- servatively low masking ratios, or (2) implementing max-heap algorithm for threshold computation. Both solutions maintain computational feasibility while preserving the benefits of uncertain masking.
Anomaly Mask We employ per-sample threshold %a computed across the prediction sequence X1.++F F to determine uncertainty masks. Global anomaly masking over Xo:7 adversely affected by training dynamics (where later samples consistently produce smaller residuals), resulting in suboptimal mask selection.
In this paper, the estimation model g is trained on the entire training set of f. diverging from some existing works in other fields that employ held-out sets train auxiliary models to prevent overfitting in the training set [34]. We prevent overfitting instead by employing a simple model (e.g.. DLinear) as g, thereby safeguarding the main model's performance. Additionally, unlike other data modalities, the patterns in time-series data often change over time. Consequently, using a holdout set may lead to underfitting of certain patterns g. particularly for datasets with strong non-stationariiyy When using a simple model as it should be trained until full convergence on the training set to avoid underfitting. Additionaly, since the sample partitioning in TSF depends on the sliding window size, each forecasting window size necessitates training a distinct model.
C.4 Selection of Masking Ratio
The masking ratios crucial hyperparameters in selective learning. We provide the default masking ratio Table 6. However, the optimal masking ratio may vary across different models. Users can perform hyperparameter search around the default masking rate to find the optimal masking ratio. For a new dataset, we recommend three-stage optimization protocol: (1) optimize the noise masking ratio achieve peak performance; (2) fix the noise masking ratio and tune the anomaly masking ratio to its optimal value; (3) conduct a local hyperparameter search within the neighborhood of these determined masking ratios.
Additionally, specific masking ratios may also be attempted as initial values:
Fu 10%, Fa 10% for stable and high-quality datasets.
* 30%, Fa = 30% for datasets containing certain levels of noise and anomalies.
Ta 90% for highly-volatile dataset.
Table 6: The default masking ratio for the datasets.
Dataset
ETThl
ETTh2
ETTml
ETTm2
Electricity
Exchange
Weather
ILI
Uncertainty Mask
30%
10%
20%
20%
10%
/
10%
10%
Anomaly Mask
30%
60%
20%
50%
10%
90%
20%
10%
D Running Cost
In this section, we analyze the computational cost introduced by selective learning from the perspective of running time and memory usage.
Complexity Analysis We first theoretically analyze of the time and space complexity of selective learning algorithm.
Time Complexity: Let B be the batch size. For uncertainty mask residual entropy updates cost O(FFN) per epoch. The complexity of the anomaly mask depends on the architecture of the estimation model. Taking a linear model as an example, the forward pass of the
18

---

estimation model requires (BBLN) complexity, and the masking process adds O(BFN).
Therefore, the complexity of the anomaly mask is +
Space Complexity: Storing residuals for timesteps requires O(TEN) space.
We measure the running time per epoch of different models trained without and with selective learning on the ETThl and ETTm2 datasets. The results in Table 7 demonstrate that selective learning maintains computational efficiency, adding acceptable training time while achieving significant performance gains.
Running Time
Table 7: Running cost of selective learning (SL). The results are averaged over 3 runs.
TimesNei
Trnnsommr
SL (s/Epoch)
Time Inc. (>Fpocch)
MSE %
who (>Fpooh)
Metric
(#Fgeoch)
(xF)joch)
(4/jecch)
(s/Epoch)
(sPpocc))
7.2
9.0
1.8
10.8%
1.8
2.0
0.2
1.7%
5.0
5.4
0.4
6.9%
10.2
11:0
0.8
13.0%
1.9
2.2
0.3
7.0%
5.1
5.6
11.5
6.8%
336
10.2
12.4
2.2
12.9%
2.1
2.3
0.2
8.1%
5.1
5.8
0.7
4.0%
720
14.1
15.7
1.6
18.9%
2.2
3.1
0.9
13.648
5,3
6,5
1.2
10.5%
Avg.
10.4
12.0
1.6
14,0%
2.0
2.4
0.4
9.398
5.1
5.8
0..7
6.9%
96
43.2
54.5
11.3
11.5%
9.7
11.0
1.3
5.1%
18.1
18.8
11.7
4.7%
ETTm22
192
50.1
56.6
6.5
8.18
10.2
11.2
1.0
9.0%
19.2
20.0
0.8
5.6%
64.2
10.9
0.9
7.2%
20.5
5.7%
100.5
12.2
10.7%
6.4%
Memory Usage Our implementation processes historical residuals on the CPU, resulting in merely 2MB of additional GPU memory allocation. Although migrating these operations the GPU would improve computational throughput, it would increase GPU memory consumption. Additionally, maintaining historical residuals in memory requires approximately 4| /N bytes of RAM (e.g.. <0.1GB IGB RAM for ETTh1 and about 7GB for Electricity when F 336).
E
Discussion
E.1
Non-generaaliablee Timesteps vs. Distribution Shift
In recent years, distribution shift in non-stationary time series has been widely studied by the research community [19, 11, 14] and shares conceptual similarities with non-generalizable timestemps. This section compares and contrasts non-general time steps with distribution shift. At their core. both concepts describe fundamental challenge in deep time series forecasting: the problem of mismatch between the data a model was trained on and the data it encounters in test, which is typically induced by changes in environmental or exogenous variables
However, the crucial distinction lies in their scale: distribution shift is typically instance- segment- level phenomenon, where the statistics of the entire dataset change gradually or abruptly over time. In contrast, non-generalizable timestep often localized, point-level issue. It refers individual or a small set of timesteps whose patterns are uncertain or anomalous that they cannot be reliably learned or predicted by the model, even if the overall data distribution remains stable.
Therefore, our proposed selective learning offers a finer-grained solution to prevent models from being affected by generallllll data.
E.2 Dynamic Masking vs. Static Masking
Static masking offers an intuitive implementation approach, for example, training an estimation model to estimate the distribution of each timestep, or employing time series anomaly detection models [40, 71, 66] to target anomalous timesteps. In contrast, dynamic masking adaptively modifies the masked timesteps during training. This approach offers two key advantages:
Static masking significantly alters the distribution of Dtrain. thereby introducing bias, whereas dynamic masking adapts the mask during training to mitigate this in expectation. Specifically, a given timestep may masked only during certain training phases and within particular lookback windows, while remaining in other contexts and stages.
19

---

For rare but critical extreme events (e.g., extreme (eather)[ 78] that are less generalizable, dynamic masking first learns the most generalizable timesteps and gradually attempts to learn timesteps previously considered anomalies. Static masking, by contrast, consistently excludes these patterns, resulting compromised forecasting capacity for extreme events.
We present comparison between dynamic and static masking in Table 8 using i'rrasformer, showing that dynamic masking yields consistently better performance, which validates our claims.
Table Comparison of static and dynamic masking.
Method
Metric
ETTh1
ETTm2
Exchange
Static masking
MSE
0.426
0,264
0.358
MAE
0.437
0.324
0.408
Dynamic masking
MSE
0.415
0.256
0.343
MAE
0.425
0.313
0.399
E.3 Capacity of Handling Clean Datasets
If selective learning is employed, then even clean datasets will be masked. This section conducts additional experiments and make discussions to investigate whether selective learning has impact on model performance on clean datasets.
To evaluate selective learning under clean conditions, construct synthetic dataset that is theoreti- cally clean (without any noise or anomalics). An ideal time series without noise or anomalies can be decomposed into trend and periodic components [39, 64]. Accordingly, combining a linear trend components with daily, weekly, and yearly sinusoidal patterns to generate multivariate time series. For each channel, both the trend slope and the amplitudes of each periodic component are sampled uniformly from specified ranges.
synthesize the dataset by
We have conducted experiments on the synthetic dataset using iTransformer as the backbone model. As shown in Table 9, uncertainty masking can reduce model performance on clean datasets (by misidentifying noise), while anomaly masking maintains This effect stems from our dynamic masking approach for anomalies: When normal timestep is mistakenly masked in epoch, it can likely be learned in subsequent epochs.
marginally improves performance.
In summary, we recommend using the anomaly mask on clean datasets, while exercising caution when applying uncertainty masking. Given that real-world time series datasets typically contain certain level of noise and anomalies, it is therefore beneficial to select an appropriate masking ratio.
Table 9: iTransformer's performance on the synthetic dataset (L 336,F 336). Better results with selective learning are in bold.
Uncertainty mask
+ Anooaly mask r.=10%
Method
iTransformer
r.=5%
r,=10%
ru=20%
ra=5%
r.=20%
MSE
0.0295 0.0838
0.0475
0.1569 0.0848
0.1640 0.1004
0.0299 0.0823
0.0294
0.0293
MAE
0.0791
0.0829
0.0833
F
Limitations and Future Work
Beyond Forecasting Task Our work currently focuses on time series forecasting tasks. However, our idea can also be applied to other time series analysis tasks, such as imputation and classification [53], guiding the model to focus more on generalizable patterns. While we highlight these potential extensions as promising directions, thorough investigation of their applicability and effectiveness remains an open question. We leave this exploration for future work.
Extreme Event Forecasting Capacity The dual-masking mechanism may filter out rare extreme events present in the training set. Although dynamic masking can mitigate this effect, the model's
20

---

predictive capability for extreme events may still be compromised. For scenarios where extreme events are critically important, we recommend fine-tuning the selective learning-trate trained model using online learning [59, 36, 22] or test-time adaptation [2, 6, 18, 52] after deployment. For example, SOLID [6] retrieves training samples similar to the current input (including potentially masked ones) to fine-tune the prediction head.
Pretraining for Time Series Foundation Model Recently, time series foundation models (TSFMs) have achieved rapid advancements [62, 9, 30, 1, 46, 41, 16). Selective learning currently focuses on in-domain forecasting. We leave this as future work. Since the TSFM is trained on samples drawn from a large-scale dataset, cannot estimate the residual entropy and lower bound at each timestep. As a result, the existing design not directly compatible with the training of TSFMs. However, thanks to the strong representational capacity of TSFMs, model (e.g., Chronos [I]. MOIRAI [62]) to directly predict the distribution of each timestep, thereby selecting the generalizable timesteps. Future work can further combine the selective learning with some model predictive methods [54] for TSFMs' robustness and efficiency improvement.
train probabilistic forecasting
G
Full Experimental Results
G.1 Full Forecasting Results
The full forecasting results are provided in Table 10 and 11 due to the page limitation of the main text. can be observed that selective learning significantly improves models' performance in all cases, demonstrating its effectiveness.
G.2 Full Results of Training Objective Comparison
The full results of the training objective comparison provided in Table 12 due to the page limitation of the main text. It is evident that selective learning consistently achieves superior performance. These results demonstrate that global alignment whole sequences, whether in shape distribution, in the temporal or frequency domain, proves suboptimal, validating the effectiveness of selective learning.
G.3
Full Ablation Results
The full ablation results are provided Table 13. The results demonstrate that the model with full selective learning consistently achieves the best performance. Removing either mask leads to significant performance degradation across all four datasets. Additionally, replacing the dual-mask mechanism with random masking reduces model performance to levels comparable to worse than the unmasked counterparts. This suggests that the effectiveness of selective learning fundamentally stems from dual-mask mechanism, which directs model attention to generalizable timesteps while filtering out non-gennrralzable ones.
H
Case Study
To showcase the effectiveness of selective learning, we provide supplementary prediction cases of five baselines across five representative datasets in Figure 6. The visualizations clearly show that selective learning can enhance models' forecasting performance and generalizability.
21

---

Table 10: Part of the full forecasting results without/with selective learning (SL). Better results are in bold. Avg. denotes the average from all prediction lengths, and A denotes the averaged improvements caused by selective learning.
Informer
Crossformer
PatchTST
Method
+SL
Metric
MSH
MAE
MSE:
MAE
MSE
MAE
MSE:
MAE
MSE
MAE
MSE:
MAE
96
1.087
0.831
0.478
0.487
0.406
0.426
0.373
0.397
0.377
0.397
0.368
0.386
192
1.262
0,917
0.492
0.490
01.488
0.455
0.417
0.422
0.417
0.421
0.412
0.413
ElThl
336
1.379
0.952
0.503
0.526
0.460
0.466
0.448
0.443
0.448
0.442
0.433
0.426
720
1.428
0.968
0.680
0.634
0.505
0.512
0.484
0.500
0.465
0.472
0.425
0.443
Avg.
1.289
0.917
0.538
0.534
0.455
0.465
0.431
0.441
0.427
0.433
0.410
0.417
A
-58.3% %
-41.8 %
-5.33% %
-5.22% %
-4.10%
-3,70%
96
3.107
1.475
1.516
0.937
0.717
0.579
0.474
0.463
0,311
0.367
0.296
0.353
ETTh2
3.707
1.697
0.736
0.610
0,381
0.369
336
2.671
1.346
1.595
0,982
0.739
0.621
0.618
0.553
0.418
0.437
0.401
0.428
720
2.543
1.348
2.065
1.175
1.113
0.784
0.935
0.707
0.443
0.464
0.431
0.452
Avg.
3.007
1.457
1.718
1.027
0.826
0.648
0.659
0.568
0.388
0.420
0.374
0.408
A
-42.9
-29.5 %
-00.2%
-12.4%
-3.61
-2.68%
96
0.443
0,446
0.304
0.350
0.307
0.360
0.301
0.353
0.294
0.343
0.290
0.333
192
0.618
0.573
0.344
0.374
0.366
0.404
0.344
0,381
0.339
0.373
0.333
0,359
ET1ml
336
0.865
0.701
0.374
0.395
0.446
0.453
0.403
0.422
0.372
0.393
0.367
0.381
720
0.941
0.757
0.431
0.433
0.578
0.503
0.424
0.420
Avg.
0.717
0.619
0.363
0.388
0.424
0.437
0.388
0.413
0.357
0.385
0.353
0.372
A
-4.3%
-37.3% %
-8.60 %
-S,49 %
-1.33%
-3.32%
96
0.334
0,443
0.226
0.345
0.281
0.356
0.194
0.291
0.175
0.261
0.164
0.251
ETTm2
192
0.729
0.676
0.367
0.464
0.374
0.450
0.262
0.341
0,236
0.306
0.218
0.289
336
1.416
0.955
0.595
0.610
0.676
0.582
0.391
0.430
0.293
0.347
0.266
0.322
720
3,460
1.601
0.786
0.690
1.019
0.724
0.631
0,570
0.379
0.402
0.361
0.384
Avg.
1.485
0.919
0.494
0.527
0,588
0,528
0,370
0.408
0.271
0.329
0.252
0.312
-66.8%
-42.6 %
-37.1
-22.7 %
-6.83%
-5.32%
96
0.300
0.389
0.266
0.364
0.137
0.236
0.134
0.228
0.139
0.235
0.139
0.234
Eletricity
192
0.311
0.399
0.286
0.384
0.162
0.260
0.147
0.241
0.153
0.248
0.152
0.245
336
0.349
0.434
0.301
0.391
(9.190
0.2883
0.169
0.271
0.168
0.267
0.166
0.258
720
0.406
0.459
0.316
0.404
0.237
0.330
0.220
0.315
0,208
0.296
0.204
0.293
Avg. A
0.342
0.420
0.292
0.386
0.182
0,277
0.168
0.264
0.167
0.262
0.165
0.258
114.4%
-8.21 %
-7.71 %
-4.87 %
-1.05% %
-1.53%
96
0.906
0.763
0.408
0.514
0.289
0..39
0.205
0.328
0.078
0.196
0.078
0.196
Exchunge
192
1.291
0.908
0.621
0.638
0.527
0.558
0.355
0.447
0.161
0.290
0.157
0.287
336
1.334
0.953
0.825
0.745
0.858
0.719
0.539
0.551
0,303
0.404
0.294
0.397
720
2.547
1.317
1.402
0.949
1.344
0.922
1.010
0.772
0.826
0.693
0.819
0.654
Avg. A
1.520
0.985
0.814
0.712
0.755
0.6499
0.527
0.525
0.342
0.396
0.337
0.384
-46.4% %
-27.8 %
-30.1 %
-19.2 %
-1.46%
-3.10%
96
0.203
0.287
0.157
0.198
0.145
0.209
0.138
0.193
0,150
0.196
0.147
0.185
Weather
192
0.303
0.216
0.190
0.184
0.195
0.190
336
0.351
0.373
0.270
0.305
0.252
0.306
0.231
0.285
0.246
0.280
0.243
0.267
720
0.491
0.465
0.448
0.435
0.318
0.363
0.298
0.336
0
0.332
0,319
0.321
Avg.
0.337
0.374
0.273
0.299
0.226
0.284
0.213
0.265
0,228
0.262
0.225
0.250
.19.1
-5.97
-1.32%
24
4,689
1.466
4,587
1.434
3.595
1.265
3.006
1.150
1,900
0.868
1.755
0.856
36
4.812
1.529
3.273
1.243
3.977
1.350
3.416
1.234
2.396
0.964
2.056
0.926
48
4,952
1.572
3.721
1.328
3.783
1.297
3.773
1.306
1.938
0.917
1.793
0.888
5.358
1.608
4,405
4.571
4.527
2.070
2.014
0.911
Avg. A
4.953
1.544
3.997
1.360
3.982
1.342
3.681
1.285
2.076
0.921
1,905
0.895
-1933%
-11.%%
-7.56%
-4.26 %
-8.26 %
-2.74%
22

---

Table 11: Part 2 of the full forecasting results without/with selective learning (SL). Better results are in bold. Avg. denotes the average from all prediction lengths, and A denotes the averaged improvements caused by selective learning.
TimesNet
iTransformer
TimeMixer
Method
Metric
MSH
MAE:
MSE:
MAE
MSE
MAE
MSE:
MAE
MSE:
MAE
MSE:
MAE
96
0.445
0.448
0.398
0.411
0.402
0.413
0.371
0.389
0.394
0.411
0.367
0.387
192
0.476
0.472
0.414
0.426
11.455
0.440
0.414
0.420
0.441)
0.442
0.410
0.415
ElThl
336
0,505
0.485
0.440
0.446
0.469
0.464
0.431
0.432
0.452
0.446
0.434
0.429
720
0.571
0.537
0.463
0.474
0.514
0.510
0.444
0.460
0.485
0.480
0.434
0.452
Avg.
0.499
0.486
0.429
0.439
0.458
0.457
0.415
0.425
0.443
0.445
0.411
0.421
A
-14.0%
-9.67 %
-9.29 %
-6.90% %
-7.11%%
-5.40%
96
0.356
0.404
0.292
0.357
0.319
0.372
0.300
0.352
0.325
0.375
0.299
0.354
ETTh2
0.427
0.351
0.394
0.375
0.378
336
0.450
0.467
0.389
0.423
0.429
0.445
0.408
0.433
0.430
0.451
0.405
0.428
720
0.505
0,500
0.439
0.459
0.460
0.474
0.445
0.467
0.457
0.472
0.443
0.462
Avg. A
0.435
0,456
0.368
0.409
0.401
0,428
0.382
0.414
0.406
0,434
0.381
0.412
-15.4%
-10.3%
-4.62%
-3.27
-6.10%
-5.07% %
96
0.329
0.375
0.298
0.342
0.305
0.358
0.295
0.342
0.298
0.350
0.287
0.337
192
0.377
0.402
0.344
0.372
0.346
0,380
0.338
0.368
0.329
0.370
0.327
0,362
ET1ml
336
0.413
0.427
0.382
0.397
0.385
0.403
0.371
0.386
0.368
0.391
0.368
0.382
720
0.464
0.453
0.419
0.424
0.446
0.441
0.423
0.422
0.431
0.426
0.422
0.416
Avg.
0.396
0.414
0.361
0.384
0.371
0.396
0.357
0.380
0.357
0.384
0.351
0.374
A
-8.84% %
-7.36 %
-3.71 %
-4.05 %
-1.54%
-2.60 %
96
0.191
0,280
0.169
0.257
0.1755
0,268
0.166
0.253
0.172
0.261
0.164
0.250
ETTm2
192
0,246
0.314
0.226
0.298
0,244
0.315
0.220
0.291
0,233
0.305
0.220
0.289
336
0.312
0,360
0.271
0.324
0.291
0,343
0.270
0.325
0.283
0.335
0.267
0.321
720
0.408
0.416
0.367
0.385
0.383
0,400
0.367
0.384
0.370
0.391
0.354
0.377
Avg.
00:899
0,343
0.258
0.316
0..733
0,332
0.256
0.313
0.265
0.323
0.251
0.309
-10.7%
-7.74% %
-6.40
-5.51 %
-5.01
-4.26
96
0.184
0.289
0.177
0.281
0.134
0.227
0.132
0.223
0.135
0.229
0.133
0.228
Eletricity
192
0.192
0.295
0.184
0,286
0.157
0.249
0.151
0.244
0.152
0.247
0.149
0.241
336
0.193
0.299
0.190
0.294
16.668
0.262
0.158
0.250
0.164
0.263
0.160
0.255
720
0.222
0.320
0.214
0.313
0.197
0.290
0.187
0.279
0.201
0.297
0.196
0.290
Avg.
0.198
0.301
0.191
0.294
0.164
0.257
0.157
0.249
0.163
0.259
0.160
0.254
A
-3.29% %
-2.33%
-4.7%%
-2.92%
-2.15%
-2.12%
96
0.109
0.239
0.091
0.218
0.088
01.211
0.082
0.203
0.080
0.199
0.078
0.197
Exchunge
192
0.182
0.312
0.165
0.299
0.173
0.303
0.162
0.293
0.166
0.298
0.157
0.287
336
0.333
0.427
0.322
0.416
0,323
0.419
0.305
0.407
0,310
0.407
0.297
0.399
720
0.904
0.736
0.875
0.720
0,870
0.720
0.821
0.693
0.834
0.694
0.808
0.693
Avg. A
0.382
0.429
0.363
0.413
0.364
0.413
0.343
0.399
0.348
0.400
0.335
0.394
-4.97%
-3.73 %
-5.78 %
-3.45 %
-3.60 %
-1.88%
96
0.167
0.222
0.160
0.207
0.161
0.210
0.153
0.194
0.152
0.207
0.152
0.204
Weather
192
0.212
0.255
0.206
0.197
0.197
0.253
0.194
336
0.265
0.300
0.257
0,289
0.249
0,280
0.243
0.270
0.249
0.293
0.245
0.287
720
0.340
0.348
0.326
0.334
0.325
0.336
0.321
0.328
0
0.350
0,312
0.338
Avg.
0.248
0.284
0.239
0.271
0.235
0.269
0.229
0.257
0,230
0.276
0.226
0.268
.3.54
-2.87
-1.74%
-2.81 %
24
2.480
1.009
1.969
0.907
1.511
0.813
1.359
0.784
1,931
0.879
1.829
0.847
36
2.815
1.095
2.405
0,979
1.929
0.929
1.696
0.847
2.430
0.971
2.113
0.902
48
2.436
1.008
2.351
0.941
2.054
0.931
1.857
0.892
2.135
0.931
2.051
0.903
2.157
2.109
0.926
Avg.
2.493
1.028
2.154
0.931
1.909
0.914
1.710
0.857
2.163
0.932
2.026
0.895
A
13.6%
-9.36%
-10.4%
-6.24 %
-6.37 %
-4.06 %
23

---

Table 12: Full comparison results between selective learning (SL) and other training objectives with iTransformer as backbone. Avg. denotes the averaged results from all prediction lengths. The best results are in bold, and the second-best are underlined.
Loss
SL
PS
FreDF
TILDE-Q Q
MSE
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.371
0.389
0.389
0.410
0.388
0.410
0.391
0.408
0.402
0.413
192
0.414
0.420
0.421
0.429
0.434
0.438
0.423
0.428
0.445
0.440
336
0.431
0.432
0.443
0.446
0.460
0.459
0.448
0.444
0.469
0.464
720
0.444
0.460
0.453
0.476
0.516
0,513
0.467
0.477
0.514
0.510
Avg.
0.415
0.425
0.427
0.440
0.450
0.455
0.432
0.439
0.458
0.457
96
0.166
0.253
0.167
0.255
0.169
0.258
0.173
0.259
0.175
0.268
ETTm2
192
0.220
0.291
0.232
0.299
0.227
0.298
0.231
0.298
0.244
0.315
336
0.270
0.325
0.287
0.336
0.274
0.330
0.278
0.331
0.291
0.343
720
0.367
0.384
0.370
0.389
0.377
0.391
0.370
0.388
0.383
0.400
Avg.
0.256
0.313
0.264
0.320
0.262
0.319
0.263
0.319
0.273
0.332
96
0.082
0.203
0.087
0.211
0.088
0.208
0.084
0.207
0.088
0.211
Exchange
192
0.162
0.293
0.180
0.303
0.185
0.305
0.171
0.301
0.173
0.303
336
0.305
0.407
0.335
0.420
0.346
0,426
0.335
0.422
0.323
0.419
720
0.821
0.693
0.861
0.700
0.886
0.712
0.884
0.724
0.870
0.720
Avg.
0.343
0.399
0.366
0.409
0.376
0.413
0.369
0.414
0.364
0.413
96
0.153
0.194
0.155
0.199
0.159
0.207
0.155
0.198
0.161
0.210
Weather
192
0.197
0.237
0.200
0.241
0.204
0.249
0.199
0.239
0.206
0.249
336
0.243
0.270
0.250
0.281
0.260
0.292
0.249
0.279
0.249
0.280
720
0.321
0.328
0.327
0.337
0.334
0.347
0.325
0.333
0.325
0.336
Avg.
0.229
0.257
0.233
0.265
0.239
0.274
0.232
0.262
0.235
0.269
Table 13: Full ablation results for selective learning with iTransformer as backbone.
Prediction Length
ETTh1
ETTm2
Electricity
Weather
Design
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.371
0.389
0.166
0.253
0.132
0.223
0.153
0.194
192
0.414
0.420
0.220
0.291
0.151
0.244
0.197
0.237
Selective Learning
336
0.431
0.432
0.270
0.325
0.158
0.250
0.243
0.270
720
0.444
0.460
0.367
0.384
0.187
0.279
0.321
0.328
Avg.
0.415
0.425
0.257
0.315
0.157
0.249
0.229
0.257
96
0.379
0.400
0.166
0.257
0.133
0.228
0.159
0.207
192
0.423
0.429
0.230
0.300
0.153
0.247
0.201
0.246
w/o Uncertainty Mask
336
0,448
0.447
0.287
0.335
0.165
0.261
0.246
0,280
720
0.495
0.497
0.377
0.394
0.195
0.287
0.321
0.331
Avg.
0.436
0.443
0.265
0.322
0.162
0.256
0.232
0.266
96
0.388
0.403
0.177
0.261
0.135
0.228
0.158
0.206
192
0.424
0.428
0.230
0.301
0.151
0.244
0.201
0.245
w/o Anomaly Mask
336
0.444
0.442
0.279
0.333
0.160
0.256
0.249
0.281
720
0.468
0.477
0.376
0.396
0.189
0.279
0.326
0.334
Avg.
0.431
0.438
0.266
0.323
0.1.9
0.252
0.234
0.267
96
0.401
0.418
0.179
0.271
0.137
0.234
0.160
0.208
192
0.452
0.452
0.240
0.312
0.158
0.256
0.203
0.249
Random Mask
336
0.471
0.466
0.294
0.346
0.169
0.266
0.258
0.284
720
0.505
0.505
0.384
0.399
0.196
0.286
0.325
0.334
Avg.
0.457
0.460
0.274
0.332
0.165
0.261
0.237
0.269
24

---

1.5
1.0
0.5
0.5
20
40
80
20
40
60
80
20
40
60
Ground truth
W/o
SL
(a) Case study the ETThl datasets with iTTransfommer as backbone.
-0.2
-0.5
-0.4
-2.0
-1.
20
40
60
80
20
40
80
20
60
80
(b) Case study the ETTm2 datasets with Informer as backbone.
20
40
40
Ground truth
w/o SL
SL
(c) Case study the Electricity datasets with Crossformer as backhone.
mmma
2.
1.8
Ground truth
w/o SL
SL
(d) Case study on the Exchange datasets with Informer backbone.
0.4
-0.25
0.5
-0.50
Ground truth
w/o SL
SL
(e) Case study on the Weather datasets with PatchTST as backbone.
Figure 6: Case study results across five datasets.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations in the conclusion and appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
The paper provides the assumptions and proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides detailed descriptions of the datasets, experimental setups, implementation details, and hyperparameters, which are sufficient to reproduce the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper provides the code in the supplemental material with sufficient instructions to reproduce the main experimental results. The code will be released publicly upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cctpublic/guit httaa//ggiiieee/eeeuuubiiieeee wwweeSubmisiiooPoo...
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The paper thoroughly describes the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper provides clear explanations of the statistical methods used to compute metrics and ensures the robustness of the reported findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tn:///nuuueeeee /nerrps.....p// cc wcc/ppblic/Eeiieeeeeeeeeeeeeeeee www.iiihhhheeceeeeeeeeenene lnes?
Answer: [Yes]
Justification:
The research aligns with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on time series forecasting problem, so there is no potential socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper credits the owners of any used assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets that are well documented alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects, IRB approvals or equivalent review are not required.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: for what should or should not be described.
32


Reviews:
[{"review_id": "A3WE12RS5B", "reviewer": "Reviewer_3Yh3", "summary": "This paper introduces selective learning for deep time series forecasting. Selective learning only calculate the MSE loss over a subset of timesteps, guiding the model to focus on generalizable timesteps while disregarding non-generalizable ones. Technically, the paper propose a dual-mask mechanism (including an uncertainty mask and an anomaly mask) to target timesteps. Extensive experiments across eight real-world datasets demonstrate that selective learning can significantly improve the performance for state-of-the-art deep models.", "questions": "Q1. Although selective learning demonstrates reduced MSE and MAE metrics, could the masking strategy potentially introduce bias to models?\n\nQ2. In Figure 5, why does it present results of Informer for 30 epochs while other models for only 10 epochs?\n\nQ3.In Assumption 4, what does $\\au$ stand for?\n\nQ4. The paper compares selective learning with other training objectives (e.g., PS loss and FreDF). If I understand correctly, selective learning appears theoretically compatible with these objectives. Could the idea of selective learning be combined with existing objectives to further enhance their performance?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. This paper proposes a novel training strategy for deep time series forecasting and offers a valuable insight: optimizing a generalizable subset of the time series is more effective than optimizing the entire sequence due to the inherent vulnerability of time series to anomalies and noise.\n\nS2. Extensive experiments and rigorous analysis demonstrate the effectiveness of selective learning. The performance gain from selective learning is significant, particularly for those models prone to overfitting (e.g., Informer, Crossformer).\n\nS3. The paper is well-written with clear explainations and is easy to follow.\n\nWeaknesses:\n\nW1. In lines 170-176, the authors state that the static masking approach is suboptimal. However, the experiment does not include a comparative analysis between the current dual-masking mechanism and static masking approach.\n\nW2. While the idea of selective learning is general, the current technical implementation of selective learning is confined to in-domain prediction task and cannot be applied to the training of time series foundation model.", "was_revised": true, "final_justification": "My major concerns on the experiments and generalization have been well addressed in the rebuttal. And after reading other reviewers' comments and authors' responses, I decide to keep the score and would like to recommend the acceptance of the paper.", "rebuttal": "*We sincerely thank Reviewer kupb for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n---\n\n1. **Comparative analysis between dynamic masking and static masking.**\n\n   Thank you for your valuable suggestion. We conduct an additional comparative analysis between dynamic masking and static masking with iTransformer as the backbone. As shown in the table below, dynamic masking achieves better performance than static masking, validating our claims. We will include this comparative analysis in the revised version.\n\n   | Method          | Metric |   ETTh1   |   ETTm2   | Exchange  |\n   | :-------------- | :----: | :-------: | :-------: | :-------: |\n   | Static masking  |  MSE   |   0.426   |   0.264   |   0.358   |\n   |                 |  MAE   |   0.437   |   0.324   |   0.408   |\n   | Dynamic masking |  MSE   | **0.415** | **0.256** | **0.343** |\n   |                 |  MAE   | **0.425** | **0.313** | **0.399** |\n\n2. **The scope of our work**\n\n   The scope of our work focuses on in-domain time series forecasting. Since uncertainty masking requires multiple prediction samples of timesteps within an epoch, the current masking mechanism cannot be directly applied to the pretraining of foundation models. However, the high-level idea of selective learning is not inherently tied to specific masking implementations and can be adapted to broader task scenarios. We will explore masking mechanisms for TSFMs in future work.\n\n3. **Could the masking strategy potentially introduce bias to models?**\n\n   Thank you for raising this important point. Selective learning improves model generalization through masking (reducing the occurrence of non-generalizable patterns), which inevitably introduces bias (e.g., degraded performance in predicting extreme events). This represents a trade-off. However, techniques like online learning and test-time adaptation can help enhance the prediction performance for extreme events that might be overlooked during training.\n\n4. **Issues about Figure 5**\n\n   Due to severe overfitting in Crossformer and TimeMixer (Fig. 5), plotting more epochs would make the selective learning curve appear nearly flat compared to the original curves. Therefore, we only display the results of Crossformer and TimeMixer for 10 epochs.\n\n5. **Explanation of Assumption 4**\n\n   We sincerely apologize for the typographical error that caused confusion. The correct version of Eq.17 in Assumption 4 should be:\n   $$\n   ||\\nabla_{\\boldsymbol{\\theta}_\\tau} \\mathcal L||<G, \\quad \\forall \\tau\\in \\mathbb{Z}^{+},\n   $$\n   where $\\tau$ is the number of iteration. We will correct this in the revised version.\n\n6. **Could the idea of selective learning be combined with existing training objectives?**\n\n   Selective learning can indeed be combined with certain training objectives. The core idea of selective learning is to mask non-generalizable timesteps, which grants it strong compatibility. For example, the table (prediction length is 336) below demonstrates that selective learning can further improve the performance of the FreDF loss.\n\n   |              |    MSE    |    MAE    |\n   | ------------ | :-------: | :-------: |\n   | iTransformer |   0.469   |   0.464   |\n   | + FreDF      |   0.460   |   0.459   |\n   | + FreDF + SL | **0.436** | **0.439** |\n\n---\n\n*Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.*"}, {"review_id": "dbIMQYbOo1", "reviewer": "Reviewer_c9bV", "summary": "This paper proposes Selective Learning, a novel training strategy for deep time series forecasting (TSF). Unlike conventional methods that uniformly optimize all timesteps via MSE loss, this method selectively computes loss only over generalizable timesteps while masking out those identified as uncertain or anomalous. The authors propose a dual-mask mechanism consisting of: an uncertainty mask using residual entropy and an anomaly mask based on residual lower-bound deviation. Extensive experiments on 8 real-world datasets and 6 popular TSF models demonstrate significant improvements, particularly for overfitting-prone architectures.", "questions": "See the Strengths And Weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a core weakness in current TSF training—overfitting on noisy or anomalous data.\n2. The experiments are comprehensive, covering diverse model backbones (Transformer, CNN, MLP), datasets with various periodicity and non-stationarity levels.\n3. The paper is very well written with clear problem setup, motivating examples and thoughtful appendices with implementation details and limitations.\n\nWeaknesses and Questions:\n\n1. Concept Drift vs. Anomalies: Many works on non-stationary TSF emphasize concept drift. In realistic scenarios, drift and anomalies often co-exist. How does the proposed method distinguish between the two? Please provide precise definitions and clarify whether “non-generalizable” includes drifted patterns.\n2. The proposed approach appears closely related to SIN [1], which also filters out unstable regions. Please clarify the technical differences and include empirical comparison to strengthen the novelty claim.\n[1] SIN: Selective and Interpretable Normalization for Long-Term Time Series Forecasting, ICML 2024.\n3. The selective learning principle overlaps with ideas from curriculum learning, hard example mining, and prioritized sampling. The paper should provide a clearer conceptual and methodological differences from these approaches in §2.2.\n4. The residual entropy and estimated lower-bound both depend on the training dynamics (e.g., optimizer, batch size, learning rate). While Theorem 1 attempts to bound the variance estimation error, empirical sensitivity analysis (e.g., different η, batch sizes) is missing.\n5. The use of a second model to estimate residual lower bounds (e.g., DLinear) could introduce biases if the estimation model underfits or overfits. Clarify the generalizability of the estimation model and whether it affects downstream learning in challenging regimes (e.g., high-noise datasets).\n6. The method relies on masking ratios (ra, ru) as key hyperparameters. These appear to be set as fixed percentages. This could lead to a situation where on a \"clean\" dataset, the model is forced to discard perfectly valid timesteps, or on a very \"noisy\" dataset, it fails to discard enough. How robust is the method for the choice of these ratios, especially across datasets with varying noise levels?\n7. The anomaly mask depends on a separate, lightweight estimation model (g(·; φ)). The paper shows that simpler models work better here, which makes sense to avoid overfitting in the estimator itself. However, the performance of the main model is now implicitly dependent on the quality and convergence of this secondary model. How does the performance of Selective Learning degrade if the estimation model g is poorly chosen or poorly trained? Is there a risk of a negative feedback loop where a bad estimator creates bad masks, which further hurts the main model?", "was_revised": true, "final_justification": "The authors have addressed my concerns in their response. Taking into account both the feedbacks from other reviewers and my own evaluation, I have decided to raise my score.", "rebuttal": "*We sincerely thank Reviewer c9bV for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n---\n\n1. **Concept Drift vs. Anomalies**\n\n   Thank you for this valuable comment. In this work, anomaly is defined as a timestep (or a range of timesteps) that significantly deviates from the predicted behavior, in line with domain convention [1]. We acknowledge that anomaly and concept drift often coexist; however, there are distinct differences between the two.\n\n   Concept drift and anomaly both originate from exogenous variable changes but differ in their scales: Concept drift typically reflects distributional shifts over periods (segment or instance level), while anomaly reflects deviations at the timestep level.\n\n   Therefore, anomaly does not necessarily indicate concept drift (as point anomaly may be insufficient to alter the overall distribution). On the other hand, drifted patterns typically contain anomalous timesteps (non-generalizable), but not all timesteps are necessarily non-generalizable.\n\n   We will include this comparison in the revised version to enhance clarity.\n\n   [1] Zamanzadeh Darban, Zahra, et al. \"Deep learning for time series anomaly detection: A survey.\" *ACM Computing Surveys* 57.1 (2024): 1-42.\n\n2. **Comparison with SIN**\n\n   Thank you for the thoughtful feedback. SIN is indeed an insightful study, but our research demonstrates significant differences in terms of:\n\n   * **Motivation.** Selective learning aims to mitigate model overfitting, while SIN seeks to improve heuristic normalization methods (e.g., RevIN) to better address distribution shift.\n   * **Technically**, SIN is a normalization method where the \"selective\" targets the statistics during normalization, enabling the model to preserve locally invariant yet globally variant statistics. Selective learning, on the other hand, is a training strategy where the \"selective\" focuses on the timesteps of time series, allowing the model to concentrate only on generalizable timesteps during optimization. The core technique of SIN is PLS-based covariance maximization, whereas selective learning employs a dual-masking mechanism to filter timesteps.\n\n   We provide an empirical comparison between selective learning and SIN in the table below. The backbone model is iTransformer and the prediction length is 336. Since SIN is not open-sourced, we reproduced it based on the details provided in the paper. The results in the table demonstrate that selective learning consistently outperforms SIN, proving the effectiveness of our work.\n\n   |        | Metric |   ETTh1   |   ETTm2   | Exchange  |\n   | ------ | :----: | :-------: | :-------: | :-------: |\n   | SIN    |  MSE   |   0.481   |   0.296   |   0.326   |\n   |        |  MAE   |   0.473   |   0.346   |   0.420   |\n   | **SL** |  MSE   | **0.431** | **0.270** | **0.305** |\n   |        |  MAE   | **0.432** | **0.325** | **0.407** |\n\n   We promise to incorporate a comparative analysis of this method in the revised version.\n\n3. **Differences with curriculum learning, hard example mining, and prioritized sampling**\n\n   Thank you for the valuable suggestion. The differences between selective learning and these three paradigms are as follows:\n\n   - *Curriculum learning* progressively increases difficulty (e.g., by expanding prediction lengths in time series tasks), as noted in Line 86.  In contrast, selective learning dynamically filters timesteps based on intrinsic data characteristics (uncertainty and anomalies), rather than relying on predetermined difficulty metrics (e.g., prediction length).\n   - *Hard example mining* and *Prioritized sampling* focus on sample-level selection, prioritizing informative or challenging instances. Our method differs in two critical aspects: (1) We operate at the timestep level, enabling finer-grained optimization of the time series forecasting model. (2) Rather than selecting based on importance or information content, we mask timesteps exhibiting non-generalizable patterns. This strategy aligns with the intrinsic properties of time series data, which are inherently susceptible to noise and anomalies.\n\n   We will incorporate a systematic review of these works and discuss the differences in the Related Work section of the revised version.\n\n4. **Empirical sensitivity analysis based on Theorem 1**\n\n   Thank you for the valuable suggestion. We conducted sensitivity analyses of the learning rate and batch size on the ETTh1 dataset with iTransformer as backbone. The results are shown in the tables below (prediction length is 336). \n\n   * *Learning rate*: An overly large learning rate (5e-3) can lead to poor estimation performance, while an overly small learning rate (1e-4) may cause the model to get stuck in a local optimum. **Overall, the model is robust to the increase in the learning rate.**\n   * *Batch size*: An overly small batch size (16) can lead to poor estimation performance, while an overly large batch size may also impair model performance (over-smoothed gradient). **Overall, the model is robust to the decrease in the batch size.**\n\n   We will include these analyses in the revised version.\n\n   | Learning rate | 1e-4  |   5e-4    | 1e-3  | 5e-3  |\n   | :-----------: | :---: | :-------: | :---: | :---: |\n   |      MSE      | 0.443 | **0.431** | 0.433 | 0.433 |\n   |      MAE      | 0.445 | **0.432** | 0.434 | 0.436 |\n\n   | Batch size |  16   |  32   |    64     |  128  |\n   | :--------: | :---: | :---: | :-------: | :---: |\n   |    MSE     | 0.433 | 0.433 | **0.431** | 0.436 |\n   |    MAE     | 0.437 | 0.434 | **0.432** | 0.439 |\n\n5. **Robustness of selective learning to the estimation model**\n\n   Thank you for raising this important concern. We notice that Questions 5 and 7 both concern the estimation model, and we address them together here.\n\n   As shown in Fig. 4, we evaluated the impact of different estimation models (including DLinear, MLP, TimeMixer, and iTransformer) on the final forecasting performance. The results demonstrate that **selective learning remains effective across a range of estimator complexities**, with only marginal variation in final model accuracy. This demonstrates that selective learning is robust to the choice of estimation model.\n\n   Additionally, to avoid instability from a poorly trained estimation model, we recommend training $g$ to full convergence before selective learning begins (Appendix B.3) and using a linear model to avoid overfitting. In this setting, experiments across all eight datasets (Tables 1&2) show **no performance degradation** attributable to the estimation model. This demonstrates the effectiveness of the current estimation setting in most scenarios (including high-noisy datasets such as ETT).\n\n  Due to the simplicity and generalizability of the linear estimation model, we believe the likelihood of such negative feedback loops is minimal in practice. Nonetheless, we will add an analysis in the revision discussing the robustness of selective learning.\n\n6. **Choice of masking ratios**\n\n   We acknowledge that the masking ratios are crucial hyperparameters in selective learning. However, for datasets with different characteristics (e.g., high-quality, noisy), we can search for appropriate masking ratios to achieve optimal results (see Appendix B.4 for details).\n\nTo facilitate this process, we provide empirically effective masking ratio settings that can serve as initialization points for tuning:\n\n   * $r_u=10$%, $r_a=10$% for stable and high-quality datasets.\n   * $r_a=30$%, $r_u=30$% for datasets containing certain levels of noise and anomalies.\n   * $r_a=90$% for highly-volatile dataset.\n\n   Noise and anomalies are often unevenly distributed within a dataset. To mitigate the risk of over-discarding clean data or under-discarding noisy data, we adopt a global masking strategy (Line 525), applying masking to all timesteps across samples at an appropriate masking ratio. Additionally, dynamic masking prevents excessive discarding of normal timesteps (Line 576). When a normal timestep is mistakenly discarded in one epoch, it can likely be learned in subsequent epochs.\n\n---\n\n*Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.*"}, {"review_id": "9JmRdhiu5J", "reviewer": "Reviewer_93xJ", "summary": "The work proposes a novel selective learning strategy that dynamically ignores spurious timesteps by applying two masks. The uncertainty mask removes any step showing unusually high entropy. Additionally to it the anomaly mask filters out observations that fall outside the lower‑bound forecast generated by a linear model $g$. Experiments show the improved performance and large applicability to several modern TSF approaches.", "questions": "- Have the authors tried non-gaussian residuals? \n- Both masks rely on noisy residuals before the backbone is warm. The authors mention a warm-up phase, can the authors give more details on that? \n- Have the authors tried a held-out validation set to train model $g$? I asked because looks a bit problematic to use the same training set for that too.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- Well written and structured work.\n- Easy integration approach, can be plugged in any TSF regression model.   \n- Interesting idea on filtering timesteps based on uncertainty and anomaly detection. \n\nWeaknesses: \n\n- Entropy mask depends on a unimodal residual fit Gaussian. How SL will behave under a mixture or non-parametric fit?\n- Early uncertainty estimation looks a limitation of the approach.", "was_revised": true, "final_justification": "I thank you the authors the answers to my concerns. I will keep my score.", "rebuttal": "*We sincerely thank Reviewer 93xJ for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n---\n\n1. **Non-Gaussian, mixture, or non-parametric fit of residual in the uncertainty mask**\n\n   Thank you for the valuable feedback. We adopt the Gaussian distribution primarily for computational tractability (since entropy becomes proportional to variance in this case, Eq. 8). Other likelihood models and estimation methods are also theoretically viable. However, in practice, we need to estimate the distribution of *each timestep* in the time series. Using complex distributions (e.g., mixture distributions) or estimation methods (non-parametric methods like KDE) can introduce significant computational overhead. In contrast, under the Gaussian likelihood model, we only need to compute the variance for each timestep in parallel. As shown in Table 1, experiments on eight real-world datasets demonstrate that the Gaussian likelihood model delivers stable performance improvements and is suitable for the vast majority of scenarios.\n\n2. **Early uncertainty estimation and warm-up phase.**\n\n   Thank you for pointing out this important issue. Conducting uncertainty estimation too early may indeed lead to larger estimation errors. Since the training loss of time series forecasting models typically exhibits its most dramatic decline in the first epoch, we skip uncertainty masking during this initial epoch to avoid introducing excessive variability to the residuals due to training, as outlined in Algorithm 1. In practical implementation, the number of warm-up epochs for the backbone model before applying uncertainty masking can be determined based on the model's convergence behavior.\n\n3. **Have the authors tried a held-out validation set to train the model?** \n\n   Thank you for the insightful comment. Some existing works indeed employ held-out sets to train auxiliary models [1], primarily to prevent overfitting in the model. In this paper, we employ DLinear (sufficiently simple) as the estimation model $g$ to prevent overfitting on the training set (Line 540), thereby avoiding any degradation in the performance of the main model. Additionally, unlike other data modalities, the patterns in time-series data often change over time. Consequently, using a holdout set may lead to underfitting of certain patterns in $g$, particularly for datasets with strong non-stationarity.\n\n   In the table below, we present a comparative analysis of two sets of results, both from iTransformer on the ETTh1 dataset. The key difference lies in the training data for $g$. The comparative results show that Exp.1 achieves better performance, which validate our claim.\n\n   We will include this analysis in the revised version.\n\n   | ID   | g trained on      |    MSE    |    MAE    |\n   | ---- | ----------------- | :-------: | :-------: |\n   | 1    | full training set | **0.415** | **0.425** |\n   | 2    | 20% held-out set  |   0.425   |   0.437   |\n\n   [1] Mindermann, S., et al. \"Prioritized Training on Points that are Learnable, Worth Learning, and not yet Learnt\". <i>Proceedings of the 39th International Conference on Machine Learning</i>. 2022.\n\n---\n\n*Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.*"}, {"review_id": "HcLgxoNZPR", "reviewer": "Reviewer_vVVG", "summary": "Deep learning models for time series forecasting suffer from overfitting because all future steps, including noisy and abnormal ones, are taken into consideration by the loss function. This work proposes a dual-mask mechanism to filter out noise and anomalies. Noise is filtered by an uncertainty mask derived from residual entropy. Anomalies are filtered by an anomaly mask derived by comparing the residual with that of a pretrained estimation model. Experiments show that the proposed approach can improve the predictive performance of deep models.", "questions": "- If assuming that the dataset contains a significant amount of noise and anomalies, does the input side also need adjustment, rather than just at the output end?\n- The current approach always filters some timestamps even when the dataset is clean. Is there some way to handle this?\n- See my questions in the #Weaknesses Section.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The idea of using selective learning for TS forecasting is interesting and novel.\n- Extensive experiments were conducted to validate the effectiveness of the proposed approach.\n- The proposed approach is agnoistic to backbone and can be used as a plug-and-play module.\n\n### Weaknesses\n- Some derivations are unclear, for example:\n  1. For the uncertainty mask, the inclusion of entropy is confusing. It's hard for me to understand the role of Equations 6 and 7. Moreover, $\\epsilon^{(i)} = 100, \\forall i$, this timestamp has an entropy of 0 accroading to the defination.\n  2. For anomaly mask, also using theoretical lower bound is reasonable, the rational of using another estimation model to detect anomalies needs more explanations, considering lightweight weak models are used here.\n\n- The proposed approach seems to be sensitive to the hyper-parameter masking ratio, which may limit its application:\n  1. Best performance is achieved only masking ratios of uncertainty and anomaly masks are jointly tuned, tuning one of them can not get the optimal performance (Figure 3).\n  2. Different datasets require different ratios (Table 6).\n\n- Other than runing time estimation in Table 7, detailed theoretical computational efficiency analysis is required.", "was_revised": true, "final_justification": "Overall, I recommend accepting this paper and will maintain my initial score of 4. My concerns about computational complexity and input adjustment are addressed. However, there are a few issues that should be addressed or at least discussed in the final version:​​\n\n​The incorporation of entropy:​​ Equations 8–9 currently measure the amplitude of residuals rather than their uncertainty. According to the formal definition of entropy, if all residuals are significant but concentrated, the entropy should be small—contrary to the current measurement. This issue is caused by the assumption of a zero-centered distribution, which is unreasonable, especially when the model is not converged.\n​Hyperparameter Sensitivity:​​ The model’s performance is highly sensitive to specific hyperparameters, particularly the uncertainty and anomaly masking ratios.\nClean Data Integration:​​ The current approach does not account for scenarios with clean data, which is another manifestation of hyperparameter sensitivity.", "rebuttal": "*We sincerely thank Reviewer vVVG for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n---\n\n1. **Clarifications of the dual-mask mechanism**\n\n   **(a) Uncertainty mask**\n\n   In the uncertainty mask, we use entropy as the measurement of uncertainty. The entropy $H(\\epsilon_t)$ quantifies the *predictive uncertainty* of timestep $t$. High entropy indicates unstable predictions across different sliding windows (i.e., the model’s output varies significantly for the same $t$ under different historical contexts).\n\n   **Eq. 6-7 Purpose**: Since calculating entropy using Eq.5 requires the distribution of residuals, we estimate this distribution through Eq.6 with given samples (the likelihood model $l(\\boldsymbol{\\psi}|\\epsilon_t)=p(\\epsilon_t|\\boldsymbol{\\psi})$). And Eq. 7 computes the entropy based on $l(\\boldsymbol{\\psi}|\\epsilon_t)$. We assume $l(\\sigma_t|\\epsilon_t)=\\mathcal N(0,\\sigma_t^2)$ for tractability, but other likelihood models could be substituted.\n\n   Our use of Eq. 8 to estimate entropy is based on the assumption that the residuals follow a Gaussian distribution. If all sampled residuals are the same, the estimated entropy would indeed be zero. However, this scenario is highly unlikely with a large sample size. Additionally, we only use the relative magnitude of residual entropy to guide the masking, while the absolute value of the entropy estimation carries no significance.\n\n   **(b) Anomaly mask**\n\n   We adopt another lightweight model as the estimator for two reasons: (1) the theoretical residual lower bound is unattainable, and (2) using the model itself as the estimator yields suboptimal performance (as shown in Figure 4). This is because complex models may have already overfitted during training, making their residual lower bounds ineffective for guiding the masking process. In contrast, using a lightweight model (e.g., DLinear) helps prevent overfitting in the estimation model.\n\n2. **Masking ratio tuning**\n\n   Thank you for your valuable comment. We agree that joint tuning of uncertainty and anomaly masking ratios is necessary for optimal performance (Fig. 3).\n\n   However, we recommend a three-stage search protocol in Appendix B.4 which reduces the search space and is computationally feasible: (1) optimize the uncertainty masking ratio to achieve peak performance; (2) fix the uncertainty masking ratio and tune the anomaly masking ratio to its optimal value; (3) conduct a local hyperparameter search within the neighborhood of these determined masking ratios.\n\n   Additionally, specific masking ratios may also be attempted as initial values:\n\n   * $r_u=10$%, $r_a=10$% for stable and high-quality datasets.\n   * $r_a=30$%, $r_u=30$% for datasets containing certain levels of noise and anomalies.\n   * $r_a=90$% for highly-volatile dataset.\n\n3. **Theoretical computational efficiency analysis**\n\n   Thank you for your kind suggestion. Beyond empirical runtime, we provide a theoretical analysis:\n\n   * *Time Complexity*: Let $B$ be the batch size.\n     * Uncertainty Mask: Residual entropy updates cost $\\mathcal O(BFN)$ per epoch.\n     * Anomaly Mask: The complexity of the anomaly mask depends on the architecture of the estimation model. Taking a linear model as an example, the forward pass of the estimation model requires $\\mathcal O(BLN)$ complexity, and the masking process adds $\\mathcal O(BFN)$. Therefore, the complexity of the anomaly mask is $\\mathcal O(B(L+F)N)$.\n   * *Memory*: Storing residuals for $T$ timesteps requires $\\mathcal O(TFN)$ space.\n\n   Overall, there exists a trade-off between the overhead and performance gain. The overhead (1.6–12.2s/epoch) is justified by significant MSE reductions (up to 66.8%). We will include this analysis in the revised version.\n\n4. **Does the input side also need adjustment?**\n\n   We appreciate the reviewer's thoughtful comment. Selective learning masks only the output during optimization to steer the model's focus toward generalizable patterns, while keeping the input end unmasked. This design stems from two key considerations: (1) First, since noise and anomalies are inevitable in real-world time series during inference, masking the input during training would undermine model robustness. In contrast, some prior works employ data augmentation (e.g., noise injection) [1] to improve robustness. (2) Second, output (can be viewed as labels) masking excludes non-generalizable patterns from loss computation, therefore enhancing predictive performance.\n\n   This demonstrates an asymmetric treatment between input and output processing: for inputs, we preserve (and even increase) noise to enhance model robustness, whereas for outputs, we aim to reduce noise to prevent overfitting in loss computation.\n\n5. **Handling clean dataset**\n\n   Thank you for raising this important issue. For theoretically clean datasets where all timesteps are generalizable, omitting masking may be a good choice. However, time series data typically contains inherent redundancy [2], such that moderate masking ratios generally do not compromise model performance.\n\n   Moreover, real-world datasets typically contain varying levels of noise and anomalies, which are the primary cause of model overfitting and the underlying rationale for the effectiveness of selective learning. For high-quality datasets with stable patterns, selecting a relatively small masking ratio can achieve better prediction performance.\n\n---\n\n**Reference:**\n\n[1] Iglesias, Guillermo, et al. \"Data augmentation techniques in time series domain: a survey and taxonomy.\" *Neural Computing and Applications* 35.14 (2023): 10123-10145.\n\n[2] Lin, Shengsheng, et al. \"SparseTSF: modeling long-term time series forecasting with 1k parameters.\" *Proceedings of the 41st International Conference on Machine Learning*. 2024.\n\n---\n\n*Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.*"}]

Author rebuttal:

Meta-review: This well-written paper proposes a new selective learning approach to training time series forecasting models that smartly focuses on more generalizable parts of the training data. It has been evaluated by 4 knowledgeable reviewers who all agreed in their final evaluations that it is acceptable for NeurIPS (2 straight accept scores, 2 marginal accepts), despite noted weaknesses (most of which the authors have managed to address in the rebuttal and discussion with the reviewers). The authors are strongly encouraged to carefully reflect on the feedback received in the final revision of the paper.
Decision: Accept (poster)