Cascaded Teaching Transformers with Data Reweighting for Long Sequence Time-series Forecasting
Authors: Haoyi Zhou
Chonghan Gao
Pengtao Xie
Jianxin Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Teaching
Reweight
Time-series
Forecast

Abstract: The Transformer-based models have shown superior performance in the long sequence time-series forecasting problem. The sparsity assumption on self-attention dot-product reveals that not all inputs are equally significant for Transformers. Instead of implicitly utilizing weighted time-series, we build a new learning framework by cascaded teaching Transformers to reweight samples. We formulate the framework as a multi-level optimization and design three different dataset-weight generators. We perform extensive experiments on five datasets, which shows that our proposed method could significantly outperform the SOTA Transformers.

Under review as a conference paper at ICLR 2023
CASCADED TEACHING TRANSFORMERS WITH DATA REWEIGHTING FOR LONG SEQUENCE TIME-SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
The Transformer-basee models have shown superior performance in the long se- quence time-series forecasting problem. The sparsity assumption self-attention dot-product reveals that not all inputs are equally significant for Transformers. In- stead of implicitly utilizing weighted time-series, we build new learning frame- work by cascaded teaching Transformers to reweight samples. We formulate the framework as a multi-level optimization and design three different weight generators. We perform extensive experiments on five datasets, which shows that our proposed method could significantly outperform the SOTA Transformers.
INTRODUCTION
The long sequence time-series forecasting (LSTF) has drawn particular attention with Transformer- based models, like electricity prediction Li al. (2019), financial predictions Zhang et al.) (2022). and weather predictions Pan et al. (2022). The pairwise self-attention allows time-series points to directly attend to each other, which benefits the future forecasting from the historical observations. The pairwise computation's 's sparsity assumption Child et al.] (2019) reveals that not every sample equally prior. If we drop unnecessary pairwise connections without violating problem settings, we can acquire stronger model with better generalization. For example, (i) Reformer Kitaev et al. (2020) uses the hashing bucket to select important query-key pairs (ii) LogSparse Transformer Li et al. (2019) only calculates attention pairs lying on the log-size step away from the diagonal. The sparsity assumption could be roughly considered as manipulating the weights within time-series, or more specifically, internal reweighting.
A key technical challenge preventing us from further performance improvement that solely relies on internal reweighting is the unavailability of specific design of time-series weights under the sparsity-oriented framework. Instead, we can perform the reweighting explicitly, that is, reweight the whole train dataset that the outlier samples can be excluded from the training. Meanwhile, assign larger dutaset-weightt to those samples belonging to the main patterns of the dataset. Inspired by the knowledge distillation Hinton et (2015) and teacher-student learning Li et al. (2014). we can teach the Transformer to learn to weighted time-series. One common method is using pseudo labels labels/Pham Pham et al. (202T) to allow student learning from teacher outputs. If we cannot assign accurate labels, the student hardly learns as well as the teacher does, phenomenon known as confirmation bias. To alleviate this drawback, we propose to reweight the inputs in a soft manner.
In this paper, we design a cascaded teaching framework. There is sequence of Transformer models, where model i teaches model + 1. Specifically, model generates pseudo time-series dataset, which is used to train model i + 1. Only the first teacher model uses the reweighted dataset, whereas other models use the pseudo time-series dataset generated by the previous model. Finally, the teacher model will update its dataset- dataset-weights weights based on the performance of student models. The experimental results show that we can significantly improve the performance a time-series prediction model by simply reweighting the dataset while maintaining the sparsity assumption the same time. More importantly, this cascaded learning framework is easily generalized to other seq-to-seq models.
The contributions summarized follows:

---

Under review as a conference paper at ICLR 2023
We propose cascaded teaching framework reweight the teacher model's dataset based on the evaluation of student models, which forces the teacher model to get rid of noise data by
generating proper weights..
We design three dataset-wiight generators to compress the trainable weigstt- parameters. Extensive experiments on three datasets (five cases) have demonstrated its improvement in time- series learning.
2
RELATED WORK
2.1 TEACHER-STUDENT LEARNING
Teacher-student learning has been investigated in knowledge distillation Hinton et al. (2013), adver- sarial robustnes: & Wagner (2017). self-supervised learningXie et al. (2020). etc. Most of these methods are based dent model with the help of a trained and fixed teacher model. In these works, the teacher model is not updated. In contrast, model. The teacher model constantly updates itself based on the teaching outcome. Teacher-student learning has been investigated in several neural architecture search works as well Li al. (2020); Trofimov et al. (2:TT); Gu & Tresp (2020). In these works, when searching the architecture of a student model, pseudo- labels generated by trained teacher model whose architecture is fixed are leveraged. Our work differs from these works in that we focus on searching the architettcc of a teacher model by letting it teach a student model where the student's 's dataset- fixed, whereas the existing works focus on searching the architecture of student model by letting it be taught by a teacher where the teacher's architecture fixed. In a recent work Pham et al. (2021) which was conducted independently of and in parallel to our work, the teacher model is updated based on student's ' performance. Our work differs from this one in that our work is based on a three-level optimization framework which searches teacher's architecture by minimizing student's validation loss and trains teacher's network parameters before using teacher to generate pseudo- labels, whereas in Pham al.] (202T) framework is based on two-level optimization which has no architecture search and does not train the teacher before using it to perform pseudo-labeling. In Such et al.] (2019), a meta-learning method developed learn a deep generative model which generates synthetic labeled data. A student model leverages the synthesized data to search its archi- tecture. Our work differs from this method in that we focus on searching the teacher's architecture via three-level optimization, while Such et al. (2099) focuses on searching the student's 's architecture via meta-learning.
escudo-labeling. In these existing methods, the focus is to learn stu-
method focuses learning a teacher model by letting it teach student
2.2 REWEIGHTING TIME-SERIES
Reweighting is a technique widely used the field of time series forecasting, but the objects of weighting diverse: features Zhao et al. (2018), fuzzy relationship Yu] (005), gradient [Zhang et al. (2019) etc. In addition to time series forecasting, reweighting is also used to periodic pattern mining Chanda et al. (017)), fault diagnosis [L al. (2017) and time-series classification (ellami & Hwang (2019). In these works, only one model is trained to generate weights on the corresponding components of the network. The difference between our method and the above works is that the dataset-weight trained by our framework are not coupled with the model, so they can be applied to other models trained on the same dataset. The weights for time-series can also be extracted with a Bayesian non-pan -on-paraettiic way /aad & Mansinghkal (2018) or even from the dataset dynamics Zhang jtt al. (202T). Our framework differs from these works in that our model reweights the entire training dataset by introducing cacherr-studet learning.
TEACHER- -STUERTTT LEARNING FRAMEWORK
Notions We start with a teacher model with parameters and a student model with parameters S. We feed the teacher model with training dataset Df = {(s J. where s, denotes the i-th input serie and t; is the corresponding outputs. Following the multi- task learning paradigm Caruanal (1997), we assign different dataset- p; e [o, 1] for each input samples (s,, tz), which forms the dataset- daightts P {p:) In order to avoid introducing prior knowledge, we initialize
2

---

Under review as a conference paper at ICLR 2023
Train Dosa (rguo)
Tann D01a (noud
(liput)
Teacher Model weights
Teacher Model weightt
Student Model weights 5
Stadent Model weigles S"
Update
Irann Dataa
Irain (oreciction)
Data (anesiction)
Loss
volid Data (predition)
Final
(prediction)
loss
Doraset
Trah
Train
(pround trucN)
Iground truth)
(psoudo laseled)
Iground truth)
The teacher model performs one-step gradient descent with weighted loss. Middle: We the teacher model with updated parameters T to generate pseudo labels and mix them with ground truth. Right: The student models perform one- step gradient descent on the pseule-labeled dataset. Then use the validation loss of the student models to update the dataset-weights.
Figure Cascaded Teaching Framework. Left:
P(A) to 1, so that all samples are treated equally in the beginning. Here we want to emphasize that the reweighting objective (s,, t,) is not single time point, but complete time-serie sample which contains several time points, depending on the input/ouput length setting. Therefore, the dataset weights have effect the multi-head attention mechanism.
Without loss of generality, we assume the end task is time-series prediction. Our proposed cascaded teaching strategy could be divided into three stages. Firstly, train Transformer as the teacher model 7 from the reweighted time-series. The training process could be formulated as:
N (A) def min P(T, ((m) min (A) s,,
where the wetiset-weights P(A) (p((A)} E [0, 11 are generated by group of parameters A {a RLa, where LA represents the length of dataset-weights parameters A. If p; is close 0, which means that (s,, t,) is important, the corresponding loss (r(A)((T, s t.) is reduced to O, which means that (s;,t) (4) is excluded from training like holds a sparsity assumption. The parameter A is fixed during this step. Otherwise, a trivial solution will be yielded where P are all set to zero. The optimally trained (A) is function of A since they are functions of the reweighted loss, which is also function of the parameters A.
Secondly, we use the previously learned T (A) to generate pseudo time-series dataset. We use another time-series dataset (unl)) {s without labels, where L,, represents the length of (unll) For each 8, E D(unt), we use 7 (AA to predict the possible outputs f, Then get pseudo- labeled time-series dataset (ipse) (T* = ((s;, f2) where I.p represents the length of ((sse) We use this pseudo-labeled time-series dataset to train another Transformer student model S. The training process is minimizing the following loss:
S (A)) min( D!! (1 T)LS (T=)))(T*))
where y E [o. 1] denotes the self-study rate. If is close to 1, then student model will study mainly on its own without consulting the "handouts" Dirse) prepared by its teacher. Here we can use the input of Dimm as the unlabeled dataset for simplicity. The optimal parameters S* (T* (T*(A)) are functions of T (A).
3

---

Under review as a conference paper at ICLR 2023
Thirdly, apply 5" (A)) to validate D(val), We update A by minimizing the validation loss. Putting all the pieces together, we get the following optimization framework:
min (AS (
s.t. (A)) min(y D!! + LSS, ((Pe))
(3)
8
(A) min) L(T, D!
In this framework, there are three optimization problems. From bottom up, the three optimization problems correspond to learning stage 1, 2, and respectively. The first two optimization problems are nested on the constraint of the third optimization problem. These three stages are conducted end-to-end in this unified framework. The solution T" (A) obtained in the first stage is used to create a pseudo time-series dataset the second stage. The time-series prediction model trained in the second stage is used to make predictions in the third stage. The parameters of the wetgset-weights generator A updated in the third stage changes the training loss in the first stage and consequently changes the solution T (A), which subsequently changes S (A)).
3.1
OPTIMIZATION ALGORITHM
In this section, develop gradient-
gradient-based based optimization algorithm solve the problem defined in 3 We approximate T (A) using one-step gradient descend w.r.t L(T, P(A), D(T)):
(*) = Te LT,, ((m))
(4)
Then we plug T" into L(S, (T*)) and get an approximated objective. S (A)) using one-step gradient descent w.r.t the approximated objective:
We approximate
S (T)) = S" = n. L
77 VSL(S, (T*)
(5)
Finally, we plug S" into the validation loss and get an approximated objective. Then we update A through the gradient descent:
A - A- na ATT(S (val)) ((al))
(6)
where
0T' aDisee as' dL (5' ((va))
V VAL(S, Dival)
0A OT @Dise) as'
anpss (1 v
(pee) HV L ((A), ((
The second-order term 2 Dive . LS, L Dival) can approximated by finite
difference Hessian matrix H S = 5 L(S", D(val)) e stands for a small number.
| Diwe L(S Dime)) V Diva L(S [(pee)]]/e.. Note that
3.2 DATA REWIIGHHEGG GENERATION
Recall that the weigstt P(A) are generated with learnable parameters A. The proposed framework allows to provide generator in various forms. Within the conventional sigmoid activation function a, we can use different dataset-weights generators to deal with different types of data noise, especially for long sequence time-series forecasting. Here, proposed three options:
(a) Identity function, pt ((a))
N/2
(i aj+NN22) )
(b) Normal distribution, P:
exp
v 2mma
2a²
N/26
N/b
2
2 aj cos
(c) Fourier series, =
Σ
aj
sin
Σ
N
N
1=1
(26+11)
4

---

Under review as a conference paper at ICLR 2023
1.0
ETTh1
www.wlwllwowwwwiaannnanmmm Manoonn M "ndnmnn
10
0.5
0.0
Fourier
www.wolsswolossssaaalaaomm Mywdaaannllllgaaaaaaoaoooaaooooa Mown nannmna
0.0
1.0
Normal
10-
0.5
0.0
1.0
Identity
10-
0.0
200
600
1000
1200
1600
Figure 2: ETTh1 dataset reweighted by the proposed three dataset-weight generators. The lighter
colors indicates lower detaset- that is, the targeted noise.
Figure presents sample of ETThl dataset reweighted by the proposed three aataset-weight gen- erators. T'ee line with lighter color indicates that the training data sampled from this area acquire lower weights (from one to zero). We can whiteness two noise areas for long sequence forecasting, around 700-800 and 1400-1500. Among the three methods, the Fourier generator has captured both data outliers with its decomposition in the frequency domain. The normal generator cannot filter outliers in fine- enn-inr manner, while the identity generator cannot generate highly discriminating weights due to training sparsity sparse training problem, where each parameter in A is trained only once every epoch. For most of the time-series Transformer converging within several epochs, A is unable to be fully trained. After the empirical evaluation in experiments, the last dataset-weights generator becomes the default option.
CASCADED TEACHING
In this section, propose generalized cascaded teaching approach for time-series prediction. There is sequence of models: I,..., K. The first model is a teacher. The K-th model is a student. For any other model, it is both teacher and student. Model teaches model + 1. The teaching mechanism is the same as described in Section 3 Given model i, we use to generate pseudo time-series dataset, then use this pseudo dataset to train model
In our framework, there are K + 1 learning stages. For 1 < k < K, the k-th learning stage corresponds to train the k-th model. The (K + 1)-th learning stage corresponds
train A via model validation. In the first stage, we train the network Ti (including encoder and decoder) of the first model by solving the following problem:
(A) min L ((A), D(mil) Ti
(8)
where A contains parameters used to generate dataset-weights in training set D(tm) In the second stage, model 1 teaches model Given the optimal parameters T (A) of model 1, we apply it to generate a pseudo-labeled dataset (TP))(TTT) in the same way described in Section 3 Then we use ((P=e((T+)) to train the network parameters Tr of model 2 by solving the following optimization problem:
(T) = min T. (T2, ((nse) (T?))
T:2
In the k-th stage, model A teaches Let (-.
denote the optimal parame-
ters of model & 1 trained at stage k: 1. We use it to generate pseudo-labeled dataset Dipse) (Tī (... (A))) and use the pseudo-labeled dataset to train the parameters Tę of the model k. This amounts solving the following optimization problem:
T min (Tr, (T (A)))) Ts
(10)
The process continues until all K models are trained. At the (K + 1) stage, we perform validation of these trained models the validation dataset and learn the architecture A of the first model by
5

---

Under review as a conference paper at ICLR 2023
Table 1: Multivariate long sequence time-series forecasting results on two datasets (four cases).
Methods
Cas-Informer
Informer
Cus-Query-Selector
Query-Selector
Role
Student
Teacher
Baseline
Student
Teacher
Baseline
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.402
0.448
0.430
0.471
0,6012
0.578
0.384
0.432
0.391
0.439
01423
0.462
E17Tb
0.478
0.499
0.534
0.718
0.425
0.460
0.423
0.458
168
0.768
0.681
0.784
0.768
1.001
0.797
0.641
0.588
0.651
0.591
0.684
0.609
336
0.999
0.752
1.002
0.758
1.304
0.938
0.786
0.678
0.805
0.688
0.850
0.704
720
1.073
0.829
1.117
0.846
1.201
0.902
1.012
0.832
1.024
0.835
1.115
0.843
24
0.613
0.626
0.609
0.611
1.792
1.078
01,088
0.491
0.391
0.481
0.412
0.486
ETTh2
48
0.974
0.796
1.089
0.831
2.099
1.168
0.790
0.704
0.784
0.702
1.407
0.932
1.959
1.185
2.157
1.203
7.376
2.365
1.145
0,855
1.125
0.839
1.739
1.013
336
2.195
1,223
2.197
1.229
4,466
1.779
2.195
1.075
2.154
1.040
2.317
1.186
720
2.532
1.342
2.481
1.286
3.326
1.516
2.767
1.399
2.610
1.324
3.066
1.308
0.268
0.341
0.346
0.320
0.385
0.300
0.364
0.313
0,410
0.335
0.388
ETTmm
48
0,389
0.440
0.378
0.426
0.461
0.466
0.440
0.458
0.440
0.453
0,473
0.470
96
01.456
0.481
0.439
0.467
0.621
0.567
0.,223
0.457
0.413
0.445
0.454
0.483
288
0.607
0.561
0.581
0.541
0,921
0.743
0.482
0.494
0.469
0.489
0.619
0.599
672
0,648
0.592
0.626
0.572
1,045
0.790
0,602
0,569
0.593
0,563
1.127
0.841
24
0,314
0.367
0,310
0.365
0.325
0.371
0,303
0.358
0.298
0,353
0.332
0.379
Weather
0.409
0.391
0,405
0.347
168
0..96
0.506
0.472
0.489
0,568
0.557
0.487
0.504
0.479
0.498
0.601
0.562
336
0.567
0.558
0.551
0.549
0.624
0.581
0.521
0.530
0.513
0.523
0.686
0.610
720
0,597
0.582
0.603
0.583
0.633
0.592
0,570
0.567
0.564
0.562
0.858
0.752
96
0.632
0.337
0.637
0.341
0.722
0.343
0.694
0.356
0.689
0.348
0.753
0.364
Traffic
192
0.623
0.334
0.633
0,346
0.708
0,383
0,700
0.361
0.694
0.356
0.763
0.381
336
0.649
0.349
0.646
0.347
0.772
0.410
0.725
0.373
0.713
0.367
0.796
0.394
720
0.676
0.394
0.671
0.389
0.874
0,470
0.774
0,388
0.755
0.374
0.870
0.409
Count
26
22
0
10
38
0
minimizing the validation loss. The corresponding optimization problem is:
K
min L (TT D(al) + Σ L
(Tr,
(11)
A
Where A stands for the cascaded teaching rate. Putting these pieces together, we get the following multi-level optimization problem:
K
min L (A), P(A), D(vat) +
Σ
L (T, D(val)
s.l
Tk = min T. (TK, D(pse) D (Tk-1))
(12)
TV = min (T2, (pse)) (TT (IPS)) Tz
T min (Ti, P(A), ((m)
From bottom to top, the K + optimization problems correspond to the learning stage 1,... K respectively. The first K optimization problems are on the constraints of the (K+1)-tt optimization problem. The K stages are performed end-to-end in a joint manner where different stages mutually influence each other. The detailed optimization algorithm can be found in Appendix.
5 EXPERIMENT
We performe experiments on 5 datasets: ETT (ETThl, ETTh2, ETTml), WTH and Traffic. Since the framework we proposed only modified the weights on dataset, it can be applied to most time series prediction models under the end-to-end deep learning paradigm. To compare with the SOTA methods, we adopt three Transformer models: Informer Zhou et al. (202T), Query-SelectooKlimel et al. (202T), Reformer Kitaev al. (2020), and CNN-based model SCINe et al. (021)) The

---

Under review as a conference paper at ICLR 2023
last two models presented in the Appendix C.1 to show the generalizability of the proposed framework. We also present a comprehensive ablation study to test the effectiveness of the dif- ferent parts of our framework. More details about parameter sensitivity will also be included in Appendix C.3
5.1 DATASETS
ETT (Electricity Transformer Temperature) provided by Zhou et al. (2021) records 2-year ETT data from two separated counties in China. The granularity of (ETTh1, ETTh2] and ETTml is
respectively hour and 15 minutes. Each data point contains 7 channels: oil temperature as target value and power load features.
WTH (Weather) contains hourly local climatological data from 2010 to 2013 for around 1600 US locations. Each data point consists of the target value "wet bulb" and other climate features. Traffic is collection of hourly data from the California Department of Transportation that describes road occupancy measured by different sensors on highways in the San Francisco Bay Area.
5.2 EXPERIMENTAL DETAILS
We perform extensive experiments on InformetZhou et al. (202T) and Query-Selector Klimek et al. (202T). Hyperparameter By default, the input length the encoder is set to 96, and the input length of the decoder is set to 48 for Informer. The other hyperparameters are consistent with the two original papers. Due to limited computing resources, the learner number is set to 2, which follows the canonical teacher-student learning paradigm. The self-study rate 7 is set to 0.2 so that the student model relies mainly on the quality of the pseudo-labeled dataset. Platform: All the models were trained/tested on 2 Nvidia V100 32GB GPU. More information about network setup can be found in the Appendix
5.3 RESULTS AND ANALYSES
The main results are summarized in table There are two comparisons of cascaded teaching's performance on Informer and Query-Selector separately. The winners are in boldface. we can see in the last row of Table [] the usage of the cascading framework outperforms the baselines in all datasets, and the improvement is more significant on ETT datasets (improved 66% in ETTh, at most). For the Informer, the student model performs slightly better than the teacher (Count 26 > 22) The goal of training helps to reduce the student's validation loss. For the Query-Selector, the student model performs worse than the teacher (Count 10 38), but their metrics very close. The different gap derives from the strong inputs' sparsity assumption in the Query-Selector model because the selected queries have weakened the effect of reweighting. For the fine-grained (FTTm1). the improvement is less obvious than hourly dataset, which indicates that reweighting is more suitable for longer inputs. is also worth noting that the performance improvement brought by the cascading framework does not decrease significantly with the increase in the prediction length, which reveals its potential for the prediction of longer sequences.
5.4 ABLATION STUDY
We design two individual ablation studies to further validate the effectiveness of the different com- ponents of the framework: detaset-weight, teacher model and student model.
5.4.1 ABLATION STUDY FOR DATASET-WEIGHTH
Unlike conventional teacher-student learning, our framework introduces the reweighiiiggt mech- anism to find valuable time-series samples. Without data-reweighting, the cascaded framework will degenerate into knowledge distillation and perform similarly to the baseline model. We perform ex- periments Cas-Informer and KD-Informer (Informer with Knowledge Distillation) Hinton et al. 2015).. The KD-Informer simply sets (A) = and removes the part of updating in the op- timization algorithm. The experimental results summarized in Table 2 As performance of the model trained by the knowledge distillation framework is consistent with the baseline. We believe the main reason is that the performance of the student model mainly depends
see, the

---

Under review as a conference paper at ICLR 2023
Table 2: Experimental results on ETT dataset with Iaf-merrrm,r KD- Informer and Informer.
Teacher Model
Methods
Cas-Informer
KD-Informer
Informer
Student Model
Role
Student
Student
Baseline
1.00
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE 2000
24
0.402
0.448
0.623
0.591
0.602
0.578
ET1h
48
0,478
0.498
0.732
0.655
0.718
0.647
0.90
168
0.768
0.681
1.134
0.812
1.001
0.797
336
0.999
0.752
1.411
0.949
1.304
0.938
0.85
720
1.073
0.829
1.317
0.926
1,201
0.902
24
0.613
0.626
1.806
1.124
1.792
1.078
0.80
48
0.974
0.796
2.147
1.192
2.099
1.168
168
1.959
1.185
7.625
2.528
7.376
2.365
0.75
2
336
2.195
1,839
4.466
1.779
Epoch
720
2,532
1.342
3.412
1,530
3.326
1,516
24
0.268
0.341
0.312
0.375
0.320
0.385
Figure 3: MSE score on test
ElTmm
48
0.389
0.440
0.466
0.468
0.461
0.466
dataset during training.
96
0.456
0.481
0.617
0.553
0.621
0.567
288
0.607
0,561
0.942
0.762
0.921
0.743
672
0.648
0.592
1.080
0.826
1.045
0.790
Table 3: Ablation study of student model's size on ETThI.
Cas-informer
Informer
Student
Teacher
Baseline
Heads
Dim
MSE
MAE
Heads
Dim
MSE
MAE
Heads
Dim
MSE
MAE
512
0.768
0.681
8
512
0.784
0.690
512
0.931
0.752
448
0.803
0.702
8
512
0.826
0.719
448
0.971
0.771
384
1.022
0.795
8
512
0.875
0.729
384
1.063
0.817
256
1.037
0.808
512
0.931
0.781
256
1.011
0.796
0.906
0.721
8
512
0.811
0.700
192
1.041
0.814
128
0.957
0.748
8
512
0.852
0.733
128
1.003
0.782
64
0.967
0.751
8 512
0.833
0.716
64
1.067
0.824
We have the prediction length 168 during experiments.
on the quality of the pseudo label generated by the teacher model, so its MSE score is close to the teacher model. The knowledge distillation framework does not improve the performance of the teacher model. Therefore it has no significant improvement compared the baseline.
5.4.2 ABLATION STUDY FOR TEACHER-STUDENT STUDENT MODEL
The teacher model generates pscudo labels to help the student learn through the dataset Díps), whereas the student model updates A by passing the partial gradient the teacher model. Therefore, the teacher model and student model could have different model sizes. We evaluate the changing of model size reducing the number heads and hidden dimensions. Table show that the teacher model can still maintain good performance when the size of the student model decreases signif- icantly, while the student model does not exhibit the same performance robustness to the teacher model size. This is because the teacher model can always improve its prediction ability by updat- ing A and Ti. Although the gradient quality of A will decrease significant improvement compared to the baseline. Conversely, the update of the student model is mainly affected by the quality of seeudo-lbbelin provided by the teacher model, and thus will have similar rate of performance degradation as the teacher model.
certain extent, there is still
5.5 PARAMETER SENSITIVITY
We perform analysis of parameter sensitivity 3 hyperparameters of the framework: the weight decay dw, the Fourier divider b, and the temperature the sigmoid function T.
Weight Decay In [(a) to the weight decay rate. Therefore a smaller dw is recommended to achieve stable performance on tasks with different prediction length.
that tasks with longer prediction length are sensitive
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation study of teacher model's size on ETTh1.
Cas-informer
Informer
Student
Teacher
Baseline
Heads
Dim
MSE
MAE
Heads
Dim
MSE
MAE
Heads
Dim
MSE
MAE
512
0.766
0.681
8
512
0.784
0.690
8
512
0.931
0.752
0.798
448
0.731
0.971
512
0.872
0.722
6
384
0.827
0.706
384
1.063
0.817
0.981
0.770
5
320
1.052
0.819
320
1.039
0.739
0.916
0.731
256
0.965
0,760
256
1.011
0.796
512
0.961
0.763
3
192
0.955
0.758
192
1.041
0.814
0.992
128
0.949
0.753
512
0.988
0.775
64
0.974
0.765
64
1.067
0.824
We have set the prediction length 168 during experiments.
NSE 1.1
0.5
Weigh dacaydecayooo
Fourer
Signold Temperature
Weight Decay.
Fourier Divider.
Sigmoid Temperature.
Figure 4: The parameter sensitivity of three hyperparameters in Cas-informer.
Fourier Divider This hyperparameter controls the amount of parameters in A. Figure 4(b) shows that the prediction task becomes more difficult, the framework requires more parameters of A to control the dataset- dataset-weights of each sample at a finer-grained level. Sigmoid Temperature In our paper, A larger sigmoid temperature 7 leads to a smoother derivative of the activation function a, which allows training of more selective for cong-sequence predictions.
sigmoid as activation function (xx) (1+e-Ta)
5.6 DISCUSSION
A detailed Analysis is given in Appendix B which theoretically shows that the dataset weights directly control the magnitude of the student model's weights updating. Therefore, we have ap- proximated gradient step for the student model S = 5 + 9o + Σi= Pigt. where denotes the batch size, and 9i are gradient terms independent of dataset-weights. So the optimization goal becomes: make the student model perform better on the validation dataset by adjusting the update magnitude Ps (i.e. weiggtts The experimental results show that the mixture of the reweighted dataset and real-world dataset can even perform better than a single real-world dataset. This is similar to teacher-student method is that we want to force the teacher to generate more valuable teaching sequences rather than simply finding which sample can make the teacher model have a lower validation loss. The latter may allow the model to perform better on specific validation set, while the former aim improve the model's prediction quality on a different dataset and find samples that better reflect the underlying features of the dataset. The teacher model trained in this way generalize better other datasets, where is the unlabeled dataset E our case. addition, this also prevents the model from taking some tricky ways reduce the loss, such as directly predicting zero for small absolute value.
results in |waang et al.) (2022). The reason for adopting
CONCLUSIONS
In this paper, proposed cascaded learning framework to reweight the training samples of the teacher model. By using one-step approximation and pseudo dataset, we successfully established a gradient flow between dataset-weights and the test loss of the last model in the framework.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
The proposed cascaded framework can help most time-series forecasting models to have better per- formance by reweighting the training dataset. This framework can be applied to many significant real-world cases, such economies and finance forecasting, climate forecasting, disease propaga- tion analysis, traffic management, etc. also reveals the latent patterns within the dataset, which is inspirational for feature engineering. Our contributions not limited to LSTF problem. It can also be applied to other tasks like long text generation, anomaly detection, and neural architecture search. In addition, the introduction of teacher-student learning results in the need for more computing sources. In our case, we use two V100 32G GPU in our experiments to achieve a training speed comparable to the baseline model. In addition, the gradient calculation between the models necessitates distributed training, which increases the difficulty of coding. Therefore, we provide decoupled distributed code for cascaded learning facilitate researchers fill in their own models in our framework.
REPRODUCIBILITY STATEMENT
We provide an open-source implementation of our cascaded framework at https:// anonymous 4open. rameter setting and analysis can be found in the Appendix.
The detailed hyperpa-
REFERENCES
Nicholas Carlini and David Wagner. Towards evaluating the robustness neural networks, 2017.
Rich Caruana. Multitask Learning. Machine Learning, 28(1):41-75, July 1997. ISSN 1573-0565. doi: 10. 10.1023/A411003376607744
Ashis Kumar Chanda, Chowdhury Farhan Ahmed, Md Samiullah, and Carson K Leung. A new framework for mining weighted periodic patterns in time series databases. Expert Systems with Applications, 79:207-224, 2017.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating Long Sequences with Sparse Transformers, April 2019.
Jindong Gu and Volker Tresp. Search for better students to learn distilled knowledge, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. 2015.
Dongseong Hwang, Khe Chai Sim, Zhouyuan Huo, and Trevor Strohman. Pseudo label is better than human label, 2022.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The Lipschitz Constant of Self-Attention, June 2021.
Nikita Kitaev, łukasz Kaiser, and Anselm Levskaya. Reformer: The Efficient Transformer, Febru- ary 2020.
Jacek Klimek, Jakub Klimek, Witold Kraskiewicz, and Mateusz Topolewski. Long-term series forecasting with query selector efficient model of sparse attention. ArXiv, abs/2107.08687, 2021.
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Blockwisely supervised neural architecture search with knowledge distillation, 2020.
Jinyu Li, Rui Zhao, Jui- Ting Huang, and Yifan Gong. Learning small-size dnn with output- distribution baasdd criteria. In Fifteenth annual conference of the international speech commu- nication association, 2014.
10

---

Under review as a conference paper at ICLR 2023
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Minhao Liu, Ailing Zeng, Qiuxia Lai, and Qiang Xu. Time series is special sequence: Forecasting with sample convolution and interaction. ArXiv,
2021.
Feiya Lv, Chenglin Wen, Meiqin Liu, and Zhejing Bao. Weighted time series fault diagnosis based on stacked sparse autoencoder. Journal of Chemometrics, 31(9):2212,, 2017.
Xiaoxin Pan, Long Wang, Zhongju Wang, and Chao Huang. Short-term wind speed forecasting based on spatial-tempont graph transformer networks. Energy, 253: 124095, August 2022. ISSN 0360-5442. doi: 10. 202.gg.2022..24422
Hicu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoc V. Le. Meta pseudo labels, 2021.
Feras Saad and Vikash Mansinghka. Temppoalll--rwweighhed chinese restaurant process mixtures for clustering. imputing, and forecasting multivariate time series. In International Conference on Artificial Intelligence and Statistics, pp. 755-764. PMI.R, 2018.
Ali Sellami and Heasoo Hwang. A robust deep convolutional neural network with batch- weighted loss for heartbeat classification. Expert Systems with Applications, 122:75-20 84, 2019.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data, 2019.
Ilya Trofimov, Nikita Klyuchnikov, Mikhail Salnikov, Alexander Filippov, and Evgeny Burnaev. Multi-fidelity neural architecture search with knowledge distillation, 2021.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2020.
Hui-Kuang Yu. Weighted fuzzy time series models for taiex forecasting. Physica A: Statistical Mechanics and Applications, 349(3-4):609-624 6244, 2005.
Chuheng Zhang, Yuanqi Li, Xi Chen, Yifei Jin, Pingzhong Tang, and Jian Li. DoubleEnsemble: A New Ensemble Method Based on Sample Reweighting and Feature Selection for Financial Data Analysis, January 2021.
Qiuyue Zhang, Chao Qin, Yunfeng Zhang, Fangxun Bao, Caiming Zhang, and Peide Liu. Transformer-base attention network for stock movement prediction. Expert Systems with Ap- plications, 202:1 17239, September 2022. ISSN 0957-4174. doi: 10.1016/.eesa222222117333
Ying Zhang, Xiaoping Wang, and Huiming Tang. An improved elman neural network with piecewise weighted gradient for time series prediction. Neurocomputing, 359: 199:199-208, 2019.
Yi Zhao, Yanyan Shen, Yanmin Zhu, and Junjic Yao. Forecasting wavelet transformed time series with attentive neural networks. In 2018 IEEE International Conference Data Mining (ICDM), pp. 1452-1457, 2018. doi: 10.1 1011 0.0201.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
11

---

Under review as a conference paper at ICLR 2023
A OPTIMIZATION ALGORITHM OF CASCADED TEACHING
In this section, we develop an optimization algorithm to solve the multi-level optimization problem. We first approximate 77* (A) using one-step gradient descent w.r.t (A, Ti, D(mi):
(A) R Ti = - L (A, D(mm))
(13)
We substitute Ti into I. (72, (TP (AT() and get an approximated objective. Then we ap- proximate T2 using one-step gradient descent w.r. the approximated loss T3 T3 T2 TV,LL (())) For the nested function (TT- (..TE()...TTA)))) (T())) approximate it using one-step gradient descent T T Tk VV L Tk, Dipso (Tx-1) In this way, plug (T/ into the validation loss:
((l) + x
Σ L (T; Dival)
We update A by gradient descend w.r.t the approximated validation loss:
K
A A A- [(A,TT D(val)) +
+AII(T, D(aa))
(14)
The above derivation makes it possible to generalize our approach in Section to the K-learner case.
B DETAILS ANALYSIS
To better understand why our cascaded framework works, we have to use the Lipschitz condition of the transformer- basedd model. As discussed in Kim et al. (202I), the scaled dot-product self- attention is Lipschitz if the input space is compact, In our case, the compact input space of the time-series dataset be written as [-M, MMBxxxX where M is the upper bound of the whole dataset and B. L, D represent respectively batch size, input length and hidden dimension. Most of the variants transformer including Informer and QS-Selector are still continuously differentiable, so the Lipschitz condition applies. Assume that the Lipschitz constant of the cascaded models is K.
In the case of two learners, the teacher model and the student model can be simplified as follows:
FT X.) = Y?" (s) = Y.S
(15)
where Fr denotes the teacher model and Fs denotes the student model. For simplicity, assume that the batch size is 3, 7=0 and after several iterations the models are close to convergence so that their gradients are lot smaller than their parameters. Now do one-step gradient descent the teacher model in Eq. (4) and use the new parameters to generate pseudo label for a specific sample ej in the unlabeled dataset E. This process can be written as:
(r = FT + + P2t2 + p3t3,e;)
(16)
where 1, = -1 3 (T, X,) denotes the parameters update brought by sample X, Since 1, < can have the following approximation:
FT = + (piti + p212 + pala) or (T(
(17)
Therefore, when we train student model on the generated pscudo-labeled dataset D(ps), the MSE loss function can be written
12

---

Under review as a conference paper at ICLR 2023
4(S,e,) = (Fs(S,ee) - ,e((),,,))"
=(Fs(S, Fr)
(18)
(p1t1 + Pat2 + psts)
OT Fr(T.e,))
After omitting the second-order terms, we have:
(y(S,e;) =130 + pilsi + p2lj2 p3lpa
8.1. 150 = (Fs(S, eÿ) - er))
(19)
0
1 = 21: JT JT (T, ( Fr(T,))
So the one-step gradient descend for student model in Eq. can be rewritten
S = 7s 0s
15(5,
=S + 9jo + P1951 + P29j2 + P305a
0
(20)
s.t.
930
S as
J0
0 9j
ns Ds 4/
Now have proven that the detaset-weights directly control the size of the corresponding update of student model parameters.
C EXTRA EXPERIMENTAL RESULTS
C.I SCINET AND REFORMER
In this section, show the performance of the cascaded framework applied to Reformer Kitaev ett al. (2020) and SCINet Liu et al. (202). a CNN-based neural network for time-series forecasting. The results summarized in table 5 As we see, the usage of the cascaded framework helps the two models to have 13% of improvement. This proves the scalability of the cascaded framework on convolutional neural networks and that data-reweighting does affect the sparse attention mecha- nism within the Transformer-basee models. We also notice that the student SCINet model performs slightly better than the teacher model. Nevertheless, since the self-study rate is set 0.2, the two models have a very similar improvement compared to the baseline model.
C.2 DATASET- WEIGHT GENERATOR
We have evaluated the could only test normal- ciimaalddistiiution-bbo based dataset-weight weight generator three datasets, among which the Exchange dataset (financial) records the daily exchange rates of eight different countries ranging from 1990 to 2016. As perform better on almost all settings except the Exchange dataset. The results confirm the statement about financial datasets section 3.2 of the paper.
with different dataset-weight generator. Due to limited time, we
see in table6 the Cas-Informer with Fourier dataset-weight generator
C.3 HYPERPARAMETERS
The structure of the time-series forecasting model within our cascaded framework is the same the original setting Informer et al. (2021), Query Selector Klimek et al. (2021), Reformer Kitaev Selector/Kimek et al. (2020) and SCINet et al. (202 1) By default, the input length of the encoder is set to 96 and the input length the decoder is set to 48. We use the Adam optimizer to train for 10 epochs with an initial learning rate of 0.0001 which is halved every 4 epochs. The dataset-weights weights
also trained by the Adam optimizer with an initial learning rate of 0.0002. Due to GPU memory limitations, we adopt batch size of 32 and hidden dimension of 512. Unless otherwise specified, the self-study
13

---

Under review as a conference paper at ICLR 2023
Table 5: Multivariate time-series forecasting results ETT dataset with SCINet and Reformer.
Methods
Cas-SCINet
SCINet
Cas- Cas-Reformer
Reformer
Role
Student
Teacher
Baseline
Student
Teacher
Baseline
Metric
MSE
MAE
MSE
MAE
MSE
MAE
||
MSE
MAE
MSE
MAE
MSE
MAE
24
0.311
0.357
0,305
0.350
0.332
0.375
0.845
0.713
0.862
0.733
0.991
0.754
BTT12 E11hb
0.367
0.398
0.430
1.123
0.906
168
0.428
0.433
0.434
0.436
0.471
0.468
L.703
1.082
1.672
1.075
1.824
1.138
336
0.706
0.578
0.711
0.583
0.738
0.601
1.916
1.193
1.937
1.215
2.117
1,280
720
0.717
0.584
0.722
0.589
0.761
0.628
2.163
1.512
2.156
1.501
2.415
1.520
24
0.186
0.289
0.189
0,292
0.223
0.315
1.403
1.567
1.387
1.549
1.531
1.613
48
0.287
0.354
0.292
0.358
0.563
0.546
1,650
1.687
1.678
1.707
1.871
1.735
0,502
0.489
0.519
0.494
0.586
0,543
4.254
1.513
4.289
1.545
4,660
1.846
336
0.604
0.541
0.621
0.558
0.702
0.603
3,271
1.245
3,338
1.274
4.028
1.688
720
1.384
0.775
1.376
0.768
1.493
0.879
4.946
1.827
1.882
5.381
2.015
0.238
0.242
0,302
0.329
0.668
0.591
0.576
0.724
0.607
48
0.384
0.403
0,380
0.399
0.414
0.416
0.923
0.618
0.937
0.623
1.098
0.777
96
0.336
0.355
0.343
0.366
0.375
0.401
1.364
0,882
1.371
11890
1.433
0.945
288
0.502
0.526
0.513
0.535
0.582
0,556
1.615
0.965
1.648
0.982
1.820
1.094
672
0.735
0.658
0.757
0.675
0.892
0.727
1.904
1.198
1.838
1.172
2.187
1.232
Count
24
10
Table 6: Experimental results on dataset-weight generator.
Methods
Cas-Informerrfourrie))
Cas-Informer(normaal)
Informer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
0.473
0.492
0.526
0.524
0.577
0.549
ETTh2 ETTh
48
0.620
0,586
0.664
0.609
0.685
0.625
168
0.877
0.714
0.912
0.735
0.931
0.752
0.957
0.725
1.077
0.821
1.128
0.873
720
1.035
0.805
1.133
0.857
1.215
0.896
24
0.633
0.638
0.735
0.673
0.720
0.665
48
1.146
0.875
1.418
0.988
1,457
1.001
168
1.915
1.046
2.732
1.348
3 3,489
1,515
2.318
1.167
2.611
1.290
2.723
1.340
720
2.479
1,336
3.124
1.422
3.467
1.473
96
0.834
0.747
0.816
0.741
0.856
0.758
Exchange
192
1.009
0,802
0.975
0.788
1.221
0.905
336
1.613
0.995
1.582
0,976
1.633
L.014
720
2.275
1.302
2.298
1.312
2.496
1.352
Count
22
rate = 0.2 and the sigmoid temperature T 5. The early stop algorithm is also adopted to prevent the model from overfitting. We have implemented the cascaded framework in Python 3.8 with Pytorch 1.10 so that the recently released distributed data parallel package can be used.
14


Reviews:
[{"review_id": "kBF0rfG6x1z", "reviewer": "Reviewer_JAsS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a teacher-student model in long-term time series forecasting. Specifically, a teacher model is learned with reweighted training set.  A student model is then trained on a mixed dataset comprising of the original training set and the pseudo-labeled dataset produced by the teacher. The student is then validated on a separated validation set, where the weights of training samples are trained. The training of both teacher and student only takes one step of gradient descent. This idea can also be extended to a cascaded teaching process.", "summary_of_the_review": "Overall I think the paper makes reasonable contribution to transformer-based forecasting, though the idea is not quite novel, and I would be happy to see it accepted.", "strength_and_weaknesses": "Strengths:\n+ The paper is well organized and the basic idea is easy to follow\n+ The motivation of selecting data samples in training set is convincing.\n\nWeakness\n- My main concern is the efficiency  and training stability of this method. The teacher-student model can be difficult to train compared to common supervised methods, and a complexity analysis can be necessary, especially in cascaded teaching\n- The Hessian approximation could be also investigated in analysis.\n- More state-of-the-art transformers (FEDformer, autoformer, etc) should be investigated\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hai2G9WalO", "reviewer": "Reviewer_eiCx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel learning framework by cascaded teaching Transformers to reweight samples. The framework is formulated as a multi-level optimization and designed with three different dataset-weight generators. It's also novel to use teacher-student framework for time series forecasting problem. The experimental results show the efficacy of proposed method.", "summary_of_the_review": "This paper proposes a novel learning framework by cascaded teaching Transformers to reweight samples. The framework is formulated as a multi-level optimization and designed with three different dataset-weight generators. The experimental results show the efficacy of proposed method.\n\nHowever, it doesn't compare with other state-of-the-art methods, such as FedFormer, Autoformer, NHITS and etc. It also didn't show that this method is general enough to extend to other transformer or non-transformer based models.", "strength_and_weaknesses": "Strengths:\n1. It's novel to propose the teacher-student cascaded teaching Transformers to re-weight samples.\n2. The formulation, especially the optimization part is derived in detail.\n\nWeaknesses:\n1. The experiments are weak. It's only compared with Informer which is not the state-of-the art method. \n2. It's not sure that whether this method is general enough for other Transformer-based models, such as Autoformer, FedFormer and etc.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fN4OzjVnp2D", "reviewer": "Reviewer_u8Gy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a time series reweighting method, integrated with knowledge distillation method, so that noisy input time series play less important roles than others during model training. The reweighting function is parameterized and trained together with the entire teacher-student models via gradient descent. The methods is generalizable and experimented with Transformer for performance evaluation.", "summary_of_the_review": "The overall novelty of this paper is not strong. The proposed method has several unclear points that may prevent understanding its design. The experiments also remain to be improved to demonstrate generalizing the method to different networks.", "strength_and_weaknesses": "Strengths\n1. The reweighting idea is reasonable for improving training quality on time series. The learned weights also provide certain interpretation on anomalous data that impair model training.\n2. The paper introduces a cascaded teaching framework with sample reweighting.\n3. The paper also describes how to generalize the model to many student models and for any base models.\n\nWeakness\n1. The overall technical novelty of this paper is not strong. Cascaded knowledge distillation has been proposed before (e.g., Improved Knowledge Distillation via Teacher Assistant, AAAI 2020). Sample reweighing is also a commonly used method for improving training quality.\n2. It is the not clear from the paper on why cascading teaching is necessary for the reweighting method. Perhaps knowledge distillation has already performed denoising for certain level. The paper should elaborate on how knowledge distillation and sample reweighting enforce each other and should be combined as an integrated method.\n3. In Eq. 2 and 3, it is unclear whether the dataset for training student models ($D^{val}$) contains labels, and whether these datasets have noises. If they have noises, should the training of the student models include another set of reweighting functions?\n4. The optimization of the model uses gradient descent, the authors may want to describe the difference between the proposed optimization method and commonly used optimizer (SGD, Adam), and whether the model can be trained by the common optimizers.\n5. The proposed framework looks generalizable, and not limited to Transformer. It thus is a little confusing on why to emphasize Transformer in this paper. It is better to demonstrate the generalizable framework by evaluating it with different base models.\n6. As for the generalization ability, since the model focuses on the time series forecasting task. It is better to discuss its generalization to other tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3r3XF5eUnF1", "reviewer": "Reviewer_2SH3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a cascaded teaching framework for long-sequence time-series forecasting using transformers. The three-level optimization framework involves updating a sequence of transformer models (teacher and students) and a set of parameters A that are used to compute sample weights. Initially, the teacher model is updated using the re-weighted time-series data. The updated teacher model is then used to generate a pseudo time-series dataset for updating the student model. Finally, the set of parameters A affecting the sample weights is updated based on the performance of the student model on a validation set. The paper empirically validates the proposed framework's effectiveness on five time-series datasets using different transformer architectures and prediction lengths.\n", "summary_of_the_review": "The proposed framework involves multiple components, thus a thorough ablation study of additional components is very important.  Currently, the compared methods in the experiments section are not convincing. However, based on the clarifications and answers to the questions in the weaknesses section, I am willing to revise my score.", "strength_and_weaknesses": "Strengths:\n\n1. This paper focuses on an interesting topic about the re-weighting of a time-series dataset so that outlier and less important samples can be excluded from the training process.\n\n2. The results show generalizability across different datasets and transformer architectures. \n\nWeaknesses:\n\n1. The experiments on different datasets do not cover one important baseline which uses sample reweighting to update the target model i.e. the teacher model, and no student models are involved. This could be formulated as an alternating optimization framework to update the target/teacher model and parameters A (used to compute sample weights). This ablation can essentially help understand the trade-off between the extra computational costs incurred to update a sequence of student transformer models and the performance improvement brought by the feedback of the student models.\n\n2. In the Related Work (Section 2.1), the authors mention that this work focuses on searching the architecture of a teacher model by letting it teach a student model where the student’s architecture is fixed. However, I could not find any results supporting the teacher architecture search in the experiments section. This point should be clarified in the main text.\n\n3. In Section 2.2, the authors mention that the dataset weights trained by the proposed framework are not coupled with the model. Have the authors studied how well the trained dataset weights transfer across different transformer architectures?\n\t\n4. Comparison among compared baseline methods and proposed framework in terms of computational costs (related to training and pseudo data generation) and running time is missing.\n\t\n5. Some sections specifically Section 3 can be improved in terms of writing and notational consistencies. Please see the first three points below.\n\n\nMinor Questions:\n1. How is the set of parameters A (used for computing the sample weights) initialized?\n\n2. Clarification about some notations such as $L_A, L_u$, and $L_p$ mentioned on page 3 is missing.\n\n3. The ordering of the inputs to the loss function L() is inconsistent at certain places, for example, Eq (1) and Eq (8).\n\n4. In Figure 2, for the dataset in consideration, how are the input-output pairs $(s_i, t_i)$ generated? To elaborate, given a longer sequence of size > 1600, what stride and window length are used to generate the smaller subsequences $(s_i, t_i)$?\n\n5. Have the authors checked as the training progresses, how well is the pseudo output  (generated by the teacher) correlated with the original training output $(t_i)$ for the same input series $(s_i)$?\n\n6. In Section 5.4.2, what is $W$? Does it denote the teacher model parameters?\n\t\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject