Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a cascaded teaching framework for long-sequence time-series forecasting using transformers. The three-level optimization framework involves updating a sequence of transformer models (teacher and students) and a set of parameters A that are used to compute sample weights. Initially, the teacher model is updated using the re-weighted time-series data. The updated teacher model is then used to generate a pseudo time-series dataset for updating the student model. Finally, the set of parameters A affecting the sample weights is updated based on the performance of the student model on a validation set. The paper empirically validates the proposed framework's effectiveness on five time-series datasets using different transformer architectures and prediction lengths.

strength_and_weaknesses: Strengths:

1. This paper focuses on an interesting topic about the re-weighting of a time-series dataset so that outlier and less important samples can be excluded from the training process.

2. The results show generalizability across different datasets and transformer architectures. 

Weaknesses:

1. The experiments on different datasets do not cover one important baseline which uses sample reweighting to update the target model i.e. the teacher model, and no student models are involved. This could be formulated as an alternating optimization framework to update the target/teacher model and parameters A (used to compute sample weights). This ablation can essentially help understand the trade-off between the extra computational costs incurred to update a sequence of student transformer models and the performance improvement brought by the feedback of the student models.

2. In the Related Work (Section 2.1), the authors mention that this work focuses on searching the architecture of a teacher model by letting it teach a student model where the student’s architecture is fixed. However, I could not find any results supporting the teacher architecture search in the experiments section. This point should be clarified in the main text.

3. In Section 2.2, the authors mention that the dataset weights trained by the proposed framework are not coupled with the model. Have the authors studied how well the trained dataset weights transfer across different transformer architectures?
	
4. Comparison among compared baseline methods and proposed framework in terms of computational costs (related to training and pseudo data generation) and running time is missing.
	
5. Some sections specifically Section 3 can be improved in terms of writing and notational consistencies. Please see the first three points below.


Minor Questions:
1. How is the set of parameters A (used for computing the sample weights) initialized?

2. Clarification about some notations such as $L_A, L_u$, and $L_p$ mentioned on page 3 is missing.

3. The ordering of the inputs to the loss function L() is inconsistent at certain places, for example, Eq (1) and Eq (8).

4. In Figure 2, for the dataset in consideration, how are the input-output pairs $(s_i, t_i)$ generated? To elaborate, given a longer sequence of size > 1600, what stride and window length are used to generate the smaller subsequences $(s_i, t_i)$?

5. Have the authors checked as the training progresses, how well is the pseudo output  (generated by the teacher) correlated with the original training output $(t_i)$ for the same input series $(s_i)$?

6. In Section 5.4.2, what is $W$? Does it denote the teacher model parameters?
	

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to follow for most of the parts, except Section 3. Please see weaknesses point 5.

Quality: Some aspects can be improved, particularly addressing the questions and comments related to Section 3.

Novelty: The idea about using a cascaded teaching framework to learn dataset weights seems novel. However, it introduces a significant computational overhead which needs to be justified in terms of relative performance improvements brought up by properly ablating different additional components involved: (i) sample weights learning (ii) cascaded training of student models on pseudo time-series data. See weaknesses point 1.

Reproducibility: The authors have open sourced the code for reproducing the empirical results.
summary_of_the_review: The proposed framework involves multiple components, thus a thorough ablation study of additional components is very important.  Currently, the compared methods in the experiments section are not convincing. However, based on the clarifications and answers to the questions in the weaknesses section, I am willing to revise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a teacher-student model in long-term time series forecasting. Specifically, a teacher model is learned with reweighted training set.  A student model is then trained on a mixed dataset comprising of the original training set and the pseudo-labeled dataset produced by the teacher. The student is then validated on a separated validation set, where the weights of training samples are trained. The training of both teacher and student only takes one step of gradient descent. This idea can also be extended to a cascaded teaching process.
strength_and_weaknesses: Strengths:
+ The paper is well organized and the basic idea is easy to follow
+ The motivation of selecting data samples in training set is convincing.

Weakness
- My main concern is the efficiency  and training stability of this method. The teacher-student model can be difficult to train compared to common supervised methods, and a complexity analysis can be necessary, especially in cascaded teaching
- The Hessian approximation could be also investigated in analysis.
- More state-of-the-art transformers (FEDformer, autoformer, etc) should be investigated

clarity,_quality,_novelty_and_reproducibility: Clarity: generally the paper is well-written, with some details unclear to me
 - What is the $x$ in (b) of section 3.2
 - In figure 2, why the weights are applied to different time steps? In my understanding the weights are consistently applied to time series sequences

Novelty: teacher-student model is not novel idea, but its application on dataset reweighting seems new to me
Reproducibility: The paper provides open-sourced implementation
summary_of_the_review: Overall I think the paper makes reasonable contribution to transformer-based forecasting, though the idea is not quite novel, and I would be happy to see it accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel learning framework by cascaded teaching Transformers to reweight samples. The framework is formulated as a multi-level optimization and designed with three different dataset-weight generators. It's also novel to use teacher-student framework for time series forecasting problem. The experimental results show the efficacy of proposed method.
strength_and_weaknesses: Strengths:
1. It's novel to propose the teacher-student cascaded teaching Transformers to re-weight samples.
2. The formulation, especially the optimization part is derived in detail.

Weaknesses:
1. The experiments are weak. It's only compared with Informer which is not the state-of-the art method. 
2. It's not sure that whether this method is general enough for other Transformer-based models, such as Autoformer, FedFormer and etc.


clarity,_quality,_novelty_and_reproducibility: The presentation is clear and novel. It's a little bit tricky on the reproducibility as it's kind of multi-level training.
summary_of_the_review: This paper proposes a novel learning framework by cascaded teaching Transformers to reweight samples. The framework is formulated as a multi-level optimization and designed with three different dataset-weight generators. The experimental results show the efficacy of proposed method.

However, it doesn't compare with other state-of-the-art methods, such as FedFormer, Autoformer, NHITS and etc. It also didn't show that this method is general enough to extend to other transformer or non-transformer based models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a time series reweighting method, integrated with knowledge distillation method, so that noisy input time series play less important roles than others during model training. The reweighting function is parameterized and trained together with the entire teacher-student models via gradient descent. The methods is generalizable and experimented with Transformer for performance evaluation.
strength_and_weaknesses: Strengths
1. The reweighting idea is reasonable for improving training quality on time series. The learned weights also provide certain interpretation on anomalous data that impair model training.
2. The paper introduces a cascaded teaching framework with sample reweighting.
3. The paper also describes how to generalize the model to many student models and for any base models.

Weakness
1. The overall technical novelty of this paper is not strong. Cascaded knowledge distillation has been proposed before (e.g., Improved Knowledge Distillation via Teacher Assistant, AAAI 2020). Sample reweighing is also a commonly used method for improving training quality.
2. It is the not clear from the paper on why cascading teaching is necessary for the reweighting method. Perhaps knowledge distillation has already performed denoising for certain level. The paper should elaborate on how knowledge distillation and sample reweighting enforce each other and should be combined as an integrated method.
3. In Eq. 2 and 3, it is unclear whether the dataset for training student models ($D^{val}$) contains labels, and whether these datasets have noises. If they have noises, should the training of the student models include another set of reweighting functions?
4. The optimization of the model uses gradient descent, the authors may want to describe the difference between the proposed optimization method and commonly used optimizer (SGD, Adam), and whether the model can be trained by the common optimizers.
5. The proposed framework looks generalizable, and not limited to Transformer. It thus is a little confusing on why to emphasize Transformer in this paper. It is better to demonstrate the generalizable framework by evaluating it with different base models.
6. As for the generalization ability, since the model focuses on the time series forecasting task. It is better to discuss its generalization to other tasks.
clarity,_quality,_novelty_and_reproducibility: See Strength and Weakness.
summary_of_the_review: The overall novelty of this paper is not strong. The proposed method has several unclear points that may prevent understanding its design. The experiments also remain to be improved to demonstrate generalizing the method to different networks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Unfortunately, the reviewers were not enthusiastic enough about this paper for it to be considered for acceptance at ICLR 2023. There are just too many papers that reviewers were much more enthusiastic about this year, and ICLR has a very low acceptance rate. The authors are encouraged to take the reviewer comments very seriously, even if there are things you disagree with, and make sure that all issues are addressed, and any potential sources of confusion are completely eliminated, in the next version of the paper. Also, please ensure that these are addressed in the initial paper submission for that next conference. 
justification_for_why_not_higher_score: see above
justification_for_why_not_lower_score: see above
Decision: Reject