Kati8/ICLR_2023_2025 huggingface dataset:
authors: Weijian Li
authors: Haozheng Luo
authors: Chenwei Xu
authors: Han Liu
reviews: 
Comments: The authors propose a generic algorithm (GA) based HPO method that aims to find good hyperparameters assuming the distribution of observations change over time. The way the proposed method works is to sequentially go through the data, as a sliding window (SWGA), and use the springs and top performing configurations as the starting population for the next iteration. The authors also propose to use TPE instead of Random Search to fill the initial population.
Strengths: * It’s a relevant problem for time series forecasting.
* It shows some potential for the sliding window strategy.
Weaknesses: - The contribution is minor. I can’t agree that using TPE instead of Random Search is an actual contribution. To fill the initial population, doing a small HPO with TPE should have better results than random configurations. It’s too obvious. Also, to show the sliding window is a better HPO strategy, it needs to first deliver better quality and second is cost effective. While the authors’ experiments (Table 1 & 2) show SWGA has better quality than GA, I am not sure the baseline of GA is properly implemented due to lacking of detail. Also, there are tons of HPO methods that can be combined with the sliding window and they can be used to show sliding window is indeed helpful.

- The technical correctness is hard to access given the current state. Many important details are missing. For example: 
  - How do the authors ensure that the only baseline GA is comparable with the SWGA? For me, to see if the sliding window helps, taking Figure 1 as an example, it should be a single GA on the average performance of 12 splits. Assume population size is M, then SWGA takes trains M*12 times and the single GA baseline also trains M*12 times.
   - Line 333: “we use seven historical timesteps to predict one timestep ahead” Does this mean the authors sample segments of size 8 from training and validation set? How many are sampled?

- The experiments, especially Table 3, does not support the contribution.Table 3 only shows HPO helps, not why sliding window or warm start is a good strategy. The part of scalability does not fit into the current story. The contribution of the paper, as claimed by the authors, are the sliding window and warm up. The focus is not distributed training or scheduling etc.

- The terminology and notations are not precise and conventional. For example:
  - Line 35 ”the model can achieve better performance on out-of-sample data with a matching distribution” What does out-of-sample data with a matching distribution mean? It’s also strange that the authors call the prediction window as out-of-sample data. It may or may not be out-of-sample.
  - Line 141, please check notation, many misusages.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This paper introduces a hyperparameter search process specifically tailored for time series forecasting, focusing on temporal distribution shifts in the data. The proposed method is based on a variation of the genetic algorithm and the sliding window validation technique. The algorithm is designed to be parallelizable, which helps further reduce the time needed for hyperparameter search.
Strengths: * The algorithm is parallelizable, significantly decreasing the computation time required to find the optimal set of hyperparameters.
* Evaluation is conducted on a sufficient number of widely known real-life datasets.
Weaknesses: * In the experiments, the proposed algorithm is only compared with the traditional genetic algorithm. It would be beneficial to evaluate other hyperparameter search techniques, such as the classic sliding window search and k-fold validation technique.
* The models evaluated in the experiments lack diversity. There are 3 tree-based models (CatBoost, LightGBM, XGBoost), 1 recurrent model (LSTM), and 1 attention-based model (Transformer).
* The authors mention high computational costs and inefficient exploration of large hyperparameter search spaces as disadvantages of commonly used techniques. However, the experiments only involve accuracy comparisons. Wouldn’t it be beneficial to demonstrate that the proposed hyperparameter search technique converges more quickly?
* In Appendix A.1, search spaces are provided for the DLinear and PatchTST models, which are not included in the experiments.
* The proposed algorithm combines variations of three widely used search techniques from the literature with minimal modifications. The novelty does not seem satisfying.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper introduces the Sliding Window Genetic Algorithm (SWGA), a distributed hyperparameter optimization method designed for time series prediction models. Key contributions include (1) a configurable sliding window technique that mitigates overfitting from distribution shifts typical in time series data, (2) a warm-up stage employing Bayesian optimization to establish a robust initial population, and (3) compatibility with distributed computing.
Strengths: 1. SWGA introduces a combination of genetic algorithms with a sliding window approach tailored specifically for time series forecasting, addressing the distribution shift and non-stationarity challenges unique to this domain.

2. The authors have provided detailed computational procedures by providing multiple algorithm boxes and demonstrated their impact on real-world applications.
Weaknesses: 1. The paper writing has too much redundancy that can be simplified or moved to the appendix. It is also not clear why the proposed method is distinguished from other hyper-parameter optimization approaches.

2. In the experiment section, there is also a lack of comparison with other hyperparameter optimization methods specifically designed for time series data.

3. The SWGA algorithm’s performance might be sensitive to parameters like window size, population size, and mutation rates. However, the paper lacks an exploration of how these parameters impact outcomes.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2