Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission presents a method for improved identification of an early-stopping point based on validation loss curves. This method is based on training a time series classifier on validation loss curves that have been labeled as "overfit" vs. "not overfit", using a collection of UCI-type datasets and various network architectures. These training examples for the time series classifiers were labeled by the authors. Experiments compare various time series classifiers and find that a k-NN-based approach, with dynamic time warping, based on a rolling window, works best. This approach to detecting overfitting (and, thus, enabling early stopping) outperforms existing methods for overfitting detection based on measuring correlation between training and validation loss curves. Further experiments, comparing to plain early stopping using various values for the patience parameter in both the proposed method and standard early stopping, show that the proposed method more accurately identifies the stopping point with minimum validation loss than the standard plain patience-based approach.
strength_and_weaknesses: + The proposed approach seems novel and is intuitively plausible
+ The results indicate that the method is preferable to existing correlation-based approaches to detecting overfitting
+ The paper is easy to follow and code is provided

- Accuracy is measured with respect to the distance to the "optimum" stopping point, not predictive accuracy of the neural network model

What matters is the predictive performance of the model, *not* how close we are to the "optimum" stopping point.

clarity,_quality,_novelty_and_reproducibility: The paper is very clear and easy to read, and there is a corresponding Github repository.
summary_of_the_review: The paper is interesting, and the proposed approach may have merit.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Detecting overfitting of deep learning networks in an efficient, accurate and non-intrusive manner is a non-trivial problem. The authors propose a method to learn a classifier that uses the history of training and validation losses to identify overfitting regions. The paper evaluates the approach using a number of loss histories of published models. They show that classification is more accurate in detecting overfitting compared to correlation methods, and that one of the classifiers KNN-DTW is also better able to identify the optimal stopping point compared to early stopping method. While there are a number of approaches to detect and prevent overfitting, this is the first paper I am aware of that uses time classification to detect and prevent overfitting. The paper does a good job collecting or reproducing the histories of models from 30+ papers. This is a useful contribution for further research in this topic. 
strength_and_weaknesses: Strengths:
1. The paper uses a simple time series classifier to learn the regions of overfitting using training and validation losses. It is a non-destructive method and can be applied easily without any modification of the training algorithm itself. 
2. The approach is tested well using a number of datasets from published models. The authors do a good job of collecting or reproducing the training and validation loss histories from 30+ papers. This is a useful contribution

Weaknesses:
1. From the paper, I struggled to infer they train a single model using all training and validation loss histories, or if they train a model per dataset. If the former, how well do they think this model generalizes to other datasets? i.e. is this a single pre-trained model that others can use off the shelf, or do other researchers need to train a model per dataset?
2. There is very little discussion of why some methods (like KNN-DTW) are able to do well, while others (like HMM-GMM and BOSSVS) perform so poorly (Table 2). As such, this feels like a brute force search method of different algorithms for time series classification. The proposed candidate KNN-DTW is an expensive algorithm to implement for inference (it is the slowest inference algorithm among the ones they tested in Table 2).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written in modular sections that define and motivate the problem, other solutions in the literature, their proposed method, and a reasonable experiments and results section. It is easy to understand and flows well. 
Quality: The work follows a standard scientific approach of problem definition, description of existing solutions, a proposal of a new method, experiments and results. The experiments section uses both simulated and real datasets. Collecting and reproducing the training and validation loss histories from 30+ papers is a good contribution
Novelty: While the problem is important, the idea is fairly obvious, and it is an incremental contribution 
Reproducibility: The authors have published the code, datasets and results on github. It seems fairly easy to reproduce.
summary_of_the_review: The paper addresses an important problem in training deep learning networks  - viz. when to stop training, and doing it in a non-intrusive manner. It proposes a fairly obvious idea of training time series classifiers on histories of training and validation losses to detect regions of overfitting, and compares it favorably against existing methods based on correlation and early stopping. A good contribution is that the paper has collected or reproduced the loss histories from a number of other published papers. This is a useful contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method to detect and prevent overfitting in training deep neural network models. The proposed method trains a time series classification model to detect overfitting based on the model's training history. The trained classifier can also be used to prevent overfitting by identifying the optimal point to stop a model's training. The authors manually labeled several hundred data points using existing datasets and test on several other data points collected from machine learning literature. Results show that the proposed approach outperforms conventional correlation-based methods on overfitting detection and slightly underperforms early-stopping for overfitting prevention but leads to lower delay, resulting in less wasted computation.
strength_and_weaknesses: Strengthes:
1. The paper is generally well written and easy to follow. The proposed method is also simple and intuitive.
2. The problem of detecting and preventing overfitting in deep learning is an important problem.
3. Experimental results show some advantages over prior methods for detecting and preventing overfitting in deep learning models.

Weaknesses:
1. The proposed method is straightforward and does not have much technical novelty/contribution. However, this is perfectly fine if the proposed method can be really useful and help facilitate training of deep learning models.
2. The experimental setup is not fully convincing. First, the test set contains only 40 data points, which is too small and thus vulnerable to randomness issues. Also, these papers are all collected in literature investigating the overfitting problem, which may results in similar choices of tasks/models and may not be able to reflect the trained classifier's real generalization ability on more realistic datasets and tasks. Also, the train histories explicitly labeled as overfitting or not and shown in the paper are probably typically overfitting examples that are very easy to distinguish. Moreover, the train set is manually labeled and require full training to generate, thus relatively laborsome.
3. The authors did not compare with intrusive methods because they require retraining, but their method also require training a classifier with labeled data. I do agree that these intrusive methods are not directly comparable with the proposed method, but it would be better that the authors at least report their performance in the experiments to help reader get a better understanding of the comparison.
clarity,_quality,_novelty_and_reproducibility: The clarity is good. The quality is ok because the method is simple and does not have major flaw. The reproducibility is ok because the  replication package is provided.
summary_of_the_review: In sum, I think the paper investigate an interesting problem. However, the technical contribution is not strong enough and the empirical experiments does not fully support the claims. Therefore I would suggest the authors to revise and improve the empirical study in the future.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tries to detect and prevent overfitting by using the training loss and validation loss history. For detecting overfitting, the paper propose a strategy to collect a labelled dataset with binary labels: overfit and no overfit. The paper shows that a time-series classifier can be trained, and empirically showed to have high average F-scores, which is better than correlation based methods. The paper further shows a way to prevent overfitting, by using a detecting method on the whole observed history and by extracting the latest history with a rolling window. If we detect overfitting, we stop training and return the epoch that has the lowest validation loss. This seemed to work well, compared with an early stopping baseline.
strength_and_weaknesses: Strengths:
- Using the training history to detect and prevent overfitting is interesting.
- The paper constructs a new dataset of overfitting/non-overfitting samples by going through many recent ML papers on overfitting.
- The experiments show that detecting overfitting is possible, and that the time-series classifiers tends to perform better than the correlation based ones.
- Practical value with experiments showing that the method can find the optimal stopping point and avoid overfitting at least 32% earlier than early stopping and achieve at least the same accuracy as early stopping.

Weaknesses:
- If I understood the paper correctly, I feel the main issue is that the paper does not directly discuss what "overfitting" is. There are two sentences that try to address what overfitting is, in the abstract and introduction: 1) the abstract mentions that "overfitting of deep learning models on training data leads to poor generalizability on unseen data", and 2) 1st paragraph of Section 1 (Introduction) mentions that "for the overfit model, after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease". The 1st one (in the abstract) can be regarded as a consequence for overfitting rather than an explanation of overfitting, so for example, the same sentence can be used for underfitting. The 2nd one (in the 1st paragraph of Section 1) is more specific, and if we use this as the definition of overfitting, it may exclude some learning curves which we may sometimes regard as overfitting or include ones that may not be overfitting. For example, if the validation loss stops decreasing after a certain number of epochs (but does not increase, just becomes flat) while the training loss keeps decreasing, the gap between training/validation loss will widen, and some people will regard this is a case of overfitting. As an extreme example, Fig 1 (b) can be considered to have some overfitting, since the validation loss is higher than the training loss and it is possible that this gap may close by increasing the number of training samples even further (which may be seen as an example of poor generalizability on unseen data due to limited training data). Another extreme example is when we continue learning for longer epochs, the validation loss can decrease again, with double descent loss curves. This is usually not regarded as overfitting, but may fall under the overfitting category based on the explanation in the 1st paragraph of Section 1.
- In my opinion, trying to directly target overfitting is difficult since different researchers may have different definitions, so adopting a specific type of overfitting which "after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease", seems like a good direction (although I wrote some issues above that perhaps should be discussed in the paper). However, if the paper adopts this direction, can we automatically label the loss history by checking if "after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease", instead of using human labels, as explained in Step 3 of Section 4.1? 
- It was also not clear to me how the two labellers decided the label of overfit/non-overfit.
- A related question is, how would we know that in papers (for constructing the real-world test dataset) with samples of overfitting, they are samples of the same type of overfitting?
- There are many papers that utilize the training history of maching learning models, but I am not sure if they were discussed. For example, "Learning Curves for Decision Making in Supervised Machine Learning - A Survey" discusses a lot of papers that use the training history or learning curves.
- For regression tasks, we usually use the same loss function for training and for evaluating, but for classification tasks, we may use the 01 loss function for evaluating the accuracy while using a different loss such as the cross entropy loss for training. This is known to cause interesting discrepancies between the behavior of the two different loss functions, e.g., discussed in Guo et al. "On Calibration of Modern Neural Networks" (ICML 2017) or Soudry et al. "The Implicit Bias of Gradient Descent on Separable Data" (JMLR 2018), and the increase of the validation loss may not be harmful in terms of the 01 loss function. Therefore, if we want to have better accuracy (i.e., lower 01 loss), is it possible that it may be better to use 01 loss instead of the cross entropy loss for validation in the proposed algorithm? (Perhaps this is not much of an issue, since the proposed method is helpful in achieving higher accuracy in the experiments, but wondering if this will make the proposed method even better.)
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear in most parts, but I had a hard time understanding how the labels are collected, especially on how they define the overfit/non-overfit labels (I wrote the details in the previous section).

Quality: I feel having more discussions on the type of overfitting the paper wants to address would raise the quality.

Novelty: The paper uses the training history to detect and prevent overfitting, which seems novel. However, the paper does not discuss other papers that utilize the training history, e.g., see the survey paper  "Learning Curves for Decision Making in Supervised Machine Learning - A Survey".

Reproducibility: A link to an anonymous github repo was provided in the paper. However, I did not check the code.
summary_of_the_review: The paper proposes an algorithm that detects and prevents overfitting, based on the labelled dataset that the paper constructed. This seems to be a new direction for research on detecting/preventing overfitting. However, discussions on how the labels were collected (or the two labellers' definition of overfitting) was not provided. A discussion about other papers that also use the training history was not discussed. Since the final target seems to be the accuracy in some of the experiments, using the validation 01 loss function seems to be a natural choice, rather than using the cross-entropy loss function. More discussions on this choice would be helpful. Overall, although the research direction is interesting, I decided to choose the score with 5.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to train time series classifiers on validation loss curves which have been labeled as "overfit" vs. "not overfit" to detect and prevent overfitting. The authors showed that their method is more accurate in detecting overfitting compared to correlation methods, and is also better at identifying the optimal stopping point compared to early stopping method on a collection of UCI datasets. Reviewer recommendations are mixed. While all the reviewers consider the proposal of learning time series classifier to detect overfitting interesting and plausible, they raise different concerns on the work, e.g., small scale of the evaluation datasets (40 data points) making the results susceptible to randomness, additional requirement to manually label validation loss curves as overfitting and non-overfitting. IMO, the paper left many questions unanswered: 1) Beyond finding exactly the optimal epoch to early stop, the generalization gap at the early stopping point detected by different methods should also be compared; 2)  It suggests that there are patterns in the loss validation curves, more than looking at the smoothed validation loss curve, that can help detect overfitting, but the paper offers neither insights nor analyses to shed light on what those pattern could be; 3) how sensitive is the method to noise in the "overfit" vs "not overfit" labels; 4) which time series classifier is used? Are the results sensitive to the specific choices of classifiers? 
justification_for_why_not_higher_score: Missing details and very limited empirical evaluation.
Decision: Reject