Using the Training History to Detect and Prevent Overfitting in Deep Learning Models
Authors: Hao Li
Gopi Krishnan Rajbahadur
Dayi Lin
Cor-Paul Bezemer
Zhen Jiang
Venue: Submitted to ICLR 2023 (2023)
Keywords: overfitting
early stopping
deep learning

Abstract: Overfitting of deep learning models on training data leads to poor generalizability on unseen data. Overfitting can be (1) prevented (e.g., using dropout or early stopping) or (2) detected in a trained model (e.g., using correlation-based methods). We propose a method that can both detect and prevent overfitting based on the training history (i.e., validation losses). Our method first trains a time series classifier on training histories of overfit models. This classifier is then used to detect if a trained model is overfit. In addition, our trained classifier can be used to prevent overfitting by identifying the optimal point to stop a model's training. We evaluate our method on its ability to identify and prevent overfitting in real-world samples (collected from papers published in the last 5 years at top AI venues). We compare our method against correlation-based detection methods and the most commonly used prevention method (i.e., early stopping). Our method achieves an F1 score of 0.91 which is at least 5% higher than the current best-performing non-intrusive overfitting detection method. In addition, our method can find the optimal stopping point and avoid overfitting at least 32% earlier than early stopping and achieve at least the same accuracy (often better) as early stopping.

Under review as a conference paper at ICLR 2023
USING THE TRAINING HISTORY TO DETECT AND PRE- VENT OVERFITTING IN DEEP LEARNING MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Overfitting occurs in deep learning models when instead of learning from the train- ing data, they tend to memorize it, resulting in poor generalizability. Overfitting can be (1) prevented (e.g.. using dropout or early stopping) or (2) detected in a trained model (e.g.. using correlation based methods). We propose a method that can both detect and prevent overfitting based on the training history (i.e., valida- tion losses). Our method first trains a time series classifier on training histories of overfit models. This classifier is then used detect a trained model overfit. In addition, trained classifier ing the optimal point to stop a model's training. We evaluate our method on its ability identify and prevent overfitting in real-world samples (collected from papers published in the last 5 years at top Al venues). We compare our method against correlation-baser detection methods and the most commonly used preven- tion method (i.e., early stopping). Our method achieves an Fl score of 0.91 which is at least 5% higher than the current perffrreron non-intrusive overfitting de- tection method. In addition, our method can find the optimal epoch and avoid overfitting least 32% earlier than early stopping and achieve at least the same rate (often better) of achieving the optimal epoch as early stopping.
be used to prevent overfitting by identify-
INTRODUCTION
Overfitting is one of the fundamental issues that plagues the field of machine learning (Nowlan & Hinton, 1992; Ng. 1997; Caruana et al., 2000; Cawley & Talbot, 2007; Erhan et al., 2010; Srivastava al., 2014; Zhao et al., 2020), which can also occur when training a deep learning (DL) model. An overfit model increases the risk of inaccurate predictions, misleading feature importance, and wasted resources (Hawkins, 2004). Figure/ I shows example training his- tories (i.e., the training and validation losses curves) of an overfit and non-overfit model. The training and validation losses of the overfit model both decrease at the beginming of the training process. Following that, the validation loss increases while the training loss decreases, resulting dation losses. Such a trend indicates that the trained model not generalizing well to new data.
Training Validation loss
(a) Overfitting
Training loss
a large gap between the training and vali-
Validation loss
Currently, the problem of overfitting is addressed by either (1) pre- venting it from happening in the first place or (2) detecting it in trained model. Overfitting prevention methods stop overfitting from happening through methods such early stopping (Morgan & Bourlard, 1989), data augmentation (Shorten & Khoshgoftaar, 2019), regularization (Kukačka et al., 2017), modifying the model by adding dropout layers (Srivastava et al., 2014) or batch normal- ization (loffe & Szegedy, 2015). Many of these methods are in- trusive and require modifying the data or the model structure and expertise to execute correctly. Furthermore, even the non-intrusive prevention methods such as early stopping incur trade-off between model accuracy and training time (Prechelt, 2012). For example, when using the early stopping method, stopping too late may
(b) Non-overfitting
Figure 1: Example training histories of overfit and non- overfit models.

---

Under review as a conference paper at ICLR 2023
improve model accuracy but also increase training time while stopping too early could result in model that performs sub-optimally.
Overfitting detection methods typically attempt to identify if trained model is overfit by retraining the model with noisy data points and observing the impact of these noisy data points on the model's accuracy (as overfit model can learn the noise reduce the impact) (Zhang et al., 2019). Al- ternatively, some detection methods check the hypothesis that the trained model and the data are independent, e.g., Werpachowski et al. (2019) check the hypothesis by comparing the test error with the estimated test error based on adversarial examples of the test set. However, similar to intrusive overfitting prevention methods, significant expertise is typically required to use existing detection methods. In addition, these methods require extra computational resources for activities such as generating adversarial examples, retraining the models, and converting the model.
In this paper, we are the first to propose a method for both overfitting detection and prevention based on training histories. Training histories have been used by researchers before to make decisions such quantitative data acquisition and model selection (van Rijn et al., 2015; Strang et al., 2018; Bornschein et al., 2020; Mohr van Rijn, 2021; 2022: Brazdil et al., 2022). Similarly, our method trains time series classifier on simulated dataset of training histories (i.e., labelled validation loss curves over epochs of training) of models that overfit the training data. Our trained time series classifier detects if trained DL model is overfitting the training data by inspecting the validation loss history (which is captured as part of the training history). In contrast to existing overfitting detection methods, our method does not incur additional resources or costs since the training history is byproduct of the training process. Additionally, our method (i.e., the trained time series classifier) can be used to prevent overfitting based on the validation losses of recent epochs (e.g.. the last 20 epochs).
While we train our method on a simulated dataset, we evaluate it on real-world dataset, collected from papers from top AI venues from the last 5 years. We collected the training histories from these papers that are explicitly labelled as overfitting or non-overfitingg by the authors as the ground truth. Our results show that our method outperforms the state-of-the-ar by at least 5% in terms of F-score for overfitting detection, with an F-score of 0.91. In addition, our method can prevent overfitting from happening at least 32% earlier than early stopping while having the same (and often better) rate achieving the optimal epoch.
BACKGROUND AND RELATED WORK
2.1 OVERFITTING
Overlitting is a well-known and explored problem in the area of machine learning (Nowlan & Hin- ton, 1992; Ng. 1997; Caruana et al., 2000; Cawley & Talbot, 2007; Erhan et al., 2010; Srivastava et al., 2014; Zhao et al., 2020). Recent research has further noted the widespread presence and impact of overfitting in the sub-fields of machine learning including reinforcement learning (Song et al., 2020), adversarial learning (Rice et al., 2020), and recommender systems (Peng et al., 2021). For recommender systems that deal with massive amounts of data every day, incremental model updates are required to catch the most recent trend. However, the incrementally updated model may overfit to the most current data and forget previously learned knowledge (Peng et al., 2021). Song et al. (2020) study the observational overfitting regime in reinforcement learning, which overfits to only a small proportion of the observation space. Furthermore, Rice et al. (2020) report that over- fitting happens more frequently in adversarial training than in traditional DL. Overfitting hurts the generalizability a trained model, but generally predicting whether a model will overfit to a certain dataset before training it is formally undecidable (Bashir al., 2020). In this paper, study how to detect if a trained model is overfit and how overfitting can be prevented from happening during the training process. Below, we give overview of existing methods to detect and prevent overfitting, and we describe the methods that we used as baselines to evaluate the accuracy of our method.
2.2 OVERFITTING DETECTION
In the field of symbolic regression, Kronberger al. (2011) propose computing Spearman's 's non-paramttric rank correlation coefficient (Spearman, 1987) between training and validation fit- ness (i.e., an evaluation metrie for the symbolic regression model) to detect overfitting. Researchers
2

---

Under review as a conference paper at ICLR 2023
have also studied how to detect overfitting by injecting noise into the training data or generating new data. They typically retrain the model that is being tested with this noisy or new data and observe the impact its performance to detect if the model is overfitting. For instance, Zhang et al. (2019) propose a Perturbation Validation (PV) method, which retrains the model after injecting different levels of noise (perturbation) into the labels. They retrain the model for each noise level and collect the training history to compute the PV measurement. The PV measurement shows how the accuracy changes in response to the injected noise and indicates that overfitting is present if the accuracy does not decrease significantly on the noisy data. Werpachowski et al. (2019) generate adversar- ial examples to detect whether an image classification model overfit to the test set. Chatterjee & Mishchenko (2020) describe method that converts machine learning models to logic circuits and detects overfitting by inspecting rare patterns of handling training samples the model. In this paper, we propose an overfitting detection method based on time series classifiers that only relies on the training history of a trained model and does not involve model conversion or retraining. We train the time series classifiers with training histories and labels indicating whether or not there is overfit- ting, hence, the classifiers can identify overfitting from the training history of a trained model. Since similar our method, correlation- well, we select them as the baseline to compare our method against and introduce it below.
methods detect overfitting based on the training history as
Correlation-basel methods. Inspired by the overfitting detection method of Kronberger et al. 1), we compute correlation metrics between the training and validation loss to detect overfitting in DL models. The idea behind this method is intuitive: the training and validation loss (similar to the training and validation fitness in symbolic regression) are expected to be strongly correlated when there no overfitting and the correlation should be weak when there overfitting. The calculated correlation metrics are compared with a threshold (more details on how we select the threshold in Section 4) to determine if there overfitting. We choose three correlation metries: Spearman, Pearson (Hauke & Kossowski, 2011), and time-lagged Pearson correlation coefficients. We calculate both Spearman and Pearson correlation coefficients since we do not know whether the relationship between training and validation loss is linear. In addition, we compute the Pearson correlation coefficient between the time-lagged version (5-epoch lagged) of the training loss and the validation loss. This method is inspired by autocorrelation (Brockwell & Davis, 2002), which computes the correlation between a time series data and a time-lagged version of itself.
2.3 OVERFITTING PREVENTION
Bejani & Ghatee (2021) identify three categories of overfitting prevention methods: passive, ac- tive and semi-active methods. Passive methods are employed before training a model, and include methods such as
optimization and model selection. For instance, Sun et al. (2017) improve the backpropagation algorithm to speed up the training process and avoid overfitting. Xu al. (2021) introduce a learning algorithm based on a probabilistic model for avoiding overfitting to the noise in the training data. Active methods prevent overfitting by either imposing noise to the data or model through methods like adding dropout layers or other regularization schemes so that models cannot memorize the patterns in the data. For instance, Dropout (Srivastava et al., 2014), simple and popular overfitting prevention method, randomly disables a part of the DL model during training prevent overfitting. Finally, semi-active methods change the model architecture during the training process. They either work by adding hidden nodes or pruning existing nodes. All aforemen- tioned overfitting prevention methods are typically intrusive (i.e., they require either modification of the model internals or data that is fed to the model) and require considerable expertise to accurately execute. For instance, using passive methods such as hyperparameter optimization to avoid overfit- ting requires expertise on the right optimization method to choose and the right parameters to tune, which is a vast area of research in itself (Bergstra & Bengio, 2012; Falkner et al., 2018; Bischl al., 2021). Similarly, active and semi-active methods such dropout or pruning require either adding layers or dynamically editing the model structure. In addition, even though these methods have been known to avoid overfitting they cannot guarantee that the model does not overfit and they typically employ methods like Early stopping (Morgan & Bourlard, 1989; Prechelt, 2012) to further predict if overfit might occur. Early stopping is widely used overfitting prevention method (which we explain below) that is non-intrusive and does not require considerable expertise to execute.
Early stopping. Early stopping stops training when there is no improvement in fixed number of epochs (indicated by the patience parameter) and returns the best epoch which has the lowest validation loss. The idea behind this method is that the training will converge become overfit.
3

---

Under review as a conference paper at ICLR 2023
Training the time series classifier
Continue training the model
Feed data (with classifier for training
Trained time series classilier
Training histories with labels
Use the whole observed history
Extract the latest history with rolling window
Overfitting detection using the training history
momes 24
Training history
Perform inference identify overfitting
N
everfittin?
Ovarliting detection using training history
Validation
Interpolate the validation losses to the same length as the data in training
Overfitting non-overfitting
Stop training return epoch that has the lowest validation loss
Overfitting prevention when training model
Figure 2: Our method for overfitting detection and prevention.
when the validation loss stops improving. However, Prechelt (2012) studied three stopping criteria of early stopping and found that using slow stopping criterion will increase the training time while producing only a small improvement in generalization. In this paper, we propose using overfitting detection methods during the training process to prevent overfitting. Hence, our method prevents overfitting based on the byproduct (i.e., the training history) of the training process. We compare our method to the early stopping method, as both methods are non-intrusive and do not require significant expertise. We also include the results of an alternative version of early stopping based on smoothed validation loss in Appendix E.
OUR METHOD
Figure 2 shows an overview of our proposed method. Our method uses time series classifier to detect and prevent overfitting. To the best of our knowledge, we are the first to use time series classification-base
method to detect and prevent overfitting. First, we collect simulated dataset (more details on how we collect the data in Section 4) that contains training histories (i.e., training and validation loss curves, however we only use the validation loss curves in our method) with labels indicating whether overfitting occurs in order to train our time series classifier. Second, we train a time series classifier on all the training histories of the simulated dataset. We evaluate six state-of- the-art time series classifiers (see Appendix A) to identify the best- nest-performing one. Finally, we use the trained time series classifier to perform both overfitting detection and prevention follows.
Overfitting detection. To detect overfitting in a trained model, we first collect its validation losses over the training epochs. We feed this loss to trained time series classifier to detect if there is overfitting. However, we cannot directly feed these validation losses to our classifier as the length of the validation losses might not be of the same length as that of the data used to train these time series classifiers. Except KNN-DTW all the studied time series classifiers expect the length of the inputs used for training for which the inference is made be the same. Therefore, we first linearly interpolate the validation losses of the model for which we need to detect overfit to the same length as the training histories used train the studied time series classifiers. We feed the interpolated validation losses to our trained time series classifier and perform inference to determine if the model overfit. Figure 4 shows how the linear interpolation process works; if we only have validation losses over 8 epochs and our time series classifier was trained over 80 epoch validation loss values, we interpolate the 8 epoch losses to 80 so that we can feed it to the trained time series classifier.
Overfitting prevention. To prevent overfitting, we feed the training history (i.e., validation loss curve) of a DL model that is being trained to our trained time series classifier during the training process. The history is fed for inference in two different ways: (1) as rolling window: we extract the latest history in a fixed window size (e.g.. the latest 20 epochs), and (2) as the whole observed history (from the first to the latest epochs). Our time series classifier detects if in the fed history
4

---

Under review as a conference paper at ICLR 2023
Datasets
Downlbad the delasens for everliting simulation
Simulate overfiting by training neural networks
Label training histories
Simulated traning cataset (419 Uraining histories)
Original data
for papers have camples of overliting
contarences and Journats
training history reproducc the training history
Roal Upiining Histoo ees)
Loss
40
8
Experiment
Evaluation
Linear interpolation
Choose threshokds for
Condatiion based methods with threehokde
Compane overfiling detection
60-
Trained time seriee classifiars
Compare overliting prevention methods
Train our mathods
Early slopping
60
80
Figure 4: An example of lin- early interpolation.
Figure 3: Overview of the experimental setup.
overfitting occurs. Similar to overfitting detection, we linearly interpolate the data before feeding it into our model. If there no overfit occurring. we continue the training and repeat the above procedure until the model has finished training. For the rolling window, we move the window by fixed step size and make another prediction. If our model detects the presence of overfit in the fed history, we return the lowest validation loss in the observed epochs as the best epoch.
EXPERIMENTAL SETUP
In this section, we introduce the datasets for training and evaluating the studied overfitting detection and prevention methods, the experiments of our study, and the evaluation metrics for the studied methods. Figure 3 shows an overview of the experimental setup.
4.1 DATASETS
Simulated training dataset. We use simulated training histories with labels to determine the thresh- old for the correlation-based methods (see Section 2.2) and to train our method as explain in Section 3. We create our simulated dataset by training neural networks with different model com- plexities to generate the training history of overfitting and non-overfitiing samples as follows:
Step Download the datasets for overfitting simulation. We download 12 datasets of real-world problems from the Proben1 (Prechelt et al., 1994) benchmark set for training neural networks (the information about the studied datasets can be found in Appendix A). These datasets were used by Prechelt (2012) simulate training histories for studying early stopping. Furthermore, all of these datasets (except the "building" one) were originally collected from the UCI machine learning repos- itory (Blake al., 1998), which has been widely used in deep learning research (Kuleshov al., 2018; Sajeev et al., 2019; Shi et al., 2021; Gadde et al., 2021). These datasets are pre- pre-partitioned into training, validation, and test (respectively 50%, 25%, and 25% of the data). addition, Proben1 partitions each dataset three times in order to generate three distinct permutations. Hence, total we collect 36 datasets from Probenl.
Simulate overfisting by training neural networks. We train Neural Networks (NNs) with various architectures on the collected 36 datasets. We do so to vary the model complexity which in turn increases the chance of producing their study. The input/output layer contains the same number of nodes as the number of input/output coefficients of the datasets (see Appendix A) and rectified linear units (ReLUs) are used for all hidden layers. The structures of the NNs are as follows: (1) 6 heehhhdddddeeee NNs with hidden nodes of 4, 8, 16, 24, 32, and (2) wwo-hidden-layer NNs with hidden nodes (represented as first
Step 2
overfitted model, in the same way Prechelt (2012) did in
5

---

Under review as a conference paper at ICLR 2023
layer hidden nodes + second layer hidden nodes) of 2+2, 4+2, 4+4, 8+4, 8+8, 16+8. We use the mean square error (MSE) as the loss function for regression problems, and cross entropy as the loss function for classification problems. Additionally, problems. To increase the likelihood of overfitting, we train these 12 neural network architectures on each dataset (of the collected 36 datasets) for 1,000 epochs, producing 432 training histories.
used SGD as the optimizer for all of these
Step 3 Label training histories. To train our proposed method and the correlation- based methods, need manually label the training history as either "overfit", "non-overfit" or "uncertain". To ensure the robustness of our manual labelling process, we follow the approach outlined by Ding al. (2020). The first and second authors of this paper independently labelled the 432 data points and discussed the results. In the first discussion round, the authors reached a 95% agreement (410 data points), and 10 data points that were labelled as "uncertain" by both authors were eliminated. In the second round, we discussed the 22 disagreements. Following the discussion, we eliminated 3 data points (labelled "uncertain" by both authors) and agreed on the labels for the remaining 19 data points. The final data set consists of 44 overfit and 375 non-overfit training histories. As an alternative (automated) approach for collecting the labels we experimented with a heuristic method, however, this method did not perform well (see Appendix H).
Real-world test dataset. To evaluate our method on real-world data, we surveyed papers from conferences and journals to collect samples of overfit and non-overfit models:
Step 1 Identify related conferences and journals. We identify related conferences and journals based the Computing Research and Education Association of Australasia (CORE') and China Computer Federation (CCF)) ranking systems. Under the CCF A rank, we have 7 conferences and 4 journals in the "Artificial Intelligence" field. Under the CORE A* rank, we have 16 conferences in the "machine learning" and "artificial intelligence" fields and 12 journals in the "artificial intelli- gence and image processing" field. We get 17 conferences and 12 journals after merging the results because of the overlap between these two ranking systems.
Search for papers that have samples of overfitting. We found 33 full papers (see Ap- pendix C) with the "overfit" keyword (which includes e.g., "overfittingg" in the title that were pub- lished at the selected conferences and journals in the last 5 years. Five papers contain samples of overfitting: P2- Chatterjee & Mishchenko (2020); P4 - Chen et al. (2021) P13 Kim et al. (2021); P17 Rice et al. (2020) and P23- Singla et al. (2021). Appendix C lists the papers and the number of collected samples of overfitting (some of them also provide samples of non-overfitting).
Step 2
Step 3 - Collect existing training history or reproduce the training history. Paper P17 shared the training history, making its replication straightforward. We replicated the other papers that provide overfitting samples to collect the training histories of these samples. We ran the code from the papers that provide replication packages (P4, P13, and P23) to generate the training history, but were unable to replicate paper P13's results. We followed the methodology to replicate the results and training history for paper P2, which did not provide replication package. In total, we collected 29 training histories of overfit models and 11 of non-overfit models (see Appendix C for details).
4.2 EXPERIMENTS
Overfitting detection. We train the time series classifiers based the simulated dataset. We per- formed a grid search with 3-fold cross validation to tune the hyperparameters for each classifier based the simulated dataset. After selecting the hyperparameters, we trained each time series classifier using the training histories and labels from the simulated dataset and saved the trained classifier. Furthermore,
search thresholds for the correlation- bbased methods on the simulated dataset. We perform a grid search for the thresholds between -I and I based on the collected simu- lated dataset select the threshold which has the best F-score.
Overfitting prevention. We reused the trained time series classifiers from the previous step to perform inference during the training process to prevent overfitting. We applied our method to the trained models in every 10 epochs (i.e., the step size) with 20, 40, 60, 80, and 100 epochs as different rolling window sizes. We set the patience values for early stopping from 5 to 1 15 epochs.
hittps://www.oom http://wwwc.rreeeeeoeooe www.eed.duc...eee.e htt.//oofreeeepoortte. hhttps:/ccot atoom. im/

---

Under review as a conference paper at ICLR 2023
Table Results of the overfitting detection methods on the real-world dataset (Prec: precision; Rec: recall; F-s: F-score; Avg F-s: average F-score), and the time cost of training the studied methods on the simulated dataset and performing inference on the real-worle dataset (per sample).
Non-overfitinng
Overfitting Rec
Avg F-s
Training time (s)
Inference time (ms)
Detection method
Prec
Rec
F-s
Prec
F-s
Spearman
0.71
0.91
0.80
0.96
0.86
0.91
0.86
2.461
0.908
Correlation based
Pearson
0.78
0.64
0.70
0.87
0.93
0.90
0.80
0.222
0,025
Autocorr
0.80
0.73
0.76
0.90
0.93
0.92
0.84
0.233
0,026
KNN-DTW
0.79
1.00
0.88
1.00
0.90
0.95
0.91
0.001
180.512
Time
HMM-GMM
0.30
0.27
0.28
0.73
0.76
0.75
0.52
99.751
17.750
series
TSF
0.77
0.91
0.83
0.96
0.90
0,93
0,88
0.311
17.209
classifier
TSBF
0.79
1.00
0.88
1.00
0.90
0.95
0.91
0.301
31.683
(ours)
BOSSVS
0.46
0.91
0.61
0.94
0.59
0.72
0.67
1.877
19.342
SAX- VSM
0.83
0.91
0.87
0.96
0.93
0.95
0.91
0.912
17,474
4.3 EVALUATION
Evaluation metries for overlitting detection. To evaluate the classification performance of over- fitting detection methods, we calculated the precision, recall, and F-score for overfitting and non- overfitting samples in the real-world test dataset. In addition,
calculated the average F-score for directly comparing the classification performance the studied methods. To evaluate the time cost of training and using the studied methods, report the training time (in seconds) of each method on the simulated dataset and the inference time (in milliseconds) on the real-world dataset.
Evaluation metrics for overfitting prevention. Ideally, overfitting prevention method returns the optimal epoch that has the optimal predictive performance for the model on the validation set and stops the training process as fast possible. We define the optimal rate of overfitting prevention method as the percentage of times the optimal epoch is identified. To evaluate the speed of the method, we define the delay as the epoch difference between the stopped epoch and the best epoch, e e.g., the delay will be 10 epochs if prevention method stops at the 123th epoch while the 113th epoch is the best one. For early stopping, the delay will be the same as the patience parameter.
5 RESULTS
Overfitting detection. Our overfitting detection approach using time series classifiers (except HMM-GMM and BOSSVS) has a better classification performance than the correlation-bassd meth- ods for overfitting detection. Table shows that our approach using KNN-DTW, TSBF, and SAX- VSM have the best F-score (0.91) on the real-world dataset, followed by TSF which outperforms the baseline methods as well. Even though our approach using BOSSVS achieves an F-score of on the simulated dataset (see Appendix B), it performs poorly on the real-world dataset. From the results both simulated and real-world datasets, the HMM-GMM time series classifier performs poorly for the problem of overfitting detection. We also note that the investigated correlation-base methods have a reasonably good performance. All of the studied correlation-based overfitting de- tection methods have F-scores above 0.8. However, our approach outperforms the best performing correlation-bar based overfitting detection approach by least 5% the studied real-world dataset. We also report the results of using perturbation validation for overfitting detection as an example of an intrusive overfitting detection method in Appendix D.
The studied time series classifiers are more computationally intensive than correlation- meth- ods for inference, yet they are still useful in practice As shown Table our method requires more time for performing inference than the correlation- inference time among the classifiers but around 20 times slower than the Spearman correlation- based method and around 700 times slower than the other two correlation-based methods. However, the speed of our method is not prohibitive in practice since overfitting detection is only executed once after the training is complete. It is also useful note that the training times of the time series classifiers in our approach are not excessive. For instance, the training times of TSF and TSBF are around 300 milliseconds and KNN-DTW, our best performing time series classifier finish train-
methods. For instance, TSF has the fastest

---

Under review as a conference paper at ICLR 2023
Patience (epoch),
10 15 20
25 30 35 40 45 50 55 60 65 70 75 80 as 90 95 100105110115
1.0
0.8
Optimal rate
Early stopping
0.4
TSF
0.2
TSBF
SAX-VSM
0.0
BOSSVS
60
100
Window size (epoch)
Figure 5: The optimal rate of our methods (using a rolling window) and early stopping with different patience values.
ing in millisecond. However, KNN-DTW requires the longest time for inference which is around 180 milliseconds for training history.
Overfitting prevention. Our method with KNN-DTW (both using rolling window and whole ob- served history) often more accurate than early stopping for overfitting prevention. Other studied classifiers do not perform as well as KNN-DTW for overfitting prevention. As Figure 5 and Table show, both our approach based on rolling window and whole observed history has higher accuracy than early stopping at identifying the optimal epoch. In particular, our approach with KNN-DTW based on rolling window is more accurate (see Figure 5) than early stopping when using up to 80 epochs as the patience parameter and window size. For example, our method with KNN-DTW obtains 78% accuracy when setting the window size to 20 epochs, while early stopping has 48% accuracy when setting the patience parameter to that number of epochs. However, we find that early stopping achieves nearly perfect accuracy when the patience is larger than 80 epochs. The reason is that 90% of the training histories in the real-world dataset have around 200 epochs, hence, large patience value makes it easy for early stopping to choose the optimal epoch. In addition, Table method with KNN-DTW based the whole observed history also obtains higher accuracy than early stopping. For instance, the KNN-DTW classifier has 95% accuracy with a me- dian delay of 43.5 epochs while early stopping has only 83% using the same number (i.e., between 40 to 45 epochs patience in Figure 5).
shows that
Table 2: The optimal rate and median delay of our overfitting prevention methods that are based on the whole observed history.
Table 3: The median delay of our overfitting preven- tion methods that are based on the rolling window with different window sizes.
Optimum rate
Median delay
Window size (epoch)
Classifier
Classifier
20
40
60
80
100
KNN-DTW
0.95
43.5
KNN-DTW
31.0
27.0
37.5
42.5
45.5
HMM-GMM
0.18
0.0
HMM-GMM
5.0
6.5
16.5
28.0
41.5
TSF
0.90
35.0
TSF
12.5
22.0
31.0
39.5
44.0
TSBE
0.83
31.0
TSBF
8.5
15.0
25.0
37.0
47.0
BOSSVS
0.65
21.0
BOSSVS
7.0
29.0
34.5
48.5
56.5
VSM
0.33
10.0
SAX-VSM
4.0
11.0
9.5
16.0
24.0
Our method using KNN-DTW and a rolling window can stop training a DL model earlier than early stopping. As shown in Table 3, with the same number of epochs for the patience parameter and window size, our method with KNN-DTW based on the rolling window (except window size 20) can save training time (i.e., there is smaller delay between the stopped epoch and the best epoch) over early stopping. For instance, when setting both the patience parameter and window size to 40 epochs, KNN-DTW and early stopping have the accuracy and KNN-DTW has
8

---

Under review as a conference paper at ICLR 2023
Traliing loss
Traiiing loss
Stopped 1155
Vallsstion
Validition loss
Earty stooping best epoch
2.0
stopping
KNN DTW best epecy
KNil DTW best spoch
MM
Optimol stopping
Stopped epoch
1.8
X
Optinal stopping point
loie UTA
Loss
Stepped spach
1.4
Stooped oF #74 epoch
156
1.0
0.6
125
150
200
50
75
300
150
175
200
Epoch
Epoch
(a) Our method stops earlier than the early stopping but both achieve the same optimal epoch.
(b) Our method stops later than the early stopping but achieves the optimal epoch.
Figure 6: Examples of overfitting prevention based on KNN-DTW (set the window size as 40 epochs) and early stopping (set the patience parameter 40 epochs).
median delay of 27 epochs while early stopping has fixed delay of 40 epochs (which is the same as the patience parameter). In comparison to the delay in early stopping, the delay between the stopped epoch and the best epoch is at least 32% shorter with our method using KNN-DTW (see Appendix F for the significance testing results). Figure 6a shows an example in which early stopping and our method both identify the optimal epoch, but our method stops 21 epochs earlier than early stopping (which stops with a 40 epochs delay). In addition, our method does not sacrifice accuracy for a shorter delay (as shown in Figure 5). Figure 6b shows an example in which our method stops later than early stopping but identifies the optimal epoch when using the same number of epochs for the patience parameter and window size. Furthermore, our overfitting detection approach can be used with zero-one validation loss, the results of our approach and early stopping with zero-one validation loss can be found Appendix G.
Among our two approaches for overfitting prevention we recommend the usage of KNN-DTW with rolling window. Though using the whole observed history may be more accurate at predicting the optimal epoch than using a rolling window for our approach, we note that we can predict the optimal epoch much earlier with the rolling window approach for a very small trade-off accuracy. As shown in the Table 3 and Figure 5, our method with KNN-DTW achieves 83% accuracy with median delay of 27 epochs and 90% accuracy with a median delay of 37.5 epochs using the window size 40 and 60 epochs respectively. However, the median delay of KNN-DTW when using whole observed history is 43.5 while using the rolling window with a window size of 80 or more epochs can achieve a higher accuracy (98% 95% accuracy) with shorter delay (42.5 vs. 43.5 epochs). In summary, we suggest using the rolling window approach since it is stops earlier with a relatively small accuracy drop using a small window (e.g., 40 epochs) and performs better than the observed whole history approach when using a large window size (e.g., 80 epochs).
6 CONCLUSION AND FUTURE WORK
We propose a non-intrusive overfitting detection and prevention method that based on time series classifiers trained on the training history of DL models. Our method (when using the KNN-DTW time series classifier) has (1) better classification performance than correlation-basel methods for overfitting detection, and (2) greater accuracy than early stopping for overfitting prevention with shorter delay. We do so using a real-world dataset of labelled training histories collected from the papers published at top AI venues in the last 5 years. Furthermore, the trained time series classifiers are included in our replication package for use by other researchers. One of the downsides of our approach is that our best performing time series classifier takes longer to perform the inference quired to detect and prevent overfit than the studied baselines. We encourage future work to optimize time series classifiers to enable overfitting detection and prevention in real-time with smaller delays.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The replication package can be found at: https: petps://githule com/ /twoo//oooso//oorrriiiii detect. This package includes the code for training and using the studied methods, notebooks for analysing the results, the simulated and real-world datasets, and the trained models. Furthermore, we are currently developing tool to use the studied methods and will make public once completed.
Experimental environment. We use Python 3.8 with TensorFlow 2.9.0 and rum experiments on Ubuntu 20.04 with Linux kernel 5.15.0. The hardware specifications are as follows: (1) NVIDIA RTX 3090 GPU with 24 GB memory (the versions of CUDA and cuDNN are 10.1.243 and 7.6.5), (2) 3.50 GHz Int Core(TM) 19-1K CPU, and (3) 64 GB RAM.
REFERENCES
Basavaraj S Anami and Venkatesh A Bhandage. A comparative study of suitability of certain fea- tures in classification of bharatanatyam mudra images using artificial neural network. Neural Processing Letters, 50(1):741-769, 2019.
Daniel Bashir, George D. Montañez, Sonia Sehra, Pedro Sandoval Segura, and Julius Lauw. An Information-Theoreti Perspective on Overfitting and Underfitting. In Marcus Gallagher, Nour Moustafa, and Erandi Lakshika (eds.), A/ 2020: Advances in Artificial Intelligence. Lecture Notes in Computer Science, pp. 347-358, Cham, 2020. Springer International Publishing. ISBN 978-3- 030-64984-5. doi: 10. 00077778-33030-6498800
Mustafa Gokce Baydogan, George Runger, and Eugene Tuv. ofg-to-features framework to clas- sify time series. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35((1):2776- 2802, 2013. doi: 10.1 00.1009/PAM...013.72..
Mohammad Mahdi Bejani and Mehdi Ghatee. A systematic review overfitting control in shallow and deep neural networks. Artifccall Intelligence Review, ((18))6391-6433 2021.
James Bergstra and Yoshua Bengio. Random search for hyper-parametee optimization. Journal of machine learning research, 13(2), 2012.
Gérard Biau and Erwan Scornet. A random forest guided tour. Test, 25(2):197-227, 2016.
Bernd Bischl, Martin Binder, Michel Lang, Tobias Piclok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Mare Becker, Anne-Laure Boulesteix, et al. Hyperparameter optimization: Foundations, algorithms, best practices and open challenges. arXiv preprint arXiv:2107.05847, 2021.
C. Blake, E. Keogh, and Christopher Merz. UCI repository of machine learning databases. 01 1998.
Jörg Bornschein, Francesco Visin, and Simon Osindero. Small data, big decisions: Model selec- tion in the small-data regime. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. JMLR.org. org, 2020.
Pavel Brazdil, Jan van Rijn, Carlos Soares, and Joaquin Vanschoren. Metalearning: Applications to Automated Machine Learning and Data Mining. Springer Nature, 2022.
Peter Brockwell and Richard Davis. Introduction to time series and forecasingg.. Springer, 2002.
Rich Caruana, Steve Lawrence, and C Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, 13. 2000.
Gavin C Cawley and Nicola LC Talbot. Preventing over-fitting during model selection via bayesian regularisation of the hyper-parameters. Journal of Machine Learning Research, 8(4), 2007.
Satrajit Chatterjee and Alan Mishchenko. Cireuit-based intrinsic methods to detect overfitting. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1459-1468.
PMLR, 13-18 Jul 2020.
10

---

Under review as a conference paper at ICLR 2023
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2021.
Houtao Deng, George Runger, Eugene Tuv, and Martyanov Vladimir. A time series forest for classification and feature extraction. Information Sciences, 239:142-153, 2013. ISSN 0020-0255. 1ttp//.00001.......
doi: https://doi.or//I 10. 101660. int.l..ss..........2.
Hui Ding, Goce Trajcevski, Peter Scheuermann, Xiaoyue Wang, and Eamonn Keogh. Querying and mining of time series data: Experimental comparison of representations and distance measures. Proc. VLDB Endow., 100:1542-1552, aug 2008. ISSN 2150-8097. doi: 10.14778/1454159. 1454226.
Zishuo Ding, Jinfu Chen, and Weiyi Shang. Towards the use of the readily available tests from the release pipeline performance tests: Are there yet? In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE '20, pp. 1435 1446, New York, NY. USA, 2020. Association for Computing Machinery. ISBN 9781450371216. doi: 10.1145/ 3377811 1.330051..
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Antorne Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11: 625 660, mar 2010. ISSN 1532-4435.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. In International Conference Machine Learning, pp. 1437-1446. PMLR, 2018.
Zhouyu Fu, Guojun Lu, Kai Ming Ting, and Dengsheng Zhang. Music classification via the bag-of- features approach. Pattern Recognition Letters, 32(14):1768-1777, 2011. ISSN 0167-8655. doi: https://doi.org/10..u 10.///00..111..... patree/. 201 1.0..00..
Sridevi Gadde, A. Lakshmanarao, and S. Satyanarayana. Sms spam detection using machine learn- ing and deep learning techniques. In 2021 7th International Conference Advanced Com- puting and Communicution Systems (ICACCS), volume pp. 358-362, 2021. doi: 10.1 10.1 ICACCSS1143..2221.4411
J. Gauvain and Chin-Hui Lee. Maximum posteriori estimation for multivariate gaussian mixture observations of markov chains. IEEE Transactions on Speech and Audio Processing. 2(2):291- 298, 1994. doi: 10.1109/89.279278.
David Hand. Principles of data mining. Drug safety, 30(7))621-62)) 2007.
Jan Hauke and Tomasz Kossowski. Comparison of values of Pearson's and Spearman's 's correlation coefficients on the same sets of data. Quaestiones Geographicae, 30(2):87-95 2011. doi: doi:
10.2478/v10117 17-01 1-0021-..
Douglas M Hawkins. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12, 2004.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Shihao Ji, B. Krishnapuram, and L. Carin. Variational bayes for continuous hidden markov mod- els and its application to active learning. JEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):522-532, 2006. doi: 10.1 (0.1109//PAM..2006.555
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):8119- 8127, May 2021.
11

---

Under review as a conference paper at ICLR 2023
Gabriel Kronberger, Michael Kommenda, and Michael Affenzeller. Overfitting detection and adap- tive covariant parsimony pressure for symbolic regression. In Proceedings of the 13th Anmual Conference Companion on Genetic and Evolutionary Computation, GECCO 11, pp. 631-638, New York, NY, USA, 2011. Association for Computing Machinery. ISBN 9781 9781450306904, doi: 81844//200188...0 2002060.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprint arXiv: 17101166., 2017.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2796-2804, PMLR, 10-15 Jul 2018.
Jessica Lin, Eamonn Keogh, Li Wei, and Stefano Lonardi. Experiencing sax: novel symbolic representation of time series. Data Mining and knowledge discovery, 15(2):007-144, 2007.
Jeffrey D. Long, Du Feng, and Norman Cliff. Ordinal Analysis of Behavioral Data. In Irving B. Weiner (ed.), Handbook of Psychology, chapter 25, pp. 635-661. John Wiley & Sons, Inc., Hobo- ken, NJ, USA, April 2003. ISBN 978-0-471-26438-5 doi: 10.
H. B. Mann and D. R. Whitney, On test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18:50-60, 1947.
Felix Mohr and Jan N van Rijn. Towards model selection using learning curve cross-validation. In 8th ICML. Workshop on automated machine learning (AutoMI.), 2021.
Felix Mohr and Jan N. van Rijn. Learning curves for decision making in supervised machine learning
A survey. CoRR, abs/2220.122500 2022. URL htttttt//aaxxi..cooo htrggabb//2200 12150.
Kumar Molugaram and G. Shanker Rao. Statistical Techniques for Transportation Engineer- ing. Henterwoort--einemmaae January 2017. ISBN 978-0-12-811555-8,
doi: 10.1016/
B978-0- 11-8-0-12-8111 1555-8.0000111
Nelson Morgan and Hervé Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. Advances in neural information processing systems, 2, 1989.
Andrew Y. Ng. Preventing "overfitting" of cross-validation data. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML '97, pp. 145-253, San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc. ISBN 1558604863.
Steven J. Nowlan and Geoffrey E. Hinton. Simplifying neural networks by soft weight-sharing. Neural Computation, 4(4):473-93,, 1992. doi: 10. 166/ne2/. 1992.4.4.473.
Danni Peng, Sinno Jialin Pan, Jie Zhang, and Anxiang Zeng. Learning an adaptive meta model-
generator for incrementally updating recommender systems. In Proceedings of the 15th ACM
Conference on Recommender Systems, RecSys '21, pp. 411--421 New York. NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384582. doi: 10.144/344023334442233
Tao Peng. Lu Liu, and Wanli Zuo. PU text classification enhanced by term frequency inverse docu- ment brequency-improve weighting. Concurrency and Computation: Practice and Experience, 26(3):728-741, 2014. doi: http:///do..rr/11..00000.0/0////
Lutz Prechelt. Early Stopping - But When? In Grégoire Montavon, Geneviève B. Orr, and Klaus- Robert Müller (eds.), Neural Networks: Tricks of the Trade: Second Edition, Lecture Notes in Computer Science, pp. 53-67, Springer, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-8 doi: 10.1007/978-3 642-6422899
Lutz Prechelt al. 1roben1: A set of neural network benchmark problems and benchmarking rules. 1994.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 8093-8104. PMLR, 13-18 Jul 2020.
12

---

Under review as a conference paper at ICLR 2023
Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, Jeff Skowronek, and Linda Devine. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen's sd indices the most appropriate choices. In annual meeting of the Southern Association for Institutional Research, pp. 1-51. Citeseer, 2006.
Shelda Sajeev, Anthony Maeder, Stephanie Champion, Alline Beleigoli, Cheng Ton, Xianglong Kong, and Minglei Shu. Deep learning to improve heart disease risk prediction. In Machine Learning and Medical Engineering for Cardiovascular Health and Intravascular Imaging and Computer Assisted Stenting, pp. 96-103. Springer, 2019.
Gerard Salton, Anita Wong, and Chung-Shu Yang. A vector space model for automatic indexing. Communications of the ACM, 18(1 1):63363,,,, 1975.
Patrick Schäfer. Scalable time series classification. Data Mining and Knowledge Discovery, 30(5): 1273-1298, 2016.
Patrick Schäfer and Mikael Högqvist. SFA: A symbolic fourier approximation and index for simi- larity search in high dimensional datasets. Proceedings of the 15th International Conference on Extending Database Technology, EDBT '12, pp. 516--527 New York. NY, USA. 2012. As- sociation for Computing Machinery. ISBN 978 9781450307901. doi: 10. 10.1145/224559622247666
Qiushi Shi, Rakesh Katuwal, P.N. Suganthan, and M. Tanveer. Random vector functional link neural network based ensemble deep learning. Pattern Recognition, 117:107978, 2021. ISSN 0031- 3203. doi: https://doi.org// 10. 00006//j.aacog2222110099999
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1 2019.
Robert H. Shumway and David S. Stoffer. Time Series Analysis and Its Applications: With R Exam- ples. Springer Texts in Statistics. Springer International Publishing, Cham, 2017. ISBN 978-3- 319-52451- 978-3-319-1 --455-2---2888 doi: 10. 10010000-7--33100 52452-8.
Vasu Singla, Sahil Singla, Soheil Feizi, and David Jacobs. Low curvature activations reduce overfit- ting in adversarial training. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 16423-16433, October 2021.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational over- fitting in reinforcement learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 1006344::41-471,, 1987.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1229-195 2014.
Benjamin Strang, Peter van der Putten, Jan N. van Rijn, and Frank Hutter. Don't rule out sim- ple models prematurely: large scale benchmark comparing linear and non-linear classifiers in openml. In Wouter Duivesteijn, Arno Siebes, and Antti Ukkonen (eds.), Advances in Intelligent Data Analysis XVII, pp. 303-315, Cham, 2018 Springer International Publishing. ISBN 978-3- 030-01768-2.
Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meProp: Sparsified back propagation for accelerated deep learning with reduced overfitting. In International Conference on Machine Learning, pp. 3299-3308 PMLR, 2017.
Jan N. van Rijn. Salisu Mamman Abdulrahman, Pavel Brazdil, and Joaquin Vanschoren. Fast algo- rithm selection using learning curves. In Elisa Fromont, Tijl De Bie, and Matthijs van Leeuwen (eds.). Advances in Intelligent Data Analysis XIV, pp. 399, Cham, 2015. Springer Interna- tional Publishing. ISBN 978-3-319-244635
Onur Varol, Emilio Ferrara, Filippo Menezer, and Alessandro Flammini. Early detection of pro- moted campaigns social media. EPJ data science, 6:1-19, 2017.
13

---

Under review as a conference paper at ICLR 2023
Zhongju Wang. Long Wang, Chao Huang, Zijun Zhang, and Xiong Luo. Soil-moisture sensor-based based automated soil water content cycle classification with hybrid symbolic aggregate approximation algorithm. IEEE Internet of Things Joumnal, 1303:14003-1011111 2021.
Roman Werpachowski, András György, and Csaba Szepesvári. Detecting overfitting via adversarial examples. Advances in Neural Information Processing Systems, 32, 2019.
Xiaopeng Xi, Eamonn Keogh, Christian Shelton, Li Wei, and Chotirat Ann Ratanamahatana, Fast time series classification using numerosity reduction. Proceedings of the 23rd international conference on Machine learning, pp. 1033-1040, 2006.
Le Xu, Lei Cheng, Ngai Wong, and Yik-Chung Wu. Overfitting avoidance in tensor train factor- ization and completion: Prior analysis and inference. 2021 IEEE International Conference on Data Mining (ICDM), pp. 1439- 1444, 2021. doi: 10. (0.1109/CCDM5162) 2021.00185.
Jie M. Zhang, Earl T. Barr, Benjamin Guedj, Mark Harman, and John Shawe- Taylor. Perturbed model validation: A new framework to validate model relevance. CoRR, abs/ 10201, 2019.
URL http: wawiiv.vvvv..///////...////aaavv org/ abs /1905 10201.
Minghang Zhao, Baoping Tang, Lei Deng, and Michael Pecht. Multiple wavelet regularized deep residual networks for fault diagnosis. Measurement, 152:107331, 2020. ISSN 0263-2241. doi:
https://doi.rr/11.101166.. h00p:/0000000.mmmmmmmememm000000
14

---

Under review as a conference paper at ICLR 2023
A STUDIED DATASETS AND TIME SERIES CLASSIFIERS
Table 4 shows the information about the studied datasets. There are 3 datasets for regression tasks and datasets for classification tasks.
Table 4: Information about studied datasets.
Dataset
Type
#In
#Out
#Examples
building
regression
14
4,208
cancer
classification
9
2
699
card
classification
51
690
diabetes
classification
8
2
768
flare
regression
1,066
gene
classification
120
3
3,175
glass
classification
214
heart
classification
35
2
920
hearta
regression
35
920
horse
classification
58
364
soybean
classification
82
19
683
thyroid
classification
21
7,200
Table introduces the studied time series classifiers. Since there has been no prior systematic research time series classifiers for training histories reference, choose the classifiers that have been reported as baselines or state-of-the-art in studies (Xi et al., 2006; Varol et al., 2017; Anami & Bhandage, 2019; Wang al., 2021)
Table Studied time series classifiers.
Classifier
Description
KNN-DTW"
Uses K-Nearest Neighbors (Hand, 2007) and Dynamic Time Warping (Ding al., 2008) as distance metric
HMM-GMM
Uses Hidden Markov Model for modeling time series data and Gaussian Mixture Model as the emissions probability density (Gauvain & Lee, 1994; Ji et al., 2006) Uses random forest (Biau & Scornet, 2016) for time series data using ensemble of time
TSF
series (Deng al., 2013)
TSBF
Time Series Bag-of-Features (Baydogan et al., 20133) extracts features based on bag-of- features approach (Fu et al.. 2011) to create random forest
SAX-VSM
Symbolic Aggregate apmroXimation transforms the data into symbolic representations (Lin al., 2007) and Vector Space Model (Salton al., 1975; Peng al., 2014) transforms them into vectors to calculate similarity for classification
BOSSVS
Bag-of-SFA Symbols in Vector Space (Schäfer, 2016) is similar to SAX-VSM but use SFA (Schäfer & Högqvist, 2012) to transform the data instead of SAX
We do not interpolate the validation loss for KNN-DTW since it does not require a constant length
input signal.
B RESULTS OF TRAINING OVERFITTING DETECTION METHODS
As mentioned in Section 4.2, we tune the hyperparameters of the time series classifiers by perform- ing a grid search with 3-fold cross validation. Table shows that all the classifiers achieve F-scores of more than 0.95 in the 3-fold cross validation except HMM-GMM which obtains only aver- age F-score of only 0.6. Table shows the results training correlation- based methods and our method the simulated dataset. Generally, the time series classifiers achieve better performance on the simulated dataset than the other methods (except the HMM-GMM classifier) and three of the classifiers
correctly identify all the data in the simulated dataset.
We notice that KNN-DTW performs well on both the simulated and real-world datasets. One pos- sible reason for the performance of KNN-DTW might be that DTW is good at measuring similarity
15

---

Under review as a conference paper at ICLR 2023
Table The F-scores of the time series classifiers in 3-fold cross validation (CV) on the simulated dataset.
Classifier
CV1
CV2
CV3
Avg
Variance
KNN-DTW
0.98
0.98
0.96
0.97
0.00
HMM-GMM
0.41
0.38
1.00
0.59
0.08
TSF
0,98
0.98
1.00
0.99
0.00
TSBF
0,98
0.98
1.00
0.99
0.00
BOSSVS
1.00
1.00
1.00
1.00
0.00
SAX-VSM
0.96
0.96
0.96
0.96
0.00
Table 7: Results of overfitting detection methods on the simulated dataset. (Prec: precision; Rec:
recall; F-s: F-score; Avg F-s: macro average F-score)
Non-overfitting
Overfitting Rec
Avg
Detection method
Prec
Rec
F-s
Prec
F-s
F-s
Spearman
0.99
1.00
0.99
0.95
0.89
0.92
0.95
Correlation
Pearson
0.99
0.97
0.98
0.79
0.93
0.85
0.92
based
Autocorr
0.97
0.94
0.95
0.59
0.73
0.65
0.80
KNN-DTW
0.99
1.00
1.00
0.98
0.93
0.95
0.97
Time series
HMM-GMM
0.95
0.55
0.70
0.16
0.75
0.27
0.48
TSF
1.00
1.00
1.00
1.00
1.00
1.00
1.00
classifier
TSBF
1.00
1.00
1.00
1.00
1.00
1.00
1.00
(ours)
BOSSVS
1.00
1.00
1.00
1.00
1.00
1.00
1.00
SAX-VSM
0.99
1.00
0.99
0.98
0.91
0.94
0.97
across curves, which aids KNN distinguishing between overfit and non-overfit samples. con- trast, HMM-GMM performs poorly both the simulated training and real-world test datasets. One possible explanation is that the extracted state models (via HMM) of the curves do not follow a Gaussian probability distribution. BOSSVS correctly identifies all the data in the simulated dataset but performs poorly on the real-world dataset (as shown in Table 1), which may be due to overfitting the simulated dataset.
C SURVEYED PAPERS
Table 8 lists the 33 papers surveyed from top AI conferences and journals. We found 5 papers out of the 33 papers provide samples overfit and fit models and collected 40 samples from these papers (as shown in Table 9).
Table 8: Information about the surveyed papers.
#P
Authors
Title
Venue
Year
P1
Belkin et al.
Overfitting or perfect fitting? risk bounds for classifica- tion and regression rules that interpolate. Fireuit-based intrinsic methods to detect overfitting.
NeurIPS
2018
Chatterjee
ICML
2020
&
Mishchenko Chatterji & Long
P3
Foolish crowds support benign overfitting.
JMLR
2022
P4
Chen al.
Robust overfitting may be mitigated by properly learned smoothening.
ICLR
2021
P5
d' d'Ascoli al.
Triple descent and the two kinds of overfitting: where & why do they appear?
NeurIPS
2020
Continued on next page
16

---

Under review as a conference paper at ICLR 2023
Table continued from previous page
#P
Authors
Title
Venue
Year
P6
Feldman
The advantages of multiple classes for reducing overfit- ting from test set reuse.
ICML.
2019
al.
P7
Feldman
Open problem: how fast can multiclass test be overfit?
COLT
2019
al.
P8
Frei et al.
Benign overfitting without linearity: neural network classifiers trained by gradient descent for noisy linear data.
COLT
2022
P9
He et al.
Sparse double descent: where network pruning aggra- vates overfitting.
ICMI.
2022
P1O
Huang et al.
Sparse progressive distillation: resolving overfitting un- der pretrain-ann Overfitting be harmless for basis pursuit, but only to degree.
ACL
2022
paradigm.
P11 Ju et al.
NeurIPS
2020
P12
Ju al.
On the generalization power of overfitted two-layer neural tangent kernel models.
ICML
2021
P13
Kim al.
Understanding catastrophic overfitting single-step adversarial training.
AAAI
2021
P14
Kochler al.
Uniform convergence interpolators: Gaussian width, norm bounds and benign overfitting.
NeurIPS
2021
P15
Liu et
Overfitting the data: compact neural video delivery via content-aware feature modulation.
ICCV
2021
P16
Mohammed & Cawley
Over-fitting in model selection with Gaussian process regression.
ICML
2017
Rice al. P17
Overfitting in adversarially robust deep learning.
ICML
2020
P18
Roelofs
meta-analysis of overfitting in machine learning.
NeurIPS
2019
al.
P19
Rozendaal et al.
Overfitting for fun and profit: instance-adaptive data compression.
ICLR
2021
P20
Russo Zou
&
How much does your data exploration overfit? control- ling bias via information usage.
IEEE Trans.
2020
Inf.
Theory
P21
Sanyal et al.
How benign is benign overfitting?
ICLR
2021
P22
Shamir
The implicit bias of benign overfitting.
COLT
2022
P23
Singla et al.
Low curvature activations reduce overfitting in adver- sarial training.
ICCV
2021
P24
Song et al.
Observational overfitting reinforcement learning. Autoencoders that don't 't overfit towards the identity. meProp: sparsified back propagation for accelerated deep learning with reduced overfitting.
ICLR
2020
P25
Steck
NeurIPS
2020
P26
Sun et al.
ICML
2017
P27
Telgarsky
Stochastic linear optimization never overfits with
COLT
2022
sindrattcclly-boundee losses on general data.
P28
Wang et al.
Benign overfitting in multiclass classification: all roads lead to interpolation.
NeurIPS
2021
P29
Webster et al.
Detecting overfitting of deep generative networks via latent recovery.
CVPR
2019
P30
Werpachowski
Detecting overfitting via adversarial examples.
NeurIPS
2019
et al.
P31
Xu al.
Overfitting avoidance in tensor train factorization and completion: prior analysis and inference. Label consistency in overfitted generalized k-means.
ICDM
2021
P32
Zhang Amini
&
NeurIPS
2021
P33
Zhang et al.
Why overfitting isn't always bad: retrofitting cross- lingual word embeddings to dictionaries.
ACL
2020
17

---

Under review as a conference paper at ICLR 2023
Table Information about collected samples from surveyed papers.
Paper
Labels for the training history in the manuscript
#Overfit
#Non-overfit
P2
"[...] the validation accuracy of nn-random is 9.73% (i.e.,
2
close to chance) confirming that it is horribly overfit"
P4
"We first observe that the robust overfitting prevails in all Baseline cases"
3
"Our methods effectively mitigates the robust overfitting" "Figure 4 [...] that is, catastrophic overfitting occurs." "Figure 6 shows that the proposed method also successfully prevents catastrophic overfitting [...]"
P13
N/A
N/A*
P17 "Figure 24 [...] We see clear robust overfitting for the smaller two options in and find no overfitting but highly regular- ized models for the larger two options [...]'
20
P23
"These results therefore validate our claim that low curvature activations reduce robust overfitting"
:: Cannot reproduce the same results the paper.
Table 10:
Results of the Perturbation Validation method for overfitting detection methods on the
simulated dataset.
Non-overfitting Recall
Overfitting Recall
Average F-score
Precision
F-score
Precision
F-score
0.89
0.85
0.87
0.07
0.09
0.08
0.47
D PERTURBATION VALIDATION FOR OVERFITTING DETECTION
Injact noise into training data at levels 0.1.1 0.2,
Retrain model with the training data
Training data a dataset
Model
Training histories (at all noise levels)
Calculate PV
PV threshold?
Overfitiig or non-overfilling
measurement
measurement
Figure 7: Our method for overfitting detection and prevention.
Zhang et al. (2019) suggest the perturbation validation (PV) assessment to determine whether a model fits the training data properly (i.e., ensure that is neither overfitting nor underfit). As shown in Figure 7, We inject three levels of noise 10.1, 0.2, and 0.3) as the original paper (Zhang et al., 2019) into the labels in the training set and retrain the model. We repeat the training for each noise level and collect the training history to compute the PV measurement. The proposed PV measurement will show how much the accuracy decreases in response the injected noise and indicate if overfitting is present. The idea behind this method is that overfit underfit models would lose accuracy more slowly when trained using the poise-injected training set than optimally-fitte models. Since the calculated PV measurement is only one value and we compare it with threshold to determine if there is overfitting. To select the threshold, we performed a grid search for the thresholds between -1 and based the F-score the simulated dataset.
The PV measurement requires retraining the models, which takes a significant amount of compu- tational time but results in poor performance in the simulated dataset (see Table 10). We do not calculate the PV measurement on the real-world dataset since retraining the models takes too long, and the performance the simulated dataset does not justify this extra effort.
18

---

Under review as a conference paper at ICLR 2023
Table 11: Results of early stopping based on the smoothed validation loss curves.
Smooth epochs
Optimum rate
Median delay
Smooth epochs
Optimum rate
Median
Patience
Patience
delay
0
0.48
20.0
0
0.90
60.0
0.38
20.0
0.93
61.0
20
60
10
0,40
20.0
10
0.93
62.0
15
0.55
22.0
15
0.90
62.0
0
0.83
40.0
0.98
80.0
5
0.85
42.0
5
0.98
81.0
40
80
10
0.83
43.5
10
0.98
82.0
15
0.88
44.5
15
0.90
81.5
E EARLY STOPPING BASED ON SMOOTHED VALIDATION LOSS CURVES
An alternate version of early stopping inspects the moving average of the smoothed validation loss curves (Shumway & Stoffer, 2017: Molugaram & Rao, 2017) to determine whether to stop the training process. After terminating the training process, early stopping returns the best epoch which has the lowest validation loss (not the smoothed value). Table 11 shows that using a smoothed validation loss curve may increase the optimal rate and slightly increases the delay. However, does not have better performance than the basic form of early stopping and cannot compete with our approach. Figure 8 shows an example which early stopping achieves the optimal epoch after using the smoothed validation loss. However, the smoothed curves also hurt the performance, such as dropping 10% of the optimal rate when early stopping using a smoothing window of 5 epochs and a patience parameter of 20 epochs. Figure 9 illustrates example of missing the optimal epoch after smoothing the validation loss curve.
F SIGNIFICANCE TEST FOR THE RESULTS OF OVERFITTING PREVENTION
To study the difference in delay across overfitting prevention approaches, we performed the Mann- Whitney test (Mann & Whitney, 1947) at a significance level of = 0.05 to determine whether the distributions of the delay epochs of early stopping and our approach are significantly different. We also computed Cliff's delta (Long et al., 2003) effect size to quantify the difference based on the provided thresholds (Romano al., 2006):
smoothing
22.
isststinn
Unitsron fitun upcch sta czech
Suz spach
12.
10-
1.0
08
0.5
86-
0.6
0
75
100
122
130
175
200
25
30
300
322
150
175
200
Spoch
Figure 8: An example in which early stopping achieves the optimal epoch based on smoothed vali- dation loss (the patience parameter is set to 40 epochs and the smoothing window size is 5).
19

---

Under review as a conference paper at ICLR 2023
smoothing
smoothing
Unidstion
20
20-
fenum
Suz spach
Stap czech
10.
10-
16.
14.
14
300
122
130
175
200
25
50
100
320
155
175
200
Fporh
Spoch
Figure 9: An example in which early stopping cannot achieve the optimal epoch based on smoothed validation loss (the patience parameter is set 20 epochs and the smoothing window size is 5).
Table 12: Significance testing results of the delays for overfitting prevention. (Ws.: Window size)
Effect Size
Cliff's value
Effect
Cliff's
Classifier
Ws.
Classifier
Ws.
P
Size
d value
20
0.000
large
-0.600
20
0.000
large
-1,000
40
0.001
medium
-0.405
40
0.000
large
-0.956
BOSSVS
60
0.000
large
-0.476
SAX-VSM
60
0.000
large
-0.868
80
0.000
large
-0.522
80
0.000
large
-0.817
100
0.001
medium
-0.449
100
0.000
large
-0.681
20
0.000
large
-1.000
20
0.000
large
-0.875
40
0.000
large
-0.966
40
0.000
large
-0.893
HMM-GMM
60
0.000
large
-0.859
TSBF
60
0.000
large
-0.742
80
0.000
large
-0.793
80
0.000
large
-0.748
100
0.000
large -0.633
100
0.000
large
-0.581
20
0.101
small 0.200
20
0.000
large
-0.525
40
0.001
medium
-0.392
40
0.000
large
-0.804
KNN-DTW
60
0.000
large
-0.485
TSF
60
0.000
large
-0.700
80
0.000
large
-0.541
80
0.000
large
-0.709
100
0.000
large
-0.517
100
0.000
large
-0.619
negligible, small, medium,
if [d] 0.144 if < |d| S 0.33
Effect size
if |d| < 0.474
large, 0.0.744< |d|
Table 12 shows that the distributions of delay of early stopping and our methods are significantly different except for KNN-DTW with window size of 20 epochs. In addition, the delays of our methods are shorter than early stopping (except for KNN-DTW with a window size of 20 epochs) with at least medium effect size.
G USING ZERO-ONE LOSS FOR OVERFITTING PREVENTION
Overfitting prevention methods may stop the training process by inspecting the validation accuracy emor (i.e., zero-one loss) rather than the loss utilized for optimizing the model. Figure 10 shows the optimal rate of our approaches as well as early stopping based on zero-one validation loss. When compared to the results in Figure 5 (which uses validation loss), the optimal rate of early stopping based on zero-one loss is quite similar. The results show that the optimal rate improves (with average of 2.5%) when the patience value is between 35 and 90 epochs but declines (with an
20

---

Under review as a conference paper at ICLR 2023
Patience (epoch)
10 15 20
25 30 35 40 45 50 55 60 65 70 75 80 as 90 95 100105110115
1.0
0.8
Optimal rate
Early stopping
0.4
TSF
0.2
TSBF
SAX-VSM
0.0
BOSSVS
60 Window size (epoch)
100
Figure 10: The optimal rate of our methods (using rolling window) and early stopping with differ- ent patience values based on zero-one validation loss curves.
average of 2.9%) when the patience value is less than 35 epochs. For our method with KNN-DTW, the results show that using zero-one loss curves increases the optimal rate, which still outperforms early stopping and other time series classifiers. However, the delay of the KNN-DTW also increases and stops later than early stopping when the window size less than or equal 40 epochs (please see Table 13 for details). Furthermore, we found that our approach with KNN-DTW has higher optimal rate but longer delay while using the zero-one loss curves (compared using the original loss curves).
Patience (epoch)
20 25 30 35
55 60 65 70
90 95 100105110115
1.0
Optimal rate
0.6
Early stopping
0.4
KNN-DIW
HMM-GGM
TSF
0.2
TSBF
SAX-VSM
0.0
BOSSVS
20
40
60
80
100
Window size (epoch)
Figure 11: The optimal rate of our methods (using rolling window) and early stopping with dif- ferent patience values. (This figure is the same Figure 5, we added it here for easy comparison to Figure 10.)
H A HEURISTIC METHOD FOR AUTOMATIC LABELLING
We developed heuristic method for labelling the training history in the simulated dataset over- fitting based on the following conditions:
The training loss and validation loss both decrease in the first inc, percentage of the train- ing history.
The training loss and validation loss both decrease the last deep percentage of the training history.
21

---

Under review as a conference paper at ICLR 2023
Table 13: The median delay and significant testing of our overfitting prevention methods based on
zero-one validation loss curves with different window sizes. (Ws.: Window size; Md.: Median delay)
Effect size
Cliff's value
Effect
Cliff's
Ws.
Classifier
Md.
P
Classifier
Md.
P
size
value
20
13.5
0.684
neg -0.050
5.0
0.000
large
-0,950
46.0
0.109
small
0.198
5.5
0.000
large
-0.976
60
BOSSVS
44.5
0.494
neg -0.087
SAX-VSM
11.5
0.000
large
-0.884
80
46.5
0.115
small
-0.202
14,5
0.000
large
-0.806
100
49.5
0.029
small -0.284
26.5
0.000
large
-0.716
20
2.0
0.000
large
-0.850
7.5
0.000
large
-0.750
40
5.0
0.000
large
-0.739
14.5
0.000
large
-0.867
HMM-GMM
0.000
large
-0.656
TSBF
27.0
0.000
large
-0.809
18.0
0.000
large -0.719
37.0
0.000
large
-0,708
100
46.5
0.001
medium
-0.431
45.0
0.000
large
-0.561
48.5
0.000
large 0.650
12.5
0.001
medium
-0.400
40
44.0
0.027
small
0.273
24.0
0.000
large
-0.726
60
KNN-DTW
47.0
0.392
neg
-0.109
TSF
31.0
0.000
large
-0.630
80
48.5
0.14
small -0.166
42.5
0.000
large
-0.596
100
56.5
0.024
small
-0.292
47.0
0.000
large
-0.479
The gap between the training loss and validation loss exceeds gapp percentage of the sum of the training and validation loss.
To select the thresholds, we performed a grid search between 10% to 50% for incp and deep, and grid search between 1% to 50% for gapp. The best performance of the heuristic method can achieve a 0.75 F-score for the overfitting samples with 0.96 precision and 0.61 recall. The result shows that the heuristic method does not work well the simulated dataset in comparison other methods (see Table 7). Furthermore, the heuristic method performs poorly on the real-world dataset, with 0.22 average F-score. Hence, human labels are still required for training the time series classifiers.
22


Reviews:
[{"review_id": "t9DVUJdYef-", "reviewer": "Reviewer_xGGx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to detect and prevent overfitting by using the training loss and validation loss history. For detecting overfitting, the paper propose a strategy to collect a labelled dataset with binary labels: overfit and no overfit. The paper shows that a time-series classifier can be trained, and empirically showed to have high average F-scores, which is better than correlation based methods. The paper further shows a way to prevent overfitting, by using a detecting method on the whole observed history and by extracting the latest history with a rolling window. If we detect overfitting, we stop training and return the epoch that has the lowest validation loss. This seemed to work well, compared with an early stopping baseline.", "summary_of_the_review": "The paper proposes an algorithm that detects and prevents overfitting, based on the labelled dataset that the paper constructed. This seems to be a new direction for research on detecting/preventing overfitting. However, discussions on how the labels were collected (or the two labellers' definition of overfitting) was not provided. A discussion about other papers that also use the training history was not discussed. Since the final target seems to be the accuracy in some of the experiments, using the validation 01 loss function seems to be a natural choice, rather than using the cross-entropy loss function. More discussions on this choice would be helpful. Overall, although the research direction is interesting, I decided to choose the score with 5.", "strength_and_weaknesses": "Strengths:\n- Using the training history to detect and prevent overfitting is interesting.\n- The paper constructs a new dataset of overfitting/non-overfitting samples by going through many recent ML papers on overfitting.\n- The experiments show that detecting overfitting is possible, and that the time-series classifiers tends to perform better than the correlation based ones.\n- Practical value with experiments showing that the method can find the optimal stopping point and avoid overfitting at least 32% earlier than early stopping and achieve at least the same accuracy as early stopping.\n\nWeaknesses:\n- If I understood the paper correctly, I feel the main issue is that the paper does not directly discuss what \"overfitting\" is. There are two sentences that try to address what overfitting is, in the abstract and introduction: 1) the abstract mentions that \"overfitting of deep learning models on training data leads to poor generalizability on unseen data\", and 2) 1st paragraph of Section 1 (Introduction) mentions that \"for the overfit model, after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease\". The 1st one (in the abstract) can be regarded as a consequence for overfitting rather than an explanation of overfitting, so for example, the same sentence can be used for underfitting. The 2nd one (in the 1st paragraph of Section 1) is more specific, and if we use this as the definition of overfitting, it may exclude some learning curves which we may sometimes regard as overfitting or include ones that may not be overfitting. For example, if the validation loss stops decreasing after a certain number of epochs (but does not increase, just becomes flat) while the training loss keeps decreasing, the gap between training/validation loss will widen, and some people will regard this is a case of overfitting. As an extreme example, Fig 1 (b) can be considered to have some overfitting, since the validation loss is higher than the training loss and it is possible that this gap may close by increasing the number of training samples even further (which may be seen as an example of poor generalizability on unseen data due to limited training data). Another extreme example is when we continue learning for longer epochs, the validation loss can decrease again, with double descent loss curves. This is usually not regarded as overfitting, but may fall under the overfitting category based on the explanation in the 1st paragraph of Section 1.\n- In my opinion, trying to directly target overfitting is difficult since different researchers may have different definitions, so adopting a specific type of overfitting which \"after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease\", seems like a good direction (although I wrote some issues above that perhaps should be discussed in the paper). However, if the paper adopts this direction, can we automatically label the loss history by checking if \"after a certain amount of training, the validation loss begins to increase while the training loss continues to decrease\", instead of using human labels, as explained in Step 3 of Section 4.1? \n- It was also not clear to me how the two labellers decided the label of overfit/non-overfit.\n- A related question is, how would we know that in papers (for constructing the real-world test dataset) with samples of overfitting, they are samples of the same type of overfitting?\n- There are many papers that utilize the training history of maching learning models, but I am not sure if they were discussed. For example, \"Learning Curves for Decision Making in Supervised Machine Learning - A Survey\" discusses a lot of papers that use the training history or learning curves.\n- For regression tasks, we usually use the same loss function for training and for evaluating, but for classification tasks, we may use the 01 loss function for evaluating the accuracy while using a different loss such as the cross entropy loss for training. This is known to cause interesting discrepancies between the behavior of the two different loss functions, e.g., discussed in Guo et al. \"On Calibration of Modern Neural Networks\" (ICML 2017) or Soudry et al. \"The Implicit Bias of Gradient Descent on Separable Data\" (JMLR 2018), and the increase of the validation loss may not be harmful in terms of the 01 loss function. Therefore, if we want to have better accuracy (i.e., lower 01 loss), is it possible that it may be better to use 01 loss instead of the cross entropy loss for validation in the proposed algorithm? (Perhaps this is not much of an issue, since the proposed method is helpful in achieving higher accuracy in the experiments, but wondering if this will make the proposed method even better.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aP8OiOvBHG", "reviewer": "Reviewer_AtDu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Detecting overfitting of deep learning networks in an efficient, accurate and non-intrusive manner is a non-trivial problem. The authors propose a method to learn a classifier that uses the history of training and validation losses to identify overfitting regions. The paper evaluates the approach using a number of loss histories of published models. They show that classification is more accurate in detecting overfitting compared to correlation methods, and that one of the classifiers KNN-DTW is also better able to identify the optimal stopping point compared to early stopping method. While there are a number of approaches to detect and prevent overfitting, this is the first paper I am aware of that uses time classification to detect and prevent overfitting. The paper does a good job collecting or reproducing the histories of models from 30+ papers. This is a useful contribution for further research in this topic. ", "summary_of_the_review": "The paper addresses an important problem in training deep learning networks  - viz. when to stop training, and doing it in a non-intrusive manner. It proposes a fairly obvious idea of training time series classifiers on histories of training and validation losses to detect regions of overfitting, and compares it favorably against existing methods based on correlation and early stopping. A good contribution is that the paper has collected or reproduced the loss histories from a number of other published papers. This is a useful contribution.", "strength_and_weaknesses": "Strengths:\n1. The paper uses a simple time series classifier to learn the regions of overfitting using training and validation losses. It is a non-destructive method and can be applied easily without any modification of the training algorithm itself. \n2. The approach is tested well using a number of datasets from published models. The authors do a good job of collecting or reproducing the training and validation loss histories from 30+ papers. This is a useful contribution\n\nWeaknesses:\n1. From the paper, I struggled to infer they train a single model using all training and validation loss histories, or if they train a model per dataset. If the former, how well do they think this model generalizes to other datasets? i.e. is this a single pre-trained model that others can use off the shelf, or do other researchers need to train a model per dataset?\n2. There is very little discussion of why some methods (like KNN-DTW) are able to do well, while others (like HMM-GMM and BOSSVS) perform so poorly (Table 2). As such, this feels like a brute force search method of different algorithms for time series classification. The proposed candidate KNN-DTW is an expensive algorithm to implement for inference (it is the slowest inference algorithm among the ones they tested in Table 2).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HovGCmffeoZ", "reviewer": "Reviewer_aLAk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to detect and prevent overfitting in training deep neural network models. The proposed method trains a time series classification model to detect overfitting based on the model's training history. The trained classifier can also be used to prevent overfitting by identifying the optimal point to stop a model's training. The authors manually labeled several hundred data points using existing datasets and test on several other data points collected from machine learning literature. Results show that the proposed approach outperforms conventional correlation-based methods on overfitting detection and slightly underperforms early-stopping for overfitting prevention but leads to lower delay, resulting in less wasted computation.", "summary_of_the_review": "In sum, I think the paper investigate an interesting problem. However, the technical contribution is not strong enough and the empirical experiments does not fully support the claims. Therefore I would suggest the authors to revise and improve the empirical study in the future.", "strength_and_weaknesses": "Strengthes:\n1. The paper is generally well written and easy to follow. The proposed method is also simple and intuitive.\n2. The problem of detecting and preventing overfitting in deep learning is an important problem.\n3. Experimental results show some advantages over prior methods for detecting and preventing overfitting in deep learning models.\n\nWeaknesses:\n1. The proposed method is straightforward and does not have much technical novelty/contribution. However, this is perfectly fine if the proposed method can be really useful and help facilitate training of deep learning models.\n2. The experimental setup is not fully convincing. First, the test set contains only 40 data points, which is too small and thus vulnerable to randomness issues. Also, these papers are all collected in literature investigating the overfitting problem, which may results in similar choices of tasks/models and may not be able to reflect the trained classifier's real generalization ability on more realistic datasets and tasks. Also, the train histories explicitly labeled as overfitting or not and shown in the paper are probably typically overfitting examples that are very easy to distinguish. Moreover, the train set is manually labeled and require full training to generate, thus relatively laborsome.\n3. The authors did not compare with intrusive methods because they require retraining, but their method also require training a classifier with labeled data. I do agree that these intrusive methods are not directly comparable with the proposed method, but it would be better that the authors at least report their performance in the experiments to help reader get a better understanding of the comparison.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BgVTM6LG2A6", "reviewer": "Reviewer_wxLr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission presents a method for improved identification of an early-stopping point based on validation loss curves. This method is based on training a time series classifier on validation loss curves that have been labeled as \"overfit\" vs. \"not overfit\", using a collection of UCI-type datasets and various network architectures. These training examples for the time series classifiers were labeled by the authors. Experiments compare various time series classifiers and find that a k-NN-based approach, with dynamic time warping, based on a rolling window, works best. This approach to detecting overfitting (and, thus, enabling early stopping) outperforms existing methods for overfitting detection based on measuring correlation between training and validation loss curves. Further experiments, comparing to plain early stopping using various values for the patience parameter in both the proposed method and standard early stopping, show that the proposed method more accurately identifies the stopping point with minimum validation loss than the standard plain patience-based approach.", "summary_of_the_review": "The paper is interesting, and the proposed approach may have merit.\n\n", "strength_and_weaknesses": "+ The proposed approach seems novel and is intuitively plausible\n+ The results indicate that the method is preferable to existing correlation-based approaches to detecting overfitting\n+ The paper is easy to follow and code is provided\n\n- Accuracy is measured with respect to the distance to the \"optimum\" stopping point, not predictive accuracy of the neural network model\n\nWhat matters is the predictive performance of the model, *not* how close we are to the \"optimum\" stopping point.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject