Title: ON SUMMARIZED VALIDATION CURVES AND GENERALIZATION

Abstract: The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks. However, this summarization tends to lose the intricacies of the per-task curves and it isn't able to reflect if all the tasks are at their validation optimum even if the summarized curve might be. In this work, we explore this loss of information, how it affects the model at testing and how to detect it using interval plots. We propose two techniques as a proof-of-concept of the potential gain in the test performance when per-task validation curves are accounted for. Our experiments on three large datasets show up to a 2.5% increase (averaged over multiple trials) in the test accuracy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum. This potential increase is not a result of any modification to the model but rather at what point of training the weights were selected from. This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics.

1. INTRODUCTION:

A validation set, separate from the test set, is the de facto standard for training deep learning models through early stopping. This non-convergent approach (Finnoff et al., 1993) identifies the best model in multi-task/label settings based on an expected error across all tasks. Calculating metrics on the validation set can estimate the model's generalization capability at every stage of training and monitoring the summarized validation curve over time aids the detection of overfitting. It is common to see the use of validation metrics as a way to stop training and/or load the best model for testing, as opposed to training a model to N epochs and then testing. While current works have always cautioned about the representativeness of validation data being used, the curves themselves haven't been addressed much. In particular, there hasn't been much attention on the summarized nature of the curves and their ability to represent the generalization of the constituent tasks.

Tasks can vary in difficulty and even have a dependence on each other (Graves, 2016; Alain & Bengio, 2016 ). An example by Lee et al. (2016) is to suppose some task a is to predict whether a visual instance 'has wheels' or not, and task b is to predict if a given visual object 'is fast'; not only is one easier, but there is also a dependence between them. So there is a possibility that easier tasks reach their best validation metric before the rest and may start overfitting if training were to be continued. This isn't reflected very clearly with the use of a validation metric that is averaged over all tasks. As a larger number of underfit tasks would skew the average, the overall optimal validation point gets shifted to a later time-step (epoch) when the model could be worse at the easier tasks. Vice versa, the optimal epoch gets shifted earlier due to a larger, easier subset that are overfit when the harder tasks reach their individual optimal epochs. We term this mismatch in the overall and task optimal epochs as a 'temporal discrepancy'.

In this work, we explore and try to mitigate this discrepancy between tasks. We present in this paper that early stopping on only the expected error over tasks leaves us blind to the performance they are sacrificing per task. The work is organized in the following manner: in §2, we explore existing work that deals with methods for incorporating task difficulty (which could be causing this discrepancy) into training. The rest of the sections along with our contributions can be summarized as:

1. We present a method to easily visualize and detect the discrepancy through interval plots in §3

2. We formulate techniques that could quantify this discrepancy by also considering the pertask validation metrics in model selection in §4.

3. We explore the presence of the temporal discrepancy on three image datasets and test the aforementioned techniques to assess the change in performance in §5 4. To the best of our knowledge, there has not been a study like this into the potential of per-task validation metrics to select an ensemble of models.

2. RELATED WORK:

Training multiple related tasks together creates a shared representation that can generalize better on individual tasks. The rising prominence of multi-task learning can be attributed to Caruana (1997) .

It has been acknowledged that some tasks are easier to learn than the others and plenty of works have tried to solve this issue through approaches that slow down the training of easier tasks. In other words, tasks are assigned a priority in the learning phase based on their difficulty determined through some metric. This assignment of priority implicitly tries to solve the temporal discrepancy without formally addressing its presence. Task prioritization can take the form of gradient magnitudes, parameter count, or update frequencies (Guo et al., 2018) . We can group existing solutions into task prioritization as a hyperparameter or task prioritization during training (aka self-paced learning).

The post-training brute force and clustering methods we propose do not fit into these categories as we believe they have not been done before. Instead of adjusting training or retraining, these methods operate on a model which has already been trained.

Task prioritization as a hyperparameter is a way to handle per task overfitting that is almost the subconscious approach for most practitioners. This would include data-augmentation and over/undersampling. An example case is in Kokkinos (2017) where they use manually tuned task weights in order to improve performance.

Task prioritization during training covers approaches where tasks dynamically change priority or are regularized in some way. For example Guo et al. (2018) takes an approach to change task weights during training based on multiple metrics such as error, perceived difficulty, and learnable parameters. The idea is that some tasks need to have a high weight at the start and a low weight later in training. In a similar direction Gradnorm (Chen et al., 2018) aims to set balance task weights based on normalizing the gradients across tasks.

Using relationships between tasks during training is another direction. Ruder (2017) discussed negative transfer where sharing information with unrelated tasks might actually hurt performance. Work by Lee et al. (2016) incorporated a directed graph of relationships between tasks in order to enforce sharing between related tasks when reweighting tasks. Task clustering has been performed outside of neural networks by Evgeniou et al. (2005) ; Evgeniou & Pontil (2004) where they regularize per-task SVMs so that the parameters between related tasks are similar.

It would be natural to use some of these methods as a baseline for our work. However, we think it would not be an equitable comparison as:

• These baseline methods are applied during training whereas ours is a post-training analysis.

• The main aspect of our analysis is only on the validation metric whereas these baselines consider a variety of different aspects of training.

• The focus of our work is on how the weights change with time, keeping all else constant, and how these changes affect the validation and test performance. The aforementioned methods modify the gradients w.r.t. several factors during the training which adds more degrees of freedom and is difficult to compare.

Regardless of task difficulty, training multiple tasks jointly with a neural network can lead to catastrophic forgetting: it refers to how a network can lose information that it had learned for a particular task as it learns another task (McCloskey & Cohen, 1989) . Multiple works have explored and tried to mitigate this phenomenon (Ratcliff, 1990; Robins, 1995; Goodrich & Arel, 2014; Kirkpatrick et al., 2017; Kemker et al., 2018; Lee et al., 2017) and it still remains an open area of research. It is highly likely that catastrophic forgetting could be causing any such temporal discrepancy; exploring the relationship between the two is an area is a very interesting direction in research and is left for future work.

3. STUDYING TEMPORAL DISCREPANCY BETWEEN TASKS:

Firstly, we define what a task is to disambiguate from its general usage in multi-task learning literature. A 'task' is predicting a single output unit out of many, regardless of the training paradigm being multi-class or multi-label or other. Tasks can be very fine-grained such as predicting the class of an image or much higher-level such image classification, segmentation etc. While our work uses the term in the former context, our motivation and findings can be applied in the latter context (which is the broader and more common context in multi-task learning) as well.

In the next two subsections, we define the term temporal discrepancy and display an example of it on CIFAR100. Then, we introduce a simple method of visualizing it on datasets with a large number of tasks that would make it difficult to analyze the per-task curves together.

3.1. TEMPORAL DISCREPANCY:

A temporal discrepancy in the validation performance refers to the phenomenon where the model isn't optimal for all of its tasks simultaneously. This occurs when there is a difference between the overall optimal epoch determined by the summarized validation metric and the epoch in which task achieves its best validation metric is higher than some arbitrary threshold, i.e., |t s -t i | > δ where t s is the optimal epoch of the summarized validation curve and t i is the optimal epoch for task i. It is most evident for the labels Sea and Lamp which undergo a drop of 7.5% and 5.7% respectively in their validation accuracy from their peak epoch to t s . Similarly, Snake also starts degrading till t s but strangely starts improving after. Conversely, Rose and Streetcar are underfit at t s as they continue to improve after.

The most noteworthy observation is that the averaged validation curve (in dotted black) completely plateaus out after the 150 th epoch. There is significant variation occurring in the per-label curves but the averaged curve is unable to represent these dynamics in the training. Selecting an optimal model off the averaged curve can be quite misleading as it represents the entire [151, 300] interval as optimal despite the labels' validation accuracies fluctuating significantly in this interval. The test performance of individual labels can wildly differ depending on which epoch is used for loading the weights for testing and/or deployment.

3.2. INTERVAL PLOTS:

It is easy to examine the per-label curves in Figure 1 as only 5% of the labels have been plotted. But when the number of tasks is high and all of them need to be plotted together to get a clearer global picture, decomposing the summarized validation curve can get very messy. Quasi-optimal validation interval plots, or interval plots for short, are a way of assessing the optimal temporality of the per-task validation performance relative to t s . It is a simple visualization method that aids in determining when and/or for how long the tasks are within the acceptable limits of the best validation performance and also which and/or how many tasks aren't within these limits near the overall optimal epoch t s .

Creating an interval plot involves finding a 'quasi-optimal' region for each task, i.e., a consecutive temporal interval in which a validation metric of the task fluctuates near its maximum with a set tolerance. The task validation curves are first smoothed out to reduce noise and the time-step (epoch) at which the task achieved its optimal validation metric is determined. Then, the number of epochs before and after this task-optimal epoch in which the task metric is greater than a threshold is calculated. This duration of epochs is the interval for the task. Given a vector of validation metrics A i for a task i, its interval τ i is given by: 2 plots the decomposed curves and the equivalent intervals for CIFAR100. The overall optimal epoch t s doesn't fall in the intervals of almost half the labels; these labels aren't at their potentially best validation performance at the early stopping point. Some intervals are notably small in duration, meaning those labels have a very sharp peak. This could imply that the validation performance is randomly high at that epoch and it'd be more suitable to shift the quasi-optimal region of these labels to a longer and/or later interval, that doesn't necessarily contain t i , as long as the validation accuracy stays within the tolerance in that interval. In this section, we present two simple techniques that consider the per-task validation metrics for selecting the best model for testing. Two aspects common to these techniques is their hand-crafted & engineered nature and their inefficiency in terms of deployment, training time and/or inference time. The aim with these techniques to assess how much potential gain in performance could be attained if we account for per-task validation metrics in selecting a model and we'd like to stress that these techniques serve as a proof-of-concept of this gain (if any). They are intended as a baseline and also a stimulus for increasing research into the effect of the subtleties of the validation curves on model performance and selection.

τ i = [t i -m, . . . t i -1, t i , . . . t i + n] ∀ a ij ≥ a iti - where t i = argmax A i , j ∈ τ i and a ij ∈ A i Figure

Brute force:

This involves loading the model with the weights from a given task i's optimal validation epoch t i and evaluating on only the samples that belong to that task. We call this particular model that has been loaded with the weights from t i as the validation-optimal model for task i. It is essentially using a "separate model" for each task during evaluation and/or deployment making this the most naive approach. It is also the most inefficient because it would (i) require storing up to N models, where N is the total number of tasks (ii) require a way to combine predictions from all N models that wouldn't be misleading during inference (iii) increase latency significantly due to overhead caused by loading and reloading the model weights (iv) scale up the inference time by a factor up to N.

Clustering Instead of having separate weights for each task, we try to cluster the set of the taskoptimal validation epochs into K clusters so that only K models are required as opposed to N. In this approach, the interval plots can also be utilized to cluster tasks that have similar interval positions and/or lengths in addition to the t i 's. Similar to brute force, this technique also involves multiple models loaded with weights from different epochs, but trades off any gain in performance for a lower number of models.

5. EXPERIMENTS:

We train variations of DenseNets (Huang et al., 2017) on three image datasets: CIFAR100, Tiny ImageNet and PadChest. All models were trained with three random seeds for model initialization and splits of the training set into training and validation sets. CIFAR100 CIFAR100 (Krizhevsky & Hinton, 2009 ) is a dataset of natural images containing 100 classes with 500 training images and 100 testing images per class. We trained a "DenseNet-BC (k = 12)" as described in Huang et al. (2017) : it has three dense blocks, a total of 100 layers and a growth rate of 12. It was trained in the exact manner as the original work, i.e. 300 epochs with dropout, weight decay of 10 -4 , SGD with Nesterov momentum of 0.9 and an initial learning rate of 0.1 that is divided by 10 at the 150 th and 225 th epochs. As we had carved out 20% of the training set as validation and didn't use data augmentation, we achieved an average test accuracy of 72.14%. In our analysis, we only use the validation curves after the 150 th epoch because the training is very noisy and brittle up to that epoch due to the use of a learning rate of 0.1 (Figures 1 and 2 ). Tiny ImageNet ImageNet (Deng et al., 2009 ) is a dataset with 1.5 million images and 1000 classes. Tiny ImageNet1 is a subset of ImageNet with images resized to 64x64 and only 200 classes. We utilized the same architecture as CIFAR100 but with a total of 190 layers and a growth rate of 40. In addition, we used a stride of 2 for the first convolution. This modified DenseNet was also trained in the same manner as above but with the hyperparameters used for ImageNet in the original work: 100 epochs, no dropout and dividing the learning rate at the 30 th and 60 th epochs instead. Rest of the hyperparameters remained the same. 10% of the training set was used as validation and we achieved an average test accuracy of 63.09%. Similar to CIFAR100, we only use the validation curves after the 30 th epoch. The interval plot from one run of training is given in Figure 3a PadChest PadChest (Bustos et al., 2019) is a medical imaging dataset of 160,000 chest X-rays of 67,000 patients with multiple visits and views available. We used the publicly available code provided by Bertrand et al. (2019) to recreate their cohort of around 31,000 samples. We trained a multilabel DenseNet-121 (Huang et al., 2017; Rajpurkar et al., 2017) on the frontal views and only those labels which have more than 50 samples (total 64 labels). With a 60-20-20 split between training, validation, and test sets, we trained for 100 epochs with Adam and an initial learning rate of 0.0001 that is halved every 20 epochs. The interval plot from one run of training is given in Figure 3b (a) Tiny ImageNet ( 200 By brute forcing the best model selection, we wanted to assess how much performance is lost when the summarized validation curve is used. This involves evaluating each task with its own set of optimal weights determined from its specific optimal epoch. This would require N models in theory but the number of validation-optimal models is actually lower as many tasks have inter-dependent learning profiles. These correlated tasks may reach their optimal validation performance at the same epoch, thus requiring only a single common set of weights for all of them. N also decreases with the total number of training epochs as that increases the probability of tasks having the same optimal epoch. On analyzing the task validation curves for the three datasets, we do find that the number of models required is much lower than N. CIFAR100 and Tiny ImageNet both required less than 60 models for brute forcing the evaluation, despite the latter having twice as many labels. Also since Tiny ImageNet was trained for one-third of the epochs, the number of models reduced drastically.

The number of models for PadChest was around 35.

The results of using a single model for each task are tabulated for the three datasets in Table 1 . On using each label's validation-optimal model for evaluating on the test, the test metric always increases in comparison to using the baseline model that uses the summarized validation curve for CIFAR100 and Tiny ImageNet. The top-1 accuracy for CIFAR100 undergoes an average and maximum increase of approximately 2.5% & 3.2% respectively. For Tiny ImageNet, an average and maximum increase of 1.72% & 1.95% is observed.

However, the average increase in PadChest is not only meagre but also within the standard deviation. This could be because the temporal discrepancy isn't very pronounced in PadChest: the overall optimal epoch does happen to be in the intervals of a very large portion of the labels (Figure 3b ). As PadChest has fewer number of outputs compared to the other two (64 vs. 100/200), there could also be a relationship between the number of outputs and the magnitude of the discrepancy's effect. These results show that accounting for the per-task validation metrics can mitigate a significant temporal discrepancy and increase the testing performance.

5.2. COMPUTATION-PERFORMANCE TRADEOFF:

We can further reduce the number of validation-optimal models required by merging the t i 's that are close to each other. We use K-means to cluster the set of task-optimal epochs into K clusters. We vary K from 2 to N to gain an insight into how much performance is lost as we reduce the number of validation-optimal models required down to one where using only one model is the same as using the summarized. For any given cluster k, we use its center as the epoch for loading the weights to evaluate the set of the tasks in k. We round off the center to the nearest integer for our analysis but the fractional part can be used to select weights after a specific batch iteration in that epoch.

The results for the three datasets are plotted in Figure 4 . There is an interesting observation that the accuracy doesn't always increase with the number of task-optimal models which is very noticeable in Tiny ImageNet Seed 2. This could be due to the noisy nature of training with SGD where the validation metric can oscillate a lot between epochs and even a shift of one epoch can cause a decrease in performance. In this work, we examine the decomposition of a model's average validation curve into its per-task curves to assess the presence of a temporal discrepancy on three image datasets. We provide a visualization method to detect if there is a disparity between the task curves and the summarized validation curve. We test two techniques that incorporate the per-task metrics into model evaluation and we find that that when per-task validation metrics are accounted for training runs that show a significant temporal discrepancy, we gain an increase in testing performance. We show that with CI-FAR100 there is room for up to 2.5% to be gained in testing accuracy and 1.72% for Tiny ImageNet if we don't use the averaged metrics as they are.

With these experiments, we aim to create more awareness of how summarized validation metrics cannot represent a model that is truly optimal for all its tasks all the time. Using averaged curves could mean that models are currently being trained oblivious of the performance being sacrificed on individual tasks. We wish to draw attention to the need of both theoretical and engineered approaches that would take the per-task validation metrics into account while training. Our experiments demonstrate that there is a potential that current state-of-the-art models could possibly be made even more optimal by ensuring all of its prediction tasks are at their validation optimums.

:

https://tiny-imagenet.herokuapp.com/

Figures and Tables:

Figure 1 displays an example of this discrepancy in CIFAR100 (only five curves plotted for clarity).It is most evident for the labels Sea and Lamp which undergo a drop of 7.5% and 5.7% respectively in their validation accuracy from their peak epoch to t s . Similarly, Snake also starts degrading till t s but strangely starts improving after. Conversely, Rose and Streetcar are underfit at t s as they continue to improve after.

Figure 1: Visualizing the presence of a temporal discrepancy in five of CIFAR100 labels (training is detailed in Section 5). The vertical black line is the best overall epoch t s . The dotted and solid curves represent the averaged validation curve and the label specific validation curves respectively. All curves have been smoothed by averaging over a sliding window of size 10.

Figure 2: Smoothed decomposed validation curves (left) and the equivalent interval plot (right) for CIFAR100 with = 2%. For both plots, the straight black line is the best overall epoch t s . On the interval plot, the blue lines indicate the interval τ i , the blue dots are the centers of the interval, and the red dots are the task-optimal epoch t i . The intervals have been sorted by their centers

Figure 3: Interval plots for Tiny ImageNet and PadChest with = 0.02. The gray/green lines and dots indicate the interval τ i and its center respectively. The black horizontal line is the best overall epoch t s and the red dots are the task-optimal epoch t i . The intervals have been sorted by their centers

Figure 4: Reducing the number of models needed to represent all per-task optimal validation points. K is the number of clusters

Accuracy/AUC on the test set for the baseline and brute forced models, averaged over 3 seeds

Table content: <table><tr><td>Dataset</td><td>Metric</td><td>Baseline</td><td>Brute-force</td></tr><tr><td>CIFAR100</td><td>Top-1 Top-5</td><td>72.14 ± 0.51 92.23 ± 0.33</td><td>74.62 ± 0.43 93.07 ± 0.26</td></tr><tr><td>Tiny ImageNet</td><td>Top-1 Top-5</td><td>63.09 ± 0.27 83.79 ± 0.26</td><td>64.82 ± 0.39 84.91 ± 0.47</td></tr><tr><td>PadChest</td><td>AUC</td><td colspan="2">0.7807 ± 0.001 0.7826 ± 0.002</td></tr></table>

Peer review (iclr2020):
The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task. At different points of training, a model*s performance on a task will vary. To make it easier to examine this, they propose displaying an interval plot. The validation accuracy for a single task will be largest at a specific epoch. For each single task, display the epochs where performance is within some threshold of the best validation-set accuracy for that task. Plots on CIFAR-100 demonstrate how some tasks are learned quickly then forgotten, while other tasks are only learned late. To examine the difference in performance, they compare the single best CIFAR-100 model, to the classifier defined by *for class k, forward input to the model with best validation accuracy for class k*. This gives about +2.5% in Top-1 performance, a similar story appears on the other 2 datasets they try (Tiny ImageNet, PadChest). This requires N models, where N is the number of classes. They also examine a clustering approach, where they cluster the N models into K models (using K-means), then forward class k to the best model out of those K models. Overall, it isn*t an especially large contribution but I felt it had some interesting points. Some comments: * Citation list feels short. The catastrophic forgetting literature seems relevant here. Rich Caruana*s multitask learning thesis (Caruana et al 1997) is also relevant. * It*s unclear how the K-means is carried out. I assume the features used for each model checkpoint are tied to its performance on individual task but it*s never spell out what distance metric is used in the K-means. * It feels strange to have no comparisons to ensemble-based baselines. The baseline here would be, for some parameter K, run K independent training runs, take the top average validation accuracy from each run, and average them together, then compare this to the K models found from K-means clustering. For small enough K (like 5 or 10) this seems like a feasible experiment, computation-wise. * In a similar vein, I wonder how this compares to the Snapshot Ensembles paper, which has a similar guarantee of doing 1 training run and giving M models for an ensemble. Is the gain from ensembling, or from directly examining which models are better at which classes? * A natural follow-up here is to use model distillation to distill the N best models for each individual class into a single model checkpoint - does this give us a better single model checkpoint? As-is, I give this paper a weak reject, but would be willing to increase if some of these experiments were tried. Edit: having read the other reviews and author comments, I still maintain a weak reject rating. I believe this paper needs further work.