On summarized validation curves and generalization

The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks . However , this summarization tends to lose the intricacies of the per-task curves and it isn ’ t able to reflect if all the tasks are at their validation optimum even if the summarized curve might be . In this work , we explore this loss of information , how it affects the model at testing and how to detect it using interval plots . We propose two techniques as a proof-of-concept of the potential gain in the test performance when per-task validation curves are accounted for . Our experiments on three large datasets show up to a 2.5 % increase ( averaged over multiple trials ) in the test accuracy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum . This potential increase is not a result of any modification to the model but rather at what point of training the weights were selected from . This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics . 1 INTRODUCTION . A validation set , separate from the test set , is the de facto standard for training deep learning models through early stopping . This non-convergent approach ( Finnoff et al. , 1993 ) identifies the best model in multi-task/label settings based on an expected error across all tasks . Calculating metrics on the validation set can estimate the model ’ s generalization capability at every stage of training and monitoring the summarized validation curve over time aids the detection of overfitting . It is common to see the use of validation metrics as a way to stop training and/or load the best model for testing , as opposed to training a model to N epochs and then testing . While current works have always cautioned about the representativeness of validation data being used , the curves themselves haven ’ t been addressed much . In particular , there hasn ’ t been much attention on the summarized nature of the curves and their ability to represent the generalization of the constituent tasks . Tasks can vary in difficulty and even have a dependence on each other ( Graves , 2016 ; Alain & Bengio , 2016 ) . An example by Lee et al . ( 2016 ) is to suppose some task a is to predict whether a visual instance ‘ has wheels ’ or not , and task b is to predict if a given visual object ‘ is fast ’ ; not only is one easier , but there is also a dependence between them . So there is a possibility that easier tasks reach their best validation metric before the rest and may start overfitting if training were to be continued . This isn ’ t reflected very clearly with the use of a validation metric that is averaged over all tasks . As a larger number of underfit tasks would skew the average , the overall optimal validation point gets shifted to a later time-step ( epoch ) when the model could be worse at the easier tasks . Vice versa , the optimal epoch gets shifted earlier due to a larger , easier subset that are overfit when the harder tasks reach their individual optimal epochs . We term this mismatch in the overall and task optimal epochs as a ‘ temporal discrepancy ’ . In this work , we explore and try to mitigate this discrepancy between tasks . We present in this paper that early stopping on only the expected error over tasks leaves us blind to the performance they are sacrificing per task . The work is organized in the following manner : in §2 , we explore existing work that deals with methods for incorporating task difficulty ( which could be causing this discrepancy ) into training . The rest of the sections along with our contributions can be summarized as : 1 . We present a method to easily visualize and detect the discrepancy through interval plots in §3 2 . We formulate techniques that could quantify this discrepancy by also considering the pertask validation metrics in model selection in §4 . 3 . We explore the presence of the temporal discrepancy on three image datasets and test the aforementioned techniques to assess the change in performance in §5 4 . To the best of our knowledge , there has not been a study like this into the potential of per-task validation metrics to select an ensemble of models . 2 RELATED WORK . Training multiple related tasks together creates a shared representation that can generalize better on individual tasks . The rising prominence of multi-task learning can be attributed to Caruana ( 1997 ) . It has been acknowledged that some tasks are easier to learn than the others and plenty of works have tried to solve this issue through approaches that slow down the training of easier tasks . In other words , tasks are assigned a priority in the learning phase based on their difficulty determined through some metric . This assignment of priority implicitly tries to solve the temporal discrepancy without formally addressing its presence . Task prioritization can take the form of gradient magnitudes , parameter count , or update frequencies ( Guo et al. , 2018 ) . We can group existing solutions into task prioritization as a hyperparameter or task prioritization during training ( aka self-paced learning ) . The post-training brute force and clustering methods we propose do not fit into these categories as we believe they have not been done before . Instead of adjusting training or retraining , these methods operate on a model which has already been trained . Task prioritization as a hyperparameter is a way to handle per task overfitting that is almost the subconscious approach for most practitioners . This would include data-augmentation and over/undersampling . An example case is in Kokkinos ( 2017 ) where they use manually tuned task weights in order to improve performance . Task prioritization during training covers approaches where tasks dynamically change priority or are regularized in some way . For example Guo et al . ( 2018 ) takes an approach to change task weights during training based on multiple metrics such as error , perceived difficulty , and learnable parameters . The idea is that some tasks need to have a high weight at the start and a low weight later in training . In a similar direction Gradnorm ( Chen et al. , 2018 ) aims to set balance task weights based on normalizing the gradients across tasks . Using relationships between tasks during training is another direction . Ruder ( 2017 ) discussed negative transfer where sharing information with unrelated tasks might actually hurt performance . Work by Lee et al . ( 2016 ) incorporated a directed graph of relationships between tasks in order to enforce sharing between related tasks when reweighting tasks . Task clustering has been performed outside of neural networks by Evgeniou et al . ( 2005 ) ; Evgeniou & Pontil ( 2004 ) where they regularize per-task SVMs so that the parameters between related tasks are similar . It would be natural to use some of these methods as a baseline for our work . However , we think it would not be an equitable comparison as : • These baseline methods are applied during training whereas ours is a post-training analysis . • The main aspect of our analysis is only on the validation metric whereas these baselines consider a variety of different aspects of training . • The focus of our work is on how the weights change with time , keeping all else constant , and how these changes affect the validation and test performance . The aforementioned methods modify the gradients w.r.t . several factors during the training which adds more degrees of freedom and is difficult to compare . Regardless of task difficulty , training multiple tasks jointly with a neural network can lead to catastrophic forgetting : it refers to how a network can lose information that it had learned for a particular task as it learns another task ( McCloskey & Cohen , 1989 ) . Multiple works have explored and tried to mitigate this phenomenon ( Ratcliff , 1990 ; Robins , 1995 ; Goodrich & Arel , 2014 ; Kirkpatrick et al. , 2017 ; Kemker et al. , 2018 ; Lee et al. , 2017 ) and it still remains an open area of research . It is highly likely that catastrophic forgetting could be causing any such temporal discrepancy ; exploring the relationship between the two is an area is a very interesting direction in research and is left for future work . 3 STUDYING TEMPORAL DISCREPANCY BETWEEN TASKS . Firstly , we define what a task is to disambiguate from its general usage in multi-task learning literature . A ‘ task ’ is predicting a single output unit out of many , regardless of the training paradigm being multi-class or multi-label or other . Tasks can be very fine-grained such as predicting the class of an image or much higher-level such image classification , segmentation etc . While our work uses the term in the former context , our motivation and findings can be applied in the latter context ( which is the broader and more common context in multi-task learning ) as well . In the next two subsections , we define the term temporal discrepancy and display an example of it on CIFAR100 . Then , we introduce a simple method of visualizing it on datasets with a large number of tasks that would make it difficult to analyze the per-task curves together . 3.1 TEMPORAL DISCREPANCY . A temporal discrepancy in the validation performance refers to the phenomenon where the model isn ’ t optimal for all of its tasks simultaneously . This occurs when there is a difference between the overall optimal epoch determined by the summarized validation metric and the epoch in which task achieves its best validation metric is higher than some arbitrary threshold , i.e. , |ts − ti| > δ where ts is the optimal epoch of the summarized validation curve and ti is the optimal epoch for task i . Figure 1 displays an example of this discrepancy in CIFAR100 ( only five curves plotted for clarity ) . It is most evident for the labels Sea and Lamp which undergo a drop of 7.5 % and 5.7 % respectively in their validation accuracy from their peak epoch to ts . Similarly , Snake also starts degrading till ts but strangely starts improving after . Conversely , Rose and Streetcar are underfit at ts as they continue to improve after . The most noteworthy observation is that the averaged validation curve ( in dotted black ) completely plateaus out after the 150th epoch . There is significant variation occurring in the per-label curves but the averaged curve is unable to represent these dynamics in the training . Selecting an optimal model off the averaged curve can be quite misleading as it represents the entire [ 151 , 300 ] interval as optimal despite the labels ’ validation accuracies fluctuating significantly in this interval . The test performance of individual labels can wildly differ depending on which epoch is used for loading the weights for testing and/or deployment . 3.2 INTERVAL PLOTS . It is easy to examine the per-label curves in Figure 1 as only 5 % of the labels have been plotted . But when the number of tasks is high and all of them need to be plotted together to get a clearer global picture , decomposing the summarized validation curve can get very messy . Quasi-optimal validation interval plots , or interval plots for short , are a way of assessing the optimal temporality of the per-task validation performance relative to ts . It is a simple visualization method that aids in determining when and/or for how long the tasks are within the acceptable limits of the best validation performance and also which and/or how many tasks aren ’ t within these limits near the overall optimal epoch ts . Creating an interval plot involves finding a ‘ quasi-optimal ’ region for each task , i.e. , a consecutive temporal interval in which a validation metric of the task fluctuates near its maximum with a set tolerance . The task validation curves are first smoothed out to reduce noise and the time-step ( epoch ) at which the task achieved its optimal validation metric is determined . Then , the number of epochs before and after this task-optimal epoch in which the task metric is greater than a threshold is calculated . This duration of epochs is the interval for the task . Given a vector of validation metrics Ai for a task i , its interval τi is given by : τi = [ ti −m , . . . ti − 1 , ti , . . . ti + n ] ∀ aij ≥ aiti − where ti = argmax Ai , j ∈ τi and aij ∈ Ai Figure 2 plots the decomposed curves and the equivalent intervals for CIFAR100 . The overall optimal epoch ts doesn ’ t fall in the intervals of almost half the labels ; these labels aren ’ t at their potentially best validation performance at the early stopping point . Some intervals are notably small in duration , meaning those labels have a very sharp peak . This could imply that the validation performance is randomly high at that epoch and it ’ d be more suitable to shift the quasi-optimal region of these labels to a longer and/or later interval , that doesn ’ t necessarily contain ti , as long as the validation accuracy stays within the tolerance in that interval .

Summary: The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task.