On summarized validation curves and generalization
Authors: Mohammad Hashir
Yoshua Bengio
Joseph Paul Cohen
Venue:  (2020)
Keywords: model selection
deep learning
early stopping
validation curves

Abstract: The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks. However, this summarization tends to lose the intricacies of the per-task curves and it isn't able to reflect if all the tasks are at their validation optimum even if the summarized curve might be. In this work, we explore this loss of information, how it affects the model at testing and how to detect it using interval plots. We propose two techniques as a proof-of-concept of the potential gain in the test performance when per-task validation curves are accounted for. Our experiments on three large datasets show up to a 2.5% increase (averaged over multiple trials) in the test accuracy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum. This potential increase is not a result of any modification to the model but rather at what point of training the weights were selected from. This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics. 

Under review as a conference paper at ICLR 2020
ON SUMMARIZED VALIDATION CURVES AND GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks. However, this summarization tends to lose the intricacies of the per-task curves and it isn't able to reflect if all the tasks are their validation optimum even if the summa- rized curve might be. In this work, we explore this loss of information, how it affects the model testing and how to detect it using interval plots. We propose two techniques as proof-of-come -f-nccept- of the potential gain in the test performance when per-task validation curves are accounted for. Our experiments on three large datasets show up to a 2.5% increase (averaged over multiple trials) in the test ac- curacy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum. This potential increase is not result of any modification to the model but rather what point of training the weights were selected from. This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics.
INTRODUCTION
A validation set, separate from the test set, is the de facto standard for training deep learning mod- els through early stopping. This non-convergent approach (Finnoff et al., 1993) identifies the best model in multi-task/label settings based on an expected error across all tasks. Calculating metrics on the validation set can estimate the model's generalization capability and monitoring the summarized validation curve over time aids the detection of overfitting. It is common to see the use of validation metrics as a way to stop training and/or load the best model for testing, opposed to training model to N epochs and then testing. While current works have always cautioned about the representativeness of validation data being used, the curves themselves haven't been addressed much. In particular, there hasn't 't been much attention on the summarized nature of the curves and their ability to represent the generalization of the constituent tasks.
every stage of training
Tasks can vary in difficulty and even have dependence on each other (Graves, 2016; Alain & Bengio, 2016). An example by Lee et al. (2016) is to suppose some task a is to predict whether visual instance 'has wheels' or not, and task b is to predict if a given visual object 'is fast'; not only is one easier, but there is also dependence between them. So there is possibility that easier tasks reach their best validation metric before the rest and may start overfitting if training were to be continued. This isn't 't reflected very clearly with the use of a validation metric that is averaged over all tasks. As a larger number of underfit tasks would skew the average, the overall optimal validation point gets shifted to a later time-step (epoch) when the model could be worse at the easier tasks. Vice versa, the optimal epoch gets shifted earlier due to a larger, easier subset that are overfit when the harder tasks reach their individual optimal epochs. We term this mismatch in the overall and task optimal epochs as a 'temporal discrepancy'.
In this work, we explore and try to mitigate this discrepancy between tasks. We present in this paper that early stopping on only the expected error over tasks leaves us blind to the performance they are sacrificing per task. The work is organized in the following manner: in $2, we explore existing work that deals with methods for incorporating task difficulty (which could be causing this discrepancy) into training. The rest of the sections along with our contributions can be summarized

---

Under review as a conference paper at ICLR 2020
1. We present method to easily visualize and detect the discrepancy through interval plots in 33
2. We formulate techniques that could quantify this discrepancy by also considering the per- task validation metrics in model selection in $4.
3. We explore the presence of the temporal discrepancy three image datasets and test the aforementioned techniques to assess the change in performance in $5
4. To the best of our knowledge, there has not been a study like this into the potential of per-task validation metrics to select an ensemble of models.
RELATED WORK
Training multiple related tasks together creates a shared representation that can generalize better on individual tasks. The rising prominence of multi-task learning can be attributed to Caruana (1997). has been acknowledged that some tasks are easier to learn than the others and plenty of works have tried to solve this issue through approaches that slow down the training of easier tasks. In other words, tasks are assigned a priority in the learning phase based on their difficulty determined through some metric. This assignment of priority implicitly tries solve the temporal discrepancy without formally addressing its presence. Task prioritization can take the form of gradient magnitudes, parameter count, or update frequencies (Guo et al., 2018). We can group existing solutions into task prioritization as a hyperparameter or task prioritization during training (aka self-paced learning)- The post-traine test-training brute force and clustering methods we propose do not fit into these categories as we believe they have not been done before. Instead of adjusting training or retraining, these methods operate on model which has already been trained.
Task prioritization as hyperparameter is way to handle per task overfitting that is almost the subconscious approach for most practitioners. This would include data-augmentation and over//ndersampling. An example case is in Kokkinos (2017) where they use manually tuned task weights order to improve performance.
Task prioritization during training covers approaches where tasks dynamically change priority or are regularized in some way. For example Guo et al. (2018) takes an approach to change task weights during training based multiple metrics such error, perceived difficulty, and learnable parameters. The idea is that some tasks need to have a high weight at the start and a low weight later in training. In a similar direction Gradnorm (Chen et al., 2018) aims based on normalizing the gradients across tasks.
balance task weights
Using relationships between tasks during training is another direction. Ruder (2017) discussed negative transfer where sharing information with unrelated tasks might actually hurt performance. Work by Lee al. (2016) incorporated directed graph of relationships between tasks in order to enforce sharing between related tasks when reweighting tasks. Task clustering has been performed outside of neural networks by Evgeniou al. (2005); Evgeniou & Pontil (2004) where they regular- ize per-task SVMs so that the parameters between related tasks are similar.
would be natural to use some of these methods as baseline for our work. However, we think would not be equitable comparison as:
These baseline methods are applied during training whereas ours is post-training analysis.
The main aspect of our analysis is only the validation metric whereas these baselines consider a variety of different aspects of training.
The focus of our work is on how the weights change with time, keeping all else constant, and how these changes affect the validation and test performance. The aforementioned methods modify the gradients w.r.t. several factors during the training which adds more degrees of freedom and is difficult to compare.
Regardless of task difficulty, training multiple tasks jointly with a neural network can lead to catas- trophic forgetring: it refers to how network can lose information that it had learned for particular task as it learns another task (McCloskey & Cohen, 1989). Multiple works have explored and tried to mitigate this phenomenon (Ratcliff, 1990; Robins, 1995; Goodrich & Arel, 2014; Kirkpatrick
2

---

Under review as a conference paper at ICLR 2020
al., 2017; Kemker et al., 2018: Lee al., 2017) and it still remains open area of research. It is highly likely that catastrophic forgetting could be causing any such temporal discrepancy; exploring the relationship between the two is an area is very interesting direction in research and is left for future work.
3 STUDYING TEMPORAL DISCREPANCY BETWEEN TASKS
Firstly, we define what a task is to disambiguate from its general usage multi-task learning litera- ture. A 'task" is predicting single output unit out of many, regardless of the training paradigm being multi-class or multi-label or other. Tasks can be very fine-grained such as predicting the class of an image much higher-level such image classification, segmentation etc. While our work uses the term in the former context, our motivation and findings can be applied in the latter context (which is the broader and more common context in multi-task learning) as well.
In the next two subsections, we define the term temporal discrepancy and display an example of on CIFAR 100. Then, introduce simple method of visualizing on datasets with a large number of tasks that would make it difficult to analyze the per-task curves together.
3.1 TEMPORAL DISCREPANCY
temporal discrepancy in the validation performance refers to the phenomenon where the model isn't optimal for all of its tasks simultaneously. This occurs when there is difference between the overall optimal epoch determined by the summarized validation metric and the epoch in which task achieves its best validation metric is higher than some arbitrary threshold. i.e., It, t, 6 where to is the optimal epoch of the summarized validation curve and fę is the optimal epoch for task i.
Figure displays an example of this discrepancy in CIFARI00 (only five curves plotted for elarity). It is most evident for the labels Sea and Lamp which undergo drop of 7.5% and 5.7% respectively in their validation accuracy from their peak epoch to fz. Similarly, Snake also starts degrading till to but strangely starts improving after. Conversely, Rose and Streetcar are underfit at t, they continue to improve after.
The most noteworthy observation is that the averaged validation curve (in dotted black) completely plateaus out after the 150th epoch. There is significant variation occurring the per-label curves but the averaged curve is unable to represent these dynamics in the training. Selecting an optimal model off the averaged curve can be quite misleading as it represents the entire [151, 300] interval optimal despite the labels' validation accuracies fluctuating significantly in this interval. The test performance of individual labels can wildly differ depending on which epoch is used for loading the weights for testing and/or deployment.
3.2 INTERVAL PLOTS
easy examine the per-label curves in Figure only 5% of the labels have been plotted. But when the number of tasks is high and all of them need to be plotted together to get a clearer global picture, decomposing the summarized validation curve can get very messy. Quasi-optimal validation interval plots, or interval plots for short, are way of assessing the optimal temporality of the per-task validation performance relative tg. It is a simple visualization method that aids in determining when and/or for how long the tasks are within the acceptable limits of the best validation performance and also which and/or how many tasks aren't 't within these limits near the overall optimal epoch t..
Creating an interval plot involves finding a 'quasi-optimal region for each task, i.e., a consecu- tive temporal interval in which validation metric of the task fluctuates near its maximum with set tolerance. The task validation curves are first smoothed out to reduce noise and the time-step (epoch) at which the task achieved its optimal validation metric is determined. Then, the number of epochs before and after this task tpst-al epoch in which the task metric is greater than a threshold is calculated. This duration of epochs is the interval for the task.
3

---

Under review as a conference paper at ICLR 2020
Temporal discrepancy in CIFAR100
90
80
Averaged
Validationaccurag
Snake
70
Lamp
Sea
60
Rose
Streetcar
50
40
100
150
200
250
300
Epoch
Figure 1: Visualizing the presence of temporal discrepancy in five of CIFARI00 labels (training detailed in Section 5). The vertical black line is the best overall epoch t.. The dotted and solid curves represent the averaged validation curve and the label specific validation curves respectively. All curves have been smoothed by averaging over a sliding window size 10.
Given a vector validation metrics A; for a task its interval T3 is given by:
Tj it, m.... 1, t.. to + n] V Oij 2 ait. where 1; = argmax A,, j and 0ij E A:
Figure plots the decomposed curves and the equivalent intervals for CIFAR100. The overall op- timal epoch doesn't fall in the intervals of almost half the labels; these labels aren't at their potentially best validation performance at the early stopping point. Some intervals are notably small duration, meaning those labels have very sharp peak. This could imply that the validation per- formance is randomly high at that epoch and it'd be more suitable to shift the quasi-optimal region of these labels longer and/or later interval, that doesn't necessarily contain t,, long as the validation accuracy stays within the tolerance in that interval.
Label valiidation curves
Interval plot
100
300
30
Va3622 on o0Ci53))
Eporh
200
100
50
100
150
200
300
Epoch
Tanks
Figure 2: Smoothed decomposed validation curves (left) and the equivalent interval plot (right) for CIFAR 100 with 2%. For both plots, the straight black line is the best overall epoch fg. On the interval plot, the blue lines indicate the interval тj. the blue dots are the centers of the interval, and the red dots the task-optimal epoch t, The intervals have been sorted by their centers
4 QUANTIFYING PERFORMANCE LOST DUE TO TEMPORAL DISCREPANCY
In this section, we present two simple techniques that consider the per-task validation metrics for selecting the best model for testing. Two aspects common to these techniques is their hand-crafted
4

---

Under review as a conference paper at ICLR 2020
& engineered nature and their inefficiency in terms of deployment, training time and/or inference time. The aim with these techniques to assess how much potential gain in performance could be attained if we account for per-task validation metries in selecting model and we'd 'dd like to stress that these techniques serve as a proof-of-concep of this gain (if any). They are intended as a baseline and also a stimulus for increasing research into the effect of the subtleties of the validation curves on model performance and selection.
Brute force This involves loading the model with the weights from a given task i's optimal vali- dation epoch 4 and evaluating only the samples that belong to that task. We call this particular model that has been loaded with the weights from 4 as the validation-optimal model for task i. is essentially using a "separate model" for each task during evaluation and/or deployment making this the most naive approach. It is also the most inefficient because it would (i) require storing up to N models, where N is the total number of tasks (ii) require way to combine predictions from all models that wouldn't be misleading during inference (iii) increase latency significantly due to overhead caused by loading and reloading the model weights (iv) scale up the inference time by factor up to N.
Clustering Instead of having separate weights for each task, we try to cluster the set of the task- optimal validation epochs into K clusters so that only K models are required as opposed to N. In this approach, the interval plots can also be utilized cluster tasks that have similar interval positions and/or lengths in addition to the f,'s. Similar to brute force, this technique also involves multiple models loaded with weights from different epochs, but trades off any gain in performance for a lower number of models.
5 EXPERIMENTS
We train variations of DenseNets (Huang et al., 2017) on three image datasets: CIFAR100, Tiny ImageNet and PadChest. All models were trained with three random seeds for model initialization and splits of the training set into training and validation sets.
CIFAR100 CIFARI00 (Krizhevsky & Hinton, 2009) is a dataset of natural images containing 100 classes with 500 training images and 100 testing images per class. We trained "DenseNet-BCC (k 12)" as described in Huang et al. (2017): it has three dense blocks, a total of 100 layers and a growth rate of 12. It was trained in the exact manner the original work, i.e. 300 epochs with dropout, weight decay of 10-4, SGD with Nesterov momentum of 0.9 and an initial learning rate of 0. that divided by 10 at the th and 225th epochs. As we had carved out 20% of the training set as validation and didn't 't use data augmentation, we achieved an average test accuracy of 72.14% 14%. In our analysis, we only use the validation curves after the 150th epoch because the training very noisy and brittle up to that epoch due to the use of learning rate of 0.1 (Figures I and 2).
Tiny ImageNet ImageNet (Deng et al., 2009) is a dataset with 1.5 million images and 1000 classes. Tiny ImageNet' is a subset of ImageNet with images resized to 64x64 and only 200 classes. We utilized the same architecture as CIFAR 100 but with a total of 190 layers and a growth rate of 40. In addition, we used stride of 2 for the first convolution. This modified DenseNet was also trained in the same manner as above but with the hyperparameters used for ImageNet in the original work: 100 epochs, dropout and dividing the learning rate at the 30th and 60th epochs instead. Rest of the hyperparameters remained the same. 10% of the training set was used as validation and we achieved an average test accuracy of 63.09%. Similar CIFAR100, we only use the validation curves after the t0 epoch. The interval plot from one run of training is given in Figure 3a
PadChest PadChest (Bustos et al., 2019) is a medical imaging dataset of 160,000 chest X-rays of 67,000 patients with multiple visits and views available. We used the publicly available code provided by Bertrand et al. (2019) to recreate their cohort of around 31,000 samples. We trained multilabel DenseNet- 121 (Huang et al., 2017: Rajpurkar et al., 2017) the frontal views and only those labels which have more than 50 samples (total 64 labels). With a 60-20-20 split between training, validation, and test sets, we trained for 100 epochs with Adam and initial learning rate of
5

---

Under review as a conference paper at ICLR 2020
0.0001 that is halved every 20 epochs. The interval plot from one of training is given in Figure 3b
100
80
80
Epoch
60
Epoch
40
20
20
Tasks
Tasks
(a) Tiny ImageNet (200 classes)
(h) PadChest (64 labels)
Figure 3: Interval plots for Tiny ImageNet and PadChest with e = 0.02. The gray/green lines and dots indicate the interval and its center respectively. The black horizontal line is the best overall epoch is and the red dots are the task-optimal epoch 4. The intervals have been sorted by their centers
5.1
BRUTE FORCE
By brute forcing the best model selection, we wanted to assess how much performance is lost when the summarized validation curve is used. This involves evaluating each task with its own set of optimal weights determined from its specific optimal epoch. This would require N models in theory but the number of validation-optima models is actually lower many tasks have inter-dependent learning profiles. These correlated tasks may reach their optimal validation performance at the same epoch, thus requiring only single common set of weights for all of them. N also decreases with the total number training epochs as that increases the probability epoch. On analyzing the task validation curves for the three datasets, do find that the number of models required is much lower than N. CIFAR 100 and Tiny ImageNet both required less than 60 models for brute forcing the evaluation, despite the latter having twice many labels. Also since Tiny ImageNet was trained for one-third of the epochs, the number of models reduced drastically. The number of models for PadChest around 35.
tasks having the same optimal
The results of using a single model for each task are tabulated for the three datasets in Table 1. On using each label's validation-optimal model for evaluating on the test, the test metric always increases in comparison to using the bascline model that uses the summarized validation curve for CIFAR100 and Tiny ImageNet. The top-1 accuracy CIFARI00 undergoes an average and maximum increase of approximately 2.5% & 3.2% respectively. For Tiny ImageNet, an average and maximum increase of 1.72% 1.95% is observed.
However, the average increase in PadChest is not only meagre but also within the standard deviation. This could be because the temporal discrepancy isn't very pronounced in PadChest: overall
Table I: Accuracy/AUC on the test set for the baseline and brute forced models, averaged over seeds
Dataset
Metric
Baseline
Brute-force
Top-1
72.14 + 0.51
74.62 + 0.43
CIFAR100
Top-5
92.23 = 0.33
93.07 + 0.26
Top-1
63.09 = 0.27
64.82 + 0.39
Tiny ImageNet
Top-5
83.79 = 0.26
84.91 + 0.47
PadChest
AUC
0.7807 = 0.001
0.7826 + 0.002

---

Under review as a conference paper at ICLR 2020
optimal epoch does happen be in the intervals of very large portion of the labels (Figure 3b). As PadChest has fewer number of outputs compared to the other two (64 vs. 100/200), there could also be a relationship between the number of outputs and the magnitude of the discrepancy's 's effect. These results show that accounting for the per-task validation metrics can mitigate a significant temporal discrepancy and increase the testing performance.
5.2 COMPUTATTON-PERIORRAACC TRADEOFF
We can further reduce the number of validation-optimall models required by merging the 4 that are close to each other. We use K-means to cluster the set of task-optimal epochs into K clusters. We vary K from 2 to to gain an insight into how much performance lost as we reduce the number of validation-optimall models required down to one where using only one model is the same using the summarized. For any given cluster k, we use its center as the epoch for loading the weights to evaluate the set of the tasks in k. We round off the center the nearest integer for analysis but the fractional part can be used to select weights after a specific batch iteration in that epoch.
The results for the three datasets plotted in Figure 4. There is interesting observation that the accuracy doesn't I always increase with the number of task-optimal models which is very noticeable in Tiny ImageNet Seed 2. This could be due to the noisy nature of training with SGD where the validation metric can oscillate lot between epochs and even a shift of one epoch can cause decrease performance.
75.0
65.0
sccuracy rate (%)
74.5
(%)
64.5
74.0
73.5
63.5
73.0
72.5
Text
63.0
72.0
Seed 1
Seed
62.5
Seed
Seed
Seed 2
Averaged
Seed
Averaged
10
20
30
40
10
20
30
40
50
60
Number clusters
Number of clusters
(a) CIFAR100
(b) Tiny ImageNet
0.784
0.782
AUC
0.780
0.778
0.776
Seed
Seed
Seed 2
Averaged
0.774
20
90
40
Number of clusters
(c) PadChest
Figure 4: Reducing the number of models needed to represent all per-task optimal validation points. K is the number of clusters

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
In this work, we examine the decomposition of a model's average validation curve into its per-task curves to assess the presence of a temporal discrepancy on three image datasets. We provide visualization method to detect if there is disparity between the task curves and the summarized validation curve. We test two techniques that incorporate the per-task metrics into model evaluation and we find that that when per-task validation metrics are accounted for training runs that show significant temporal discrepancy, we gain an increase in testing performance. We show that with CI- FARIOU 100 there is room for up 2.5% to be gained in testing accuracy and 1.72% for Tiny ImageNet we don't t use the averaged metrics as they are.
With these experiments, aim to create cannot represent a model that is truly optimal for all its tasks all the time. Using averaged curves could mean that models are currently being trained oblivious of the performance being sacrificed individual tasks. We wish to draw attention to the need of both theoretical and engineered approaches that would take the per-task validation metrics into account while training. Our experiments demon- strate that there is potential that current state-of-the-ar models could possibly be made even more optimal by ensuring all of its prediction tasks are their validation optimums.
awareness of how summarized validation metrics
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations, 2016.
Hadrien Bertrand, Mohammad Hashir, and Joseph Paul Cohen. Do Lateral Views Help Automated Chest X-ray Predictions? 2019 In Medical Imaging with Deep Learning Abstract.
Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia- PadChest: A large chest x-ray image dataset with multi-labe annotated reports. 2019 arxiv.
Rich Caruana. Multitask Learning. 1997 Machine learning. doi: 10.
Zhao Chen, Vijay Badrinarayanan, Chen- Lee, and Andrew Rabinovich. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. In International Con- ference on Machine Learning, 2018.
J Deng, W Dong, R Socher, L.-J. Li, K Li, and L Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition, 2009.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In International Conference on Knowledge Discovery and Data Mining, New York, New York, USA, 2004. ACM Press. doi: 10.11455/014053 1014067.
Theodoros Evgeniou, Charles A. Micchelli, and Massimiliano Pontil. Learning Multiple Tasks with Kernl Methods. Journal of Machine Learning Research, 2005.
William Finnoff, Ferdinand Hergert, and Hans Georg Zimmermann. Improving model selection by nonconvergent methods. 1993 Neural Networks. doi: 10. 10.1016/S0893 6080(05)8011)2))))
Ben Goodrich and Itamar Arel. Unsupervised neuron selection for mitigating catastrophic forgetting in neural networks. In 2014 JEEE 57th International Midwest Symposium on Circuits and Systems (MWSCAS). 2014 ieeexplore.ieee.og doi: 10.1
Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. 2016 arXiv.
Michelle Guo, Albert Haque, De An Huang, Serena Yeung, and Li Fei-Fei. Dynamic Task Pri- oritization for Multitask Learning. In European Conference on Computer Vision, 2018. doi:
17.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. In Computer Vision and Pattern Recognition, 2017.
8

---

Under review as a conference paper at ICLR 2020
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L Hayes, and Christopher Kanan. Measur- ing Catastrophic Forgetting in Neural Networks. Thirty-second AAAI Conference on Artificial Intelligence. aaai.org, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Brrbska-Barwinse Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences, 2017 volume 114. doi: 10. 10.nas 161 16116331111 1835114.
lasonas Kokkinos. UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017 IEEE. doi: 10. 10.110//CPR...07....7
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Asymmetric multi-task learning based on task relatedness and loss. International Conference of Machine Learning, 2016.
Sang-Woo Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang. Overcoming Catastrophic Forgetting by Incremental Moment Matching. I Guyon, U V Luxburg, Bengio, H Wallach, R Fergus, Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 2017.
Syoung-Tak
Michael McCloskey and Neal J Cohen. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In Gordon H Bower (ed.), Psychology of Learning and Motivation, volume 24. 1989 Academic Press. doi: 10. 1411//007997421
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang. Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. CheXNet: Radiologist-Level Pneumonia Detection Chest X-Rays with Deep Learning. 2017 arxiv.
Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. 1990 Psychological review. doi:
Anthony Robins. Catastrophic Forgetting, Rehearsal and Pseudorehearsal. 1995 Connection sci- ence. doi: 10.1080005440995503331
Sebastian Ruder. An Overview of Multi Task Learning in Deep Neural Networks. 2017 arXiv.


Reviews:
[{"review_id": "r1xyacKWcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to do/contribute. Be positive and generous.\nThe paper proposes a new method for model selection in the case of classification with multiple labels. The method suggests not relying on average accuracy over all labels to choose a model but choosing multiple models depending on the label and apply them to the related samples.\n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\nReject.\n- I don't think the paper is contributing something new to the literature. Also there seems to be some confusion re: what accuracy means and is used in some places instead of precision.  (eg in. Sect. 4.1)\n\nProvide supporting arguments for the reasons for the decision.\n- One question I had was to think about is how one would know what model to apply to what samples if they don't know a priori the labels of these samples. This seems to be something your suggested approaches rely on.\n\nProvide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n- \"valid\" is used often instead of validation.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgjwrSBtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task.\n\nAt different points of training, a model's performance on a task will vary. To make it easier to examine this, they propose displaying an interval plot. The validation accuracy for a single task will be largest at a specific epoch. For each single task, display the epochs where performance is within some threshold of the best validation-set accuracy for that task. Plots on CIFAR-100 demonstrate how some tasks are learned quickly then forgotten, while other tasks are only learned late.\n\nTo examine the difference in performance, they compare the single best CIFAR-100 model, to the classifier defined by \"for class k, forward input to the model with best validation accuracy for class k\". This gives about +2.5% in Top-1 performance, a similar story appears on the other 2 datasets they try (Tiny ImageNet, PadChest). This requires N models, where N is the number of classes. They also examine a clustering approach, where they cluster the N models into K models (using K-means), then forward class k to the best model out of those K models.\n\nOverall, it isn't an especially large contribution but I felt it had some interesting points. Some comments:\n* Citation list feels short. The catastrophic forgetting literature seems relevant here. Rich Caruana's multitask learning thesis (Caruana et al 1997) is also relevant.\n* It's unclear how the K-means is carried out. I assume the features used for each model checkpoint are tied to its performance on individual task but it's never spell out what distance metric is used in the K-means.\n* It feels strange to have no comparisons to ensemble-based baselines. The baseline here would be, for some parameter K, run K independent training runs, take the top average validation accuracy from each run, and average them together, then compare this to the K models found from K-means clustering. For small enough K (like 5 or 10) this seems like a feasible experiment, computation-wise.\n* In a similar vein, I wonder how this compares to the Snapshot Ensembles paper, which has a similar guarantee of doing 1 training run and giving M models for an ensemble. Is the gain from ensembling, or from directly examining which models are better at which classes?\n* A natural follow-up here is to use model distillation to distill the N best models for each individual class into a single model checkpoint - does this give us a better single model checkpoint?\n\nAs-is, I give this paper a weak reject, but would be willing to increase if some of these experiments were tried.\n\nEdit: having read the other reviews and author comments, I still maintain a weak reject rating. I believe this paper needs further work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg_5_6hYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nModel validation curve typically aggregates accuracies of all labels. This paper investigates the fine-grained per-label model validation curve. It shows that the optimal epoch varies by label. The paper proposes a visualization method to detect if there is a disparity between the per-label curves and the summarized validation curve. It also proposes two methods to exploit per-label metrics into model evaluation and selection. The experiments use three datasets: CIFAR 100, Tiny ImageNet, PadChest.\n\nLimitations\nThe paper is very preliminary in nature. It does not compare with other related work. For example,  the task prioritization during training approach where tasks dynamically change priority or are regularized in some way. It will be good to see how the proposed approach compare with other related ones (three listed in related work) in the literature. \n\nOther approaches not mentioned in the paper can be more effective. For example, \nFocal Loss for Dense Object Detection, https://arxiv.org/abs/1708.02002\nmight automatically handle the problem to a large extent.\n\nHow much the problem is due to label imbalancing? If label imbalancing is one main problem, please first address it.\n\nThe proposed approach is not very interesting, brutal force and clustering. They are very straightforward.\n\nOverall, the quality is far below the bar of ICLR.\n\nComments not affect rating\n\nThe paper uses \"task\" which is not defined. It is confusing until much later in the paper that it just refers to \"class\".\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers reached a consensus that the paper is preliminary and has a very limited contribution. Therefore, I cannot recommend acceptance at this time.
Decision: Reject