Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summarize what the paper claims to do/contribute. Be positive and generous.
The paper proposes a new method for model selection in the case of classification with multiple labels. The method suggests not relying on average accuracy over all labels to choose a model but choosing multiple models depending on the label and apply them to the related samples.

Clearly state your decision (accept or reject) with one or two key reasons for this choice.
Reject.
- I don't think the paper is contributing something new to the literature. Also there seems to be some confusion re: what accuracy means and is used in some places instead of precision.  (eg in. Sect. 4.1)

Provide supporting arguments for the reasons for the decision.
- One question I had was to think about is how one would know what model to apply to what samples if they don't know a priori the labels of these samples. This seems to be something your suggested approaches rely on.

Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
- "valid" is used often instead of validation.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary
Model validation curve typically aggregates accuracies of all labels. This paper investigates the fine-grained per-label model validation curve. It shows that the optimal epoch varies by label. The paper proposes a visualization method to detect if there is a disparity between the per-label curves and the summarized validation curve. It also proposes two methods to exploit per-label metrics into model evaluation and selection. The experiments use three datasets: CIFAR 100, Tiny ImageNet, PadChest.

Limitations
The paper is very preliminary in nature. It does not compare with other related work. For example,  the task prioritization during training approach where tasks dynamically change priority or are regularized in some way. It will be good to see how the proposed approach compare with other related ones (three listed in related work) in the literature. 

Other approaches not mentioned in the paper can be more effective. For example, 
Focal Loss for Dense Object Detection, https://arxiv.org/abs/1708.02002
might automatically handle the problem to a large extent.

How much the problem is due to label imbalancing? If label imbalancing is one main problem, please first address it.

The proposed approach is not very interesting, brutal force and clustering. They are very straightforward.

Overall, the quality is far below the bar of ICLR.

Comments not affect rating

The paper uses "task" which is not defined. It is confusing until much later in the paper that it just refers to "class".

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task.

At different points of training, a model's performance on a task will vary. To make it easier to examine this, they propose displaying an interval plot. The validation accuracy for a single task will be largest at a specific epoch. For each single task, display the epochs where performance is within some threshold of the best validation-set accuracy for that task. Plots on CIFAR-100 demonstrate how some tasks are learned quickly then forgotten, while other tasks are only learned late.

To examine the difference in performance, they compare the single best CIFAR-100 model, to the classifier defined by "for class k, forward input to the model with best validation accuracy for class k". This gives about +2.5% in Top-1 performance, a similar story appears on the other 2 datasets they try (Tiny ImageNet, PadChest). This requires N models, where N is the number of classes. They also examine a clustering approach, where they cluster the N models into K models (using K-means), then forward class k to the best model out of those K models.

Overall, it isn't an especially large contribution but I felt it had some interesting points. Some comments:
* Citation list feels short. The catastrophic forgetting literature seems relevant here. Rich Caruana's multitask learning thesis (Caruana et al 1997) is also relevant.
* It's unclear how the K-means is carried out. I assume the features used for each model checkpoint are tied to its performance on individual task but it's never spell out what distance metric is used in the K-means.
* It feels strange to have no comparisons to ensemble-based baselines. The baseline here would be, for some parameter K, run K independent training runs, take the top average validation accuracy from each run, and average them together, then compare this to the K models found from K-means clustering. For small enough K (like 5 or 10) this seems like a feasible experiment, computation-wise.
* In a similar vein, I wonder how this compares to the Snapshot Ensembles paper, which has a similar guarantee of doing 1 training run and giving M models for an ensemble. Is the gain from ensembling, or from directly examining which models are better at which classes?
* A natural follow-up here is to use model distillation to distill the N best models for each individual class into a single model checkpoint - does this give us a better single model checkpoint?

As-is, I give this paper a weak reject, but would be willing to increase if some of these experiments were tried.

Edit: having read the other reviews and author comments, I still maintain a weak reject rating. I believe this paper needs further work.
review_title: Official Blind Review #3

Meta-review: The reviewers reached a consensus that the paper is preliminary and has a very limited contribution. Therefore, I cannot recommend acceptance at this time.
Decision: Reject