Venue: ICLR 2020 Conference

Reviews:

final_score: rating: null
review_content: NB this was an emergency review although there is now an R3. I hope the perspective from an additional reader of the paper will be helpful for authors and ACs, but I know there is little time left so it is understandable that authors may not be able to respond.

Summary:

This paper takes a look at the prevailing way of evaluating classifiers through the use of a held-out test set, raising the question of whether this is the best way to be comparing models. It highlights drawbacks of the existing approach, and proposes an alternative which captures more precisely how classifiers can differ even if they have the same accuracy overall. The main proposal is to build a test set adaptively in a manner that captures how classifiers disagree, as measured by the wordnet tree.

Experiments are carried out considering a number of popular models trained on ImageNet, with the unlabelled set of potential test images obtained from Flickr. This is used to re-rank the classifiers, with shifts up to -4.

Review:

I am borderline towards accept.

I find that the strength of the work is the analysis of several recent CNN architectures in a way that moves beyond fixed test set accuracy, and evaluates models through the lens of how their predictions differ. In this sense it is a worthwhile contribution, similar in spirit to work like Recht et al. and Hendrycks et al. in seeking to see how over-reliance on benchmarks can be misleading. However as a general framework it leaves something to be desired, being quite ad-hoc (use of wordnet, choice of k, collecting labels). So I see it as more of a one-off experiment designed to evaluate their hypothesis rather than a methodology people can easily start using to move beyond test accuracy alone.

I think it's good for the community to take a second look at evaluation methodologies, and whether we can be missing something by only caring about being top of the leaderboard, and the experiments support the claims that something is being missed by considering test set accuracy alone, finding more subtle differences between popular models, such as VGG16bn (accuracy 73.36) being re-ranked above ResNet101 (despite higher accuracy 77.37).

"A classifier that is more likely to be falsified is considered worse." How robust is this principle? Would it be thrown off by some pathologically bad classifier in the competition? Would it lead to the Galileo of classifiers being condemned for speaking the truth?

Does it actually solve the problems mentioned? Eg adversarial attack, domain differences. Still need to collect the unlabelled images, which are from a given distribution (Flickr images).

To be useful to the community a good replacement or complementary ranking to accuracy should be easy to report for any new model proposed. But the proposed procedure is a bit unwieldy, it requires having some set of competing classifiers, wordnet classes, and labelling new images for any new classifier added (which may need domain experts). For it to work fairly in practice there'd need to be some sort of evaluation server and annotation effort. Maybe for Imagenet it would be worth it but it's not very general. Alternatively, can any of the principles or lessons learned be used to design a better evaluation protocol but without the expense of manual labelling for each new classifier, such as re-weighting a fixed test set, or to generate some benchmark test sets like ImageNet-A.

"careful inspection of the selected images may suggest potential ways to improve" any examples where this was found?

Feedback:

"we believe a much larger test set in the order of millions or even billions must be used" I think this goes against the rest of the paper (ie a small high quality test set is ok) - and even with infinite budget annotating billions of test images from the same distribution would be of little use.

Presentation: best not to rely on red/green colours being visible (colourblindness or mono printer) add some symbol too
review_title: Comment/Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method to compare existing classifiers, which does not use fixed test set and adaptively sample it from an arbitrarily large corpus of unlabeled images. The main idea seems similar to adopting active learning for the test set selection.
 
One of the main advantage is that it can select a sample set from an arbitrarily large unlabeled images. However, to compare different classifiers, the proposed algorithm still needs humans to annotate the selected dataset, which is very expensive compared with traditional methods.  

Since this paper select the top-k images in D, if k is large the annotating for S will be very tedious, however if k is relatively small the method seems very sensitive to selected examples, which will make the comparison not totally convincing.
 
The authors invite five volunteer graduate students to annotate the selected example. However, for many categories, it’s nor easy for normal people to distinguish. So the experiments in this paper is also not convincing. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper points out that the traditional way of model selection is flawed due to that the validation/test set is often small. The authors also attribute the existence of adversarial examples to the small validation/test set, which I agree to some degree. Hence, the authors proposed an alternative approach to comparing different classification models by the notion of inter-model discrepancy. 

The main idea is reasonable, but it requires that the models to compare all perform reasonably well. Otherwise, some poorly performed models could lead to near-random or adversarial inter-model discrepancies, failing the proposed approach. 

Another potential issue is that the proposed approach cannot handle training set bias. If all models are biased in similar ways (e.g., toward a particular class or domain), they will not reveal informative discrepancies for the images over which they all make similar mistakes. 

Another question which is not answered in the paper is the number $k$ of images to select for each pair of classifiers. Is this number task-dependent? Is it related to the number of classes? What is a general guideline for one to choose this number $k$ given a new application scenario? 

The unlabeled set is not "unlabeled" in essence. If my understanding was correct, it cannot contain open-set images which do not belong to any of the classes of interest. It is also nontrivial to control that the images contain only one salient object per image.

Hence, while I agree with the authors that existing approaches to comparing deep neural network classifiers could be improved, I think the proposed solution is not a good alternative yet. 


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper proposed a novel image classifier comparison approach that went beyond one fixed testing set for all. Instead, for a pair of classifiers to be compared, it advocated to sample their "most disagreed" test set from a large corpus of unlabeled images. The level of disagreement was measured by a semantic-aware distance derived from WordNet ontology. Because of the efficacy of such "worst-case" comparison, the needed set size is very small and thus minimizes the human annotation workload. 

The proposed MAD competition distinguishes classifiers by finding their respective counterexamples. It is therefore an "error spotting" mechanism, rather than a drop-in replacement of standard test accuracy. I feel the approach to implicitly assume that the classifiers to be compared are already "reasonably accurate"; since if not, both classifiers might be easily falsified by certain trivial examples, making the "disagreed examples" not as meaningful. If that is true, I would suggest the authors to make this hidden assumption clearer in the paper

The idea shows clear liaison to the "differential testing" concept in software engineering besides the cited work of perceptual quality assessment. The idea has a cross-disciplinary nature and is fairly interesting to me. I can see the paper to be of interest to a quite broad audience and can motivate many subsequent works. 

One minor comment: for images in "Case III", the authors considered them "contribute little to performance comparison between the two classifiers" and therefore did not source labels for them. However, since the authors adopted an affinity-aware distance, two incorrect predictions can still be compared based on their semantic tree distances to the true class.

review_title: Official Blind Review #2

Meta-review: This paper proposes a new way of comparing classifiers, which does not use fixed test set and adaptively sample it from an arbitrarily large corpus of unlabeled images, i.e. replacing the conventional test-set-based evaluation methods with a more flexible mechanism. The main proposal is to build a test set adaptively in a manner that captures how classifiers disagree, as measured by the wordnet tree. As noted by R2, this work has the potential to be of interest to a broad audience and can motivate many subsequent works. 

While the reviewers acknowledged the importance of this work, they raised several concerns: (1) the proposed approach is immature to be considered for benchmarking yet (R1,R4), (2) selecting k and studying its influence on the performance ( R1, R3, R4), (3) the proposed approach requires data annotation which might not be straightforward -- (R3, R4).  The authors provided a detailed rebuttal addressing the reviewer concerns.

There is reviewer disagreement on this paper. The comments from R3 were valuable for the discussion, but at the same time too brief to be adequately addressed by the authors. The comments from emergency reviewer were helpful in making the decision. AC decided to recommend acceptance of the paper seeing its valuable contributions towards re-thinking the evaluation of current SOTA models.

Decision: Accept (Poster)