I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively
Authors: Haotao Wang
Tianlong Chen
Zhangyang Wang
Kede Ma
Venue:  (2020)
Keywords: model comparison

Abstract: The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training. On the other hand, the trained classifiers have traditionally been evaluated on small and fixed sets of test images, which are deemed to be extremely sparsely distributed in the space of all natural images. It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations. Inspired by efficient stimulus selection for testing perceptual models in psychophysical and physiological studies, we present an alternative framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather than comparing image classifiers using fixed test images, we adaptively sample a small test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy. Human labeling on the resulting model-dependent image sets reveals the relative performance of the competing classifiers, and provides useful insights on potential ways to improve them. We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition. Codes can be found at https://github.com/TAMU-VITA/MAD.

Published as a conference paper at ICLR 2020
I AM GOING MAD: MAXIMUM DISCREPANCY COM-
PETITION FOR COMPARING CLASSIFIERS ADAPTIVELY
Haotao Wang
Tianlong Chen
Department of Computer Science and Engineering Texas A&M University htwang@ter www.ettauueec..
Department of Computer Science and Engineering Texas A&M University wi w jp 61 9 tamu. edu
Zhangyang Wang
Kede Ma
Department of Computer Science and Engineering Texas A&M University at sllaswang@ttmm edu
Department of Computer Science City University of Hong Kong kede. ma @ tyuut...u... edu.. hk
ABSTRACT
The learning of hierarchical representations for image classification has experi- enced impressive series of successes due in part the availability of large- scale labeled data for training. On the other hand, the trained classifiers have traditionally been evaluated on small and fixed sets of test images, which are deemed to be extremely sparsely distributed in the space of all natural images. It is thus questionable whether recent performance improvements on the exces- sively re-used test sets generalize to real-world natural images with much richer content variations. Inspired by efficient stimulus selection for testing pereep- tual models in psychophysical and physiological studies, we present alterna- tive framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather than comparing image classifiers us- ing fixed test images, we adaptively sample a small test set from an arbitrarily large corpus of unlabeled images the classifiers, measured by the distance over WordNet hierarchy. Human la- beling on the resulting model-dependent image reveals the relative perfor- mance of the competing classifiers, and provides useful insights on potential ways to improve them. We report the MAD competition results of eleven Ima- geNet classifiers while noting that the framework is readily extensible and cost- effective add future classifiers into the competition. Codes can be found
to maximize the discrepancies between
petpss//ggie com TAMAT-ITAAAAAA
INTRODUCTION
Large-scale human-labeled image datasets such as ImageNet (Deng et al. 2009) have greatly con- tributed the rapid progress of research in image classification. In recent years, considerable effort has been put into designing novel network architectures (He et al. 2016 Hu et al.) 2018) and ad- vanced optimization algorithms (Kingma & Ba 2015) improve the training image classifiers based on deep neural networks (DNNs), while little attention has been paid to comprehensive and fair evaluation/comparison of their model performance. Conventional model evaluation methodol- ogy for image classification generally follows a three-step approach (Burnham & Anderson 2003). First, pre-select number of images from the space of possible natural images (i.e., natural im- age manifold) to form the test set. Second, collect the human label for each image in the test set to identify its ground-truth category. Third, rank the competing classifiers according to their goodness of fit (e.g., accuracy) the test set: the with the best result is declared the winner.
significant problem with this methodology the apparent contradiction between the enormous size and high dimensionality of natural image manifold and the limited scale of affordable testing (i.e.. human labeling, or verifying predicted labels, which is expensive and time consuming). As a result, typical "large-scale" test set for image classification allows for tens of thousands of natural

---

Published as a conference paper at ICLR 2020
VGG16BN: bubble
VGG16BN: traffic light
ResNet34: shower curtain
ResNet34: balloon
VGG168N: pretzel
VG616BN: guncomole
ResNet34: carousel
ResNet34: mortor
(0)
(b)
(c)
Figure 1: Overview of the MAD competition procedure. (a): A large unlabeled image set of web scale. (b): The subset of natural images selected from (a) on which two classifiers JVGG16BM and ResNer34 in this case) make different predictions. Note that collecting the class label for each image in this subset may still be prohibitive because of its gigantic size. (c): Representative ex- amples sampled from top-k images on which VGGI6BN's and ResNet34's predictions differ the most, quantified by Eq. 3 Although the two classifiers have nearly identical accuracies on the ImageNet validation set, the proposed MAD competition successfully distinguishes them by finding their respective counterexamples. This sheds light on potential ways to improve the two classifiers or combine them into a better one. The model predictions are shown along with the images, where green underlined and red italic texts indicate correct and incorrect predictions, respectively.
images to be examined, which are deemed to be extremely sparsely distributed in natural image manifold. Model comparison based on a limited number of samples assumes that they are sufficiently representative of the whole population, an assumption that has been proven to be doubtful in image classification. Specifically,, |Recht et al. (2019) found that minute natural distribution shift leads to large drop in accuracy for broad range of image classifiers on both CIFAR- 10 (Krizhevsky) 2009) and ImageNet (Deng al., 2009), suggesting that the current test sets may be far from sufficient to represent hard natural images encountered in the real world. Another problem with the conventional model comparison methodology is that the test sets are pre-selected and therefore fixed. This leaves the door open for adapting classifiers to the test images, deliberately or unintentionally, via extensive hyperparameter tuning, raising the risk of overfitting. As a result, it is never guaranteed that image classifiers with highly competitive performance such small and fixed test sets generalize to real-world natural images with much richer content variations.
In order reliably measure the progress in image classification and to fairly test the generalizability of existing classifiers in a natural setting, we believe that it is necessary to compare the classifiers on much larger image collection in the order of millions or even billions. Apparently, the main challenge here is how to exploit such a large-scale test set under the constraint of very limited budgets for human labeling, knowing that collecting ground-truth labels for all images is extremely difficult, if not impossible.
In this work, we propose an efficient and practical methodology, namely the MAximum Discrepancy (MAD) competition, to meet this challenge. Inspired by Wang & Simoncelli (2008) and Ma et al. 2019), instead of trying to prove an image classifier to be correct using a small and fixed test set, MAD starts with large-scale unlabeled image set, and attempts to falsify classifier by finding a set of images, whose predictions are in strong disagreement with the rest competing classifiers (see Figure D. A classifier that is harder to be falsified in MAD is considered better. The initial image set for MAD to explore can be made arbitrarily large provided that the cost of computational prediction for all competing classifiers is cheap. To quantify the discrepancy between two classifiers on one image, we propose a weighted distance over WordNet hicrarchy (Miller) 1998). which is more semantically aligned with human cognition compared with traditional binary judgment (agree
2

---

Published as a conference paper at ICLR 2020
disagree). The set of model-dependent images selected by MAD are the most informative in discriminating the competing classifiers. Subjective experiments on the MAD test set reveal the relative strengths and weaknesses among the classifiers, and identify the training techniques and architecture choices that improve the generalizability to natural image manifold. This suggests potential ways to improve classifier to combine aspects of multiple classifiers.
We apply the MAD competition to compare eleven ImageNet classifiers, and find that MAD verifies the relative improvements achieved by recent DNN-based methods, with a minimal subjective test- ing budget. MAD is readily extensible, allowing future classifiers to be added into the competition with little additional cost.
THE MAD COMPETITION METHODOLOGY
The general problem of model comparison in image classification may be formulated as follows. We work with the natural image manifold X, upon which define class label f(z) y for every x E X, where y = (1.2..... c} and c is the number of categories. We assume a subjective assessment environment, in which human subject can identify the category membership for any natural image x among all possible categories. assumed, each of which takes natural image x as input and makes prediction of f(x)), collectively denoted by (J(zz) J The goal is to compare the relative performance of m classifiers under very limited resource for subjective testing.
group of image classifiers F = [ are also
The conventional model comparison method for image classification first samples a natural image set D { 1n k= C X. For each image T& E D. we ask human annotators provide the ground- truth label f(x)) E V. Since human labeling is expensive and time consuming, and DNN-base classifiers are hungry for labeled data the training stage (Krizhevsky et al., 2012). n is typically small in the order of tens of thousands (Russakovsky al.) 2015). The predictions of the classifiers are compared against the human labels by computing the empirical classification accuracy
Acc(f; D)
1
Σ [[f(()= ZED
f()), for i 1,...,,7. m.
D
The classifier Si with a higher classification accuracy is said to outperform ʃj with a lower accuracy.
As an alternative, the proposed MAD competition methodology aims to falsify classifier in the most efficient way with the help of other competing classifiers. classifier that more likely to be falsified is considered worse.
2.1 THE MAD COMPETITION PROCEDURE
The MAD competition methodology starts by sampling an image set 'D (ax Yi from the natural image manifold X. Since the number of images selected by MAD for subjective testing is indepen- dent of the size of D, we may choose to be arbitrarily large such that D provides dense coverage of (i.e., sufficiently represents) X. MAD relies on a distance measure to quantify the degree of discrepancy between the predictions of any two classifiers. The most straightforward measure is the 0-1 loss:
do (f(())) = =(f))))))))))))))))))))))(())()) + fx(2).
Unfortunately, it ignores the semantic relations between class labels, which may be crucial in distin- guishing two classifiers, especially when they share similar design philosophies (e.g., using DNNs as backbones) and are trained on the same image set (e.g., ImageNet). For example, misclassifying dog of other species is clearly more acceptable compared with misclassifying as a "wwatrrmeon".. We propose to leverage the semantic hierarchy WordNet (Miller 1998) to measure the distance between two (predicted) class labels. Specifically, we model WordNet as a weighted undirected graph EVVEET Each edge e = (u, E E connects a parent node u of a more general level (e.g., canine) to its child node of more specific level (e.g., dog), for u, v E V. A nonnegative weight w(e is assigned to each edge e = (u, v)) to encode the semantic distance
"chihuahua"
Although WordNet tree-structured, child node in WordNet may have multiple parent nodes.
3

---

Published as a conference paper at ICLR 2020
Figure 2: Comparison of weighted and un- weighted distances. In the sub-tree of Word- Net, we highlight the shortest paths from "fountain" " "church" and from "drake" to "American coot" in green and red, respec- tively. The semantic distance between the two aquatic birds is much shorter than that between the two constructions of completely different functionalities (verified by our in- ternal subjective testing). The proposed weighted distance is well aligned with hu- man cognition by assigning a much smaller distance the red path (0.0037) compared with the green one (0.0859).
entity
structure, construction
fountain
church, church building
aquatic bird
American coot, marsh hen
drake
between u and v. A larger w(e) indicates that u and v are semantically more dissimilar. We measure the distance between two labels as the sum of the weights assigned to the edges along the shortest path P connecting them
de (.(()))))((()))))) = Ew(e).
(3)
PEP
Eq. (3) reduces to the standard graph hop distance between two vertices by setting w(e) = We design we) be inversely proportional the tree depth level of the parent node (e.g-, w(e) x 2 2 In other words, we prefer the shortest paths to traverse the root node (or nodes with smaller I) as a way of encouraging (x) and f, (z) to differ in a more general level (e.g.. vehicle rather than watercraft). Figure shows the semantic advantages of our choice of weighting compared with the equal weighting. With the distance measure at hand, the optimal image in terms discriminating fi and f, can be obtained by maximizing the discrepancy between the two classifiers on D
xx
= arg max dew (f((1)) fz(x)).
(4)
=ED
The queried image label f(**) leads to three possible outcomes (see Figure D
Case I. Both classifiers make correct predictions. Although theoretically impossible based the general problem formulation, it is not uncommon in practice that natural image may contain multiple distinet objects (e.g., guacamole and mortar). In this case, fr and Sj successfully recognize different objects in x*, indicating that both classifiers tend to perform at a high level. By restricting 'D to only contain natural images with a single salient object, we may reduce the possibility of this outcome.
Case II. fo (or f,) makes correct prediction, while fy (or fo makes incorrect prediction. In this case, MAD automatically identifies a strong failure case to falsify one classifier, not the other; a clear winner is obtained. The selected image x=' provides the strongest evidence in differentiating the two classifiers as well as ranking their relative performance.
Case III. Both classifiers make incorrect predictions in a multiclass image classification problem (i.e,& 2 3). Although both classifiers make mistakes, they differ substantially during inference, which in turn provides strong indication of their respective weaknesse Depending the subjective experimental setting, x* may be used to rank the classifiers based on Eq. (3) if the full label (x*) has been collected. As will be clear later, due to the difficulty in subjective testing,
collect partial label: "x" does not contain f((x=) nor f, ("" In this case, only Eq. | can be applied, and x* contributes less to performance comparison between the two classifiers.
"This is in stark contrast to natural adversarial examples in ImageNet-A Hendrycks et al. 2019). where different image classifiers tend to make consistent mistakes. For example, VGGT6BN and ResNet34 make the incorrect predictions on 1 149 out of 7, 500 images in ImageNet-A.
4

---

Published as a conference paper at ICLR 2020
Algorithm 1: The MAD competition
Algorithm 2: Adding a new classifier into the MAD competition
Input: An unlabeled image set D, group of image classifiers
Input:
An unlabeled image set D, the pairwise dominance matrix B E I for F = [f: i= new classifier fn+1 to be ranked, distance measure dw
F= { i= to be ranked,
distance measure d defined over WordNet hierarchy
Output: A global ranking vector
rE TER m
Output: A global ranking vector
S + 0, I for
r R R=++
+ 1 to m do
S
0
Compute classifier predictions x D}
B'
B 0"
0
E [[(++))((+1))
end
2 Compute the predictions of the new image classifier {fm+1 (r)), E D
for i +- to do m do Compute the distances using Eq.
for j ++
3 for <- 1 to m do
Compute the distances using Eq. (3)
D
{dw ff D} Select top-k images with k largest distances to form S(i)) S (-SS)
5 Select top-k images with & largest
distances to form S (1,m+1)
S = USUS(immt
7 end
end
8 Source human labels for S
end
9 for +- to m do
12 Source human labels for S
aim++1 Acc(fi: S (t.m+ 0m+11 Aco(S S (1.m+1))
Compute the pairwise accuracy matrix A
with 0.j = Acc(f(; S(t,)) using
end
Eq. [I
Update the pairwise dominance matrix B' with
14 Compute the pairwise dominance matrix B with byg d.j/aj;
b 1/ = fs.m+1/ami /mm+1.s Compute the global ranking vector E R=+1 using Eq. (5)
15 Compute the global ranking vector r using Eq. (5)
In practice, to obtain reliable performance comparison between fi and fj, we choose top-k images in D with k largest distances computed by Eq. (3) to form the test subset S(L,)- MAD runs this game among all (?2) distinct pairs classifiers, resulting in the final MAD test set S (S The number natural images S is at most m(m - 1)k/2, which is independent of the size n of D. In other words, applying MAD to a larger image set has no impact the cost of human labeling. In scenarios where the cost of computational prediction can be ignored, MAD encourages to expand D to cover many "free" natural images as possible.
We now describe our subjective assessment environment for collecting human labels. Given an image S, which is associated with classifiers fi and f5, pick two binary questions for human annotators: "Does x contain an fi (x))?" and "Does x contain an (z))"" When both answers no (corresponding Case III), we stop querying the ground-truth label of because it difficult for humans to select one among e classes, especially when e is large and the ontology of classes is complex.
After subjective testing, we first compare the classifiers in pairs and aggregate the pairwise statistics into a global ranking. Specifically, we compute the empirical classification accuracies of fü and fy on S(ij) using Eq. (T). denoted by a;y and ajs. respectively. When & is small, Laplace smoothing is employed to smooth the estimation. Note that 0:j + azi may be greater than one because of Case The pairwise accuracy statistics of all classifiers form matrix A, from which we compute another matrix B with byj = auj an/aa,. indicating the pairwise dominance of f, over fz- We aggregate the pairwise comparison results into a global ranking E R using Perron rank (Saaty & Vargas 1984)-
Ba1
lim
(5)
1TB"1'
5

---

Published as a conference paper at ICLR 2020
where 1 is an m-dimensional vector of all ones. The limit of Eq. (5) is the normalized principal eigenvector of B corresponding to the largest eigenvalue, where r, > 0 i -,... m and Σ, P2 = 1. The larger r; is, the better f. performs in the MAD competition. Other ranking aggregation methods such as HodgeRank (Jiang et al. 2011) may also be applied. We summarize the workflow
of the MAD competition in Algorithm
Finally, it is straightforward and cost-effective to add the m + 1-th classifier into the current MAD competition. No change is needed for the sampled S and the associated subjective testing. The additional work is to select a total of mk new images from 'D for human labeling. We then enlarge B by one along its row and column, and insert the pairwise comparison statistics between fm+1 and the previous m classifiers. An updated global ranking vector r E R can be computed using Eq. (5). We summarize the procedure of adding a new classifier in Algorithm
APPLICATION TO IMAGENET CLASSIFIERS
In this section, apply the proposed MAD competition methodology to comparing ImageNet classifiers. We focus on ImageNet (Deng et al. 2009) for two reasons. First, it is one of the first large-scale and widely used datasets image classification. Second, the improvements ImageNet seem to plateau, which provides an ideal platform for MAD to distinguish the newly proposed image classifiers finer.
3.1 EXPERIMENTAL SETUPS
Constructing D Inspired by Hendrycks et al. (2019), we focus the same 200 out of 1, 000 classes to avoid rare and abstract ones, and classes that have changed much since 2012. For each class, we crawl a large number of images from Flickr, resulting images. Although MAD allows us to arbitrarily increase n with essentially no cost, we choose the size of D to be approximately three times larger than the ImageNet validation set to provide a relatively easy enviromment for probing the generalizability of the classifiers. As will be clear in Section 3.2] the current setting of n is sufficient to discriminate the competing classifiers. To guarantee the content independence between ImageNet and D, we collect images that have been uploaded after 2013. It worth noting that no data cleaning (e.g., inappropriate content and near- duplicate removal) is necessary at this stage since only need to ensure the selected subset S for human labeling is eligible.
total of 168, 000 natural
Competing Classifiers We select eleven representative ImageNet classifiers for benchmark-
ing: VGG16BN (Simonyan & Zisserman| (014)) with batch normalization (loffe & Szegedy)
2015), ResNet34, ResNet101 (He et al. 2016), WRN101-2 (Zagoruyko & Komodakis (016))
CesNext101-32xx (Xic et al.) 2017). SE-ResNet-101. SENet154 (Hu et al.] 2018), NASNet-A-
Large (Zoph et al. 2018), P-ASNet---Large (Liu et al. 2018), EfficientNet-B7 (Tan & Le| 2019). and WeL-
(Mahajan al. 2018). Since VGG16BN and ResNet34 have nearly identical accuracies on the ImageNet validation set, it is particularly interesting to see which method generalizes better to natural image manifold. We compare ResNet34 with ResNet101 to see the in- fluence of DNN depth. WRN101-2, ResNeXt101-3224 SE-ResNet-101 101 are different improved versions over ResNet-101. We also include two state-of-the-arr classifiers: W-L-ResNeXt1101- 32x48 and EfficientNet-B7 on Instagram data, while the latter makes use of compound scaling. We use publicly available code repositories for all DNN-based models, whose top-1 accuracies on the ImageNet validation set are listed in Table for reference.
The former leverages the power of weakly supervised pre-training
When constructing S using the maximum discrepancy principle, we add another constraint based on prediction confidence. Specifically, candidate image x associated with fi and f, filtered out if does not satisfy min(p;(r), P3 (2)) 2 T, where pa(x)) is the confidence score (i..., probability produced by the last softmax layer) of fi(x) and T is predefined threshold set to 0.8. We include the confidence constraint for two main reasons. First, if fa misclassifies x with low confidence, it is highly likely that is near the decision boundary and thus contains less information on improving the decision rules of f.. Second, some images D do not necessarily fall into the 1, 000 classes in ImageNet, which are bound to be misclassified (a problem closely related to out- of-distribution detection). If they are misclassified by f, with high confidence, we consider them as hard counterexamples of ft. To encourage class diversity in S, retain maximum of three images
Constructing

---

Published as a conference paper at ICLR 2020
34:
34s
3e:
Dutch oven EfficientNNet-BB manhole cover
spider web
mailbox, letter box EfficientNet-BB8 manhole cover
EfficientNet-877 manhale cover
ResNet101:
ResNet101:
ResNet1011:
sundial
doormat
sundial
NAsNet-A-Lorge:
NASNet-A-Lorge- manhole cover
NASNet-A-Lorge: barbell
manhole cover
(a)
(b)
Figure 4: Visual comparison of images selected by MAD and in the ImageNet validation set. (a): "manhole cover" images selected by MAD along with the predictions by the associated classifiers. (b): All "manhole cover" images in the ImageNet validation set. The MAD-selected images are visually much harder, which contain diverse and non-trivial distortions, e.g., occlusion, shading and unbalanced lightening, complex background, rare colors, and untypical viewing points. In contrast, ImageNet images mainly include single center-positionss object with a relatively clean back- ground, whose shape, color and viewing point are common.
with the same predicted label by ft. In addition, we exclude images that are non-natural. Figure visually compares representative "manhole cover" images in S and the ImageNet validation set (see more in Figure 6)
VGG16BN nan 0.57 0.1
0.37 0:33
02
02
02
0:3
Collecting Human Labels As described in Section 2.1 given an image S, human an- notators need to answer two binary questions. In our subjective experiments, we choose = 30 and invite five volunteer graduate students, who are experts computer vision, to label total of 11 x 10 x 30/2 1, 650 images. If more than three of them find difficulty in la- beling (associated with fi and fj), it is dis- carded and replaced by x' E D with the k + 1- th largest distance d (ff( (z'")) Majority vote is adopted to decide the final label when disagreement occurs. After subjective testing, we find that 53.5% of annotated images belong Case II, which form the cornerstone of the subsequent data analysis. Besides, 32.9% and 13.6% images pertain to Case and Case III, respectively.
ResNer34
03
ResNec101 0.43
0.6 nan 001 033
01
04
031
WAN101-2 0.73
0.7
0.4
nen
0.4
0.63 0.53
020
0.33
0.43
011
ResNext101-32xx 0.63
0.53 0.53
0.33
0327 02
0.17
SE- RE-ReNNet-101-- 0.67
SCNet154 0.47
0.67
0.57
0.67
0.5
0.6
pan
0:33
0.37
0.4
017
LASNet-A-Large 0.4 0.63
0.57
0.
0.43 0.5
nan
02
0.17
INASSLarge
0.67 0.7
0.4 0.4
0:38
0:13
WSL
0.53
nan
EfficienNNe-077 0.7
0.77
0.73
0.63
0.6
0.51
0.8
0.53
0.63
0.37 nan
VOG168N
RecNet34
Reslet101
WRN10 WRN101-2 -1-2
32x4
SE Beseeeet-10
SENe1154
NASRet-AL A-Lagge
Pagg WSL
atficientee-81
Resteexro1
accuracy matrix with brighter colors indicating higher accuracies (blue numbers). WSL--ResNeXt101-33 -32 abbrevi- ated to WSL for neat presentation.
Figure 3: Pairwise
3.2 EXPERIMENTAL RESULTS
Pairwise Ranking Results Figure shows the pairwise accuracy matrix A in the current MAD competition, where larger value of entry (a brighter color) indicates higher accuracy in the subset selected together by the corresponding row and column models. An interesting phenomenon we observe is that when two classifiers f; and fy perform at similar level on S(1,) (i.e., AAc(ff)- Acc(f,) is small), max(Ace(fi) Ac)(f,)) is also small. That is, more images on which they both make incorrect but different predictions (Case III) have been selected compared with images falling into Case I. Taking a closer look images in ((,))) may reveal the respective model biases of f;
7

---

Published as a conference paper at ICLR 2020
Models
ImageNet top-1 Ace
Ace Rank
MAD Rank
A Rank
WSL--ResNXXr101-2222 x48 (Mahajan et al. 2018
85,44
EfficientNet-B7 (Tan & Le 2019
84.48
2
PNASNet-S-Large (Liu 2018
82,74
3
-4
NASNet-A-Large (Zoph et 2018
82.51
4
0
SENct154 (Hu et al. 2018)
81.30
5
2
WRN101-2 (Zagoruy & Komodakis 2016
78.85
0
SE-ResNet- T0T (Hu of 2018
78.40
2
ResNext101-32.:4 (01e [Xie al. 2017
78.19
8
0
ResNet101 (He al.7 2016
77.37
VGG16BN Sumonyan & Zisserman 2014
73.36
ResNet34 He al. 2016
73.31
Table 1: Global ranking results. A smaller rank indicates better performance.
and fj. For example, we find that WSL-ResNext100--22x02 tends to focus on foreground objects, while EfficientNet-B7 attends more to background objects (see Figure (7). We also find several
common failure modes of the competing classifiers through pairwise comparison, e.g., excessive reliance on relation inference (see Figure 8), bias towards low-level visual features (see Figure and difficulty in recognizing rare instantiations of objects (see Figures 4 andd 6).
Global Ranking Results We present the global ranking results by MAD in Table where we find that MAD tracks the steady progress in image classification, verified by reasonable Spearman rank-order correlation coefficient (SRCC) of 0.89 between the accu- racy rank on the ImageNet validation set and the MAD rank on our test set Moreover, by looking at the differences between the two rankings, we obtain a number of interesting findings. First, VGGI6BN outperforms not only ResNet34 but also ResNet101, suggesting that under similar computation budgets, VGG-like networks may exhibit better generalizability to hard samples than networks with residual connections. Second, both networks equipped with the suuezze-and-extractio mechanism, i.e., SE-ResNNet1001 and SENet1 54, up by two places in the MAD ranking. This indicates that explicitly modeling dependencies between channel-wise fea- ture maps seems quite beneficial image classification. Third, for the models that exploit neural architecture search, NASNet-A-Large is still ranked high by MAD; interestingly, the rank of PNASNet-5-Large drops lot. This implies MAD may prefer the global search strategy used in NASNet-A-Laue Larage to the progressive cell-wise search strategy adopted in PNASNet-5-Lang though the former is slightly inferior in ImageNet top-1 accuracy. Last but not least, the top-2 performers, WSI--ReNNext10-333
1.0
0.8
0.6
Un 0.4
0.2
0.0
10
15
20
25
30
Figure 5: The SRCC values between top-30 and other top-k rankings, k (1,
al-
and EfficientNet-77, are still the best in MAD competition (irrespective of their relative rankings), verifying the effectiveness of large-scale hashtag data pre- training and compound scaling in the context of image classification.
Ablation Study We analyze the key hyperparameter in MAD, í.e., the number of images in S(.J) selected for subjective testing. We calculate the SRCC values between the top-30 ranking (as reference) and other top-k rankings with & 21,2,.. 29}. As shown in Figure 5 the ranking results are fairly stable (SRCC > 0.90) when l: > 15. This supports our choice of 1: 30 since the final global ranking already seems enter stable plateau.
DISCUSSION
We have presented new methodology, MAD competition, for comparing image classification mod- els. MAD effectively mitigates the conflict between the prohibitively large natural image manifold that we have to evaluate against and the expensive human labeling effort that aim to minimize. Much of our endeavor has been dedicated to selecting natural images that are optimal in terms of distinguishing or falsifying classifiers. MAD requires explicit specification of image classifiers to
8

---

Published as a conference paper at ICLR 2020
be compared, and provides an effective means of exposing the respective flaws of competing clas- sifiers. It also directly contributes to model interpretability and helps us analyze the models' focus and bias when making predictions. We have demonstrated the effectiveness of MAD competition in comparing ImageNet classifiers, and concluded a number of interesting observations, which were not apparently drawn from the (often quite close) accuracy numbers on the ImageNet validation set.
The application scope of MAD is far beyond image classification. It can be applied to computational models that produce discrete-valued outputs, and is particularly useful when the sample space is large and the ground-truth label being predicted is expensive to measure, Examples include medical and hyperspectral image classification (Filipovych & Davatzikos 2011, Wang et al., 2014), where signification domain expertise is crucial to obtain correct labels. MAD can also be used to spot rare but fatal failures in high-cost and failure-sensitive applications, e.g., comparing perception systems of autonomous cars (Chen et al., 2015) in unconstrained real-world weathers, lighting conditions, and road scenes. addition, by restricting the test set 'D to some domain of interest, MAD allows comparison of classifiers in more specific applications, e.g., ine-grained image recognition.
We feel it important to note the limitations of the current MAD. First, MAD aims at relatively com- paring models, and cannot give an absolute performance measure. Second, an "error spotting" mechanism, MAD implicitly assumes that models in the competition are reasonably good (e.g., Ima- geNet classifiers); otherwise, the selected counterexamples may be less meaningful. Third, although the distance in Eq. (3) is sufficient to distinguish multiple classifiers in the current experimental set- ting, it does not yet fully reflect human cognition of image label semantics. Fourth, the confidence computation used to select images is not perfectly grounded. How to marry the MAD competition with Bayesian probability theory to model uncertainties during image selection an interesting di- rection for future research. Due to the above issues, MAD should be viewed as complementary to, rather than a replacement for, the conventional accuracy comparison for image classification.
Our method arises as natural combination of concepts drawn from two separate lines of research. The first explores the idea of model falsification as model comparison. [Wang & Simoncelli (2008)
introduced the maximum differentiation competition for comparing computational models of contin-
uous perceptual quantities, which was further extended by Ma et al. (2019). |Berardino et al. (2017)
developed computational method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans. MAD, on the other hand, is tailored to applications with discrete model responses and relies on semantic distance measure to compute model discrepancy. The second endeavour arises from machine learning literature on generating
adversarial examples (Szegedy al. 2013. Goodfellow et al. 2015; Madry et al.) 2018) and eval- uating image classifiers on new test sets (Geirhos et al.) 2019) Recht et al. 2019; Hendrycks & Dietterich] 2019) [Hendrycks et al., 2019). The images selected by MAD can be seen as form of natural adversarial examples as each of them is able to fool at least one classifier (when Case I eliminated). Unlike adversarial images with inherent transferability MAD- images emphasize on their discriminability of the competing models. Different from recently created test sets, the MAD-selected set is adapted to the competing classifiers with the goal of minimizing human labeling effort. In addition, MAD can also be linked to the popular technique of differential testing (McKeeman 1998) software engineering.
mislead most classifiers,
REFERENCES
Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Eigen-distortions of hierarchical representations. In Advances in Neural Information Processing Systems, pp. 3530- 3539. 2017.
Kenneth P Burnham and David R Anderson. Model Selection and Multimodel Inference: A Practical CepommtiionThhoreeti Approach. Springer Science & Business Media, 2003.
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. DeepDriving: Learning affordance for direct perception in autonomous driving. In IEEE International Conference on Computer Vision, pp. 2722-22222222 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 245,255,220 2009.

---

Published as a conference paper at ICLR 2020
Roman Filipovych and Christos Davatzikos. Semi-supervised pattern classification of medical im- ages: Application to mild cognitive impairment (MCI). Neurolmage, (533):110--1133 2011.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174. 2019.
Jie Hu, Shen, and Gang Sun. Squeeze-and-ex-eee Vision and Pattern Recognition, pp. 7132-71411 2018.
networks. IEEE Conference on Computer
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye. Statistical ranking and combinatorial hodge theory. Mathematical Programming, 127(1):203-244 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In European Conference on Computer Vision, pp. 19-34, 2018.
Kede Ma, Zhengfang Duanmu, Zhou Wang. Qingho Wu, Wentao Liu, Hongwei Yong, Hongliang Li, and Lei Zhang. Group maximum differentiation competition: Model comparison with few samples. IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear, 2019.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits weakly supervised pretraining. In European Conference Computer Vision, pp. 181-196, 2018.
William M McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107 1998.
George A Miller. WordNet: An Electronic Lexical Database. MIT Press, 1998.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? arXiv preprint arXiv: 1902, 1081 2019.
10

---

Published as a conference paper at ICLR 2020
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115 (3):211-252, 2015.
Thomas L Saaty and Luis G Vargas. Inconsistency and rank preservation. Journal of Mathematical Psychology, 28(2):205-214 1984.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2013.
Mingxing Tan and Quoc V Le. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint
2019.
Zhangyang Wang, Nasser M Nasrabadi, and Thomas S Huang. Semisupervised hyperspectral clas- sification using task-driven dictionary learning with Laplacian regularization. IEEE Transactions Geoscience and Remote Sensing, 53(3):1161-1173 2014.
Zhou Wang and Eero Simoncelli. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. Journal of Vision, 8(12):1-13, 2008
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 1492-1500, 2017.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018.

---

Published as a conference paper at ICLR 2020
A
MORE VISUALIZATION RESULTS
Due to the page limit, we put some additional figures here.
VGG168N: sea slug
WSL: broccoli
ResNext1101322+4 broccoli
EfficientNet-B plate
Cocaa
VG6168N: rugby ball
VGG16BN: nugby
EfficientNet- B7:
soccer ball
EfficientNet B7: soccer ball
(o)
Figure 6: Visual comparison of "broccoli" and "soccer ball" images (a) selected by MAD and (b) in the ImageNet validation set.
WSL: broccoli
WSL: broccoli
WSL: broccoli
WSL: cauliflower
EfficientNet B7: hot pot
EfficientNet-B1 B7: plate
EfficientNet-B8 BB: plate
EfficientNet B7: plate
(a)
(b)
(c)
(d)
Figure 7: Examples of network bias. WSL-ResNex1101-22244 (WSL) tends to focus on foreground objects, while EfficientNet-Bi B7 attends more to background objects.
12

---

Published as a conference paper at ICLR 2020
ResNet101: snorkel
V6616BN: basketball
WRN-101-2: toaster
WRN-101-2: disk brake
23
(a)
(b)
(c)
(d)
Figure 8: Examples of relation inference. (a): Snorkel is correlated underwater environment. (b): Basketball is correlated to basketball court. (c): Toaster is correlated to toasted bread. (d): Disk brake correlated freewheel and spokes. Similar with how humans recognize objects, would be reasonable for DNN-based classifiers to make predictions by inferring useful information from object relationships, only when their prediction confidence is low. However, this the case in our experiments, which show that classifiers may make high-confidence predictions by leveraging object relations without really "seeing" the predicted object.
ResNer101: cardoon
ResNet101: nail
ResNeXt101: geyser
ResNet34: swimming cap
(a)
(c)
(d)
Figure 9: Examples of network bias to low-level visual features, such as color, shape and texture, while overlooking conflicting semantic cues. An ideal classifier is expected to utilize both low-level (appearance) and high-level (semantic) features
when making predictions. ResNeX:101-30 is abbreviated to ResNeXt101 for neat presentation.
13


Reviews:
[{"review_id": "r1xYhv5J5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out that the traditional way of model selection is flawed due to that the validation/test set is often small. The authors also attribute the existence of adversarial examples to the small validation/test set, which I agree to some degree. Hence, the authors proposed an alternative approach to comparing different classification models by the notion of inter-model discrepancy. \n\nThe main idea is reasonable, but it requires that the models to compare all perform reasonably well. Otherwise, some poorly performed models could lead to near-random or adversarial inter-model discrepancies, failing the proposed approach. \n\nAnother potential issue is that the proposed approach cannot handle training set bias. If all models are biased in similar ways (e.g., toward a particular class or domain), they will not reveal informative discrepancies for the images over which they all make similar mistakes. \n\nAnother question which is not answered in the paper is the number $k$ of images to select for each pair of classifiers. Is this number task-dependent? Is it related to the number of classes? What is a general guideline for one to choose this number $k$ given a new application scenario? \n\nThe unlabeled set is not \"unlabeled\" in essence. If my understanding was correct, it cannot contain open-set images which do not belong to any of the classes of interest. It is also nontrivial to control that the images contain only one salient object per image.\n\nHence, while I agree with the authors that existing approaches to comparing deep neural network classifiers could be improved, I think the proposed solution is not a good alternative yet. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgKMIsTFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel image classifier comparison approach that went beyond one fixed testing set for all. Instead, for a pair of classifiers to be compared, it advocated to sample their \"most disagreed\" test set from a large corpus of unlabeled images. The level of disagreement was measured by a semantic-aware distance derived from WordNet ontology. Because of the efficacy of such \"worst-case\" comparison, the needed set size is very small and thus minimizes the human annotation workload. \n\nThe proposed MAD competition distinguishes classifiers by finding their respective counterexamples. It is therefore an \"error spotting\" mechanism, rather than a drop-in replacement of standard test accuracy. I feel the approach to implicitly assume that the classifiers to be compared are already \"reasonably accurate\"; since if not, both classifiers might be easily falsified by certain trivial examples, making the \"disagreed examples\" not as meaningful. If that is true, I would suggest the authors to make this hidden assumption clearer in the paper\n\nThe idea shows clear liaison to the \"differential testing\" concept in software engineering besides the cited work of perceptual quality assessment. The idea has a cross-disciplinary nature and is fairly interesting to me. I can see the paper to be of interest to a quite broad audience and can motivate many subsequent works. \n\nOne minor comment: for images in \"Case III\", the authors considered them \"contribute little to performance comparison between the two classifiers\" and therefore did not source labels for them. However, since the authors adopted an affinity-aware distance, two incorrect predictions can still be compared based on their semantic tree distances to the true class.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxHV7JPjB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method to compare existing classifiers, which does not use fixed test set and adaptively sample it from an arbitrarily large corpus of unlabeled images. The main idea seems similar to adopting active learning for the test set selection.\n \nOne of the main advantage is that it can select a sample set from an arbitrarily large unlabeled images. However, to compare different classifiers, the proposed algorithm still needs humans to annotate the selected dataset, which is very expensive compared with traditional methods.  \n\nSince this paper select the top-k images in D, if k is large the annotating for S will be very tedious, however if k is relatively small the method seems very sensitive to selected examples, which will make the comparison not totally convincing.\n \nThe authors invite five volunteer graduate students to annotate the selected example. However, for many categories, it’s nor easy for normal people to distinguish. So the experiments in this paper is also not convincing. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new way of comparing classifiers, which does not use fixed test set and adaptively sample it from an arbitrarily large corpus of unlabeled images, i.e. replacing the conventional test-set-based evaluation methods with a more flexible mechanism. The main proposal is to build a test set adaptively in a manner that captures how classifiers disagree, as measured by the wordnet tree. As noted by R2, this work has the potential to be of interest to a broad audience and can motivate many subsequent works. 

While the reviewers acknowledged the importance of this work, they raised several concerns: (1) the proposed approach is immature to be considered for benchmarking yet (R1,R4), (2) selecting k and studying its influence on the performance ( R1, R3, R4), (3) the proposed approach requires data annotation which might not be straightforward -- (R3, R4).  The authors provided a detailed rebuttal addressing the reviewer concerns.

There is reviewer disagreement on this paper. The comments from R3 were valuable for the discussion, but at the same time too brief to be adequately addressed by the authors. The comments from emergency reviewer were helpful in making the decision. AC decided to recommend acceptance of the paper seeing its valuable contributions towards re-thinking the evaluation of current SOTA models.

Decision: Accept (Poster)