I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively

1 INTRODUCTION . Large-scale human-labeled image datasets such as ImageNet ( Deng et al. , 2009 ) have greatly contributed to the rapid progress of research in image classification . In recent years , considerable effort has been put into designing novel network architectures ( He et al. , 2016 ; Hu et al. , 2018 ) and advanced optimization algorithms ( Kingma & Ba , 2015 ) to improve the training of image classifiers based on deep neural networks ( DNNs ) , while little attention has been paid to comprehensive and fair evaluation/comparison of their model performance . Conventional model evaluation methodology for image classification generally follows a three-step approach ( Burnham & Anderson , 2003 ) . First , pre-select a number of images from the space of all possible natural images ( i.e. , natural image manifold ) to form the test set . Second , collect the human label for each image in the test set to identify its ground-truth category . Third , rank the competing classifiers according to their goodness of fit ( e.g. , accuracy ) on the test set ; the one with the best result is declared the winner . A significant problem with this methodology is the apparent contradiction between the enormous size and high dimensionality of natural image manifold and the limited scale of affordable testing ( i.e. , human labeling , or verifying predicted labels , which is expensive and time consuming ) . As a result , a typical “ large-scale ” test set for image classification allows for tens of thousands of natural images to be examined , which are deemed to be extremely sparsely distributed in natural image manifold . Model comparison based on a limited number of samples assumes that they are sufficiently representative of the whole population , an assumption that has been proven to be doubtful in image classification . Specifically , Recht et al . ( 2019 ) found that a minute natural distribution shift leads to a large drop in accuracy for a broad range of image classifiers on both CIFAR-10 ( Krizhevsky , 2009 ) and ImageNet ( Deng et al. , 2009 ) , suggesting that the current test sets may be far from sufficient to represent hard natural images encountered in the real world . Another problem with the conventional model comparison methodology is that the test sets are pre-selected and therefore fixed . This leaves the door open for adapting classifiers to the test images , deliberately or unintentionally , via extensive hyperparameter tuning , raising the risk of overfitting . As a result , it is never guaranteed that image classifiers with highly competitive performance on such small and fixed test sets can generalize to real-world natural images with much richer content variations . In order to reliably measure the progress in image classification and to fairly test the generalizability of existing classifiers in a natural setting , we believe that it is necessary to compare the classifiers on a much larger image collection in the order of millions or even billions . Apparently , the main challenge here is how to exploit such a large-scale test set under the constraint of very limited budgets for human labeling , knowing that collecting ground-truth labels for all images is extremely difficult , if not impossible . In this work , we propose an efficient and practical methodology , namely the MAximum Discrepancy ( MAD ) competition , to meet this challenge . Inspired by Wang & Simoncelli ( 2008 ) and Ma et al . ( 2019 ) , instead of trying to prove an image classifier to be correct using a small and fixed test set , MAD starts with a large-scale unlabeled image set , and attempts to falsify a classifier by finding a set of images , whose predictions are in strong disagreement with the rest competing classifiers ( see Figure 1 ) . A classifier that is harder to be falsified in MAD is considered better . The initial image set for MAD to explore can be made arbitrarily large provided that the cost of computational prediction for all competing classifiers is cheap . To quantify the discrepancy between two classifiers on one image , we propose a weighted distance over WordNet hierarchy ( Miller , 1998 ) , which is more semantically aligned with human cognition compared with traditional binary judgment ( agree vs. disagree ) . The set of model-dependent images selected by MAD are the most informative in discriminating the competing classifiers . Subjective experiments on the MAD test set reveal the relative strengths and weaknesses among the classifiers , and identify the training techniques and architecture choices that improve the generalizability to natural image manifold . This suggests potential ways to improve a classifier or to combine aspects of multiple classifiers . We apply the MAD competition to compare eleven ImageNet classifiers , and find that MAD verifies the relative improvements achieved by recent DNN-based methods , with a minimal subjective testing budget . MAD is readily extensible , allowing future classifiers to be added into the competition with little additional cost . 2 THE MAD COMPETITION METHODOLOGY . The general problem of model comparison in image classification may be formulated as follows . We work with the natural image manifold X , upon which we define a class label f ( x ) ∈ Y for every x ∈ X , where Y = { 1 , 2 , . . . , c } and c is the number of categories . We assume a subjective assessment environment , in which a human subject can identify the category membership for any natural image x among all possible categories . A group of image classifiers F = { fi } mi=1 are also assumed , each of which takes a natural image x as input and makes a prediction of f ( x ) , collectively denoted by { fi ( x ) } mi=1 . The goal is to compare the relative performance of m classifiers under very limited resource for subjective testing . The conventional model comparison method for image classification first samples a natural image set D = { xk } nk=1 ⊂ X . For each image xk ∈ D , we ask human annotators to provide the groundtruth label f ( xk ) ∈ Y . Since human labeling is expensive and time consuming , and DNN-based classifiers are hungry for labeled data in the training stage ( Krizhevsky et al. , 2012 ) , n is typically small in the order of tens of thousands ( Russakovsky et al. , 2015 ) . The predictions of the classifiers are compared against the human labels by computing the empirical classification accuracy Acc ( fi ; D ) = 1 |D| ∑ x∈D I [ fi ( x ) = f ( x ) ] , for i = 1 , . . . , m. ( 1 ) The classifier fi with a higher classification accuracy is said to outperform fj with a lower accuracy . As an alternative , the proposed MAD competition methodology aims to falsify a classifier in the most efficient way with the help of other competing classifiers . A classifier that is more likely to be falsified is considered worse . 2.1 THE MAD COMPETITION PROCEDURE . The MAD competition methodology starts by sampling an image setD = { xk } nk=1 from the natural image manifold X . Since the number of images selected by MAD for subjective testing is independent of the size of D , we may choose n to be arbitrarily large such that D provides dense coverage of ( i.e. , sufficiently represents ) X . MAD relies on a distance measure to quantify the degree of discrepancy between the predictions of any two classifiers . The most straightforward measure is the 0-1 loss : d01 ( fi ( x ) , fj ( x ) ) = I [ fi ( x ) 6= fj ( x ) ] . ( 2 ) Unfortunately , it ignores the semantic relations between class labels , which may be crucial in distinguishing two classifiers , especially when they share similar design philosophies ( e.g. , using DNNs as backbones ) and are trained on the same image set ( e.g. , ImageNet ) . For example , misclassifying a “ chihuahua ” as a dog of other species is clearly more acceptable compared with misclassifying it as a “ watermelon ” . We propose to leverage the semantic hierarchy in WordNet ( Miller , 1998 ) to measure the distance between two ( predicted ) class labels . Specifically , we model WordNet as a weighted undirected graph G ( V , E ) 1 . Each edge e = ( u , v ) ∈ E connects a parent node u of a more general level ( e.g. , canine ) to its child node v of a more specific level ( e.g. , dog ) , for u , v ∈ V . A nonnegative weight w ( e ) is assigned to each edge e = ( u , v ) to encode the semantic distance 1Although WordNet is tree-structured , a child node in WordNet may have multiple parent nodes . church , church building fountain drakeAmerican coot , marsh hen structure , construction aquatic bird .. . . . . ... ... . . . ... entity Figure 2 : Comparison of weighted and unweighted distances . In the sub-tree of WordNet , we highlight the shortest paths from “ fountain ” to “ church ” and from “ drake ” to “ American coot ” in green and red , respectively . The semantic distance between the two aquatic birds is much shorter than that between the two constructions of completely different functionalities ( verified by our internal subjective testing ) . The proposed weighted distance is well aligned with human cognition by assigning a much smaller distance to the red path ( 0.0037 ) compared with the green one ( 0.0859 ) . between u and v. A larger w ( e ) indicates that u and v are semantically more dissimilar . We measure the distance between two labels as the sum of the weights assigned to the edges along the shortest path P connecting them dw ( fi ( x ) , fj ( x ) ) = ∑ e∈P w ( e ) . ( 3 ) Eq . ( 3 ) reduces to the standard graph hop distance between two vertices by setting w ( e ) = 1 . We design w ( e ) to be inversely proportional to the tree depth level l of the parent node ( e.g. , w ( e ) ∝ 2−l ) . In other words , we prefer the shortest paths to traverse the root node ( or nodes with smaller l ) as a way of encouraging fi ( x ) and fj ( x ) to differ in a more general level ( e.g. , vehicle rather than watercraft ) . Figure 2 shows the semantic advantages of our choice of weighting compared with the equal weighting . With the distance measure at hand , the optimal image in terms of discriminating fi and fj can be obtained by maximizing the discrepancy between the two classifiers on D x ? = arg max x∈D dw ( fi ( x ) , fj ( x ) ) . ( 4 ) The queried image label f ( x ? ) leads to three possible outcomes ( see Figure 1 ) : • Case I . Both classifiers make correct predictions . Although theoretically impossible based on the general problem formulation , it is not uncommon in practice that a natural image may contain multiple distinct objects ( e.g. , guacamole and mortar ) . In this case , fi and fj successfully recognize different objects in x ? , indicating that both classifiers tend to perform at a high level . By restricting D to only contain natural images with a single salient object , we may reduce the possibility of this outcome . • Case II . fi ( or fj ) makes correct prediction , while fj ( or fi ) makes incorrect prediction . In this case , MAD automatically identifies a strong failure case to falsify one classifier , not the other ; a clear winner is obtained . The selected image x ? provides the strongest evidence in differentiating the two classifiers as well as ranking their relative performance . • Case III . Both classifiers make incorrect predictions in a multiclass image classification problem ( i.e. , c ≥ 3 ) . Although both classifiers make mistakes , they differ substantially during inference , which in turn provides a strong indication of their respective weaknesses2 . Depending on the subjective experimental setting , x ? may be used to rank the classifiers based on Eq . ( 3 ) if the full label f ( x ? ) has been collected . As will be clear later , due to the difficulty in subjective testing , we collect a partial label : “ x ? does not contain fi ( x ? ) nor fj ( x ? ) ” . In this case , only Eq . ( 1 ) can be applied , and x ? contributes less to performance comparison between the two classifiers . 2This is in stark contrast to natural adversarial examples in ImageNet-A ( Hendrycks et al. , 2019 ) , where different image classifiers tend to make consistent mistakes . For example , VGG16BN and ResNet34 make the same incorrect predictions on 3 , 149 out of 7 , 500 images in ImageNet-A . Algorithm 1 : The MAD competition Input : An unlabeled image set D , a group of image classifiers F = { fi } mi=1 to be ranked , a distance measure dw defined over WordNet hierarchy Output : A global ranking vector r ∈ Rm 1 S ← ∅ , B ← I 2 for i← 1 to m do 3 Compute classifier predictions { fi ( x ) , x ∈ D } 4 end 5 for i← 1 to m do 6 for j ← i + 1 to m do 7 Compute the distances using Eq . ( 3 ) { dw ( fi ( x ) , fj ( x ) ) , x ∈ D } 8 Select top-k images with k largest distances to form S { i , j } 9 S ← S ⋃ S { i , j } 10 end 11 end 12 Source human labels for S 13 Compute the pairwise accuracy matrix A with aij = Acc ( fi ; S { i , j } ) using Eq . ( 1 ) 14 Compute the pairwise dominance matrix B with bij = aij/aji 15 Compute the global ranking vector r using Eq . ( 5 ) Algorithm 2 : Adding a new classifier into the MAD competition Input : An unlabeled image set D , the pairwise dominance matrix B ∈ Rm×m for F = { fi } mi=1 , a new classifier fm+1 to be ranked , distance measure dw Output : A global ranking vector r ∈ Rm+1 1 S ← ∅ , B′ ← [ B 0 0T 1 ] ∈ R ( m+1 ) × ( m+1 ) 2 Compute the predictions of the new image classifier { fm+1 ( x ) , x ∈ D } 3 for i← 1 to m do 4 Compute the distances using Eq . ( 3 ) { dw ( fi ( x ) , fm+1 ( x ) ) , x ∈ D } 5 Select top-k images with k largest distances to form S { i , m+1 } 6 S = S ⋃ S { i , m+1 } 7 end 8 Source human labels for S 9 for i← 1 to m do 10 ai , m+1 = Acc ( fi ; S { i , m+1 } ) am+1 , i = Acc ( fm+1 ; S { i , m+1 } ) 11 end 12 Update the pairwise dominance matrix B′ with b′i , m+1 = 1/b ′ m+1 , i = ai , m+1/am+1 , i 13 Compute the global ranking vector r ∈ Rm+1 using Eq . ( 5 ) In practice , to obtain reliable performance comparison between fi and fj , we choose top-k images in D with k largest distances computed by Eq . ( 3 ) to form the test subset S { i , j } . MAD runs this game among all ( m 2 ) distinct pairs of classifiers , resulting in the final MAD test set S = ⋃ S { i , j } . The number of natural images in S is at most m ( m − 1 ) k/2 , which is independent of the size n of D. In other words , applying MAD to a larger image set has no impact on the cost of human labeling . In scenarios where the cost of computational prediction can be ignored , MAD encourages to expand D to cover as many “ free ” natural images as possible . We now describe our subjective assessment environment for collecting human labels . Given an image x ∈ S , which is associated with two classifiers fi and fj , we pick two binary questions for human annotators : “ Does x contain an fi ( x ) ? ” and “ Does x contain an fj ( x ) ? ” . When both answers are no ( corresponding to Case III ) , we stop querying the ground-truth label of x because it is difficult for humans to select one among c classes , especially when c is large and the ontology of classes is complex . After subjective testing , we first compare the classifiers in pairs and aggregate the pairwise statistics into a global ranking . Specifically , we compute the empirical classification accuracies of fi and fj on S { i , j } using Eq . ( 1 ) , denoted by aij and aji , respectively . When k is small , Laplace smoothing is employed to smooth the estimation . Note that aij + aji may be greater than one because of Case I . The pairwise accuracy statistics of all classifiers form a matrix A , from which we compute another matrix B with bij = aij/aji , indicating the pairwise dominance of fi over fj . We aggregate the pairwise comparison results into a global ranking r ∈ Rm using Perron rank ( Saaty & Vargas , 1984 ) : r = lim t→∞ 1 t t∑ α=1 Bα1 1TBα1 , ( 5 ) where 1 is an m-dimensional vector of all ones . The limit of Eq . ( 5 ) is the normalized principal eigenvector of B corresponding to the largest eigenvalue , where ri > 0 for i = 1 , . . .m and ∑ i ri = 1 . The larger ri is , the better fi performs in the MAD competition . Other ranking aggregation methods such as HodgeRank ( Jiang et al. , 2011 ) may also be applied . We summarize the workflow of the MAD competition in Algorithm 1 . Finally , it is straightforward and cost-effective to add the m + 1-th classifier into the current MAD competition . No change is needed for the sampled S and the associated subjective testing . The additional work is to select a total of mk new images from D for human labeling . We then enlarge B by one along its row and column , and insert the pairwise comparison statistics between fm+1 and the previous m classifiers . An updated global ranking vector r ∈ Rm+1 can be computed using Eq . ( 5 ) . We summarize the procedure of adding a new classifier in Algorithm 2 .

Summary: The paper proposed a novel image classifier comparison approach that went beyond one fixed testing set for all. Instead, for a pair of classifiers to be compared, it advocated to sample their "most disagreed" test set from a large corpus of unlabeled images. The level of disagreement was measured by a semantic-aware distance derived from WordNet ontology. Because of the efficacy of such "worst-case" comparison, the needed set size is very small and thus minimizes the human annotation workload.