SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Relative portion per group.
Figure 20: Top-1 errors caused by missing multi-label annotations. Surprisingly, the portion of these errors is the highest on the "others" group. In absolute terms, "organisms" and "others" result in roughly the same constant amount of errors, despite the much larger number of organism classes (410 vs. 68). For "artifacts", it first increases and then starts dropping, possibly due to more accurate models learning to leverage IMAGENET labeling biases.
(b) Abs. number per group.
Figure 21: Multi-label errors due to fine-grained misclassifications. The portion of fine-grained errors on "others" is larger than those on "artifacts" but smaller than "organisms".
(a) Relative portion per group.
(c) Relative portion for all sam
Figure 22: Fine-grained out-of-vocabulary multi-label errors. The portion of errors follows a similar trend for "artifacts" and "others", while the sudden sharp drop in absolute numbers for "artifacts" is absent for "others".
(a) Relative portion per group.
Figure 23: Multi-label errors identified as non-prototypical instances. The number of samples identified as non-prototypical by Vasudevan et al. (2022) is inherently small (36) and the majority of them are artifacts, leading to noisy results. As the models get better at other types of errors, the relative portion of multi-label errors due to non-prototypical images increases.
(a) Relative portion per group.
Figure 24: Multi-label errors caused by spurious correlations. In relative terms, “artifacts” and “others” follow a similar trend which is noisier for the “others” group.
Finally, in Fig. 25 we present the ratio of unexplained model failures (MLF) over multi-label accuracy (MLA) and standard top-1 accuracy. Again, subfigures (a) and (b) are organized per group, while (c) and (d) show the results for all samples.
Figure 25: Ratio of unexplained model failures (MLF) over multi-label accuracy (MLA - (a) and (c)) and standard top-1 accuracy ((b) and (d)). In general, our pipeline explains the biggest portion of errors made on "organisms" and the lowest portion of errors on "artifacts". However, better models quickly close this gap. For the best models, our pipeline explains up to  $85\%$  of all multi-label errors and  $95\%$  of all top-1 errors.
C Analysis of IMAGENET-A
To demonstrate the applicability of our automated error analysis pipeline to other datasets, we now apply it to IMAGENET-A. IMAGENET-A was introduced by Hendrycks et al. (2021) as an IMAGENET-like test set of "Natural Adversarial Examples". It contains 7500 images from 200 IMAGENET classes, picked to fool a set of ResNet-50 classifiers, while still constituting high-quality, single-entity images, clearly depicting an entity of
Figure 26: IMAGENET-A top-1 Accuracy vs IMAGENET top-1 Accuracy grouped by architecture (left) and training data size (right).
the target class. A comparison of IMAGENET and IMAGENET-A accuracy is shown in Fig. 26. In this section, marker color encodes model parameter counts, i.e., larger models have darker markers.
As IMAGENET-A images were selected to be challenging for IMAGENET classifiers at the time, we see that many classifiers with IMAGENET accuracies below  $80\%$  have almost 0 accuracy on IMAGENET-A. In fact, 801 of the 962 models we consider have an accuracy of less than  $30\%$  on IMAGENET-A. As classifiers improved ( $>80\%$  top-1 IMAGENET accuracy) IMAGENET-A accuracy catches up rapidly. Among other improvements, the main driver seems to be the model parameter count. Note that due to the much lower IMAGENET-A accuracy (of some models), the x-axis of all further plots starts at  $0\%$  rather than  $60\%$ , as before.
As IMAGENET-A uses IMAGENET classes, we keep the split into organisms (77 out of 200 classes,  $38.5\%$ ), artifacts (104 / 200,  $52\%$ ), and other classes (19 / 200,  $9.5\%$ ).
Class Overlap Due to the shared class structure, we can use the same analysis as for IMAGENET and show the results in Fig. 27. The 200 classes in IMAGENET-A were selected from the 1000 IMAGENET classes to avoid class overlap and label ambiguity. Thus, as expected, we see a much lower overall amount of such errors (below  $0.8\%$  instead of up to  $8.5\%$  for IMAGENET). In particular, for artifacts and other classes, we see no
Figure 27: Portion (left) and number (right) of top-1 errors caused by class overlap for organisms (green). For the artifacts and other groups we observe no errors.
class overlap errors at all. Interestingly, for organisms, we still observe a small number  $(\leq 22)$  of errors caused by class overlap. We observe that errors become more frequent (albeit still rare) for models with higher accuracy. As with IMAGENET this is due to more accurate models classifying more images correctly, including to overlapping classes.
Multi-Label Annotations In contrast to IMAGENET, IMAGENET-A was designed to only include single-entity images. Thus, we skip the multi-label analysis for IMAGENET-A and use top-1 accuracy/errrorather than MLA accuracy/errorfor all further analyses.
Fine-Grained Classification Errors Due to the shared class structure, we can apply the same analysis including superclasses for fine-grained classification errors as onImagenet. We show results in Fig. 28.
As with IMAGENET, the group for which we observe most of these errors are organisms (explaining up to  $47\%$  of the group's errors). However,
Figure 28: Portion (left) and number (right) of top-1 errors caused by fine-grained misclassifications by group - organisms (green), artifacts (red), and others (blue).
the portion of explained errors is much lower than for IMAGENET, where up to  $88\%$  of MLA errors for organisms can be explained by fine-grained classification errors. We see the percentage of errors increase linearly with accuracy. Looking at absolute counts, especially for organisms, we see that fine-grained errors initially become more prevalent as model accuracy increases before dropping off as models more frequently also succeed in the fine-grained class distinction.
Fine-Grained Out-of-Vocabulary
Errors Again, we can use the same superclass definition, CLIP embedding and open world classifier as for IMAGENET to identify fine-grained OOV errors. We generally observe slightly larger portions of OOV errors than on IMAGENET and a much clearer linear dependence on accuracy (see Fig. 29). Notably, the superlinear increase observed on IMAGENET (see Fig. 11) is absent here. Looking at absolute counts, after an initial increase and stagnation, we observe a
Figure 29: Portion (left) and number (right) of top-1 errors identified as fine-grained  $OOV$  by group - organisms (green), artifacts (red) and other (blue).
linear decrease with model accuracy. In contrast to IMAGENET, we again do not observe a slope change.
Non-Prototypical Instances Our analysis pipeline for IMAGENET uses the manual annotation of non-prototypical instances from (Vasudevan et al., 2022). As we do not have these available for IMAGENET-A we skip this analysis step here. However, for standard IMAGENET we only observe a small number of such errors ( $\leq 2\%$  of MLE) and note that the IMAGENET-A images were manually selected to only include high-quality images (Hendrycks et al., 2021). Thus, we expect their portion to be even smaller here.
Spurious Correlations Again following our IMAGENET approach for spurious correlation errors we obtain the results in Fig. 30. At first glance, we observe much less spurious correlations than for IMAGENET (6.2% compared to 15% for artifacts). However, we observe a more linear instead of super-linear increase in the portion of errors explained by spurious correlations, leading to this reduction being smaller for less accurate models.
Figure 30: Portion (left) and number (right) of top-1 errors identified as spurious correlations by group - organisms (green), artifacts (red) and other (blue).
Model Failures Looking at the remaining failures, we see that we can only explain up to half of the errors for the best-performing models. Not only are top-1 and MLA error rates much higher than for IMAGENET but also a much larger portion of the remaining errors constitutes severe model failures, highlighting that IMAGENET-A is a much harder dataset.
Figure 31: Portion for remaining failures over IMAGENET-A top-1 (left) and IMAGENET MLA (right).
However, analyzing Fig. 31, we ob
serve that both the IMAGENET-A top-1 and IMAGENET MLA accuracy are good predictors for the portion of model failures which decreases with increasing model accuracy. This highlights again and perhaps surprisingly that IMAGENET MLA underreports progress in model performance, even on particularly challenging subdistributions like IMAGENET-A.
D Extended Examples on Fine-Grained OOV Error Classification
In this section we provide further step by step examples of the procedure that determines if a given model mistake is categorized as a fine-grained OOV error. The sample in Fig. 32 is confirmed to be fine-grained OOV, the one in Fig. 33 is not fine-grained and the one in Fig. 34 is rejected as OOV.
GT: suit  
Pred: bicycle-built-for-two  
Pred: superclass: motor_cycle
(4) GT: backpack bag
(1)GT:suit suits
(2)GT:suit suits
(3)GT:unicycle motor_cycle
(5)GT:suit suits
(6) GT: moped motor_cycle
(9) GT: bicycle-built-for-two motor_cycle
Figure 32: Fine-grained OOV confirmed. The evaluated validation sample (top) has a ground-truth IMAGENET label (GT) suit, but the model predicts bicycle-built-for-two which falls in the "motor_cycle" superclass. Below, we show the top-10 training images that are the most visually similar to the validation sample (according to CLIP similarity) together with their IMAGENET labels and the superclasses of these labels. The labels of images 3, 6, 9 and 10 are in the same superclass as the predicted label (motor_cycle), hence we proceed with the procedure. After obtaining the in-vocabulary and OOV label proposals, as described in Sec. 3.4, the proposals with the highest probability according to CLIP are: safety bicycle, ordinary bicycle, velocipede and bicycle, all of which are OOV, confirming that the error is indeed fine-grained OOV.
(7)GT: traffic light No superclass
(8)GT: plastic bag No superclass
(10) GT: bicycle-built-for-two motor_cycle
GT: wall clock clock
GT: airship, analog clock  
Pred: jinrikisha  
Pred superclass: open_car
GT: tram train
GT: bell cote religious_castle_palace
GT: cab car_bus_truck
GT: tram train
GT: tram train
Figure 33: Fine-grained OOV rejected. The evaluated validation sample (top) has ground-truth IMAGENET multi-labels (GT) airship and analog clock, but the model predicts jinrikisha, arguably attempting to classify the carriages on the street. The predicted label is in the "open-cart" superclass. Next, we show the top-10 training images that are the most visually similar to the validation sample (according to CLIP similarity) together with their IMAGENET labels and the subclasses of these labels. None of the labels is in the "open-cart" superclass, thus we terminate the procedure and reject the mistake as a possible fine-grained OOV error.
GT: umbrella No superclass
GT: tram train
GT: tram train
Figure 34: Fine-grained OOV rejected. The evaluated validation sample (top) has a ground-truth IMAGENET label (GT) barrow, but the model predicts shopping basket, arguably attempting to classify the basket in the cart. The predicted label is in the "box" superclass. Next, we show the top-10 training images that are the most visually similar to the validation sample (according to CLIP similarity) together with their IMAGENET labels and the superclasses of these labels. The labels of images 3 and 4 are in the same superclass as the predicted label (box), hence we proceed with the procedure. After obtaining the in-vocabulary (IV) and OOV label proposals, as described in Sec. 3.4, the proposal with the highest probability according to CLIP is shopping cart, which is in-vocabulary, therefore, the mistake is not categorized as a fine-grained OOV error. The other most likely proposals are: shopping basket (IV), bushel basket (OOV), basket (OOV) and handbasket (OOV).
E Mistake Examples by Error Type
GT: weasel, polecat, mink, black-footed ferret Pred: polecat
GT: screen, monitor  
Pred: monitor
GT: laptop, notebook  
Pred: notebook
GT: projectile, missile  
Pred: missile
GT: Indian elephant, tusker  
Pred: tusker
GT: coffee mug, cup  
Pred: cup
GT: Siberian husky, Eskimo dog  
Pred: Eskimo dog
GT: maillot, tank suit  
Pred: tank suit
GT: cassette player, tape player Pred: tape player
GT: sunglass, dark glasses  
Pred: dark glasses
GT: bathtub, tub, washbasin  
Pred: tub
Figure 35: Class overlap: examples of common prediction errors (Pred) due to overlapping classes. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT: monastery, altar Pred:altar
GT: medicine chest, washbasin, shower curtain Pred: shower curtain
GT: English foxhound, beagle Pred: beagle
GT: paper towel, dock  
Pred: dock
GT: green snake, green mamba  
Pred: green mamba
GT: partridge, ptarmigan  
Pred: ptarmigan
GT: ground beetle, tiger beetle Pred: tiger beetle
GT: minibus, school bus  
Pred: school bus
GT: computer mouse,mousetrap Pred:mousetrap
GT: lumbermill, chainlink fence Pred: chainlink fence
GT: ice cream, chocolate sauce  
Pred: chocolate sauce
GT: computer mouse, desktop computer Pred: desktop computer
GT: cucumber, zucchini  
Pred: zucchini
GT: alp, ibex  
Pred: ibex
GT: container ship, dock, drilling platform  
Pred: drilling platform
Figure 36: Multi-label: examples of common prediction errors (Pred) due to multi-label errors. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT: computer mouse, monitor desk, desktop computer, screen Pred: desktop computer
GT: space bar, computer keyboard Pred: computer keyboard
GT: honeycomb, pillow, mailbag  
Pred: pillow
GT: plate, burrito  
Pred: burrito
GT: shoji, patio, tub sliding door Pred: sliding door
GT:china cabinet Pred:wardrobe
GT: water jug Pred: pitcher
GT: hare Pred: wood rabbit
GT: monastery, stone wall  
Pred: castle
GT: cauliflower  
Pred: broccoli
GT: red wolf Pred: coyote
GT: tabby cat  
Pred: Egyptian cat
GT: moving van Pred: tow truck
GT: bloodhound Pred: redbone
GT:trimaran Pred:catamaran
GT: tobacco shop  
Pred: barbershop
GT: sloth bear  
Pred: American black bear
GT: squirrel monkey  
Pred: titi monkey
GT: grey fox Pred: kit fox
Figure 37: Fine-grained: examples of common prediction errors (Pred) due to fine-grained errors. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT: teapot  
Pred: coffeepot
GT: crate Pred: apiary
GT: book jacket  
Pred: comic book
GT: window screen, cup  
Pred: eggnog
GT:whippet Pred:Saluki
GT: Norwich terrier Pred: cairn
GT: sunglass, dark glasses  
Pred: miniature schnauzer  
OOV: Other dog breed
GT: microwave  
Pred: wardrobe  
OOV: Cupboard or cabinet
GT: suit  
Pred: bicycle-built-for-two  
OOV: Single-person bicycle
GT: velvet Pred: cardigan OOV:Garment
GT:cup,china cabinet Pred: toyshop OOV:Other shop or store
GT: cannon Pred: aircraft carrier OOV: Military vessel
GT: wardrobe  
Pred: sliding door  
OOV: Wardrobe door
GT: car wheel, grille  
Pred: beach wagon  
OOV: Other car model
GT: lumbermill  
Pred: crane  
OOV: Lifting device or facility
GT: dock  
Pred: aircraft carrier  
OOV: Guard ship or facility
GT: car wheel  
Pred: sports car  
OOV: Coupe or other car model
GT: teddy bear  
Pred: toyshop  
OOV: Gift shop or other store
GT:plate Pred: ice cream OOV: Ice-cream cake or dessert
GT:hammer,hook Pred: shovel OOV:Other tool / shovel type
Figure 38: Fine-grained OOV: examples of common prediction errors (Pred) due to fine-grained OOV errors. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT: tiger cat  
Pred: lynx  
OOV: Baby ocelot
GT: strainer  
Pred: cucumber  
OOV: Other vegetable
GT: paddle  
Pred: canoe  
OOV: Rowboat
GT: sea snake, coral reef  
Pred: pufferfish  
OOV: Other fish
GT: cradle, barn, hay  
Pred: altar  
OOV: Altar crib
GT: holster  
Pred: revolver  
OOV: Semiautomatic pistol
GT: pencil box, purse  
Pred: pillow
GT: padlock  
Pred: shield
GT: breastplate  
Pred: water jug
GT: flute Pred: plunger
GT: marmoset  
Pred: tarantula
GT: water jug Pred: espresso maker
GT: badger Pred: wombat
GT: hair slide  
Pred: pinwheel
GT: English foxhound  
Pred: Staffordshire terrier
GT: green lizard  
Pred: buckeye
GT: airliner Pred: wreck
GT: lipstick Pred: beaker
Figure 39: Non-prototypical: examples of common prediction errors (Pred) due to non-prototypical instances. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT: nipple Pred: candle
GT: chest, necklace  
Pred: radio
GT: bonnet, lipstick, saltshaker  
Pred: bucket
GT: jack-o'-lantern  
Pred: traffic light
GT: vestment Pred: altar
GT: washbasin Pred: medicine chest
GT: tank suit, maillot  
Pred: bathing cap
GT: otterhound  
Pred: briard
GT: chain Pred: hook
GT: bookcase  
Pred: bookshop
GT: grille  
Pred: police van
GT: ski mask, alp  
Pred: ski
GT: strawberry  
Pred: trifle
GT: flowerpot Pred: greenhouse
GT: wool Pred: bonnet
GT: chain, suspension bridge  
Pred: swing
GT: water jug Pred: water bottle
GT: traffic light, crash helmet Pred: bicycle-built-for-two
GT: bulletproof vest, police van Pred: military uniform
GT: kuvasz  
Pred: Samoyed
GT: warplane, pole, chain Pred: aircraft carrier
GT: pole Pred: traffic light
Figure 40: Spurious correlations: examples of common prediction errors (Pred) due to spurious correlations. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
GT:Windsor tie Pred:suit
GT: tobacco shop  
Pred: street sign
GT: wardrobe
GT:washer
GT: backpack
Pred: unicycle
GT: loggerhead
Pred: scuba diver
Pred: refrigerator
Pred: vending machine
GT: soccer ball
Pred: potter's wheel
GT: pay-phone
Pred: cash machine
GT: letter opener
Pred: pedestal
GT: missile.projectile
Pred:obelisk
GT: hatchet, chain mail
GT: pop bottle
GT:overskirt
Pred: apron
GT: dumbbell
Pred: abacus
GT: library
Pred: lab coat
Pred: pencil sharpener
GT: pig
Pred: greenhouse
Pred: swing
Pred: confectionery
GT: suit, umbrella
GT: frying pan
Pred: tray
Figure 41: Model failures: examples of common prediction errors (Pred) classified as model failures. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENet label is listed first.
GT: bannister
Pred: crib
GT: dark glasses, sunglass
Pred: sunscreen
GT: passenger car
Pred: studio couch
GT: thimble
Pred: microphone
F Experimental Setup
In this section, we provide further details on our evaluation.
We publish our code and detailed instructions on how to reproduce our results at https://github.com/eth-sri/automated-error-analysis.
F.1 Computing the Multi-Label Accuracy
When computing the models' multi-label accuracies, we closely follow the recommendations by Shankar et al.  $(2020)^{3}$ . In particular, we discard the set of problematic images, then compute per-class accuracies and finally average them to obtain the multi-label accuracy. We consider a model prediction to be correct if it is marked as correct or unclear in the multi-label dataset.
F.2 Datasets and Labels
We consider the validation set of the ILSVRC-2012 subset of IMAGENET (Deng et al., 2009; Russakovsky et al., 2015), available under a non-commercial research license. More concretely, we use the subset of this validation set labeled by Shankar et al. (2020) and then Vasudevan et al. (2022), with the labels being available under Apache License 2.0. We further evaluate our pipeline on the ImageNet-A dataset (Hendrycks et al., 2021) available under MIT License.
F.3 Summary of Evaluated Models
Table 3 contains a list of all models we considered in this study and a subset of their metadata. The models were obtained from multiple sources: Torchvision $^6$ , torchhub $^7$ , HuggingFace $^8$ , and timm $^9$ . They can all be automatically downloaded and evaluated using our open-sourced code. After collecting all model outputs (6 days for ImageNet and 1 day for ImageNet-A on a single GeForce RTX 2080 Ti GPU), running our error analysis pipeline on all models takes 12 to 24 hours using a single GeForce RTX 2080 Ti GPU for ImageNet and ImageNet-A respectively.
Table 3: List of evaluated models and a subset of their metadata.
No	Model ID	Source	Arch	Dataset
1.	alexnet	torch	cnn	ImageNet-1k
2.	bat_resnext26ts.ch_in1k	timm	cnn	ImageNet-1k
3.	convformer_b36.sail_in1k	timm	cnn	ImageNet-1k
4.	convformer_b36.sail_in1k_384	timm	cnn	ImageNet-1k
5.	convformer_b36.sail_in22k_ft_in1k	timm	cnn	ImageNet-21k
6.	convformer_b36.sail_in22k_ft_in1k_384	timm	cnn	ImageNet-21k
7.	convformer_m36.sail_in1k	timm	cnn	ImageNet-1k
8.	convformer_m36.sail_in1k_384	timm	cnn	ImageNet-1k
9.	convformer_m36.sail_in22k_ft_in1k	timm	cnn	ImageNet-21k
10.	convformer_m36.sail_in22k_ft_in1k_384	timm	cnn	ImageNet-21k
11.	convformer_s18.sail_in1k	timm	cnn	ImageNet-1k
12.	convformer_s18.sail_in1k_384	timm	cnn	ImageNet-1k
13.	convformer_s18.sail_in22k_ft_in1k	timm	cnn	ImageNet-21k
14.	convformer_s18.sail_in22k_ft_in1k_384	timm	cnn	ImageNet-21k
15.	convformer_s36.sail_in1k	timm	cnn	ImageNet-1k
16.	convformer_s36.sail_in1k_384	timm	cnn	ImageNet-1k
17.	convformer_s36.sail_in22k_ft_in1k	timm	cnn	ImageNet-21k
Table continues onto next page
<sup>3</sup>https://www.tensorflow.org/datasets/catalog/imagenet2012_multiplelabel
4Fore more details see https://image-net.org/download
<sup>5</sup>https://github.com/hendrycks/natural-adv-examples
<sup>6</sup>https://pytorch.org/vision/stable/models.html
<sup>7</sup>https://pytorch.org/hub/
<sup>8</sup>https://huggingface.co/models
<sup>9</sup>https://github.com/huggingface/pytorch-image-models