Beyond Accuracy: What Matters in Designing Well-Behaved Image Classification Models?
Authors: Robin Hesse
Doğukan Bağcı
Bernt Schiele
Simone Schaub-Meyer
Stefan Roth
Venue: Accepted by TMLR ()

Abstract: Deep learning has become an essential part of computer vision, with deep neural networks (DNNs) excelling in predictive performance. However, they often fall short in other critical quality dimensions, such as robustness, calibration, or fairness. While existing studies have focused on a subset of these quality dimensions, none have explored a more general form of "well-behavedness" of DNNs. With this work, we address this gap by simultaneously studying nine different quality dimensions for image classification. Through a large-scale study, we provide a bird's-eye view by analyzing 326 backbone models and how different training paradigms and model architectures affect these quality dimensions. We reveal various new insights such that (i) vision-language models exhibit high class balance on ImageNet-1k classification and strong robustness against domain changes; (ii) training models initialized with weights obtained through self-supervised learning is an effective strategy to improve most considered quality dimensions; and (iii) the training dataset size is a major driver for most of the quality dimensions. We conclude our study by introducing the QUBA score (Quality Understanding Beyond Accuracy), a novel metric that ranks models across multiple dimensions of quality, enabling tailored recommendations based on specific user needs.

Published in Transactions on Machine Learning Research (01/2026)
Beyond Accuracy: What Matters in Designing
Well-Behaved Image Classification Models?
Robin H
rhesse@mpiiinf de
Mar Planck Institute for Informatics, SIC
Doğukan Bager*
batpiiannnbcci@sttud.e htw.::asssuuuudddddddmeeee wwwdarastddddd....mmmmmm de
Deportment of Computer Science
Technical Universily of Durmstadt
Zuse School ELIZA
Bernt Schicle
schiele@@mpiiinf.mppp
Max Planck Institute for Informaties, SIC
Zuse School ELIZA
Simone Schaub-Meyer
Department of Computer Science Technical University of Dormstadt
Aessian.AI
Stefan Roth
Department of Compuler Science Technical University of Dormstadt hessian.Al AT
Zuse School ELIZA
Reviewed on OpenReview: https: /tpp:///oppeeeeeeeeeee net/ forum? E7HDtLCoo66
Abstract
Deep learning has become essential part of computer vision, with deep neural networks (DNNs) excelling in prediclive performance. However, they often fall short in other critical quality dimensions, such as robustness, calibration, or fairness. While existing studies have focused on a subset of these quality dimensions, none have explored a more general form of "well-behhvvddness" DNNs. With this work, we address this gap by simultaneously study- ing nine different quality dimensions for image classification. Through a large-scale study, we provide bird's-eye view by analyzing 326 backbone models and how different training paradigms and model architectures affect these quality dimensions. We reveal various insights such that. (i) vision-language models exhibit. high class balance on ImageNet-Ik classification and strong robustness against domain changes; (ii) training models initialized with weights obtained through self-supervised learning is an effoctive stratogy to improve most considered quality dimensions; and (iii) the training dataset size is a major driver for most of the quality dimensions. We conclude our study by introducing the QUBA score (Quality Understanding Beyond Accuracy), novel metric that ranks models across multi- ple dimensions of quality, enabling tailored recommendations based specific user needs
Project page: https://vieinf. www.ggtww......oneeeeeeeeeeeeeee of Darmstadt.
"equal contribution; wwrk done while Technical University

---

Published in Transactions on Machine Learning Research (01/2026)
Accuracy
Accunacy
Adv. Rob
Parame
Adv. Rob
Parame
Shape
Shape Bis
C.Rob.
Biss
C-Rob.
00D Rob.
Obj-
0OD
Objj
Focus
Rua.
Foces
Class
Civer
Cal. Error
Balance
Error
Balance
self-SL fine tuning
Standard
vs.
ViL
models
Figure 1: Visualization of Iwo of our main results. We compare nine different quality dimensions for popular backbone models trained with standard supervised learning (SL) against the corresponding backbones trained after initialization with weights obtained through sclf-supervised learning (left) and when utilized in a vision-language (ViL) model (right). Axis units indicate the distance (in standard deviations) to the mean (0 line) of each quality dimension; see Eq- (1) and its explanation for details. Please refer Tab. 2 (e) and (i) for raw values and Sec. 3 for an interpretation of the results.
1 Introduction
Today's computer vision research heavily shaped by advances in the field of deep learning. While deep neural networks (DNNs) excel at predictive performance, often mcasured via the accuracy, has been shown that they are flawed across various other quality dimensions, such as robustness (Goodfellow al., 2015 Hendrycks & Dietterich 2019), calibration (Guo et al. 2017 and fairness (Du al. 2021). To address these challenges, the scientific community started various parallel streams of research focusing individual aspects of DNN quality, developing mostly orthogonally. We argue that this orthogonal development is somewhat surprising, as the overarching goal for most. applications should be the implementation of more weellbehaved networks that excel in many quality dimensions (Liu et al. 2023). While some works study the relationship between subset of quality dimensions, e.g., how accuracy and calibration relate (Guo et al. 2017 Minderer et al. 2021), ; or how adversarial training (Goodfellow et al. 2015) improves calibra- tion GGrabinski et al. 2022), we are not aware of any work that studics broad range of quality dimensions simullaneeussyy. Consequently, it is largely unknown how model improvements in one direction affect other quality dimensions. Here, we close this gap by studying how 326 backbone models perform along nine different quality dimensions for ImageNet-Ik [Russakovsky et al., 2015) image classification. By doing so, we analyze how diffcrent training paradigms and model architectures can be used to improve these quality dimensions, uncover unknown connections between quality dimensions, and give recommendations on what models to use based on specific user needs. We expect our findings be highly relevant for advancing the development of classification models that not only excel in accuracy but also across wide range of DNN quality dimensions.
recommenoov
Our contributions can be summarized as: (1) We introduce a novel benchmark to measure a broad range of quality dimensions simultaneously, which is compatible with any DNN / bbckboon that performs ImageNet- 1k (Russakovsky et al. 2015) classification. (2) In a large-scale study, we evaluate how 326 backbone models from prior work perform along nine considered quality dimensions. (9) We use this to analyze how different training paradigms and architectural changes can be utilized to improve the different quality dimensions. Among other things, find that self-supervised pre-training followed by fine-tuning improves most quality dimensions and that vision-language models achieve high fairness (here measured the class balance)
ImageNet-1k 1k classification while being fairly robust against domain changes (sce Fig. D (4) Building on trends in related work that examine relationships between individual quality dimensions (e.g., Guo el al., 2017 Miller et al. 2021 Minderer et al. 2021), we analyze the relationships among all considered quality dimensions. (5) We conclude our study by introducing a novel QUBA score (Quality Understanding Beyond Accuracy) that ranks models across multiple dimensions of quality. We use this score recommend top- performing models tailored to diverse needs.
2

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Overview of our considered DNN quality dimensions for image classification. Arrows indicate if higher (t) or lower (4) is better. If a quality dimension is computed over multiple datasets or metrics, we use the geometric mean.
Quality Dimension
Description (visual illustrations are provided in Appendix A
Accuracy (t)
Fraction of correctly classifed (clean) images
Adversarial Robustness (1)
Fraction of correctly classified images after an FGSM or PGD attack (normalized by clean accuracy)
C-Robustness (f)
Fraction of correctly classified images after corrupting images (normalized by clean accuracy)
OOD Robustness (*)
Fraction of correctly classified images from different domains (normalized by clean accuracy)
Calibration Error (L)
Misalignment of the output confidence Staudard deviation of the accuracies and average confidences across all individual classes
the true probability a correct classification
Class Balance (t)
Object Focus (f)
Fraction decisions that are based on foreground and not on background
Shape Bias (1)
Fraction of decisions that based shape and not texture Number of parameters
Parameters ())
2
Evaluating quality beyond accuracy
We go beyond accuracy by exploring the general weell-behavedness" of DNNs. While this term is inherently ill-defined and task-dependent, we use it informally as the performance across the nine quality dimensions choson in this study. Specifically, we consider (1) accuracy; three robustness metrics: (2) adversarial robust- ness, (3) corruption robustness, and (4) out-of-domain robustness; (5) calibration error; (6) fairness measured via class balance; two dimensions concerned with shortcut-learningg (7) object focus and (8) shape bias; and (9) computational cost measured via the number of parameters. These dimensions encompass a wide range and attracted considerable attention in related work, which is why we regard
of DNN qualities/propent them as particularly important. We evaluate these quality dimensions simultaneously by merging corre- sponding evaluation protocols into a single, comprehensive benchmark. To enable a large-scale study with a feasible computational load, we only select protocols that require ImageNet-1k (Russakovsky et al. 2013) classificatton without model fine-tuning. While there are other important dimensions, such as explainabilin and dut-of-iistributiin detection capabilities, they are challenging evaluate as they are coupled to specific detection methods that might vary across backbones (Hesse et al.] 2023b) thus, we excluded them. Also, note that all evaluation protocols merely proxies for the targeted dimension; they do not necossarily reflect true performance in that dimension, nor do they capture the full complexity of the underlying behavior. Consequently, different evaluation protocols may lead to different conclusions, which is worth keeping in mind when interpreting our findings.
We now outline related work, the considered quality dimensions, and how we mcasure them in this work; see Tab. D for a summary and Appendix A for more details.
Accuracy. The success of DNNs is largely driven by their superior accuracy, first showcased in 2012 by AlexNet (Krizhevsky et al. 2012) on ImageNet-1k (Russakovsky et al. 2015). This marked the beginning of the "deep learning era," leading to increasingly powerful models (Dosovitskiy et al., 2021 He et al. 2016 Liu
et al. 2022b Simonyan & Zisserman 2015). To measure a model's accuracy, we report the top-1 accuracy on the ImageNet-Il 1k evaluation split.
Adversarial robustness. DNNs are vulnerable to adversarial atlacks, ie.,, small perturbations in the input space (Szegedy et al.] 2014). This vulnerability can be reduced by training with adversarial exam- ples [Goodfellow et al.J 2015 Madry et al., 2018) or by defensive distillation (Papernot et al.) 2016). To assess adversarial robustness, we measure the geometric mear of the accuracies after applying two popular attacks, FGSM (Goodfellow et al. 2015) and PGD (Madry et al. (2088). To reduce the dependence on the clean accuracy of the model, we report adversarial robustness relative to the clean ImageNet-Ik accuracy. In Appendix C.4 we compare adversarial robustness mcasure to Autokttack (Croce et al. 2021), yiclding very similar conclusions.
Corruption robustness. DNNs are susceptible to common image corruptions such as JPEG com- pression and contrast changes (Hendrycks & Dietterich, 2019). which can be reduced with special kinds of data augmentations (Hendrycks et al. 2020) or self-supervised learning (Hendrycks al. 2019). To
2Th geometric mean ensures that metrics of different scales contribute equally without one overshadowing the others.
3

---

Published in Transactions on Machine Learning Research (01/2026)
assess model's robustness to common corruptions (C-robustness), we measure the mean accuracy on ImageNet-C (Hendrycks & Dietterich 2019), i.e., the TmageNet evaluation split with different corruption types of increasing strength. To normalize C-robustness and to be consistent with our other robustness met- rics, we again report the top-1 accuracy on the corrupted data relative to the clean ImageNet- 1k accuracy.
OOD robustness. Out-of-domain (OOD) robustness is concerned with the generalizability of a model to OOD data (e.g., Geirhos et al., 2019 Hendrycks et al.) 2021 Wang et al., 2019). Contrary to adversarial and corruption robustness, datasets to assess the OOD robustness exhibit stronger visual domain shifts and contain new data samples. We assess OOD robustness by reporting the geometric mcan of the relative accuracy on five established OOD datasets: ImageNet-R (Hendrycks et al. 2021). ), OntgeNeetSSkeoo (Wang
et al. 2019), as well as
Edge, and Silhouette from Geirhos et al. (2019).
Calibration error. Calibration measures how well model's output confidence reflects the probability of correct prediction. Guo et al. (0117) found DNNs to be poorly calibrated, spurring the developments of approaches like deep ensembles Lakshminarayanan et al.) 2017) and label smoothing (Müller et al. 2019
We report the geometric mean of the expected calibration error (ECE) (Guo et al. 2017 Nixon et. al. 2019) and the adaptive calibration error (ACE) NNxon et al. 2019).
Class balance. A well-behaved model should behave fairly. Fairness can, e.g., be improved by the class- wise weighting of the cross-entropy loss (Benz et al. 2020). As fairness has multiple facets (Verma & Rubin 2018) and there is no standardized fairness metrie, we rely on simplified notion of "class balance": no class should be particularly favored or disadvantaged (Benz et al. 2020 Kuzucu et al.) 2024). More specifically, we evaluate the class balance of a model in two ways: (1) the standard deviation of ImageNet-1k class accuracies, similar to Croce el al. (202T), and (2) the standard deviation of average class confidences, similar to Kuzucu et al. (2024). To align with other metrics, we subtract these values from 1, so higher scores indicate greater class balance, and we aggregate both measures using the geometric mean. While aiming to approximate fairness, we note that ImageNet- 1k itself exhibits dataset biases, and thus our conclusions about fairness may not fully reflect real-world concerns.
Object focus. ImageNet-traine trrineed DNNs rely on background features to the extent that they can be fooled by changing the background (Xiao et al. 2021 Zhu et al., 2017]. This can be avoided by training on images where the background signals are decorrelated from the class labels (Xiao et al. 2021). Similar to Xiao et al.) 2001), we measure object focus by assessing accuracy drops when replacing backgrounds with those from other classes.
Shape bias. ImageNet-trained CNNs exhibit a texture bias rather than a shape bias (Geirhos et al. 2019 Since this can hurt generalizability and robustness, models with increased shape bias are preferred and have been developed (Nam et al. 2021 Shi et al., 2020). The shape bias is measured using images with a conflict between the shape and texture cues e-g., an image of cat (shape) with the skin of an elephant (texture). These images are fed into the model to determine whether it predicts based on texture (texture bias) shape (shape bias) (Geirhos et al.) 2019).
Parameters
Unlike the other quality dimensions, a higher shape bias is not inherently better, as some applications may benefit from a stronger focus on texture. Nonetheless, we include shape bias as a dimension and, for consistency with related work and since is more in line with human vision (Geirhos et al. 2019 Wang al. 2020), assume that higher values are preferable. This assumption does not alfect the core of our analysis. Further, in Sec. 5 we introduce the QUBA score, where the shape bias weight can be adjusted or even inverted to reflect specific preferences.
FLOPs
Memory
Parameters
FLOPs
Memory
Figure 2: Rank correlation ma- triz for the considered metrics on computational cost for our full model zoo of 326 models. All entries have a p-value below 0.05, indicating statistical signif- icance.
Deep neural networks (DNNs) should be memory-effiiiett and fast to reduce resource consumption and operate sustainably. Since actual memory and runtime performance depend heavily specific imple- mentations and hardware, we follow established practice and use the num- ber of model parameters as a hardware- and
Parameters.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Adv. Rob.
C-Rob.
OOD Rob.
Cal. Error
0.04
0.6
0.8
1
0.4
0.75
0.4
0.02
0.2
0.25
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.9
Class Balance
Obj. Focus
Shape Blas
Parameters in Mil.
0.9
0.9
0.6
750
500
0.4
0.8
0.2
250
0.7
0.7
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.9
Figure 3: Different quality dimensions (y amis) vs. accuracy (x anis). To reduce clutter in the plots, we only plot representative models instead of our full model 20o; please refer to the project page for interac- tive plots with all models. To emphasize the effect of different training strategies and model architectures, we group models visually: the training dataset size is marked by symbols within each marker (no sym- bol for ImageNet-1k, dot (-) for ImageNct-21k, star (+) for large-scale datascts); different training strate- gies by shapes (standard supervised training as squares m, adversarial training as circles self-supervised (pre-)training as triangles 4, semi-supervised training as diamonds +, A[1,2,3] training as pentagons )); and different architectures by color (blue for CNNs, orange for Transformers, green for B-cos models, and yellow for vision-language (ViL) models).
proxy for computational cost (Kaplan et al. 2020 Tay et al., 2022). To
validate its suitability as a proxy, we visualize the Spearman's 's rank correlation for 326 models (see Ap- pendix E.3 between the number of paramcters, required memory, and thcoretical FLOPs in Fig. 2 All three measures are strongly correlated (correlation coefficients 0.85), confirming that the number of pa- rameters is reliable proxy. Thus, for practical purposes and ease of comparison in future studies (memory consumption is implementation dependent, and the computation for the thcorctical mumber of FLOPs neods to be adjusted for novel model components), we adopt the number of parameters as our primary metric. However, we note that parameter count, a alone cannot. fully capture the efficiency of model and should be interpreted with caution (Dehghani et al. 2022). Various studies are concerned with reducing the number of parameters or the computational cost of DNNs while keeping the accuracy as high as possible (Hesse al.
2023a Li et al. 2022b Tan & Le 2019
2021
3 What makes a model more well-behaved?
Equipped with the above quality metrics, we now study how different design choices affect DNN quality. This not only sheds light on quality beyond accuracy of the current state of the art for image classification, but also facilitates the development of more well-behaved DNNs in the future. In Tab. D we compare the average of each quality dimension for different setups. Since not all backbones can be considered in each configuration (e.g., not all have adversarially trained variants), we ensure fair comparisons by selecting the subset of models that is consistently available for all configurations of the respective setup (with no minor modifications; see Appendix D.I.. In Fig. 3 we plot the different quality dimensions for selected models against accuracy, distinguishing bet ween model groups
Experimental setup. We use 326 publicly available models in our large-scale study. See Appendix E.3 for overview, including numerical results and implementation details for each quality dimension.
5

---

Published in Transactions on Machine Learning Research (01/2026)
3.1 Different training strategies
Training dataset size. We compare models trained on ImageNet-lk RRusskkvsky et al. 2015 to those trained on the larger ImageNet-21k (Deng et al.) 2009) dataset in Fig. 3 no symbol us. dot (.)) and Tab. 2 (a) and (b). Training on a larger dataset improves nearly all quality dimensions for CNNs and Transformers, particularly accuracy, C-robustness, and calibration likely by promoting more general and less overfitted features. Interestingly, with larger training dataset, the adversarial robustness of Transformers decreases.
Conclusion: Training on a larger dataset improves most of the considered quality dimensions.
Adversarial training. We compare adversarially trained models against the corresponding standard supervised models in Fig. (circles vs. squares
and Tab. 2 (c). Adversarial training (AT) improves
2021: Wang al. 2020 OOD robustness (Eugstrom al., The latter is in line with results of Benz et al.
shape bias (Engstrom et al. 2019 Geirhos el. al.
2019), and, expectedly, adversarial robustness. Accuracy and class balance significantly worsen with AT,
(2021a). Interestingly, we observe trend of (Grabinski et al., 2022) that found evidence 2016 exhibit improved calibration errors.
(2020); Xu ct al.
increasing calibration error with AT, which exlends findings in
that adversarially trained ResNets (He et al.
Conclusion: AT improves adversarial/OOD robustness and shape bias. It impairs accuracy and class balance.
Self-supervised training. Self-supervised learning eliminates the need for dataset annotations and thus allows for training on significantly larger datasets. We compare models initialized with weights obtained through self-supervised learning to standard supervised models in Fig. 3 (triangles A us. squares m) and Tab. 2 (d) and (e). We consider self-supervised models in two standard transfer settings: (i) models where only the final classification layer trained (linear probing, LP) and (ti) models that are fully fine-tuned on ImageNet-1k (F2E). We analyze how each approach affects the different quality dimensions. LPP models (Tab. 2 (d)) generally underperform compared to supervised models, except in OOD robustness, calibration, and shape bias likely due to the larger gap between training and testing distributions. The reduced object focus is notable, as Caron et al. (2021) found that supervied Transformers produce attention maps that closely align with objects. This suggests that attention maps may not serve as reliable explanations, finding consistent with (Hesse el. al. 2023b 2024). The slight parameler difference stems from DINOv2 (Oquab)
et al. 2024 using a 14 x 14 patch size instead of the original 16 x 16.
On the other hand, fine-tuning (E2E) self-supervisod models (Tab. 2 (c)) improves most quality dimensions (except calibration) probably due larger pre- pre-training datasets typically used for self-supervised learning and smaller domain gap than LP. The improvement in class balance is particularly surprising, expected the larger training datasets used for self-supervised training to have much stronger class imbalances than the evenly distributed ImageNet dataset. We hypothesize that the class balance is nonetheless improved because the self-supervised models are pre-trained without any class information, resulting in features less tailored to specific classes. As a result, these features yield a more balanced distribution of class accuracies and class confidences.
Self-supervised models with linear probing perform worse than supervised models in most quality dimensions. Fine-tuning self-supervised models in an end-to-end fashion improves most quality dimensions.
Conclusion:
Semi-supervised training. We also measure how remi-supervised training (Xie et al. 2020 Yalniz et al. 2019), i.e., training on combination of labeled and unlabeled data, compares to supervised training in Fig. 3 (diamonds us. squares m) and Tab. P (f). Among the dimensions with statistically significant changes, semi- supervised training has similar effects self-supervised training with F2E fine-tuning, probably also due
6

---

Published in Transactions on Machine Learning Research (01/2026)
Average quality dimensions for models with different configurations. We evaluate various setups, focusing on different training strategies (a-g) and different architectural choices (h-j). In cach configuration, we report the average score for each quality dimension across the models associated with that configuration. As different models are available for different sctups, each sctup considers distinct selection of models being compared. As a result, both the models and their total number (indicated by the number beside each setup) vary across setups to maintain fair basis for comparison. The number of asterisks represents the statistical significance of differences in the average scores of a quality dimension across configurations within each sctup: *** for < 0.05, * for p < 0.1, and for < 0.2, based on t-test results.
Table 2:
Setup
Configuration
Acc.t
Adv. Rob. 1
C-Rob,-
OOD Rob.
Error
Cal.
Balance
Class
Obj- Focus
Shape, Bias
Panams.
in Mil.
(IN-21k)
0.0026
0.33
(h)
16 Trandorrars (IN-1k)
0.81
0.21
0.68
0.61
0.00
0.79
0.94
0.38
85
Transformers (IX-21k)
0.84***
0.17"
0.75*
0.69
0. 0.0027
0.80*
0.95
0.42
85
(c) Supervised models Adversarially trained modele
0.0056
0.60
0.74"*a
0.03
0.0006
0.78
0.72
(d)
13 Supervited models
0.81
0.20
0.66
0.58
D. 0.0047
0.81
0.98
0.34
69
Salf-aupervien models (I.P)
0.75
0.10***
0.61
0.58
0.0028
0.75**
0.85**
0.40*
81
(c) Supervised models
0.16
0. 0.0084
0.79
0.93
Self-auperviodt models (E2B)
0.24 ***
0.78
0.79
0.0046
0.81
0.06
0,30
(8)
18 Supervised models
0.50
0.18
0.58
0.00
0.0045
0.78
0.92
0.24
28
Somi-ouperviaee models
0.82
0.20
0.40
0.59
0.0059
0.80
0.98
0.20
28
(x) Supervined models
0:12
0.50
0.82
0.0064
0.78
0.0/
0.25
supervized models (800 epochu)
0.31"
0.0030
0.75"
0.02
0.27**
A2 supervined models (300 spochu)
0.41
0.30"
0.0026
0.75"
0.03
0.25
AB supersised modele (100 cpocho)
0:32
0.04
0.27
0.0048
0,76
0.91
0
(b)
46 CNNs
0.51
0.11
0.62
0.64
0.6045
0.79
0.92
0.29
40
Transformers
0.51
0.20
0.00
0.02
0.0046
0.50
0.9g
0.32
40
(1)
12
Standard models
0.77
0.07
0.88
0.56
0.0038
0.77
0.93
0.27
B-coa medels
0.02
0.0116
0.75*
0:27
()
24
Standard modela
0.81
0.18
0.02
0.66
0.0044
0.79
0.93
0.35
152
modela
0.74
0.10*
0.80
1.00
0.0337***
0.00
0.93
0.50
275
to the combination of a large-scale training dataset and relatively close training and testing domains. Only C-robustness is negatively affected statistically significantly.
Conclusion: Semi-supervised training improves accuracy, adversarial robustness, class balance, and shape bias. Only C-robustness is clearly impaired.
[1,2,33 training. Wightman et al. 2021) introduce several training strategies, termed A1, A2, and A3, which incorporate best practices for training DNNs e.g., multi-label classification objectives, data aug- mentation techniques, and the use of advanced optimizers. Most importantly, the three strategies vary in their training duration: A1 is trained for 600, A2 for 300, and A3 for 100 epochs. We compare the train- ing strategies to standard supervised models in Fig. 3 (pentagons + us. squares m) and Tab. 2 (g). While some training strategies of the standard supervised models might overlap with the A[1,2,3] training, the long training of A1 is not utilized in any of the standard models. The accuracy is slightly increasing for the setups with increasod training times (A[1,2): statistically insignificant). Interestingly, adversarial robustness signif- icantly improves with the A[1,2,3] training, while C-robustness and OOD robustness decrease. We believe the improved training enhances adversarial robustness by expanding the distance between decision bound- aries and data points, but reduces generalizability by encouraging "overfitting" to the training distribution. Calibration error decreases for the sctups with increased training times (A[1,2)), which extends findings of Minderer et al. (2021) that showed that calibration error increases with longer training when measured only on BiT models (Kolesnikov et al., 2020). Class balance is reduced for all the setups; object focus remains fairly stable, and the shape bias increases with longer training, confirming results of Hermann et al. 2020
Conclusion: Adversarial robustness, calibration, and shape bias improve with longer training times. C/00DDrrbustness and class balance are impeded.

---

Published in Transactions on Machine Learning Research (01/2026)
3.2 Different model designs
Now that we have covered various training strategies and their effect on different quality dimensions, we analyze the effect of specific architectural choices.
Is the time of CNNs over? We compare models based on convolutions (CNNs) and attention (Trans- formers) in Fig. 3 (blue us. orange and Tab. 2 (h). Since Vision Transformers @Dsoviiskly et al, 2021) were introduced only in 2020, exclude CNNs proposed before 2020 and compare only newer CNN architectures with Transformers for a fairer evaluation. To further improve the fairness of our comparison, we make sure that we have an equal number of CNNs and Transformers from different setups (e.g., adver- sarial training) and only compare them when they have a similar number of parameters (within a 1-million difference). Despite our efforts to ensure a balanced comparison, this setup gives us less control over certain variables than our other experiments. Therefore, these results should be interpreted with cantion. Remark- ably, CNNs and Transformers perform equally in accuracy. However, Transformers outperform CNNs in all
the other quality dimensions. Our results robustness nicely complement those of Bai et al. (2021) p who
compared the robustness of CNNs and Transformers but considered only ResNet50 (He et al. 2016 and De1T-S/16 [Tovvrn et al. 2021a).
Conclusion: Transformers consistently outperform CNNs across almost all quality dimensions.
B-cos transform. Initially introduced to improve interpretability, the B-cos transform (Bhlle et al, 2022 can substitule the linear transformations in DNN. It encourages the weights to align with the input and potentially affects the model beyond the improved interpretability. We thus analyze B-cos models in Fig. 3 [green and compare them to the corresponding standard models in Tab. (i). Besides shape bias and the number parameters, all considered quality dimensions drop significantly when using the B-cos transform. A potential reason for this is the inductive bias of weight-input alignment, limiting the model's expressiveness.
Conclusion: The B-cos transform negatively affects most the considered quality dimensions.
Vision-language (ViL) models. With ViL models becoming increasingly relevant, we study their perfor- mance across the considered quality dimensions Fig. (yellow [) and compare them to their corresponding backbones trained in a supervised fashion in Tab. 2
(i). Please note that Tu ct al. (2023) conducted a sim-
ilar study, however, focusing exclusively on CL.IP models (Radford et al. 2021] and covering slightly different set of quality dimensions. Since ViL models perform zero-shot classification by mapping the 1000 ImageNet-1k class labels into their fcature space and then predicting the class label closest to the fcature embedding of the given image, their accuracy is nolably lower than that of the supervised models. Also, they contain significantly more parameters due to the additional language encoder. They exhibit decreased adversarial robustness and C-robustness while strongly improving OOD robustness (Radford et al.) 2021 At first glance, one might attribute the improved OOD robustness the models being trained on signifi- cantly larger datasets that include domains similar those in the OOD datasets (Liu et al., 2023) While this is certainly factor (Mayilvahanan et al.
2025 closer look reveals that ViL models still outperform
other models trained on similarly large datasels (see Appendix C.2). suggesting that they oller advanlages beyond just dataset size. While Minderer et al.
2021)) found that CLIP is fairly well calibrated when trained
on Teb (WTT) (Radford et al., 2021 Tu et al. (2023) found that CLIP calibration can decrease when trained on other datasets. We extend their finding by observing that other ViL models also exhibit significantly worse calibration than standard models. Class balance and shape bias improve by a large margin the former probably for similar reasons as for self-supervised models.
Conclusion: ViL models excel in OOD robustness, class balance, and shape bias. However, they fall behind accuracy (zero-shot), calibration, and parameters.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Class Balance
Accuracy
Adv. Rob
000 Rob.
Cal. Error
Obj Focus
Shape Blas
Parameters
C-Rob
Accuracy
Adv. Rob.
C-Rob.
0.5
0OD Rob.
Cal. Error
Class Balance
Obj. Focus
-0.5
Shape
Parameters
Figure 4: Rank correlation matrix for the considered qualily dimensions among our full model zoo. All non-crossed-out entries have p-value below 0.05, indicating statistical significance. Crossed-out entries correspond to p-values above 0.05 and are therefore not statistically significant.
4 Relationships between quality dimensions
Comparison to related work. While most previous work is concerned with improving quality dimen- sions, there are also studies examining their relationships. However, not all relationships have been explored, and prior studies used fewer and older models, which can lead to contradictory findings (see Appendix C.3 To address this gap, investigate the relationship between numcrous quality dimensions for our extensive model zoo, plotting the Spearman's 's rank correlation matrix for all nine considered quality dimensions across 326 models in Fig. ! Our analysis confirms that accuracy is positively correlated with OOD robustness (Miller et al.] 2021), ob- ject focus (Xiao et al. 2021), shape bias (Hermann et al. 2020), and with the number of parameters Liu ct al. 2025). The number of parameters positively correlates with OOD robustness (Liu et al.) 2025 and
please refer to Appendix [C.1 for correlation matrices of specific model subgroups.
adversarial robustness (Madry el al. 2018 Nakkiran 2019). Increasing the shape bias improves ad adversarial
robustness (Geirhos et al. 2021 Jo & Bengio, 2017). accuracy (Geirhos et al., 2019), and OOD robust-
ness
(Geirhos et al. 2019 Accuracy and calibration error exhibit negative correlation, aligning with
Minderer et al. (0021 who found negative correlation in more recent Transformer models, and contradict-
ing Guo et al. (2017) who observed a positive correlation between these metrics in older backbone models.
Unlike hypothesized by Tsipras et al. (2019) and confirming Yang et al. (2020). accuracy and adversarial robustness are positively correlated. Contrary to Liu et al. (2025) and Jo & Bengio (2017), adversarial robustness is not statistically significantly correlated with C-robustness and OOD robustness. While Gra-
binski al. (0222) found that adversarial training improves calibration, find statistically significant correlation between adversarial robustness and calibration.
Conclusion: We provide a bigger picture of related work using our extensive model zoo to validate known quality relationships, resolve conflicting findings, and extend recent findings regarding a link between adversarial and OOD robustness / C-robustness calibration.
Discovering new relationships While cannot discuss all findings in Fig. some insights to knowledge have not been reported for backbone models in image classification. For example, accuracy and class balance are strongly correlated, meaning higher-accuracy models have less discrepancy between the best and worst-perfondt perrotrerformin classes. Further, object focus is strongly correlated with all quality dimensions but the calibration error, rendering models with improved object focus an interesting research dircction. This may be because models with greater object focus capture fewer surface-level statistical regularities and instead develop higher-level conceptual understanding (Jo & Bengio 2017). Interestingly, calibration error is only

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: QUBA score and quality dimensions for the five top- perfooffonnng models, The configuration lists the architecture, training dataset, and training paradigm. iidiccttss models trained with knowledge distillation.
Model
Configuration
QUBA, Score
Acc.T
Adv.
C-Rob.t
OOD,
Roh.
Rvror
Cal.
Ralance T
Cleas
Obj. Pocus
Shape, Bila
Pacama, in MIL.
CNN, IFT- 10T--00M
Neectent
Hinton 2015
0.25
0.0048
INT: Transformer IN1,,
MisneB
Ryral et A
2023
welf-S1 (R26)
0.85
0.55
0:23
0.76
0.76
0.0130
0.93
0.94
0.34
51
ComrNeXeV22:1
ONN, IN21 set--5 (R26)
Woo
2023
0.96
0.87
0.28
0.79
0.82
0.0023
0.81
0.96
0.40
98
[tyalt
(122))
0.85
0.24
0.74
0.0130
0.03
BVA02-B/1
Pang
202th
Transformer, IN21K, self-St, (R22K)
1.08
0.88
0.21
0.86
0.0058
0.83
0.97
0.34
87
statistically significantly correlated with OOD robustness, class balance, and shape bias, highlighting the need for dedicated calibration research. Lastly, most considered quality dimensions (excluding the number of parameters and calibration) improve together, indicating that developing models that excel in wide range of quality dimensions is feasible. This observation aligns with Liu et al. (2023), who argue that many desirable properties of trustworthy machine learning are underpinned by shared foundations, suggesting that improvements in one aspect may benefit others.
Conclusion: Accuracy and class balance are strongly correlated, object focus is strongly correlated with most quality dimensions, calibration error is not correlated with most quality dimensions, and there are only a few trade-offs between the considered dimensions.
5 Which backbone to use?
We conclude our analysis by ranking models to provide recommendations on the best model choices. Ranking models across multiple quality dimensions is non-trivial task with no fiessile-itts-all solution, as user priorities vary depending specific needs. Nonetheless, aim to identify models that perform well across wide range of dimensions and, thus, require effective way to summarize the different quality scores with flexible weightings to reflect different user nceds.
Probably the most straightforward way to summarize our results would be to take the average of all quality dimensions. However, given that these dimensions have vastly different ranges and scales, this approach would not treat all dimensions fairly. Another alternative would be to compute the mcan rank: for cach quality dimension, the models are ranked, and then the geometric mean of these individual ranks is calculated. However, using ranks has two key limitations. First, ranks uniformly distributed, whereas the raw scores are not, meaning that the difference in mean rank between two models would not accurately capture the actual difference in their model quality. Second, if future studies introduce new models, the set of models will change, altering most of the rankings. As a result, mean ranks would no longer be consistent across different papers.
QUBA score. To address these issues, we leverage intriguing property of our large model zoo: its size makes it representative of a broad range of models, enabling us to estimate meaningful mean μ4 and standard deviation a, for each quality dimension i (we exclude the bottom and top 10% models to reduce outlier sensitivity). We then express each model's quality scores smodel in terms of how many standard deviations they deviate from the mcan. The final QUBA score (Quality Understanding Beyond Accuracy) for model is the weighted arithmetic mean of these scores:
model 14
s
QUBA moddll
Σ
w
(1)
i=
Wi
By default, we use a balanced weighting where the three robustness dimensions are weighted at w; 1/s prevent them from overshadowing the results and, similarly, assign w; 1/2 to object focus and shape bias.
10

---

Published in Transactions on Machine Learning Research (01/2026)
both are related to shortcut learning. All other weights set to 1 (different weightings analyzed below). Since calibration errors and the number of parameters should be as small as possible, they are multiplied by -1 before computing the mean, so that higher values indicate better performance. Intuitively, the QUBA score reflects how many standard deviations a model deviates from the "average model" across the considered dimensions.
Our approach solves both limitations of the mean rank: the distances now have a consistent and meaningful interpretation, and the score can be calculated independently of the considered model set (since the mean and standard deviation for each quality dimension are assumed to be fixed).
To validate that our model zoo is large enough to produce reliable estimates of the mean and standard deviation for each quality dimension - and to assess the robustness of the QUBA score we randomly sample 100 models from our full model zoo and compute the QUBA mean and standard deviation. This process is repeated five times. For cach of the five resulting QUBA variants, we rank all 326 models and compute the rank correlation between the resulting rankings. The average rank correlation is very high (0.97), indicating that the QUBA rankings are stable and not overly dependent on the specific subset of models used.
The best models. We report results for the top five QUBA score models in Tab. 3 Of our 326 models, EVA02-B/14 (IN2Ik) (Fang et al. 2023) achieves the best QUBA score. Compared to the other top- performing models, achieves the highest accuracy. C-robustness, OOD robustness, and object focus. It lags behind in adversarial robustness, calibration, class balance, shape bias, and the number of parameters. The second-best model, Hiera-B-Plus (Ryali et al. 2023), ranks lower in accuracy and calibration but performs well in the other dimensions, excelling in class balance and shape bias. In third place, the convolutional model ConvNeXtV2-BB (IN21k) (Woo et al.) good rcsults in all other dimensions but the paramcter count. The last two models, Hiera-B (Ryali et al. 2023) and NeficienNNet-B6 (Xie el al. 2020 have particularly low parameter count. Remarkably, all five models have been trained semi- or sueevsuerr making these promising training paradigms for developing well-behaved models. Our analysis highlights that even the five top-performing models vary strongly among the quality dimensions, highlighting the need to consider wide range of quality dimensions simultancously in the design process of new models.
2023) leads in adversarial robustness and calibration while achieving
Conclusion: The models with the highest QUBA scores excel across various quality dimensions, with each model showcasing distinct strengths.
A closer look at popular models. There are many popular models that did not make it into the top five above. We here go over some of the most. popular models and briefly discuss their performance according to the considered quality dimensions (see Tab. 4). SwinV2-b/12ton (Liu et al. 2022a) is the best supervised model, particularly excelling in accuracy and object focus, while having quite a large number of parameters. DINOv2-B-res --rregB--rgg (LP) (Darcet al.] 2024] exhibits very good calibration and achieves good results in most other metrics. VIT-b/1G-MAE (E2F) (He et al., 2022 is in dimension particularly good bad. Although ViT-b/16 (Dosovitskiy et al. 2021) and ResNet50 (He et al. 2016) are still two of the most popular backbones, they perform quite poorly, with QUBA ranks of 124 and 214, respectively. The ResNet50 has a comparably low number of parameters. CL.IP-I/114 (Radford et al. 2021 suffers particularly in the calibration error and the number of parameters. On the other hand, exhibits a high shape bias. Based on these findings, we suggest that the vision community should recousider its selection of canonical backbone models.
Widely used models such as ResNet50 and ViT underperform in several of the evaluated quality dimensions. This suggests that the vision community should critically reconsider its choice of canonical backbone modcls.
Conclusion:
Different weightings. As outlined above, different practitioners could have different requirements on their models, depending on the task at hand. To reflect this in our analysis, in Fig. 5 we plot the five top-performing models when weighting one (group) of the considered quality dimensions twice as much
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: QUBA score and quality dimensions for particularly popular models that did not make in the top five. The configuration lists the architecture, the training dataset, and the training paradigm.
Model
Configuration
QUBA Seoret Maakk]
Aco.t
Advit Rtb. 1
C-Rob.t
OOD,
Rob.
Cal. Error
Class Balance
Obj Focux
Shape, Bins
Pareuss, In Mil.
CLIP-1/114
Redford et nl.
2021
Vil, WiT-
Radford
2021
-0.65/263
0.76
0.32
0.76
1.04
0.0110
0.89
0.94
0.50
427
NosNot50
2016
CNN, INIK,
-0.31/214
0.76
0.03
0.51
0.50
0.0021
0.38
0:22
25
VIT-bb/16
Donovitekly at ml
2021
Transformer INIK, 8L
0.20/124
0.81
0.18
0.06
0.56
0.0084
0.79
0.98
6.40
80
He ct ml.
2:7-b/16-MAE
2022
Transformer.
INIk, self-81 (E2E)
0.30/84
0.84
0.25
0.71
0.58
0.0040
0.80
0.95
0.36
80
OINOv2-Par
Daanoet
202.
LVD142
0.74/25
0.85
0.79
0.79
0.0011
0.49
Suin Riil
Liu 2022
Tramfformer. 1N213, SL
0.00/8
0.86
0.26
0.81
0.81
0.0040
0:41
the other dimensions when computing the weighted mean for the QUBA score. EVA02-B/14 (IN21k) (Fang ct al. 2023 leads in five setups, highlighting its versatility and quality beyond accuracy. The top five models remain fairly consistent when emphasizing accuracy, robustness, calibration, and shorteut learning, though their ranking within the top five varies slightly. Interestingly, the Hiera (Ryali et al. 2023) model family, self- supervised models only trained ImageNet-lk, dominates strongly when focusing class balance. Besides for class balance, the training datasct and architecture quite heterogencous for most of the setups. For the training paradigm, semi- and self-supervised learning dominate.
Conclusion: When focusing on specific quality dimensions, the top five models remain fairly stable. For class balance, the Hiera model family is dominating.
Limitations. Naturally, our work comes with limitations. First, while we focus on image classification, which certainly relevant field, some downstream tasks rely on the evaluated backbone models for other purposes, and there is guarantee that our findings will directly translate. Second, similar to the previous point, our analysis is limited to models trained on ImageNet- lkk and we cannot guarantee that the results generalize to other datasets. However, extending our analysis to another dataset is challenging: assumning an average training time of only 10 hours per model, retraining all 326 models on another dataset would require 3260 hours (~ 136 days) of compule, which is infeasible with our compule resources. Further, Image 1k remains highly relevant, with numerous impactful papers focusing primarily on it and its variations. Third, we acknowledge that there are numerous different protocols to assess diffcrent dimensions of DNN quality. While cannot include all protocols, (iii) easy-to-use (requiring no fine-tuning) protocols. However, our benchmark can oasily be adapted or even extended with other protocols and quality dimensions. Fourth, to provide a comprehensive bird's bird's-eye view, we prioritize breadth over depth here. While we briefly discuss various findings and why they might occur, this means we cannot. provide detailed analyses for specific observations indeed, studying theoretical connections for just two quality dimensions often the scope of full paper (c.g.. Minderer et al.j 2021; Tsipras et al., 2019 Xiao et al. 2021p. We rather consider this paper as ground paving the way for future research to conduct more fine- grainrai in-depth investigations. Fifth, our chosen evaluation protocols capture only certain aspects of the evaluated dimensions, and different protocols might yield different conclusions. That said, our benchmark design and its simple applicability allow us to conduct one of the largest studies to date. We thus argue that our design choices are justified and that our analysis makes numerous valuable contributions.
aimed for (i) representative selection of (ti) established and
6 Conclusion
In this work, we provide a bird's-eye view of nine quality dimensions for ImageNet-lk image classification across 326 vision backbones by conducting one of the largest studies to date. This broad perspective allows us to examine how various training strategies and model architectures impact these dimensions, finding that larger training datasets and self-supervised pre-training followed by end-to-end fine-tuning enhance almost all
12

---

Published in Transactions on Machine Learning Research (01/2026)
2.6
1000
5020
EVA22 8/14 [2nk c0c
Dit Saall
Contirsty teieerr tons ran pom
Hice Roll na Faros
Hier' Relltee eagieet
Werer 2023
tie per Sell COrE
Herrs-uli (eg Polusa 2ON Ellicet DNX
200
(Roil te 20100
Trayvi 21M NA S. 2M/2
2.2-
Hos Pho (or
PMA TTrg
conr va Wo
QUBA Score
1.8-
Heoo B (hlll
Hes il liatl
Hhe 10 Phs
BVAio 14
Fitrent t8
[Ball
Books 8,16
1.4
Elioett
EWMe
EUGGo STA
0.6
Equal
Acc.
Rob.
Cal.
Class
Shortcut Params.
Weight
Balance
Learning
Figure Top five QUBA score models under different weightings. We report the top five models when weighing specific (groups of) quality dimensions twice as strongly. See Fig. 3 for color and marker details.
measured quality dimensions. Additionally, we explore the relationships between these quality dimensions, providing novel insights such that object focus is strongly correlated with most of the considered dimensions. Our analysis is rounded off by ranking models based on our proposed QUBA score, which is possible due to our large model zoo. We highlight that no single model is universally superior and instead provide recommendations on which models excel for specific requirements. To conclude, we encourage researchers to consider a broad range of quality dimensions together, rather than focusing on individual ones, to foster the development of more well-behaved image classification models. Our work facilitates this by offering an easy-to-use use benchmark, along with comprehensive analysis of how design choices influence these quality dimensions, their interrelationsiis, and how models can be ranked across multiple quality dimensions.
Acknowledgments
RH and SR have received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programmn (grant agreement No. 866008). SSM has been funded by the Deutsche Forschugsgemeiisscaatt (DFG, German Research Foundation) project No. 529680848. Further, SR was supported by the DFG under Germany's Excellence Strategy (EXC 3066/1 "The Adaptive Mind," " project No. 533717223). Additionally, SR and SSM have received funding from the DFG under Germany's Excellence Strategy (EXC-3057/1 "Reasonable Artificial Intelligence", Project No. 533677015). Moreover, DB is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
References
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégon. XCTT: Cross-covariance image transformers. NearIPS, pp. 20014-20027, 2021.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In ECCV, volume 12368, pp. 484 501, 2020.
Yutong Bai, Jieru Mei, Alan L. Yuille, and Cihang Xie, Are transformers more robust than CNNs? NeurIPS, pp. 26831-26843, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furn Wei. BEIT: BERT pre-training of image transformers. In ICLR, 2022.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Robustness may be at odds with fairness: An empirical study class-wise accuracy. In NewrIPS, pp. 325-3-3-1 2020.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Wieland Brendel and Matthias Betlgee Approximating CNNs with Bag-ofllocll-Feature models works surprisingly well on ImageNet. In ICLR, 2019.
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment is all we need for interpretability. In CVPR, pp. 10319 10328, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9630-9640, 2021.
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. CrossViT: Cross-attention multi-scale vision trans- former for image classification. In ICCV, 347-356, 2021a.
Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. PaLl: A jointly-scaled multilingual language-image model. In ICLR, 2023.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In 1CCV, pp. 9620-9629, 2021b.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilaarco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt,, and Jenia Jitsev. Reproducible scaling laws for contrastive languago-image learning. In CVPR, pp. 2818-28201 2023.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pp.
2017.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In ICML, volume 119, pp. 2196-2205, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chi- ang, Prateek Mittal, and Matthias Hein. RobustBench: standardized adversarial robustness benchmark. In NewrIPS Datasets and Benchmarks, 2021.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, In JCLR, 2024.
Stéphane d' A'Acooli Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. In ICML, pp. 2286-2296, 2021.
Mostafa Dehghani, Yi Tay, Anurag Arnab, Lucas Beyer, and Ashish Vaswani. The efficiency misnomer. ICLR, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. DaVIT: Dual attention vision transformers. In ECCV, pp. 74-92, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Mengnan Du Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: A computational perspective. IEEE Intell. Syst., 36(4):25 34, 2021.
Logan Engstrom, Andrew Tlyas, Shibani Santurkar, Dimitris Brandon Tran, and A Aleksander Madry. Adver- sarial robustness as a prior for learned representations. arXi::1966.0080 009450000......0.0000000... [stat.MI.J, 2019.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024a.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVArrobustness met limits of masked visual representation learning at scale, In CVPR, pp. 19358-19365 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation [or neon genesis. Image Vis. Compall. 149:105171, 2024b.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M. Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannanch Hajishirzi, Ali Parhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In NewrIPS Datasets and Benchmanks, 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trainee CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wich- mann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NewrIPS, pp. 23885-23899, 2021.
Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopad- hyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, and Tom Goldstein. Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks. In NewrIPS, 2023.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Julia Grabinski, Paul Gavrikov, Janis Keuper, and Margret Keuper. Robust models are less over-confident. In NeurIPS, 2022.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeVIT: A vision transformer ConvNet's clothing for faster inference. In ICCV, pp. 12239-12249, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, pp. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 15979-15988, 2022.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks, In CVPR, pp. 558-567, 2019.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS, pp. 15637-15648, 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Takshminarayanan. Aug- Mix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Dan Hendrycks, Sleven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. Tn ICCV, pp. 8320-8320 2021.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, pp. 11916-11925, 2021.
Katherine I.. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. In NewrIPS, 2020.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Content-adaptive downsampling in convolutional neural networks. In CVPR Workshops, pp. 4544 4553,, 2023a.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. FunnyBirds: A synthetic vision dalaset for part-based analysis of explainable AI methods. In ICCV, pp. 3981-3991, 2023b.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Benchmarking the attribution quality of vision models. In NeurIPS, 2024.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. Xr:: 1503.000..22 02533 [L], 2015.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang. Liang-Chiel Chinh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for MobileNet In ICCV, pp. 1314-1324, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitationn networks. In CVPR, pp-
2018.
Gao Huang, Zhnang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp- 770-778 2017.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and less than 1MB model size. arXin:1622.00 07366 [cs.CV], 2016.
Jason Jo and Yoshua Bengio. arXis:1711.1156: [cs.LG], 2017.
Measuring the tendency of CNNs to learn surface statistical regularities.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CollR, 2020.
Hoki Kim. Torchattacks: A Py'Torch repository for adversarial attacks. trXii:2010.01930 fcs. LGJ, 2020.
Donald E. Knuth. Two notes on notation. The American Mathematical Monthly, (9(5):403-42)) 1992.
Alexander Kolesnikov, Lucas Beyer, Xiaolua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In ECCV, pp. 491-507, 2020.
Alex Krizhevsky, Ilya Sulskever, and Geolfrey E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, pp. 1106-1114, 2012.
Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. Uncertainty as a fairness measure. J. Artif. Intell Res., 81:307 335, 2024.
Balaji Lakshminarayannn, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, pp. 6402 6413, 2017.
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. MVTTv2: Improved multiscale vision transformers for classification and detection. In CVPR, pp. 1794 4804, 2022a.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. EffiiienFormmr: Vision transformers at MobileNet speed. In NewrIPS, 2022b.
Chang Liu, Yinpeng Dong, Wenzhao Xiang, Xiao Yang, Hang Su, Jun Zhu, Yuefeng Chen, Yuan He, Hui Xue, and Shibao Zheng. A comprchensive study on robustness of image classification models: Benchmarking and rethinking. Int. J. Comput. Vis., 133(2):567-589, 2025.
Haoyang Liu, Maheep Chaudhary, and Haohan Wang. Towards trustworthy and aligned machine learning: A data- centric survey with causality perspectives. arXio:237.168511 (ss. 2023.
Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9992-10002, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Niug, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling up capacity and resolution. In CVPR, pp. 11999 1200,, 2022a.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Zhuang Liu, Hanzi Mao, Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A CouvNet for the 2020s. In CVPR, pp. 11966-11976, 2022b.
Peter Lorenz, Dominik Strassel, Margret Keuper, and Janis Keuper. Is nobustbench//antotttae suitable benchmark for adversarial robustness? CoRR, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. In ICLR, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient CNN architecture design. In ECCV, pp. 122 138, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddäus Wiedemer, Evgenia Rusak, Attila Juhos, Matthias Bethge, and Wieland Brendel. In search of forgotten domain generalization. In ICLR, 2025.
John Miller, Rohan Thori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization. In ICML, pp- 7721 7735, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In NewrIPS, pp. 15682-15694, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NewrlPS, pp. 4696-4705, 2019.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. oiv:
orX:::1911.008: 1901.005320 /cs.LG], 2019.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, pp. 8690 8699, 2021.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Chassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, pp. 38-41, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Trans. Mach. Learn. Res., 2024, 2024.
Nicolas Papernol, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, pp. 582- 597, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Mar- tin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorrch: An imperative style, high-performance deep learning library. In NewrlPS, pp. 8024-8035, 2019.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furn Wei. BEIT v2: Masked image modeling with vector- quantized visual tokenizers. arxi::2008.00 06366 (cs.CV] 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. Tn ICML, pp- 9748-8763 2021.
Tlija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pp. 10425-10433, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpa- thy, Adilya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recoguition challenge. Int. J. Comput. Vision, 115(13):211 2522, 2015.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Chailanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkubandhu Chowdhury, Omid Poursaeed, Judy Hoffiaan, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A 2023.
hierarchical vision transformer without the wells-snd-whisllss. In ICML, pp.
Hadi Salman, Andrew Ilys,, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NewrIPS, 2020.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. MobileNet Inverted residuals and linear bottlenecks. In CVPR, pp. 4510 4520, 2018.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In NewrIPS, 2022.
Baifeng Shi, Dinghuai Zhang, Qi Dai, Zhanxing Zhu, Yadong Mu, and Jingdong Wang. Informative dropout for robust representation learning: shape-bias perspective. In JCML, pp. 1828-8839, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Naman Deep Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for ImageNet: Architectures, training and generalization across threat models. In NewrIPS, 2023.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, augmentation, and regularization vision transformers. Trans. Mach. Learn. Res., 2022.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning In ICCV, pp. 433-852,42 2017.
Christian Szegedy, Wojeiech Zaremba, llya Sulskever, Joan Bruma, Dumitru Erhan, lan J. Goodsellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zhigniew Wojna. Rethinking the Inception architecture for computer vision. In CVPR, pp. 2818-2826, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, Inception-ResNet and the impact residual connections on learning. AAAT, pp. 4278 4284, 2017.
Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICML, volume 97, pp. 6105-6114, 2019.
Mingxing Tan and Quoc V. Le. V2f:cientNetV2: Smaller models and faster training. In ICML, pp. 10096-10106, 2021.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-aware neural architecture search for mobile. In CVPR, pp. 2820 2828, 2019.
Yi Tay, Mostafa Dehghami, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning trans- formers. ICLR, 2022.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Commun. ACM, 59(2):64477, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347-10357, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 32 12, 2021b.
Hugo Touvron, Matthieu Cord, and Hervé Jégon. DeiT III: Revenge of the VIT. In ECCV, pp. 516 533, 2022.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénalf, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual lasgunl encoders with improved semantic understanding, localization, and dense features. arXiv:2502. 14786 (cs.CV), 2025.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy. In ICLR, 2019.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (CLIP). In NewrIPS, 2023.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan C. Bovik, and Yinxiao Li. MaxViT: Multi-axis vision transformer. ECCV, pp. 159 479, 2022.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Amurag Raujan. FastViT: A fast by brid vision transformer using structural reparameterization. In ICCV, pp. 3772- 2023.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Ravileja Vemulapalli, and Oncel Tuzel. MobileCLIP: Fast image-text models through multi-modal reinforced training. In CVPR, pp- 15963-15974, 2024.
Sahil Verma and Julia Rubin. Fairness definitions explained. In Fair WCre pp. 1-7, 2018.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, pp. 10506-10518 2019.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR, pp. 8681-8691, 2020.
Ross Wightman, Hugo Touvron, and Hervé Jégou. ResNet strikes back: An improved training procedure in timm. trXiv:2110004470 CV., 2021.
Sanghyun Woo, Shoubhik Debuath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vNeXt V2: Co-designing and scaling convnets with masked autoencoders. In CVPR, pp. 16133 161/2 2023.
Mitchell Wortsman, Gabriel Ihaarco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, pp. 23965- 23998, 2022.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tiny VIT: Fast pretraining distillation for small vision transformers. Tn ECCV, pp- 68- 85, 2022.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Iyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR, 2021.
Qizhe Xie, Minh-T Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR, pp. 10684 10695, 2020.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 5987 5995, 2017.
Han Xu, Xiaorui Liu, Yaxin Li, Anil K. Jain, and Jiliang Tang. To be robust be fair: Towards fairness in adversarial training. In ICML, pp. 11492 -1501, 2021a.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. ICLR, 2024.
Weijian Xu, Yifan Xu, Tyler A. Chang, and Zhuowen Tu. Co-scale Conv-attentional image transformers. In ICCV, pp. 9961-9970, 2021b.
1. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXic:1905.005. [cs. Cv., 2019.
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look accuracy vs. robustness. In NewrIPS, 2020.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Yang You, Igor Gilman, and Boris Ginsburg. Large batch training of convolutional networks, arXie: 033888 [Vs 2017.
Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. InceptionNeXt: When inception meets ConvNeXt, In CVPR, pp. 5672-5683, 2024.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. In ICCV, pp. 538 547, 2021.
Sangdoo Yum, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regu- larization strategy train strong classifiers with localizable features. In ICCV, pp. 6022 6031, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
Xiaohua Zhai, Basil Mustala, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pp. 11941 11952, 2023.
Zhuotun Zhu, Lingxi Xie, and Alan L. Yuille. Object recognition with and without objects. In IJCAL, pp. 3609-3615, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In CVPR, pp. 8697 8710, 2018.
20

---

Published in Transactions on Machine Learning Research (01/2026)
"Toy Poodle"
Adv. attack
"Bucket"
Figure 6: Illustration of adversarial attack.
A Details on the considered quality dimensions
In the following, we provide more details on the considered quality dimensions and the corresponding evalu- ation protocols. We let f be the model of interest and use the ImageNet-Ik evaluation split with N images {an n Ee.....N belonging to one C' classes {ea, 11.... C n I....N for most protocols.
Accuracy. To measure the predictive performance of the considered models, let denote the Iverson bracket (Kuuhh 1992) and report the ImageNet- 1k top-1 accuracy
1 N
A
f((x)) cn)
(2)
Adversarial robustness. To assess the adversarial robustness of a model, we use the two popular attacks, FGSM (Goodfellow et al. 2015) and PGD (Madry et al. 2018). In the Fast Gradient Sign Method (FGSM), adversarial examples are generated by computing the sign of the gradient of the cross-entropy loss L with respect to the original imput, scaling with a small factor e, and adding the result to the original image (see Fig. Gb. Formally, we obtain the FGSM accuracy (FGSM-A) via
N FGSM-A
Σ [f(tn)
=cl cnl.
(3)
N R=1
with 2n = x 6 signn sign(V.C). Projected Gradient Descent (PGD) extends FGSM by applying it repeatedly, yielding
N PGD-A
Σ (()!!) = cn n=
(4)
N
with in (1+1) = (0) + c sign V 20 C) and ((0) = z. We use e = 8/253 and I = 10 (Kim 2020 To reduce the dependence the clean accuracy of the model, report adversarial robustness relative to the accuracy A from Eq. ( We combine the results from the two attacks using their geometric mean (GM), resulting in the final adversarial robustness
AR = GM FGSM-A PGD-A A
A
Corruption robustness. To assess a model's robustness to common corruptions (CR) like JPEG com- pression or contrast changes (see Fig. 7 we measure the accuracy on ImageNet-C (Hendrycks & Dietterich 2019), i.e., the ImageNet evaluation split with different corruption types of incrcasing strength. We here fol- low Hendrycks Dietterich (2019) and the standard mean instead of the geometric mean to summarize the results for different corruption types and strengths. To normalize the C-robustness and to be consistent with our other robustness metrics, we deviate from Hendrycks & Dietterich (2019) and again report the top-1 accuracy on the corrupted data (Acor)) relative to the clean ImageNet- 1k accuracy (A), yielding
CR = AC A
(6)
OOD robustness. To measure the out-of-domain robustness a model, we report the goometric mean of the relative accuracy (normalized by A) on five out-of-domain datasets. Specifically, we use ImageNet- R (Hendrycks et al. 2021), Neagehht-S (Wang al. 2019), well Stylized- ImageNe, Edge, and
21

---

Published in Transactions on Machine Learning Research (01/2026)
Naise
Blur
Brightness
Contrast
JPEG
Pixelate
Snow
Fog
Figure 7: Erample corruptions from ImageNet-C [endrrykk & Dietterich 2019
ImageNet-R
ImageNet- Sketch
Stylized- ImageNet
Edge
Silhouette
Figure 8: Example images from the considered OOD datasets Wang et al. 2019).
GGeirhos et al. 2019 Hendrycks et al. 2021
Silhouette from Geirhos et al. 2019). For MaageeNet-Sketch and Stylized- Stylized-Imagehle ImageNet, we use the versions also usod in Goirhos et al. (2021) Please refer to Fig. g for example images of the different datascts.
Calibration error. Calibration means that the output confidence of model faithfully reflects the prob- ability of the prediction being correct. We use two established metrics for measuring the calibration error (CE). The expected calibration error (RCE) (Guo et al., 2017 Nixon et al. 2019h divides the predictions into bins based on the output confidence of the model and compares how well the confidences conf(b) of the predictions in that bin are aligned with the accuracy As of the predictions that bin:
B Σ
ECE
Thh N
As cco(((](
(7)
with ne denoting the number of predictions in bin b. Since common criticism of the ECE is the use of fixed bin range, we additionally report the adaptive calibration error (ACE) (Nixon et al. 2019) that measures the discrepancy between Arc and conf(r, c), ie,, the accuracy and confidence of images in the adaplive calibration range for class label c:
C
ACE
Ar. conf((r,)
(8)
CR c=r==l
As in Guo et al. 22017 Nixon et al. 2019 use 15 bins for both protocols and again report the geometric mcan (GM) of both errors, i.e.,
CE GM(ECE,ACE)
Class balance. We consider model fair if none of the classes is classified less well than the others Benz et al. 2020 We evaluate the class balance of accuracies (FAce) of model similar to Croce et al. (2021)
and subtract the standard deviation of TmageNet- 1k class accuracies from (this ensures that higher scores indicate a higher class balance; the standard deviation cannot exceed 1):
1
F FAc 1
Σ
(A A)2
(10)
22

---

Published in Transactions on Machine Learning Research (01/2020)
Unfair model
Fair model
0.8
Accuracy
0.6
0.4
0.2
Class
Class
Class
Claas
Figure 9:
Visual illustration of the class accuracies of fair model vs. unfair one, both with equal average accuracy.
Original
Mixed- Same
Mixed- Rand
Figure 10: Example images from Xíao et al. 2201 to estimate the object focus.
with Ac denoting the accuracy for images of class c. Intuitively, a high value indicates that the accuracies of each class are similar, and thus, the model behaves fairly, as illustrated in Fig. g Similar Kuzueu let al. (2024), we also consider model fair if the average confidence (target softmax outputs) for each class is balanced. We compule the class balance of confidences (FCont) of model by subtracting the standard deviation of ImageNet- 1k average class confidences from 1:
1 V
Fonn =1
E (Confe Conf)?:
(11)
with Confa denoting the average confidence for images of class and Conf denoting the average confidence for all images. The final class balance score (F) is the geometric mean (GM) of Foe and F Foonf, i.e.,
F = GMM(Faccc Acc, FCont).
(12)
Object focus. To compute the object focus (OF), we first compute the background focus BF AMixHn-SAMI: AMINED-RAN (Xiao et al.] 2021). MIXED-RAND a dataset where image backgrounds are substituted with backgrounds from random classes and, therefore, contain class information (see Fig. I0 MIXED-SAME is dataset where image backgrounds are substituted with backgrounds from the same class to account for editing artifacts in MIXED-SAME. Intuitively, we measure the drop in accuracy when changing the image background with the background from another class to assess the model focuses on background signals. Next, we compute the inverse of the background focus to obtain the object focus OF
BF.
Shape bias. Geirhos et al. (019)) showed that ImageNet-trained CNNs exhibit strong texture bias, meaning that decisions are formed on the basis of texture information rather than shape information. As stronger shape bias is said to be advantageous for robustness and more in line with how humans form decisions, we follow Geirhos et al. (209)) and report the shape bias (SB) as follows:
Σ [f(#m)
(shape)
SB
(13)
(shape)] Σ m=1 (f((ām) = c
+ [ā = Cm e (texturee))
with F, being synthetically generated images with a texture-shape cue conflict, i.c. where the shape from one class of (shape) and the texture is from another class en
(texture) e.g., as in Fig. 11
Parameters. As memory efficiency and inference time depend highly on the implementation and hardware used, impeding future comparisons, we report the number of parameters proxy.
23

---

Published in Transactions on Machine Learning Research (01/2026)
Shape image
shape cue conflict
image
Figure 11: Example images from Geirhos al. 2019 to estimate the shape bias.
Adherett
Acouncy
Archiactures:
Dutasets:
ImageNet--k G Ima
Large Dataset [Laion28, JFT-300M,
Training:
Supervised learning
training
Self-superiied leaming
Semi-supervised leamiin
X-Axie:
Accuracy
Adversorial Rebusiness
Comupttin Rebustnoss
000 Robusiness
Calibration Error
Class Balance
Deject Focus
Shape Blas
Parameters
QUBA
Accuracy
Adversorial Robusiness
Comupptin Robusttnes
000 Robasinass
Calibration Enar
Clasa Belance
Object Focus
Shape
Parameters
Figure 12: Preview of our interactive plot. can be found in the supplement under interactive_plot. html or on our project page.
B Interactive scatter plot
In Fig. 3 of the main paper, we only plot representative subset of models to reduce clutter. the supplement, we additionally include an interactive plot, called interactive_pe patplpplooohhe. .tmml, that can be opened with standard browsers. includes all 326 models and allows the filtering of the models based on the training dataset, training paradigm, and architecture. Also, different quality dimensions can be chosen for the æ and y-axis to visualize different relationships. Hovering the cursor over marker reveals tooltip displaying the model's name and scores for the considered quality dimensions, offering detailed performance insights at glance. A preview of the interactive plot is included in Fig. 12
C Additional experiments
C.1 Relationships in model subgroups
During our analysis, we noticed that the rank correlation matrices for certain subgroups of the models can change compared the correlation matrix of all models shown in Fig. X of the main paper. This has important implications: First, it is crucial to look at many models possible to make general statements,
24

---

Published in Transactions on Machine Learning Research (01/2026)
Class Balance
Class Balance
Class Balance
Accuracy
Adv. Rob
000 Rob.
Cal. Error
Ob) Foais
Shape Bias
Parameters
Accuracy Adv. Rob.
000 Rob.
Cal. Error
Obi. Facus
Shape Blas
Parameters
Azcuracy Adv. Rob.
ODD Rob.
Cal. Error
Obl Focus
Shape Bias
Parametora
C-Rob.
C-Robb
C-Rob.
Accuracy
Adv. Rob.
C-Rob.
0.5
0OD Rob.
Cal. Error
Class Balance
Obj. Focus
-0.5
Shape Bias
Parameters
CNN (n=170)
CNN INIK (n=140)
CNN IN21k (n=16)
Accuracy
Adv. Rob.
C.Rob.
0.5
OOD Rob.
Cal. Error
Class Balance
Obj. Focus
.055
Shape Blas
Parameters
Transformer (n=106)
Transformer INIK (n=84)
Transformer IN21k (n=19)
Figure 13: Rank corvelation malrices for model subgroups. We investigate the rank correlations of different quality dimensions for all CNNs (top left), all CNNs trained on ImageNet-lk (top middle), all CNNs trained ImageNet-21k (top right), all Transformers (bottom left), all Transformers trained on ImageNet- (bottom middle), and all Transformers trained on ImageNet-213. (bottom right). Crossed-out entries indicate p-value above 0.05, and thus, are not statistically significant.
which shows that our confirmation of (or contradiction to) existing results with a large model zoo is a valuable contribution. Second, general statements should be taken cautiously, as they do not necessarily apply to all model groups. We thus continue our analysis by studying the correlation matrices for different model subgroups. In Fig. 13 we plot the correlation matrices for all CNNs and Transformers, respectively. Overall, both matrices are similar to the matrix for all models (Fig. 4 of the main paper). However, for CNNs, C-robustness and OOD robustness are negatively correlated with adversarial robustness, while the opposite holds for Transformers. We observe that the calibration error is positively correlated with accuracy for CNNs and negatively for Transformers (however, both correlations are not statistically significant). This also aligns with another large-scale study that found that the calibration error is docrcasing with more accuracy for recent state-of-the-art Transformers (Minderer et al., 2021). Notably, accuracy and shape bias are strongly correlated for CNN-based models, while they exhibit only weak correlation for Transformer models. Moreover, for Transformers, an increased number of parameters is quite strongly correlated with desirable properties of all other quality dimensions, which is less pronounced for CNNs.
Given that a larger fraction of the newer Transformer models were trained on ImageNet-21kk, whereas most CNNs were trained on ImageNet-Ik, some of the correlations could also be due to the training dataset size. To account for this, in Fig. [3 further plot the correlation matrices for CNNs and Transformers trained exclusively on ImageNet-1k 1k and ImageNet-2lk,, respectively. Interestingly, the negative correlation between adversarial robustness and C-robustness/ OODD robustness for CNNs is only apparent when trained on ImageNet-1k but not when trained on ImaaeNee-21kk. Calibration error and OOD robustness have a strong negative correlation for ImageNet-21k CNNs while having almost no correlation for ImageNet- 1k CNNs (however, not statistically significant). Generally, the correlations for CNNs are much more pronounced for the models trained on the larger dataset. For Transformers, the statistically significant correlations are quite similar for models trained on ImageNet-1k and ImageNet-21k.
25

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Top 15 models with the highest OOD accuracy. Vision-language (ViL) models are clearly dominating the list.
Model
Configuration
OOD,
Acc.
SixLIP2-/118
Pelannen St
2026
ViL. WebLI (Chen 2023
(E2E)
0.57
MesCLLP--1/11 15
VIL. MetaCLir-aro
2022
0.85
MobiaCLiP-B (L.7
2021
VIL, DataCom
2224
se1f-S1.
0.55
CLIP.L/14- Commm
Prug el nL.
2024
VIL, DFN2B [Pan
(K2E)
0.54
SigL/P-1/16
(Zhal 6s al:
2078
VIL, WebLl (Grme Laaa 20755
0.53
CLPP.1/144-
LuMiis ot al
2023
VII. DataCompXt: (owath of sl
2023
self--31 (822R)
0.53
MobiaCLIP-B
Wazu et Bi.
ViL, DataCompDR
2024
vel6-SL
0.53
SigLiP2-b/116
ViL, WebL1 (Chan el al Puree
(E2E)
0.82
2022
Wortoman at al
2022 2022
ViL, Leiou28
2022).
sol--L
0.51
CLP-LL/14-Lann
VIL, Laten
self-SL (E28)
0.80
SigLIP-b/16 Zhat et
VIL. WebLi
0.50
C.MP.ComNNeTTr Nexrops
Wortsman
DataCom Lalondt Scmeppaar
solt-St
D.P--1//C-DAeCCOmpX PX1,
(822)
wTT-/M--DINDe2
zelf-81. (RAR)
McwcLLI-B/1B
Xu
MetaCLIP-44M
C.2 OOD robustness for models trained on large-scale datasets
In Sec. 3 of the main paper, in the paragraph about vision-language (ViL) models, we state that vision- language models outperform other models trained on similarly large datasets when it comes to oft-of-domain robustness. In Tab. []] we support this statement with numerical results. To this end, we report the "raw" OOD accuracy, i.e., the OOD accuracy without normalizing by the clean accuracy, for the 15 models with the highest OOD accuracy. The best. ViL model achieves an OOD accuracy of 0.87 while the best self- supervised model (pre-)trained vision-language models clcarly dominate in the list (14 out of 15). These results suggest that the incrcased robustness of ViL models is not only due to the increased dataset size but also due to other factors that are likely linked to the language part of the models.
large-scale dataset only achieves
OOD accuracy of 0.79. Further,
C.3 Reproducing conflicting results
We report several results that extend findings from related work using smaller model pools. To verify these results, we reproduce their experiments with similar model pools in Tab. G When using comparable models, we successfully replicate most of their findings, indicating that the discrepancies arise from the limited model pools related work. This underscores the importance our large model zoo.
C.4 Comparison between our adversarial robustness protocol and AutoAttack
In the main paper, we employ a combination of FGSM and PGD adversarial attacks to assess adversarial robustness. However, within the communily, alternative evaluation standards have emerged, such as Au tack a widely adopted adversarial benchmark introduced in RobustBench (Croce et al. 2021). Tt. consists of an ensemble of four parameter-fie attacks: two PGD variants using cross-entropy and differennce-f-llgits ratio losses, targeted FAB attack (Croce & Hein 2020 and a black-box Square attack (Andrinshchenko) et al. 2020
We opted for a more straightforward setup, as we found the success rate of AutoAttack to be excessively high for our purposes - nearly all non-adversarially robust models are reduced below 0.1 accuracy, effectively collapsing. Moreover, prior work has criticized Auto for producing perturbations that significantly alter the input images, making adversarial examples casily detectable, and for exhibiting scnsitivity to image resolution (Lorenz et al., 2021).
Nevertheless, given that AutoAttack remains important benchmark in adversarial robustness research, we include a comparison between the results obtained using AutoAttack and those produced by our proposed evaluation prolocol in this section. First, we evaluate how different training paradigms and architectures (cf. Sec. 3 affect adversarial robustness under both our protocol and AutoAttack, as shown in Tab. 7 For nearly all configurations, the overall conclusions remain consistent. Only in sctups (c) and (f) do we observe improvements adversarial robustness when using our protocol, while robustness under Attak remains unchanged (though these differences are not statistically significant). Next, compare the correlation
26

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Comparison of our results that amtrrdict/extend findings from related work. the main paper, we highlight a few findings that diverge from results in related work. To verify these discrepancies, we reproduce their findings using similar model pool. In doing so, we find that most of their conclusions hold when considering similar models.
Fioding
models related work
Reproduction simii-
findiog
Used modols
work
models
Adversarial Train- lng improves cali-
ReNNe1s, ResNet50 Be
es nh.
2010
WRN-50-2
Average ECE
Adversorial training ims- pairs colibration w when cousidering u laroad and diverse ant models
See Tab. [c)
Eagorayko Komodaku 2010
Staudard Modelo; 0.2722 Robust Mudelo: 0.0876
bralion Sztot
w
blacks at
ResNett, RenNer30. WRN--0-2 from Saiman al 2020 condacted additiponl
Conclusion: We can repro- duce thetr Rading when ua-
NOTE: Grabmnsk
pertment
model pool
Longer Ing PDP sl
srain-
Five selEtrrined versions of BiT-L-R50x1 and BiT-L-
Average ECE
Longer training im- prove calibration eeror when considering broad and diverse net of medels
See Tab.
worsons
R101x3 (Mindeser et all [021
Standard modele: 0.0396
(c)
Minderer
NOTR: Stire the mesd or Rimilr chacknoints are not pub-
A[I] modele: 0.0515
2021
hely available, we use the A[1,2] (Wightman et al 2021
Aj2] modele: 0.3025
versions of ReoNei18, ReaNei34, ReaNeeau, RCINe To
Conclusion: We can repro- duce their finding when un- ing stmilar model pool
and ResNet152. which are at leass somewhat shadlar to
Kolumlov
2020 for zepreduction
Ancuracy poe-
Net1e [liangg al. 2017 RecNet152 Iee al.
Accuracy ECC
We found that negatively correlated with calibration 0.05) whon com- sidering a broad and di- varse set of modell
Pattin model soo
Ittvaly correlated
2016
RoeNec152 (Iie all 2016
with GGuo
celibration
KOTE Guo es
conducted edditional experiments wish non- non-Inogee Imogengge monela
2017
0,7882 & 0.00
DamgNee161 (Honny ol. 2017): 0.7711 & 1.06
2017
Conclusion: We can not re- producs their fondag when nsing stmilar model pool
Adversorial
VCG18
Eimonvas
2015
VGG15
3
Corzelation belween ad- versarial roboxtnare and c-roburtnar:: 0.6547 (p
found siguificunt correlations betemen robust- nees and e-robustaess GOD no- bustnese (p=0 15) when comsidering e broad and diveres sel of models diverse
Eatire model
bustnese tively corralated
VGG
Xarrss All
2021
X
adversarial 0.0). Correiation between
with Cirobustuess
20211
XoTT-L Ai
OF
ResNet50
10
and 0OD robust-
20710
ReeNettor He
2010
ResNec152
10
adversarial robustness and
(p=0.84)
Boes for plven
A5
41
2016)
Wide Raldd-BoNan Nacin
Salmar es Al
2120)
OOD sohastners: 0.5842 (p=0.0)
urchilclure
L1u
[Huaai et al 2017
2024
C1
ml
2011
DemseNe
(Suuag et nl. 2017
Conclusion: We con re- produce their lioding when using stmilar modal pool
Coun CounNoN
2022
Cony
ConvNeX
NeN
Logv
Lau
2022b)
ConvAra
LaUl
2022) Vidrr Digor
VIT-
dorovitayy
VII (AAE
4022
2021
Swin-b
Lis
(21k)
Liu
(212)
Ne150
Mocoy]]
3
Cacc
A5 al
2021
1-21-
YoAR At
2021
T2T-19
YOAR
2012
T2T-24
YOBA
Swin-L
2021
matrices among our full model zoo (cf. Sec. with the two different adversarial robustness protocols in Fig. 14 Again, all the statistically significant conclusions remain the same. Lastly, we compare how model rankings change (cf. Sec. 5 under the two different evaluation protocols for adversarial robustness. While the rank correlation between the two adversarial robustness metrics remains fairly high (0.64), the overall ordering of models changes substantially. This shift occurs because, under AutoAttack, nearly the entire model 2oo collapses reducing the mean/standard deviation of adversarial robustness from 0.2/0.11 to 0.04/0.03. Consequently, the adversarially trained models that still achieve relatively high accuracies of around 0.6-0.7 deviate by many more standard deviations from the mean. As a result, the adversarial robustness score dominates the overall ranking, with only adversarially robust models appearing among the top five (see Table S Although this effect could casily be mitigated by reweighting our proposed QUBA score, as discussed in Sec. 5 we consider this finding an indication that AutoAttack is not the most suitable evaluation protocol for adversarial robustness within our benchmark. Nevertheless, aside from the unweighted rankings which must be weighted according to requirements in any case almost none of the conclusions in our work would differ under Autokt
D Experimental details
The code provided the supplemental material gives detailed instructions on how use our proposod benchmark and how to reproduce our main results. To include new models, one only needs to add the respective model file and weights.
27

---

Published in Transactions on Machine Learning Research (01/2026)
Cass Balanca
Class Balance
Accuracy
Adv. Rob.
000 Rob
Cal. Emror
Obi Focus
Shape Blas
Parameters
Accuracy
AutoAttack
OOD Rob
Cal. Eroo
Obj Focus
Shape Blas
Parameters
C-Rob.
C-Rob
Accuracy
Accuracy
0
Adv. Rob.
AutoAttack
C-Rob.
X
0
0
C-Rob.
0OD Rob.
OOD Rob.
Cal. Error
Cali.
Balance
Fairness
Obj. Focus
Obj. Focus
-0.5
Shape Bias
Shape Bias
Parameters
Parameters
Figure 14: Rank correlation matriz for the considered quality dimensions among our full model 200, comparing our adversarial robustness protocol to AutoAttack. We investigate differences of the rank correlations among our models with our adversarial robustness metric (left) and AutoAttack (right). Crossed-out entries indicate a p-value above 0.05, and thus, are not statistically significant. All the statistically significant conclusions remain unchanged.
Table Average adversarial robustness and AutoAttack for models with different configurations. We evaluate various sctups, focusing different training strategies (a-g) and different architectural choices (h-j). In each configuration, we report. the average score for both metrics across the models associated with that configuration. As different models are available for different setups, each setup considers a distinct selection of models being compared. As a result, both the models and their total number (indicated by the number beside each setup) vary across setups to maintain a fair basis for comparison. The number asterisks represents the statistical signíficance of differences the average scores of a quality dimension across configurations within each setup: for p 0.05, for p < 0.1, and for p 0.2, based on t-test results.
Sotup
Conffguration
Adv.,
Auto-
CNNs (IN-1K)
0.14
0.03
CNN: (IN-21k)
0.15
0.04
(b)
Transfarmess (IN-1k)
0.21
0.09
Transformess (IN-21k)
0.17*
0.05
(c)
Suparsteed models
0.12
0.05
Adversatially trained models
0.52
0.68*..
(d)
13 Supersieed models
0.20
0.08
Self-oupervieed models (I.P)
0 0.10
0.06
(e) Supersteed models
0.16
0.07
Salf-zupersieed models (F28)
0.24
0.07
(8)
13 Supervised model
0.18
0.04
Semi-ouper vised models
0.20
0.04
Supersieed models
0.03
supervised models (600 epoche)
0.47
0.05*
AG emperviedd models (300 epoche)
0..11 0.41
0.05*+-
Ad enpervtedd models (100 epoche)
0.32***
0.05"*
(b)
40 CNNs
0.11
0.11
Traneformers
0.20
Standard models Brooe models
0.08
2.00
0.00
()) Standard models
0.09
Viston- Viston-lanngngg langnage modele
0.01
To ensure comprehensive comparisons in future work, we publish the code and results of our model zoo (https://visinf wtwps//ff...ttuu....//eeeeoeoeee htt../oooooacccuuaaaeeeee the considered quality dimensions easily for their model. In the next subsections, we will describe additional experimental details.
This will allow practitioners to evaluate and compare
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: QUBA score and quality dimensions for the top fiee performing models using AutoAttack as metric for adversarial robustness. The configrration lists the architecture, training datasct, and training paradigm. indicates models trained with knowledge distillation.
Model
Configuration
QUBA Score
Acc.f
Rob.
C-Rob.t
OOD, Rob.
Cal. Brror
Cless Balance
Obj. Focue
Shape,
(Baggh
Transformer, IN1k,
Swin-B
Liu et ml
2026
Transformer, IN1k, AT
1.39
0.77
0.74
0.85
0.74
0.0083
0.79
0.94
0.73
87
IN1k, AT
Liu cl
CouvNeX
2025
CNN, IN1k, AT
1.45
0.77
0.73
0.84
0.62
0.0075
0.79
0.95
0.73
88
ONN, INIK, AT
D.1 Comparisons
In Tab. 2 of the main paper, we compare the average of each quality dimension for different setups. We report the models that have been used for each comparison in Tab. 9
29

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Models used for each comparison in Tab. of the main paper. In some setups, the same model appears multiple times within configuration because the corresponding configuration contains multiple models that need to be compared to that single model (e.g., there might be two different ViI. models using the same backbone). This duplication ensures that the final average accounts for these cases correctly.
Setup
* of Models
Conaguration
Modela
(b)
14
GNNs (IN-18)
VobiloNatt3-- (ooward et all (20
ReeNet50 He es
2016 ResNet301 [e
2010
(ann Loi 20211 BeeientNett-22aa
2021). BBickattii-
Con-XCXt-1
Loo et
2022D
ConrNeXX-s Lot ct na (0225)) CoN
NoX-L Clan
(2226).
3 CopyNeXi
NcXUV
CNNs (IN-21la)
Ne1N0111
BilicieuNNttwa
Tan
Thnn
2021
Ciu
2022h
MMi
2022h)
2029h
CoaeknXFl
A0225
Woo
2023
ConvADXIVE
Woo
2023).
CouvXeXtVy
B WWo al
2023
CouvNeXIV2
Woo
2023
Tramnfrmers (IN-1k)
De.33.o
Touvrou
DeiT3-u
Tourron et al
DeTaab Touvron
Tonyzon
SwinV2 2-071
(2022a) YinyYrlakky Wa
(Wa
Tonvron
20211
VTT-571B
Unsovissklty
Dirlie Darts
Togvoo
209Ta
Transformers (IN-212)
Touvron
Hlmnvron at Aizzi
Flunvrch
2022
Tonvion
SwinV2.b/ 12511 (La
2021
Thyy V12-ITky111 10
We at al. 2092) Tin TC wa (122),
VIT
Stemer
A1 m
3029
VIT-b/16
Minier et arall (2222))
(220)))) VIT-1/16 (Steiher of (PPP).
b/T2
Stelner
4 m
2022)
VIT-I/10
Sieiner et ah ahipue2) BurT-b [Bao st at
4022 EVA02-B/
FaRA
21 ml
20240
EVA02-8/11
14 [Fang et (024
Suparvieed madels
Zaporuyko
[ifa
Conv
CoRyNoXLLL
20221
ComyNeXr (L
Cony N
Cony
(2022h
ViT-b/1B
(oooovaaaayy
202
Dellor
2021a
Adveriatially trained modela
W
(Sltennr al 2020
Ressetu Eanna de
2020
20225
CeavXeXi-E Liu d.
2022),
CenrNeXi-L
(025)
CousNeXt-1 (Singh
2021
Nexts pingh dt
2022 CoxNext-l Mingh ot All
2023
Cony NeXI-1
M S1
2023
VIT-h/1B
Singh at Al 2020
VIT-a/18
Siaga dt Al
Suparvieed models
2021
2021n
Doriss
Hior-l
Iflera
V/H//16 (Dnovitakiy
VIT-b/16 /66
2011
aill flarvron
Pozial
Sell-aupervised model (LP)
Hezo-B
(Ryad at d. 2023 Vrr-b7iG H.E
al
2022 ReNNeL50-DINO CCrron et
2021
(Cave ol 2021),
(Ouuk
20/44 ViT-
6/14 DIEDDNND2LL.PP Ornak,
202 (TTha/11.DNNO221 (0min el P074
VTT-a/16.
DINO
(Caron et al 2022 J Hirre (Eyal: et ai 2029) Hiernes liyah at al 2l1es)
V-T-1/14-
DINOwI-Rg-Li
(Patee of al. 2024) VI1-B/14-DINUN Be--//1-DNN02-ee.....
Darrect et mi
2024
VIT-a/14-
DINDv2-rag-1.l Faaret At
25
Supervised modela
MVTTv2-b
et
VIT-b/16
(Deoviikkiy
2021) ReaNei50 [e
2010
Vri-b/16 [Dokovirakly
2021
CowVAnXFT (bm (a ComNNeXe-1
(0222)
2022
ConvNeXt-L Lan
ConvNeXE-T Lau al
20122 ConvNeXt-13
2022b
ConvNeXt-L Las al (2222) VTT-//88 (ttmeer
ViT-h/Ig Daxos
of (021) VIT-1y10 Seei all 2027)
al 2023
PA1 Staher Pi al (029), VIT-7166 (Joavitakly ds
Vii-b716
Desovitad nt
2031 Depioe Jouvron of al 2023
Vpari Hooordayy di al
202
papy
cl 0. 2021 Derss (rayvoo el
202 S Vri-E/16 ( [hooul el
2022
b/C
Doowwilkly et al 202)) ViT-b/32
(osoiithki et el [021
VeT-
Husovitokiy
At
ml
Self-mpaariied modela (E2E)
Maall
2023
ViT-b/IG.MAE He et
BerNetho--DNC Caron it
Bo- PPan
BVA02-3/14
Fuun
EVA02-B
20240) EVA02-1714
2024
2025)
BTIEDINDv2 PT (Dquas
202
VTI-1/14-
DINOv2-reg-IP
Darces
2000)
Unrrett
2011)
ViT-s/14-
DINO-2-ag-LP
Darcel
2024
BLiP-B/18-0pcsAba AL-PT. Vinton- Enceder
CCherti al.
2023 CLAPBB/
Vrrum Rncader (Chrrti es Al
2023] ChiPP- SaaP-R//aco/ppa A1A
CChrrti Bt al.
2023
(Chert: at
2021
Suparvized models
NetcientNet-Bi
EffccinnNet-Bi
2018
2019). Edtimenr
&
2019
RecNat50
(011.) Re (tin
2017. RecNatis U1
MeoNcX1100-222 Aie el
2017)
ReaNeXiplrsstee 2 Ale
2019
Sctsi-uperrised models
Nei-Bo
[Xi
2020)
EllcionnNet-1 [ie et
M020
EllicionNNet-P
Xie
e1
(0200 Notte Ki 2020) BlTeeeal Not-B Xie Xie al 2020) Ellicied
B5
Xie et al 20200 EftetentNa--B Bi
Ale al. 2020) Eshelenthar-
Ale at al 201
Bea Yannis et al 2019) KonNeXtsi
Salaiz et all
2019)
ResNer IA (aalad ct
2018
NeeNNeXTi 1A TAinle es al. 9018), RerreXtouRzz
Vanii es
2011)
30

---

Published in Transactions on Machine Learning Research (01/2026)
(x)
19
Superviacd modela
Neii (Tun & Le 2021 EdBeientNek-222
(Ta &
Le
2021 16N
Bu
2019), RogNeto-55l Radowvorii
2020).
[Rodowvovis
20220
loghet-y-lont [aadorvvvin at HarNes.v-aag
[Eadosavantc An
HotNetior
al (2019] RerNerls
2018
Set
R4sNet34
or (101g ReaNetōod
Rcmivel30 [He al porto
[Xle 2017
TTan
supervised modele
EitecionNee (Wixktmas SeNatiS3 (Wightman
2021
(Wishtanng
2023
ToogNety-igg (Wightmnn
[022]]
Sgf [Wightean al (222). ToENe--y-18gt Wightmat et (012) atgght-v--22et Wighht
(212))) HerRettoL (Wightman
12221 Heades (iighmms
TosNdloo W Wighhman es all 2021) Rea veag (V lighomnn at al Poll NorNatsth [W (ighhrmh
At mll
2021 Neeer Wirntman at al 2021 MeaNee Mineepr (Wishtman et all A0P3
EitecioniC- B0 [Wihtmmn ci af 2024
Edrdem Net-Bi Willtman et n. 2021 EliikeaNN
B2 (Wislimn cl ml (222)) Blickeatee--
Wighiman di al
N041 EibocentNet (Wight-
Rsee et al 2021
supervise models
EmectintiN Grard [Wightman
BffiiietNet--2-- (Wishtman
ScNot155 (Wightman a. 2021
HogNeb.y-ggf (Wishtman nt nll (2021 Reganty Wighrmal
Apf
Wiphrman
2021)
(Wis
Ner [Vightman
(Wichmann
B2 (Wightman (22z)) EifcestNee-B0 Wighemat
EihelenENate NeieenNNet-B.. Wiight-
A3 supervise models
202 Hegkatty Spf (Wightman at at (222) MorNe--y-15rt [Wighrmat at Al (222)) woy (Wirhs- Noe C1 or (222)) Recketo (Wightmen ot 12121 Ressers Wightman et ol
DflcccatNei.22.55 (Wichtann S4 (Wishtman of Al 2021 "eggRat---llt (Wightman at al 2021
EincemiNG -V2d (Wiskiman
al 2021
Rot
1 Viggllman mann es 2021 ReaSenn Sen A Ninommn et 207 Rernetiin mi
At
2021] Net [Wirhmad et (021) ReeNeX (im Py (Wishtman et
EdticiniNG B0 (Wihtman BllicicalNc-- Wightmas
2021) Edbiian Net-Bi Wiihtman
(Wishtman m. (221)
(b)
46
CNN$
RogNet-y-400uuf (Rademnvovic of 2020 RogNetsy.800mmf (Radoaveoiic ci al.
2020 Reg-
welefee (RadooooRaooaa (Radorawa al (010] RegNot.w. RRdlorvva ei 2030
RegNot.y. Bud
(Ra
dowavovic et al (2022)) RerNe-y--ir (Eadaravovic dt al Air20) DiirrntNes-v225 1255-22-5 TTaa
Lo 20121 Eiseentkattv22 [Tan & Le 2021
UnarKext-
LLid et al 2lisdb) ConyNeXt
20/22)).
ConvNeXTiL
at
9029h)
Repnatsy NaN-y----p ipl Harinzaweie
20200]
ConvNe
202th RegNaryy
(Ladoeewoeiz Al. (1220 ConvNeXe
Lit es ml. 2022b] ConvAeXS- Liu et
20122h).
RegNat-y. Hadoravviic
2020)
(Rakensviic
2020
KeyNet
(Radosavon voss
(Raioavvvvle
shil ComyNeXe
Next-
ConyNeXI-E
y-500af
v2-8 (Tan (202)] IntiinNNnt---2
UGnVNeXT
Li
2022h)
CosvXeXt-5
(11225) ConcNeXT-I it
(0223) ConYNEXE-T Lale
20230) CanvAeXtes Metts Lovt at 2092h)
RegNetsy
[Raimnoovvi et al
2020. Blicent
V2-5 TTan & Le 202 Elr Ii
Tan le
2071). UmavNeXtii I (our at al 20226 [b)
EdflcientNessvve (an & 212)) MiheleeNNat-11
Fan K Kln 20211 NoX--1 (Tm
21225 ConvNeXi-5 [Lu el dn 2020
ConNeXX-BB
Singh ci de 2048 CoavNeXeV1-1 Woo
21 0l 2093]
Trandformers
2021). Deff-s Touvron at
2021a Caff
Touvron al.
LaViT
225
(Graham
2021)
LaVIT-364
Grabhsm at all 2022
XCTI-m2 11--
at
Dm (ibozion all 2022a)
MarViich
Swinst
an (2y1))
Crea-Vi1-10
CoaT-t-ite
CooT-miilise
DeTT-
De/T3-i
2022
SwinV2-175
SwinV2a/5
Swinya-b77 (Li
20225
SwinV2-4/1 (Lis
Muzza SwinVaay11 I
Swis
20/22a
TImoVTL-SAPTE We (i2 TuyVEr-TEMT
Tun
[Wo
2022
Vibaz110 10 Brenn
al 2023) DaVil"I Ding
2022
VaVeT- Tinng ot Al 20r2
DAViThh (Ting or (ing Mnsepionnnet-
2022
AncentionNeeee Na at Mi
Pan VI- (Varh at 2023)
Var
01
2023
DefTaa (Tourron &
2022
SwiuV2 272
2022
Tin
21M
[W et 6l 2022) Vilse
(Steloor cl al
2022
(Seenee diner et 03 2022
Swin-b
Diu <t a. (0M
Swinob (Li al
2020
Hierae
Ryali et al
2022
12
Standard models
ResNet15
[Ne
2016
ReaNet34
He
Ree Not50
He
RecNet152
[[
2010)
[in
(200)
Net121
[Tnang
DemscNetl01
[aanx
2017)
Houng
2011
61 ml 2017)
VTT-5/16 Diotorizkly
2021
ConvXcXeT
20220). Con-NcXI-
B 145 st 31
2022h
B-cos modele
Balvars
Monnl Sr all
2022]
TooNet34
Boble at al
2022 NetNet50
Sphle et
20222
BesNet101
(Soble et al
20122)
Net (Boble et bi. (2022)
Net Pobble
ct
2022)
Deneevarto [10
Pozz
Dendave fiol mo
2022
Donse
At 2022
VIT-b/ff (IMble
2022
ConNeXt-II
(MMll at
ContNeXttT
[Mhil
24
Standard modele
ReeNet50
He
2016
ReeNaa111
He
2018
VrT-b/i8
Dorovitelly
2021
VyT-b/32 (Uorovitadiy
2021
Faxt Vutter--aal Vart 2023
Furve SeEVer-a32
2021
Pac Vaal Yoou
2023
byl Uosovink dr 2021),
VIT-D71C Dowo
vitakiy
at al. Poz
Anny NeXel Lin ot al 20222) Ver4r16 Donvitakly at 209
7/10
Unsovitoiiyy dt a.
2021 CoavNext-D
Lac 2022h
COIvEEXLL Lite di 21P22
NoX-L Lag cl mi
2022
VIT-b/16
Donentsk e di 2021) V:T-5/1B Coloviooi
2021]
Digeriskly et al HWo)) Verii
Booutskky ei Bi. 2011)
2021). VIL-MIS Bosonitakl at 2021 ViT-rio Doeevitkit by
VII-5718
Intobizky et
Vr1-7716 Daseribayy it
Vision-language
Taaloond er
202
(Vou
MeinChi
LI2. (R 2124)
CLIP- NoX
W20ps
Schehmann
(o22 Weztaman sl.
Ar22
Chil N
Schuamaan
N022
Wartsman al
202 ULIP-LoReNEXE
Schecmaan
2026 Wortsman
al
Gadie st
2628
ULIP La-LaooV
Sch Schakmann
Kr
9029
Wortem an al 2092)
C1.1-177 1ettoommmmmmm Commmmmnaaaaa Xa. (Pang
MA0 (Pp-epp (laddard stal
2021
[Gadre et
20
L/14-Laim2B [Scin) di mL
Wortomee dt d.
2022)
Ch PeIePPo ool XL-
DFK28 [Fang d1 ol. 20244 Sip
Jachannec C.M. 2020 SigLiP2-/116
Technmec
st al
2025)
31

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: Correlation matrix with corresponding p-values (in parentheses) for all models (numerical results for Fig. in the main poper).
C-Rob.
Parameters
(-00)
Adv. Rob.
(.00)
.44
(.00)
1.00
.02
(-00)
(.84)
(.00)
OOD Rob.
(-00)
.35
(-15)
-0.08
(.00)
.80
1.00
(00)
(.00)
Class Balance
(.00)
-53
(.14)
(.00)
.73
(.00)
76
(.00)
-18
(.00)
1.00
Obj Foc.
(-00)
(-00)
(:00)
(.00)
(:11)
(.00)
(.00)
Shape Bixs
(.00)
-26
(.00)
-17
(.00)
-61
(.00)
62
(.00)
-24
(.00)
.44
(.00)
.52
(.00)
1.00
(.00)
(:00)
(000
(.05)
(.00)
(.00)
(.00)
(.00)
D.2 Training self-supervised models
To increase the number of models we can use to compare self-supervised and supervised models, we trained additional models for the end-to-end fine-tuning setup (E2E) and the linear probing setup (LP). For the E2E sctup; we fine-tune the official pre-trained DINO CCrrn et al. 2021) checkpoints for ResNet50 (He et al. 2016) and ViT-b/16 (Dosovitskiy et al.] 2021 For both models,
follow the training procedure of He et al. (2022), as also done in Goldblum et al (2023). Specifically, we train the ResNet50 for 100 epochs with an AdamW [Loshchilov & Hutter 2017) optimizer using a batch size of 128, a weight decay of 0.05, and a learning rate of 0.001 with a cosine scheduler. We use 5 warm-up epochs with learning rate of 0.0001. For the ViT-b/16, we follow the same procedure but with a learning rate of 0.008. Furthermore, we also fine-tuned the official pre-trained DINOv2 (Oquab et al.) (2024) checkpoints for ViT-s/14, ViT-b/14, and ViT-I/14 as well as their respective version with register tokens (Darcet et al.) 2024 For training all DINOv2 models, we follow Touvron et al. (2022) done in Oquab et al. (2024). To be more precise, we trained each model for 50 epochs with a batch size of 128 for the small variant and batch size of 64 for the base and large variants. We used an Adamn (Loshchilov & Hutter 2017) optimizer, a weight decay of 0.02, and a learning rate of 0.0003. Additionally, we use 5 warm-up epochs with a learning rate of 10-6. We also use the ThreeAugment augmentation method that was introduced in (Touvron et al. 2022 together with color jitter and CutMix (Yun al.) 2019).
For the LP setup, train four additional models, ViT-b/16 (with Masked Autoencoder (He et al. 2022) pre-training) and Hiera-ting teteetttn//mmllll base (Ryali et al. 2023). We only train the classification head of each model and follow the training of He et al. (2022), consistent with Ryali et al. (2023). We train lincar layer for 90 epochs with LARS You et. al. 2017) oplimizer using a learning rate of 0.2 and a batch size of 512.
E Numerical results
E.1
Correlation matrix in the main paper
As our visualization of the correlation matrix in Fig. of of the main paper summarizes much information and partly relics on color vision, we additionally report the numorical results in Tab. 10
E.2
Mean and standard deviation of each quality dimension
For our proposed QUBA score described Sec. 5 of the main paper, we compute a representative mean and standard deviation for each quality dimension. In Tab. 11
we report these values.
32

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Mean and standard deviation for each quality dimension as described Sec. of the main paper.
Quality Dimension
Mean
Standard Deviation
Accuracy
0.80
Advensarial Robustness
0.19
0.11
C-Robustaess
0.53
0.23
OOD Robustness
0.57
0.15
Calibration Error
0.0045
0.0027
Class Balance
0.78
0.02
Object Focus
0.93
0.02
Shape Bias
Parameters in Mil.
55
43
E.3
Model zoo
We use 326 models in our large-scale study. For a comprehensive overview models, their configuration (architecture, training dataset, and training paradigm), their QUBA score, and their scores for each quality dimension, please refer Tab. [2] where models are listed in the order of increasing QUBA score. Each model most models they are, and do not change them
is implemented in PyTorch (Paszke et al., 2019); in any way (except. for training some of the sell-supervised models to perform ImageNet-1k classification, cf. Appendix D.2p. The selected models were chosen based on several criteria. We only considered models that are publicly accessible and free to use for academic research. We aimed to include the most popular models and models achieving high ImageNet-lk accuracies. We additionally included some particularly interesting models with designs that differ substantially from the more widely established models.
33

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: Overview of our model zoo and the corresponding numerical results. The configuration of each model lists the architecture, the training dataset, and the training paradigm. "SL" refers to supervised learning and "AT" to adversarial training. "LP" and "E2E" indicate if a self-supervised model has been adjusted for ImageNet-Ilk 1k classification by only training lincar layer (lincar probing, LP) or by fino-tuning the full network the ImageNel-Ik dalaset (E2E). Models sorted according to their QUBA score in increasing order.
Model
Confguuratin
QUBA, Score
Acc.t
Adv. Rob.
C-Rob.T
OOD, Rob.
Cial Error
Class Balance
Obl- Focus
Shape, Bina
Parama, MLL.
SigLIP3-/16
VIL.
WebLi Chen
+3.55
0.82
0.80
1.06
0.0388
0.98
(Tschannnn
2028
2020
(226)
Bagt
-3.01
0.48
0.00
0.15
0:25
0.0052
0.81
0.04
Betlage
Rupooboraky
Zhoi
VIL.
WebLi Chen
-2.52
0.80
0.23
0.72
1.04
0.0384
0.90
0.97
0.63
652
2023
Rt a1
2023
Bag
CLIP-L11 2
ViL.
2.45
0.75
0.01
0.71
1.07
0.0370
0.89
0.95
0.34
428
Schahmann es ml
Laton Schuamann
At
2024 GolPal
6E
C1.1--114
V1b.
DataCom-
2.10
0.79
1.05
0.0380
0.80
0.96
DataCompXI
DX1
Gadre
(Cadee
2023
soleaL (E2E)
2029
-2.09
CL-P--//33
Radh
Vil.
2200
0.63
0.05
0.64
1.12
0.0318
0.98
0.86
0.38
151
Fand et Al
2021
WIT Badford
6L
m
202
CLID-ConeNe-t-I1 (3el
0027
2023 selest
C1.1P-ConeMan
2.05
0.77
0.08
0.74
1.04
0.0371
0.89
0.95
0.58
382
p-320pp
Schmbb
Lalon2 Schuhmann
200
MetaC1.PP-1
VIL.
PIP.
+2.03
0.79
0.09
0.50
1.07
0.0352
0.80
0.97
0.58
425
Xu c1 D2 2024
400M0 Xa
2024
Coanm
DFN21
(E2E)
20245
Vit.
Webti Chhbn
Chen
+1.98
0.75
0.12
0.07
1.04
0.0350
0.80
0.96
0.55
375
E22
SquceeeNes
Loa-
CNN
-1.97
0.58
0.01
0.32
0.45
0.0013
0.74
0.77
0.21
Molu
2010
IN1L
Rusonkoysky
61 m1
2014
31
flrande
CNN
11.87
0.65
0.01
0.25
0.31
0.0041
0.72
0.85
0.05
15
Beslage
IN11
Ruoookovaky
6L al
2010 -B
Hicro-S
+1.57
0.65
0.60
0.0058
0.79
IN1k
Rusenkovkyy
2015 6ol...
Mlara-
Byal
Trandarmes.
+1.85
0.57
0.03
0.47
0.56
0.0044
0.76
0.78
0.38
Rusonooraky
NesNel18 18
Böhle
Boos INkk
11.81
0.69
0.00
0.11
0.42
0.0190
0.79
0.58
0.22
12
Aunenoovaky
2010
Sic
2022
CLIP-BiB-
1..78
0.70
0.62
1.08
0.0357
0.52
Laica2B
Sclanhaan
manns et
34

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Rob.
G-Rob..
Balance
Focus
Shape,
Params; MiL."
ResNet5o.-DT
-1.78
ShoffaNet-v2--th
CNN.
-1.62
0.61
0.01
0.35
0.46
0.0028
0.32
0.85
0.27
Ma 65
2015
INTK
lurenoovakyy
A1
m1
2019
VGGI1
0.0014
Zleserrhnn
IN1K
2015
2hat
Chan snleS
-1.58
0.78
0.10
0.63
1.00
0.0373
0.80
0.95
0.48
203
Rt
2023
Wight
CNN
-1.55
0.88
0.17
0.04
0.31
0.0057
0.78
0.53
0.14
1N1k
Luannoovakyy
m1
2015
5o5Net152
11855
0.0207
IN1k
Rusookorake
[PP
B/16
ViL.
MLPP
11064
0.72
0.03
0.68
1.09
0.0365
0.99
0.92
0.54
160
Xa et NJ.
2024
4003
Xm
2024
MobisCLIP-S
VIL.
DalaCompDR
-1.08
0.68
0.00
0.15
0.55
0.0826
0.89
0.58
0.66
64
Voou ol all 2024
18 Wesu
Ne
Beon
+1.01
0.41
0.0208
IN1k
Kusenovvkky
2019 SL
1.49
1.07
0.0867
DuteCompP1
(ETE)
VGB1
Nimonvan
CNN IN1E
11.49
0.72
0.02
0.40
0.43
0.0017
0.75
0.56
0.11
135
Ziseeraian 2010
Sueeloovaky
65 m
2010 S1
V-Cil-be
0,70
0.0017
0.87
P1/16-
Red
VIL.
1.45
0.68
0.03
0.64
1.11
0.0258
0.88
0.92
0.48
100
Bibble
Rooe
1..4
0.72
0.46
0.0121
0.74
0.88
0.30
20222
IN
Rusonkovaky
81 m1
2019 SL
MobilcCL.PPP
DotaCompDR
VGG11 9
Simanvan
CNN
1444
0.72
0.03
0.41
0.44
0.0017
0.75
0.87
0.12
144
Siaeersana 2010
IN1k
Ruseloovkky
5L m.
2010
VIT-L/1G-MAB
0.44
0.0078
0.85
He 2022
IN1k
Rusekovaky
2015 0el.33
Moaz Net.875
1.337
0.71
0.06
0.47
0.53
0.0185
0.81
0.88
0.21
IN1k
VGC13-be
31
CNN
-133
0.72
0.02
0.40
0.45
0.0017
0.75
0.88
0.12
183
Zinuer
INIL
Rusonkovsky
2011
6L m1
2014 SL
MobdeCC.1P
DataCompDR.
CLIP--R1B-
Vdl
DPNSn Fang
1.35
0.75
0.04
0.65
1.03
0.0374
0.90
0.94
0.31
150
Pool
6L
2024
- D2B Pang
1225
21124a
MobileCiIP
ViL.
DataCompDR
1.225
0.77
1.08
0.0372
0.94
Vaen
10
Vam
2024
+.22
0.68
0.40
0.18
0.0041
0.75
0.84
0.23
Rusonboraky
VGG1 G-bm
3h
CNN
11223
0.73
0.02
0.45
0.45
0.0020
0.75
0.59
0.11
138
manmynn Ziaser
INT Rowesoovsky
2024
St al
2010 Sic
35

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Rob.
G-Rob..
OOD,
Balance
Focus
Shape,
Params, MiL."
2e
Bees,
-1.17
0.44
0.0099
V1L
DataCompDi
1.38
0.77
0.01
0.79
1.10
0.0375
0.90
0.95
0.82
150
Vau at ai
1B
Vam at
2024
2021
201-3L
Net-Bo
0022
VGC19-ba
CNN
1.11
0.74
0.02
0.48
0.50
0.0021
0.76
0.89
0.16
144
manaan
Alrenn
2015
CNN
+1.10
0.73
0.51
0.57
0.0145
0.78
0.89
0.21
Mi
2015
Luannoovaky
m1
2015
0.0117
IN1
Ruszokoraky
Ne
Wixbs
CNN 1N1k
+1.08
0.73
0.27
0.04
0.29
0.0051
0.75
0.86
0.18
22
maB ml
2021
61 m1
2015
-3
NetieetNet-BI
0.0048
2021
Ne0
Beos
-1.06
0.75
0.02
0.17
0.48
0.0088
0.74
0.91
0.26
14
Bobble
N11
Rupoolovaky
2015
+1.05
0.79
0.54
0.0161
Bobble
IN1k
Huwnnovaky
2022
2019 SL
ResNet18
Iue
Rupoooorakrr
Vol
CNN IN1K
11.04
0.68
0.03
0.49
0.56
0.0017
0.74
0.84
0.32
Howard
Bureslovaky
2014
61 m1
2019
CNN IN11
-0.99
0.78
0.22
0.02
0.21
0.0137
0.79
0.59
0.16
12
WW I Limonn es ml
Rusenlovaky
2015
Nat 8
CNN
-0.95
0.71
0.33
0.0022
0.59
IN1k
2019
VYT-1/18
Doretts
Trandormes
-0.93
0.80
0.54
0.50
0.0032
0.78
0.82
0.35
304
Aily
ROCA!
Rusnoboraky
Mias
Not-13 Moa-1Not11
Tan
CNN IN1k
-0.93
0.77
0.13
0.53
0.01
0.0155
0.81
0.91
0.19
al 2019
Horesoovsky
65 al
2019 21
Bolile
ResNetid
He
CNN IN1k
-0.85
0.70
0.05
0.47
0.56
0.0018
0.74
0.88
0.24
12
2010
Roreslovaky
At
2019
VTT-1/32
Dosovtt:
Prandormes
-0.52
0.77
0.55
0.65
0.0030
0.57
2021
IN1k
21 m
2019
Boar:
-0.78
0.74
0.65
0.55
0.0042
0.89
0.44
Rugsobovaky
V2 Sam
CNN IN1k
-0.77
0.72
0.02
0.44
0.50
0.0018
0.75
0.55
0.18
Hier at al 2018)
Ht x1
2015 S1.
Net-e
CNN 1N1k
-0.75
0.77
0.18
0.02
0.20
0.0040
0.75
0.91
0.15
9
Wighttmn es
Horenlovaky
NecNot50 NotNotoo-DIN
ONN
-0.74
0.75
0.07
0.32
0.0201
0.84
0.92
26
Carron
1
Insealoovalee
00211
Valnis
-0.73
0.73
0.35
0.0060
0.50
36

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
Rob.
G-Rob..
Balance
Focus
Shape,
Params;
MiL.
ShufflNette22
-0.72
0.57
0.0133
NewNNtt0
Wizht-
CNN.
00.71
0.78
0.34
0.05
0:31
0.0063
0.75
0.91
0.14
26
maa
2021
IN
lovesoovvyyy
6t m1
2015
BusNoXe
Rtannkovaky
0021
CLIP-ReoNccJ0 160
VIL.
0.70
0.60
0.29
0.45
1.00
0.0109
0.94
0.89
0.26
102
Baaford
2021 salES
De220
Bear
+0.69
0.75
0.18
0.49
0.0039
0.74
0.91
0.27
Boble
1N m1
(uwnnoovaky
2022
2015
VrT-1/32
Trenffrmee,
-0.67
0.63
0.0022
2 Nci-B0
CNN IN1k
-0.60
0.77
0.18
0.02
0.23
0.0026
0.74
0.99
0.18
Wightman es mi
2021
6t m1
2015 -2
Mete-Net-1
2010
RosNeL101 H1 2023
Boble
Bees
-0.65
0.77
0.00
0.17
0.49
0.0042
0.73
0.98
0.28
44
IN16
Rupsolovaky
2015
Net
ONN
-0.05
0,70
0.54
0.0032
0.91
Baugedy
IN1k
Rusenoovaky
2010
2019 SL
A1
-0.65
0.0110
CNN
-0.65
0.74
0.03
0.52
0.58
0.0026
0.75
0.87
0.26
Howard
mi
IN1k
Kueeloovky
2001
65 m1
2010
S1
155
-0.65
0.81
0.0078
RoxNetsx Netoy 400ml
CNN Luannoovaky
.0.63
0.74
0.02
0.48
0.54
0.0015
0.75
0.86
0.22
A1
Nat23
Wight+
CNN
-0.61
0.76
0.05
0.36
0.0051
0.74
0.90
IN1
Rusonoovaky
81 m1
2019 A1
Ro1Ne
-0.99
Re-Net3
Wirht
CNN
-0.58
0.76
0.35
0.03
0.37
0.0026
0.73
0.91
0.23
92
mua et ml
2021
IN11
huoploovkyy
05 m1
2010
722
NosNo
-0.55
0.73
0.66
0.0020
0.90
IN1k
201
Netecient
CNN
.0.33
0.77
0.21
0.02
0.24
0.0027
0.74
0.93
0.26
Wiihhtma es
Ruzenlowaky
2019
Dee T-e
Tousron
Prandormas
-0.52
0.72
0.61
0.55
0.0053
0.77
0.90
0.21
202
IN1L Rusonovaky
Rt m
2014 31
-0.52
ReeNet50d
CNN INkk
-0.48
0.79
0.27
0.04
0.24
0.0030
0.75
0.93
0.14
96
[Wightman 45 m1
Invesdovvde
20241
65 m1
2010
A3
Balman
Rusebovaky
2020
201
Reg N2t
ONN Inzesloodyy
-0.44
0.81
0.05
0.00
0.54
0.0026
0.77
0.92
0.27
145
at
2013
Nat50
Wight
CNN 1k
-0.43
0.80
0.05
0.32
0.0033
0.73
0.89
26
At al
2019
Heo
Tranformer,
-0.42
0.73
0.59
0.0051
0.50
StNat 1
Hn
CNN. IN1k
-0.22
0.81
0.36
0.09
0.35
0.0091
0.80
0.93
0.31
115
Ht Al
2014
Inneslowwvy
xl
2013 31
37

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Advi
G-Robb.
Balance
Focus
Shape,
Params, Mil."
S2
CNN,
-0.41
0.0088
WRN-101-2
CNN.
-0.4
0.79
0.05
0.57
0.54
0.0023
0.76
0.94
0.28
197
Zaporaylo
Euzenoovaky
NomocRAD
M011
Ner---t---dpt
CNN
-0.38
0.79
0.03
0.27
0.0050
0.75
0.94
0.16
21
Wighhtmn
Nkk
2021
SeNuta5d
+0.38
0.82
0.08
0:26
0.0025
0.76
0.54
Reg
CNN
-0.35
0.82
0.52
0.07
0:30
0.0022
0.76
0.94
0.18
145
WWightman e5 xl
NNk
2021
64 m
2010
A3
IN21k
Dong
V Ascoli
Transformez Innensowayyy
-0.77
0.73
0.15
0.62
0.68
0.0051
0.77
0.91
0.27
201M
Net-tt
CNN
+0.38
0.79
0.03
0.26
0.0021
0.74
0.82
Wightman
Ni
2021
At m
2019
NopNettyy-0OO 800mm
-0.35
0.78
0.52
0.0017
0.89
NeeNNe550
Yainte
CNN.
-0.31
0.79
0.16
0.07
0:33
0.0022
0.77
0.90
0.23
26
xl
2019
VPCC100M
homes
nt
21
At
Reg
Bo
CNN
-0.33
0.79
0.28
0.0015
0.76
0.90
21
Mosavoxie
1Nkk
2020
2019
2e
-0.33
0.74
0.54
0.51
0.0017
0.52
0.22
[Huang
Re-Net50
He
CNN
-0.31
0.75
0.03
0.51
0.50
0.0021
0.75
0.93
0.29
96
Ht all 2016
IN1k
64 m
2010
Resebovvky
CNN
-0.29
0.51
0.47
0.04
0.27
0.0024
0.76
0.94
0.17
84
Wigghtman es
Inveslowdyy
28 Net-entN
CNN
-0.29
0.79
0.03
0.30
0.0022
0.74
0.82
0.27
W Wightman
Nk
0021
61
2019
Net.et
-0.29
0.80
0.03
0.25
0.0033
0.58
Ne=NeXe101-Give
CNN
-0.28
0.83
0.35
0.69
0.61
0.0196
0.85
0.95
0.21
84
Nie at
2017
INkk
8t m1
2013
S1
Zagorayko
ResNet50
Wightt
CNN
-0.29
0.80
0.49
0.08
0.31
0.0027
0.74
0.92
0.19
26
man
S
Wight
CNN
+0.28
0.82
0.07
0.29
0.0032
0.76
0.94
0.30
115
1N
covalyy
2015
BrTM- 50 Kalcusikov
122 Deng
-0.28
0.84
0.0018
2009
Rox NoxNokk 32xt
CNN INLk
-0.27
0.82
0.51
0.09
0.91
0.0029
0.76
0.94
0.31
145
Wi Samma #5 m.
2021
m.
2013
MobslaNerVa3.3
2010
He
CNN.
-0.24
0.77
0.04
0.57
0.62
0.0023
0.76
0.92
0.29
44
2010
IN1L
Rupoobovaky
2015
38

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
Rob.
G-Rob..
Balance
Focus
Shape,
Mil."
ResNet50d
-0.24
0:27
0.0028
S4
Wizht
CNN. IN
-0.2
0.82
0.69
0.08
0:32
0.0035
0.75
0.83
0.30
115
2021
nt m
2015
DemseNat180
Rusnnkovkky
2017
2019
Vin-VTT-DMC
Treasfrmncy Inzesoovayyy
00.21
0.79
0.15
0.64
0.68
0.0110
0.80
0.98
0.24
6
Ht
CNN
-0.21
0.80
0.39
0.57
0.0028
0.77
0.82
0.27
84
Radouavooic
Nikk
2023
ml 2015
50-92x4
-0.21
0.27
0.0004
VcT-b/L6-DINO
Transformer.
00.21
0.78
0.13
0.68
0.40
0.0012
0.77
0.91
0.36
87
et
IN1k Reeekovaky
AF
NeeNNoeeSn-222
CNN
-0.21
0.78
0.34
0.53
0.0028
0.76
0.83
0.22
Xie 2017
1ikk
lovely
A1
-0.21
0.34
0.0029
0.04
11 325d
CNN
-0.21
0.79
0.07
0.59
0.55
0.0028
0.76
0.94
0.28
89
Kie 2017
INkk
Invesoovskyy
201
RusNatbOd
0021
RoxNeXe50-22.de
CNN
-0.20
0.51
0.69
0.05
0.31
0.0034
0.73
0.92
0.36
25
Wiigttmnn es
Insessooveyy
Rte 101-5--2ed
CNN
[G] Yalniz
-0.19
0.84
0.05
0.30
0.0121
0.55
0.42
Vaints
2019
2019
1N11
61
CNN
-0.17
0.80
0.46
0.15
0.36
0.0038
0.76
0.92
0.29
65
Wighttman es ml
IN1k
101
0.81
0.0028
20211
1k
Wight
-0.17
0.82
0.08
0.33
0.0029
0.75
0.33
0.23
man
Tuannoovvayyy
VJT-b/92
Doso:
Transormess (iove/oovsky
-0.18
0.78
0.18
0.58
0.56
0.0038
0.77
0.91
0.82
88
Pibetlyy
IN1k
2021
2013
-0.16
Wight
CNN
-0.15
0.81
0.49
0.07
0.32
0.0028
0.75
0.93
0.28
44
mag 0t ml
2021
IN1k
husoloovaky
DINO
0.0012
N
Covely
X021
NeiiientNee.BBB
-0.14
0.78
0.77
0.80
0.0034
0.59
IN1k
Rusznovvkky
BITM-
CNN 6E M1
I21 Deng
-0.14
0.85
0.04
0.67
0.72
0.0016
0.80
0.95
0.28
280
Not 5252
2009
Kole Mow on m.
20200
50-5224d
0221
Rox L-
CNN
-0.12
0.78
0.08
0.64
0.64
0.0018
0.76
0.91
0.27
11
39

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Rob.
G-Rob..
Balance
Focus
Shape,
Panams, Mil."
Rog BR
-0.12
0.0023
He
CNN.
0..1
0.78
0.06
0.57
0.55
0.0023
0.76
0.94
0.28
60
sl
2016
IN
nt m
2015
Net
2021
ConyNeXX
Liu
CNN
-0.10
0.84
0.12
0.72
0.67
0.0064
0.81
0.95
0.31
198
2022h
CNN
+0.09
0.77
0.39
0.57
0.0025
0.76
0.54
0.28
Husns
1k
2015
Nee
0.0049
Resookoraky
LB
CNN
-0.08
0.79
0.11
0.97
0.66
0.0037
0.78
0.89
0.24
Xie ml
2020
300M 300MeHinton Hinton
It
8
2015
pun
81
2017
cambe
Xcaption
Chollet
CNX IN1k
-0.08
0.79
0.57
0.52
0.00311
0.77
0.53
0.22
2017
Rucenkovaky
At a1
2019
CNN.
-0.05
0.79
0.03
0.55
0.52
0.0018
0.37
0.91
0.27
19
Ic dogavoniic
NNkk
Wig (ighmmann
IN1k
Rureolovekyy
2021
Wisht-
CNN
-0.05
0.82
0.63
0.07
0.34
0.0028
0.74
0.84
0.31
man
Net
CNN
-0.08
0.82
0.28
0.05
0.36
0.0022
0.76
0.94
0.23
63
W Wightman
Rusenoovaky
0021
2019
Co
0.49
0.0069
Net
CNN
-0.05
0.81
0.27
0.04
0.28
0.0022
0.73
0.95
0.24
12
Wightman et mi
IN1k
Insenoovaky
2021
85 m1
2015 A2
0.0052
0021
2015)
Coa T-leit
Xu
Treasformay (oorenoovvyyy 2015
.0.04
0.78
0.12
0.00
0.66
0.0058
0.79
0.98
0.26
2021b
nt
XC1T-126-16
Al
Transformes
-0.04
0.83
0.22
0.74
0.72
0.0037
0.80
0.93
0.34
189
20221
IN1k Hueenovakk At al
2011)
Transfrmec, IN1h
-0.01
Rumoooorky
201
CNN IN1k
-0.00
0.77
0.08
0.57
0.09
0.0018
0.76
0.95
0.27
20
Huana
m.
Ausenlovaky
2017
6L m.
2015
0.0018
Resokoroky
2015)
RosNetcs.1Gam 16at
CNN (tannoovvayy
0.01
0.82
0.68
0.05
0.30
0.0025
0.76
0.95
0.36
84
Wightman es
Ee Bangedy
ONN NIL
0.77
0.55
0.0017
0.76
0.94
0.30
Kuoonoovkky
2010
2019 SL
4gf
0.25
0.0021
Rugeokovaky
CNN IN1k
0.04
0.52
0.58
0.03
0.29
0.0024
0.75
0.94
0.26
39
pl catmon es m.
Kwnnkovaky
2021
65 m.
2010
Nci-BB
0.0023
2019
40

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Advi
G-Rob..
Balance
Focus
Shape,
Params; MiL."
NeeiientNet.B
0.83
0.0041
T3-
TORvion
Trandormer,
0.05
0.85
0.32
0.82
0.76
0.0031
0.81
0.96
0.30
304
It sl
2024
INik
8t m1
2015
Net-rr
Rtssnkovaky
BYTM- ReaNeeee
CNN
I621 Deug
0.06
0.80
0.01
0.65
0.66
0.0012
0.78
0.91
0.19
26
Kalawni at
m1
2008
Netd b
Yainis
CNN
IG13
0.51
0.20
0.05
0:44
0.0022
0.77
0.53
0.29
26
2019
2011
1811
H1
m0
20M
51.
Net-B
CNN. IN1K
0.08
0.82
0.34
0.08
0.30
0.0023
0.76
0.93
0.28
19
Wighttman es ni.
Rusekoovakyy
AT
0.79
0.70
0.0074
0.94
2025
1
0.81
0.04
0.25
0.0022
0.78
0.04
htman
Tuannlovalyy
So Wightman es xl
CNN IN1k
0.08
0.82
0.62
0.08
0.90
0.0028
0.75
0.94
0.26
89
2021
HE m1
2013
Trnad
Net-4
CNN IN1k
0.10
0.82
0.88
0.00
0.34
0.0028
0.76
0.95
0.24
19
Lgl Sitmon es m.
Twooloovkyy
7
0.0078
Bingh
1
lunealoovekyy
20245
2015
0.83
0.65
0.80
0.0077
0.52
IN1k
Ruaznkovakyy
Swink2.s/S /
Liu
Tramsrrmer.
0.11
0.82
0.04
0.68
0.68
0.0044
0.80
0.92
0.18
28
2022
INTk Rusonbovaky
Bt x1
2013 S1c
NoX-N
Liu
CNN:
0.11
0.78
0.69
0.65
0.86
0.0078
0.79
0.95
0.73
198
CL m.
IN1k
Rupoolovoky
6L al
2019 A1
Eecoption
CNN
0.80
0.64
0.0014
0.94
Riogedy
IN1
Puyly
2015)
Coa --m--lite
Xa
Trandfrmer,
0.14
0.79
0.15
0.03
0.57
0.0051
0.79
0.93
0.27
11
Rtwnnkovaky 2019
Swinek
Liu es ol.
Treasfomes.
0.14
0.81
0.05
0.66
0.09
0.0637
0.79
0.92
0.22
28
2021
IN1k
Rusonkovaky
6t al
2014 S1
Trandormer. [NJA Rwooloroky
0.17
0.82
0.21
0.69
0.63
0.0096
0.81
0.94
0.26
11
Wi <t al
2022
6L al
2019 SB
VITV1
0.85
0.0053
0.65
IN1k
Rurenlovaky
2015
Nt
0.18
0.84
0.12
0.70
0.66
0.0088
0.82
0.95
0.36
2022h
Inzealovvdyy
Net--
Zoph
CNN INLL Ruseokovaky
0.19
0.83
0.88
0.58
0.0029
0.79
0.94
0.24
2018
At al
2014 S1.
41

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Advi
G-Rob..
Balance
Focus
Shape,
Params, MiL.
VIT-b/16
Transformn,
0.0034
Inception-
CNN. INik
0.20
0.80
0.15
0.63
0.56
0.0022
0.78
0.95
0.26
56
Nat22
Esegedy
It
sl
2017
nt al
2015 Si
Rusnnovakyy
2019
Swineb
Liu et o1.
Treasformcz
0.28
0.84
0.15
0.71
0.00
0.0030
0.80
0.98
0.28
88
3021
nt
2015
EVAU2-t
Pang
Frandormer,
0.51
0.85
0.58
0.0051
0.79
0.53
0.21
2024b
IN21k
Dong
2008
sole3L (122))
CoavNoXe-8
0.76
0.0081
Binch
IN1k
Resookoraky
LBT
CNN
0.23
0.84
0.22
0.67
0.07
0.0061
0.81
0.92
0.26
66
Jan &e Le
2010
NN1k
lovenoovayyy
6t m
2015
0.0050
DINOv2-PT
LVD142m
(Qqueb
20241
ViTT./11 /1
Siugh
Treasformeg
0.26
0.73
0.49
0.63
0.72
0.0061
0.78
0.94
0.72
23
at
Vwank22./8
Liu
Transformss
0.28
0.84
0.04
0.71
0.59
0.0038
0.80
0.93
0.26
50
20222
IN1k Kuwenoovaky
Ht m1
2015) M1
Ellician
Ruseolovaky
VIT--/16 1 HE
Steinor
Traneformer.
0.26
0.86
0.24
0.84
0.77
0.0011
0.81
0.96
0.54
304
m
2022
IN21k
Dong
[onng et
2009
3L
LeViT-266
0.65
0.0022
Rusnbovaky
Treasformey Kuannlovvayyy
0.26
0.81
0.14
0.63
0.57
0.0014
0.77
0.94
0.21
12
At
2013
Peicient Net-
CNN IN
0.84
0.50
0.0065
0.92
Tuo & Le 2019
Rusonkovaky
R1 ml
2019) SL
Tranformerr
0.47
Reicient Net.
CNN
0.27
0.83
0.09
0.65
0.32
0.0027
0.90
0.92
0.24
92
lam & Le 2021
IN1L
Rwoeloovkky
6L m. 1010 -01
VIT-I-144 2
0.87
0.0010
LVD142m
1024
2021 0e/.3T
Swinus
Trandarmes,
0.27
0.83
0.08
0.70
0.57
0.0025
0.80
0.93
0.28
50
Rternkovaky
Tiny ViT-VVT-N MLT-DM//2
Trassormer. IN21 (Dong 2009
0.28
0.81
0.09
0.67
0.66
0.0041
0.79
0.92
0.38
Wi al
2022
36
Trandormar
0.30
0.83
0.24
0.69
0.62
0.008R
0.82
0.94
0.26
22
Roou cl al 2025
[NJ Rwooloroky
6L al
2019 S3
0.83
0.68
0.0038
0.93
2021
IN1k Rurnnoovaky
2011)
Rfficient Net.vv-S
CNN
IN31k Deng
0.31
0.83
0.06
0.68
0.13
0.0840
0.80
0.93
0.34
La 2021
Ht al
2008
Piiceent Pormer-17
Pandformss
0.83
0.70
0.88
0.0023
0.78
0.84
0.23
Li al. 2022b
1 luwrnoovvve
2019
Net
0.0065
0.27
42

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
Advi
G-Rtbb.
OOD,
Error
Balance
Focus
Shape,
Params; MiL."
CNN,
0.72
0.0043
0.52
0.31
0.83
0.19
0.55
0.59
0.0054
0.91
0.92
0.25
30
kusznlovaky
Net
CNN
1221k Deng
0.85
0.73
0.13
0.0048
0.96
0.38
118
Too & Le 2021
cl
2009
BYFM-
1 Deng
0.82
0.61
0.71
0.0011
0.79
0.54
0.23
Del
ToRvyon
Prandormer,
0.34
0.80
0.22
0.71
0.59
0.0043
0.79
0.94
0.31
22
H1
INJk Ruenelovaky
61 a.
2010
VIT-L/14.
0.87
0.88
0.55
0.0030
DINOv2.repe.PP
(YD14mm
Dangget
2021
2024
Touvmon
Trandarmer
0.34
0.87
0.35
0.80
0.87
0.0023
0.82
0.98
LeVID-384 ham al
Treasfomerr IN1k Rurenlovaky
0.34
0.83
0.19
0.67
0.63
0.0023
0.78
0.94
0.24
39
2021
Rt al
201M S1.
Ctn-NeXtt-B
Inception Nexttt Yuu et
Prandormer,
0.35
0.82
0.21
0.68
0.84
0.0082
0.81
0.94
0.31
28
2024
IN1k Ruseslovaky
At
201M 31
ComoNeXeVVS.
CNN
0.82
0.05
0.0068
0.90
0.94
Woo
2023
IN1k
Invenloovllyy
1225
0.78
0.64
0.80
0.0069
0.95
2
14
Trandormer,
0.36
0.85
0.44
0.80
0.82
0.0035
0.81
0.96
0.34
215
HE xl
20224
INIK Ioweeoovkky
6L m.
2010
OpenAL-PT>
WIT-00M RRdford
Viston
202
CChertt
V-T-b/16--AR
Trandarmeer Kuookboraky
0.38
0.84
0.25
0.71
0.58
0.0048
0.80
0.95
0.36
VLT-b/16 al 2023
Singh
Transfommrr INTK RooreNovvkyy
0.36
0.77
0.55
0.69
0.50
0.0072
0.79
0.95
0.77
87
SL a. 2019
2014
De
Touvzon
Treasfomecy IN1k Rozealovayy
0.37
0.82
0.23
0.75
0.02
0.0035
0.80
0.95
0.39
87
202
at
2019 31.
PiT
lloo
Trandormes. INkk Lovealoovvey m
0.81
0.61
0.0042
0.79
0.93
0.31
9021
2019
Inception
Trandfrrme Rugnobovaky 2015
0.84
0.74
0.72
0.0081
0.82
0.54
0.35
PyT
Beo et. al
Trandfomer. INkk Noveky
0.40
0.52
0.25
0.74
0.62
0.0637
0.90
0.94
0.33
74
RD21
m.
2010
2011
SL
TransDrmez,
Ruggobovaky
Sfficenn Pormerall
Vaaddormer..
0.41
0.53
0.20
0.69
0.65
0.0020
0.78
0.94
0.24
31
Li 0s al.
20224
NI Atwenlovakyy
AL
m
2015
43

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Rob.
G-Rob.1
Balance
Focus
Shape,
Params, MiL."
Transformr,
0.87
0.0040
V-T-5/14-
Frandormer.
0.42
0.81
0.05
0.68
0.71
0.0011
0.78
0.93
0.35
24
DINOv2-reg-LPP
LVD142m Uguaa
Dances
2021 #01-31
0024
VutViT-
0.0083
Vaet
2023
IN1k
TOEvrOn
Prandarmer, bossaaoovayyy
0.43
0.81
0.15
0.74
0.59
0.0023
0.77
0.95
0.31
2015
Net-entNettv22-.
CNN Rusenkovaky
0.43
0.85
0.15
0.71
0.53
0.0058
0.82
0.95
0.26
64
Tao & Lo 2021
IN1k
At al
2013)
Efflcien
CNN
IN21 Deng
0.0048
VTT 16 It
Steinor
Trazsformer,
0.45
0.81
0.03
0.71
0.45
0.0010
0.78
0.94
0.38
22
4022
NN21k
Dong
2000
SL
CuaViT-bb
0.0043
Ascoll
InceptiinnNoiie Yu el al 2024
Treasformce,
0.46
0.84
0.28
0.71
0.67
0.0072
0.82
0.95
0.29
49
IN1k Rustolovaky 2015
VIT-S/14-
Trandfornce
0.81
0.70
0.0052
0.79
0.96
11N0x2-reg-ll
LVD142m (qqubb 2024
Duceei
2
Tranformag
42
Qquanb
Do Ascoli m
Fransformer,
0.48
0.51
0.25
0.73
0.81
0.0048
0.80
0.94
0.34
28
IN1k
Rwonkovaky
VIGG//--IINO
0.83
0.0089
0.94
Nik
2021
2015
VIT-5/14-
Tranformer, Oquab
0.70
0.0054
DINO.2-PT
CNN
IN Deng
0.50
0.82
0.12
0.68
0.83
0.0027
0.79
0.94
0.32
16
Woo et sl
2020
m
2009
#01-31
1226
ViT-VTT-2
0.0077
Ruspboovkky
2015)
CLIP-BS2.
ViL.
0.00
0.83
0.17
0.70
0.52
0.0020
0.79
0.93
0.46
88
Laion28B-PP
Viasen Encoder
Rt
Coa T-ee Iite
Xm
Trandormer,
0.52
0.82
0.30
0.70
0.59
0.0038
0.80
0.93
0.30
90
Ht xl 20215
INik Invvee
nt
al
2013 Bic
VLT-L/14
Fransformez. 12D Orgaab
0.52
0.88
0.45
0.87
0.90
0.0044
0.83
0.98
0.83
310
DINGV---rap--.PP
At
VrT-
Ding
Tranformes IN Rusnnoovay 6t al
0.54
0.83
0.72
0.82
0.0038
0.80
0.94
0.34
2022
2019)
Tranffrmer,
0.72
0.0045
CrostVeT- 18icChee at m. 202
Trandormer, INkk Innennoovvyy
0.54
0.83
0.26
0.74
0.82
0.0038
0.80
0.94
0.36
44
Bt al
2015
Rusmkkvokk 2011)
Conn Nox
Liu
CNN
1821
Dean
0.66
0.53
0.16
0.71
0.09
0.0687
0.80
0.95
0.29
29
2009
44

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Advi
C-Rob.1
Balance
Focus
Shape,
Params, MiL."
ConrNeXeV221
0.0028
Max V
Trandformer
0.59
0.84
0.26
0.74
0.86
0.0037
0.81
0.53
0.28
31
4022
INik kowknoovaky
At al
2019
0.0037
151(Chee (Chen
INkk Hooreoooakyy
2022
2015
D-T-b-14-DINDD
Prandormar,
0.59
0.85
0.10
0.77
0.50
0.0011
0.80
0.94
0.45
90
Orumbb at
Uquab
2win Tai al
frnnsforner
0.59
0.87
0.29
0.82
0.80
0.0638
0.82
0.97
0.42
197
2022
IN21k Dong 31.
2009
GowvNeXeVa.rr
Meiiien
CMN,
JP
0.61
0.84
0.18
0.67
0.77
0.0045
0.81
0.93
0.27
12
Nie et
2020
300M
Himbonn
pmm
2a6/6/
Lin
Prandormer.
0.61
0.84
0.26
0.75
0.62
0.0033
0.90
0.94
0.30
50
21 m.
IN1k
Ruooloovaky
2010
CeavNeXX
IN21
0.81
0.88
0.0016
20231
2009
Del
Touvzon
Trensformerr Dong
0.63
0.83
0.28
0.67
0.70
0.0033
0.80
0.92
0.35
22
2022
DaVITTb 2022
Ding
Prandormer IN1k loseeloovey ct al
0.85
0.75
0.68
0.0020
0.80
0.94
0.35
2019
Touvzan
Trandformes 2015
0.84
0.79
0.70
0.0038
0.95
0.43
Hiera-T:
Ryalt
Trandormer,
0.64
0.85
0.32
0.81
0.78
0.0127
0.83
0.95
0.43
214
Ht xl 2023
INIk
21
2019 GOIPS1
1E26
Coa-T-me-lie
0.84
0.06
0.0037
0.85
2021b
IN1k
201M
CNN:
JPT
0.65
0.85
0.20
0.72
0.77
0.0050
0.81
0.92
0.28
Nite 2020
300M
Sum
ViT-VTT- M-M16
Transformer.
0.65
0.83
0.11
0.70
0.73
0.0039
0.80
0.95
0.38
11
Wo et al 2022
N221 Dens et
CLIP-BIG-
0.85
0.72
0.82
0.0024
0.94
OpenAI-PT-
WTT [aadford
Viaton- Encoder
2021
Cherti
Tranforacc,
ConvNoXeV2-11
CNN.
IN Deng
0.68
0.87
0.32
0.80
0.81
0.0023
0.81
0.97
0.45
198
Woo us ml. 2023
BL ml
2006
24
0.0022
Ruonnoovky
2015)
D m Touvzon
Transformez, Deng
0.69
0.85
0.24
0.69
0.71
0.0632
0.81
0.98
0.36
39
MV
Transformes
0.70
0.84
0.34
0.75
0.75
0.0048
0.80
0.98
0.29
35
2m
IN1k Rusenoovaky 6t ml
2011 S1.
Towvzon
Tranformec, Rugenbovaky
45

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
QUBA,
Advi
G-Rob..
Balance
Focus
Shape,
Params, MiL."
0.57
0.0027
V-T-B/14-
Trandformet.
0.74
0.55
0.12
0.79
0.79
0.0011
0.80
0.94
0.49
90
DINOv2erep-iP
12D142m (Qquab
Danges
66
Con
ONN
1N21k Deng
0.84
0.72
0.73
0.0026
0.80
0.85
0.35
Woo
2023
2006
Ding
Tranfformer
0.84
0.75
0.70
0.0022
0.55
2015
MVITv2-h
14
Prandrrmer
0.76
0.84
0.34
0.75
0.73
0.0035
0.81
0.95
0.33
52
It xl
20225
IN1k
m 2010
CanvNnXeVVas
2023
IN1k
Russolovaky
Rffictent
CNN:
0.82
0.85
0.20
0.75
0.90
0.0048
0.82
0.94
0.31
Nie 2020
Himton
Sum
CaiTTx224 Towvzon HC
Treasformez.
0.82
0.85
0.80
0.77
0.64
0.0022
0.80
0.95
0.29
47
20210
IN1k
Jovoly 2015
0.86
0.79
0.50
0.0035
0.96
2022
INkk
Lurenloovayyy
2015
Hicro-J
Treaffrmer,
0.0114
Rusogkoraky
Ti-h
Towvzon
Frandfomerr [
0.83
0.88
0.28
0.73
0.76
0.0027
0.81
0.95
0.40
87
St m
2000
CeavNeXXL
IN21 Tronsformee,
0.0022
20221
BelT-b Bao el ol. 2092
0.84
0.85
0.07
0.75
0.80
0.0036
0.81
0.96
0.58
86
N21
Doug
VI-T-B/14-
Trandormee
0.86
0.81
0.82
0.0045
DINO-2-rog-22
LVD148 42m Ququnb
Pucoci
2024
2024
122
VIT-B/14-
Tranformer
Qquanb
CourNeXt-B
Liu
CNN.
I821 Deug
0.85
0.89
0.20
0.78
0.82
0.0019
0.81
0.98
0.38
89
20220
6L ml
2009
BVA02.g
0.0030
2024D
N21
Dong
vole31 (82E))
VIT-21 M/16
Transfarmer,
0.90
0.85
0.13
0.73
0.80
0.0834
0.81
0.95
0.37
21
Wi al 2022
Dong
Swin 2222/1
Trandormes.
0.85
0.81
0.83
0.0040
0.96
Lio
2022
IN21 Dong
2009
3b
Nei
CNN.
JFT-
0.98
0.57
0.29
0.77
0.96
0.0064
0.53
0.95
0.44
66
Xic et ml 2020
5003 Hipton
A1
200
Sun
Sovaley
Siiiien BB
CNN.
JPT
0.94
0.88
0.25
0.77
0.83
0.0045
0.82
0.95
0.36
Nie al 2020
300M
300M{1Inston [1inton
Bt
AL
Dmn
HT
46

---

Published in Transactions on Machine Learning Research (01/2026)
Confguration
Advi
G-Robb.
Balance
Focus 1
Shape,
MiL."
Hicra-B
Ryali
Transformer,
0.0150
ConrXeXeV2-B
CNN.
1k Deng
0.98
0.87
0.25
0.79
0.82
0.0023
0.81
0.58
0.40
89
Woo es al.
2020
2009
0.0130
Ryall
3023
IN
Pans
Trandarmer,
0.88
0.81
0.66
0.0039
0.57
0.34
Dang (825)
47


Reviews:
[{"review_id": "ekw9620vqI", "reviewer": "Reviewer_kA2D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the performance quality of various models on the ImageNet-1k dataset. It employs multiple evaluation metrics and directs its analysis toward several aspects: model behavior when the amount of training data is varied, behavior under self-supervision, and behavior under domain shifts.\nA major goal of the paper is to evaluate performance beyond accuracy. The paper presents extensive and comprehensive results (and I would like to strongly emphasize this point).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is highly practical in nature. The material is divided into two parts: detailed results are presented in the appendix, while the main paper focuses on describing the experiments, formulating hypotheses, summarizing results, and presenting conclusions.\nIn the abstract, the paper makes three claims, all of which are supported by the results. The extensive findings presented in the appendix reinforce the conclusions regarding all three aspects: domain shifts, self-supervised learning, and variations in the amount of training data.\nI find the proposed metric, QUBA, to be relevant and the built conclusion also, interesting.", "audience_yn": "Yes", "audience_explanation": "The paper delves deeply in explaining robustness with respect to classification. While the problem of image classification is of secondary importance today, yet many model trained on ImageNet are still used as starting point in further training. This paper helps to understand the consequences of pra-training", "requested_changes": "None. While there are some phrases which made be more clear, he overall text is understandable and it represent the authors. In my view, the paper is fine", "broader_impact_concerns": "It is not the case", "additional_comments": "While I have highlighted the strengths in the previous section, the most notable weakness is the limitation of the evaluation to ImageNet and image classification only. Nevertheless, the amount of experimentation conducted to support the conclusions is impressive, and asking the authors to broaden the scope would be unreasonable and would likely dilute the paper’s message. In my view, extending the scope should be the focus of future work. Therefore, I believe the paper is well-balanced and appropriate in its current form.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fOsXkcDBou", "reviewer": "Reviewer_wn2T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the limitations of evaluating deep neural networks (DNNs) for image classification solely based on accuracy. The authors systematically study nine different quality dimensions—accuracy, adversarial robustness, corruption robustness, out-of-domain robustness, calibration error, class balance, object focus, shape bias, and parameter efficiency—on a large zoo of 326 backbone models. They analyze the effect of different training paradigms (supervised, adversarial, self-/semi-supervised, vision-language models) and architectures (CNNs, Transformers, B-cos, etc.) on these dimensions.\n\n**Strengths:**\n\nVery broad and systematic evaluation across multiple dimensions.\n\nComprehensive experimental study with 326 models, offering a “bird’s-eye view” that is rare in the field.\n\nThe QUBA score provides a practical tool for the community.\n\nWell-written, clear figures and tables to support conclusions.\n\n\n\n**Weaknesses:**\n\nAll evaluations are limited to ImageNet-1k and its variants; it is unclear if the findings generalize to other domains or tasks.\n\nSome dimensions (e.g., interpretability, OOD detection beyond classification) are excluded.\n\nWhile the breadth is excellent, depth of analysis on individual surprising findings is limited.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are consistently backed by large-scale empirical evidence. The authors use a diverse set of models and standardized evaluation protocols, minimizing cherry-picking. Statistical significance testing is reported in the comparisons, and the proposed QUBA score is validated for stability. While the scope is limited to ImageNet-1k, within that scope the evidence is convincing and the methodology is rigorous.", "audience_yn": "Yes", "audience_explanation": "The findings are highly relevant to both academic researchers and practitioners in machine learning and computer vision. The paper addresses the community’s over-reliance on accuracy as a single metric and provides actionable insights into robustness, fairness, and calibration. The proposed benchmark and QUBA score can serve as useful tools for model selection and design, making the paper broadly interesting to TMLR’s readership.", "requested_changes": "Critical: Clarify the limitations regarding generalisation beyond ImageNet-1k.\n\nStrengthening (not critical): Provide deeper discussion of some surprising results (e.g., why self-supervised linear probing reduces object focus despite attention maps suggesting otherwise).\n\nStrengthening (not critical): Discuss whether QUBA score weightings could be made adaptive or user-configurable in practice, possibly with a released toolkit.", "broader_impact_concerns": "The paper mainly benchmarks existing models and does not propose new architectures or training pipelines. Thus, the ethical risks are limited. However, as the study evaluates fairness only through class balance on ImageNet-1k, a dataset with its own biases, the conclusions about fairness may not fully capture real-world concerns. The authors should briefly acknowledge dataset bias as a limitation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cC1gCy0ZA9", "reviewer": "Reviewer_UDNH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies image classification models beyond the traditional focus on accuracy, introducing a broad evaluation of nine quality dimensions including robustness, calibration, fairness, object focus, shape bias, etc. The authors analyze many backbone models under different training paradigms and architectures, revealing interesting trade-offs and synergies. They find that self-supervised and semi-supervised training generally improve multiple quality dimensions, while adversarial training strengthens robustness but harms accuracy and fairness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are fair within the scope of the paper, the paper comes with a clear self-noted limitation writing mentioning that these conclusions might not generalize.", "audience_yn": "Yes", "audience_explanation": "Audience working in deep learning computer vision will be interested.", "requested_changes": "1. The paper has offered some extensive scope and some interesting conclusions. Unfortunatley, some of the conclusions might not be objective enough, for example, when the authors mention the advantages of self-supervise training, what are exactly the setting of the self-supervised training? The authors offered some details in LP and E2E, but these read like the models are still fine-tuned in ImageNet, in this case, is it really the advantages of SST that leads to the performances, or is it more like a clear initalization of the parameters (obtained through SST) that lead to the performances. How much can we credit to SST, instead of fine-tuning with a clever set of initializations. \n   - to answer this, the authors might need to offer more detailed discussions in the configurations of fine-tuning after SST, and probably need to demonstrate the performance differences along the convergence of fine-tuning after SST. \n\n2. training on large dataset can help seems to be a very under-whelming conclusion. Similiarly, the transformer and cnn argument. \n   - In addition, it has been a long discussing/debating topic of how to measure the performances w. this kind of questions. When the authors mention that training on large dataset can help, they are measuring pure performances. On the other hand, people might get different opinions if they measure the gap between other performances vs. i.i.d test accuracy. \n\n3. Following the above comment, a more clear discussion of the measurement and a more clear note that the conclusion will depend on the measurement might be necessary. \n\n4. The authors might have to use autoAttack for adversarial attack, which is a standard of the community now. \n\n5. Given the scope of this paper, it's well understood that the authors will not be able to cite every paper, here are some suggestions: \n     - in terms of shape bias and mitigation [1, 2, 3]\n     - in terms of AT improves shape bias [3, 4]\n     - a relevant work trying to offer a full package view of many of these factors of ML and computer vision, but from a more methodology view [5]\n\n\n[1] The origins and prevalence of texture bias in convolutional neural networks \n\n[2] Measuring the tendency of cnns to learn surface statistical regularities \n\n[3] High-frequency component helps explain the generalization of convolutional neural networks \n\n[4] Learning perceptually-aligned representations via adversarial robustness\n\n[5] Towards Trustworthy and Aligned Machine Learning:A Data-centric Survey with Causality Perspectives", "broader_impact_concerns": "None.", "additional_comments": "Overall, i think it's a nice reading, but some claims and writing needs to get further tighten.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)