Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement
Authors: Bryan Bo Cao
Lawrence O'Gorman
Michael Coss
Shubham Jain
Venue: ICLR 2025 Poster (2025)
Keywords: Few-Class
lightweight
small neural network
benchmark
scaling law
image similarity
convolutional neural network
CNN
transformer
Primary area: datasets and benchmarks
TL;DR: We propose a Few-Class neural network benchmark for model selection, along with a novel SimSS metric for evaluating dataset difficulty, accompanied by in-depth analyses.

Abstract: We propose Few-Class Arena (FCA), as a unified benchmark with focus on testing efficient image classification models for few classes. A wide variety of benchmark datasets with many classes (80-1000) have been created to assist Computer Vision architectural evolution. An increasing number of vision models are evaluated with these many-class datasets. However, real-world applications often involve substantially fewer classes of interest (2-10). This gap between many and few classes makes it difficult to predict performance of the few-class applications using models trained on the available many-class datasets. To date, little has been offered to evaluate models in this Few-Class Regime. We conduct a systematic evaluation of the ResNet family trained on ImageNet subsets from 2 to 1000 classes, and test a wide spectrum of Convolutional Neural Networks and Transformer architectures over ten datasets by using our newly proposed FCA tool. Furthermore, to aid an up-front assessment of dataset difficulty and a more efficient selection of models, we incorporate a difficulty measure as a function of class similarity. FCA offers a new tool for efficient machine learning in the Few-Class Regime, with goals ranging from a new efficient class similarity proposal, to lightweight model architecture design, to a new scaling law. FCA is user-friendly and can be easily extended to new models and datasets, facilitating future research work. Our benchmark is available at https://github.com/bryanbocao/fca.

Published as a conference paper at ICLR 2025
Few-Class Arena: LECTION OF VISION MODELS AND DATASET DIFFI-
A BENCHMARK FOR EFFICIENT SE-
CULTY MEASUREMENT
Bryan Bo Cao" & Shubham Jain
Department of Computer Science
Stony Brook University Stony Brook, NY, USA
(boccao, jaan)accssstooybroooo... stt.yytrooooee.... edu
Lawrence O'Gorman' & Michael Coss Nokia Bell Labs
Murray Hill, NJ, USA
(larry. alarry.o.goman o.gorman, mi ke coss) @noiiasbeeiiaaebelll com
ABSTRACT
We propose Few-Class Arena (FCA), as a unified benchmark with focus on testing efficient image classification models for few classes. A wide variety of benchmark datasets with many classes (80-1000) have been created to assist Computer Vision architectural evolution. An increasing number of vision models are evaluated with these many-class datasets. However, real-world applications often involve sub- stantially fewer classes of interest (2-10). This gap between many and few classes makes it difficult to predict performance of the few-class applications using mod- els trained the available many-class datasets. To date, little has been offered to evaluate models in this Few-Class Regime. We conduct a systematic evaluation of the ResNet family trained ImageNet subsets from 2 to 1000 classes, and test a wide spectrum of Convolutional Neural Networks and Transformer archi- tectures over datasets by using our newly proposed FCA tool. Furthermore, to aid an up-front assessment of dataset difficulty and more efficient selection of models, we incorporate difficulty measure as a function of class similarity. FCA offers new tool for efficient machine learning in the Few-Class Regime, with goals ranging from new efficient class similarity proposal, to lightweight model architecture design, to new scaling law. FCA is user-friendly and can be easily extended to new models and datasets, facilitating future research work. Our benchmark is available https: /tt///gtttuu com/bry http://bbabbboooooeeeeeee
INTRODUCTION
The de-facto benchmarks for evaluating efficient vision models are large scale with many classes (e.g. 1000 in ImageNet (Deng et al., 2009), 80 in COCO (Lin et al., 2014), etc.). Such benchmarks have expedited the advance of vision neural networks toward efficiency (Tan & Le, 2019a; Tan & I.,, 2021; Sinha & El-Sharkawy, 2019; Sandler et al., 2018; Howard et al., 2019; 19; landola et al., 2016; Ma al., 2018; Mehta & Rastegari) with the hope of reducing the financial and environmental cost of vision models (Patterson et al., 2021; Rae et al., 3021). More efficient computation is facilitated by using quantization (Gysel al., 2018; Han al., 2015: Leng al., 2018), pruning (Cheng al., 2017; Blalock et al., 2020; Li et al., 2016; Shen et al., 2022), and data saliency (Yeung et al., 2016). Despite efficiency improvements such as these, many-class datasets are still the standard of model evaluation.
"ook done during internship at Nokia Bell Labs. +Corresponding authors.

---

Published as a conference paper at ICLR 2025
34 341K
ML. 5K
S0L.a 201
Ms. 132
102
100
90
80
ACC.
70
70
60
50
75
40
40
5
10
30
30
600
800
1000
1000
No
(a) Accuracies for sub-models (blue) and full models (red).
Zoomed window shows accuracy values and range for full and sub-models in few- class range.
100
100
90
Top-1 Acc. (8)
80
70
80
100
60
60
70
50
40
60
40
90
10
10
30
200
400
600
800
1000
200
400
600
800
1000
No.
Na.
(c) Zoomed window shows (c.1) drop of accu- racy Nez decreases, (c.2) accuracy scales with model size for full models in the few-class range.
(d) Zoomed window shows (d.1) rising accu- racy as NeL decreases, (d.2) accuracy does not scale with model size for sub-models in the fow-class range.
Figure I: Top-1 accuracies of various scales of ResNet, whose model sizes are shown in the legend, and whose plots vary from dark light by decreasing size. Plots range along number of classes NeL from the full ImageNet size (1000) down to the Few-Claxs Regime. Each model tested on subsets whose Ne1 classes are randomly sampled from the original 1000 classes. (a) Plots for sub-models trained on subsets of classes (blue) and full models trained on all 1000 classes (red). (b) Zoomed window shows the standard deviation of subset's accuracies is much smaller than for the full model. (c.1) Full model accuracies drop when Nez decreases. (c.2) Full model accuracies increase model scales up in the Few-Class Regime. (d.1) Sub-model accuracies grow as NeL decreases. (d.2) Sub-model accuracies do not increase when model scales up in the Few-Class Regime.
Real-world applications, however, typically comprise only a few number of classes (e.g, less than 10) (Shao et al., 2020; A. Delplanque, 2022; Cai et al., 2021) which we termed Few-Class Regime. This introduces a crucial research question: what is the simplest baseline model capable of meet- ing performance criteria within this Few-Class Regime? To deploy vision model pre-trained on large datasets in a specific environment, it requires the re-evaluation of published models or even retraining to find an optimal model in an expensive architectural search space (Scheidegger al., 2019).
One major finding is that, apart from scaling down model and architectural design for efficiency, dataset difficulty also plays vital role in model selection (Scheidegger et al., 2021) (described in Section 4.3).
Figure 1 summarizes several key findings under the Few-Class Regime. On the bottom left graph in red are accuracy results for range of number of classes Ne1. for what we call the "full model", that is ResNet models pre-trained on the full 1000 classes of ImageNet (generally available from many websites). On the bottom right in blue are accuracy results for what we call "sub-models", each of which is trained and tested on the same NCL. where this number of classes is sampled from the full dataset down to the Few-Class Regime. Findings include the following. (a) Sub-models attain higher upper-bound bound accuracy than full models. (b) The range of accuracy widens for full models at few-classes, which increases the uncertainty of a practitioner selecting a model for few classes. In contrast, sub-models narrow the range. (c) Full models follow the scaling law (Kaplan
2

---

Published as a conference paper at ICLR 2025
et al., 2020) in the dimension of model size larger models (darker red) have higher accuracy from many to few classes. (d) Surprisingly, the scaling law is violated for sub-models in the rew-Class Regine (see the zoomed-in subplot) where larger models (darker blue) do not necessarily perform better than smaller ones (lighter blue). From these plots, our key insight is that, instead of using full models, researchers and practitioners in the Few-Class Regime should use sub-models for selection of more efficient models.
However, obtaining sub-models involves computationally expensive training and testing cycles since they need to be converged on each of the few-class subsets. By carefully studying and comparing the experiment and evaluation setup of these works in the literature, we observe that, how models scale down to Few-Class Regime is rarely studied. The lack of comprehensive benchmarks for few- class research impedes both researchers and practitioners from quickly finding models that are the most efficient for their dataset size. To fill this need, we propose a new benchmark, Few-Class Arena (FCA), with the goal of benchmarking vision models under few-class scenarios. To our best knowledge, FCA is the first benchmark for such purpose.
We formally define Few-Class Regime as a scenario where the dataset has a limited number of classes. Real-world applications often comprise only a few number of classes (e.g. Ne1 10 or 10% classes of a dataset). Consequently. Few-Class Arena refers to a benchmark to conduct research experiments to compare models in the Few-Class Regime. This paper focuses the image classification task, although Few-Class Regime can generalize to object detection (Chen et al., 2019), segmentation (Contributors, 2020) and other visual tasks (described in the Appendix).
Statement of Contributions. Four contributions are listed below:
To the best of our knowledge, we are the first to explore the problems the Few- Tew-Class Regime and develop benchmark tool Few-Class Arena (FCA) facilitate scientific search, analysis, and discovery for this range of classes.
We introduce a scalable few-class data loading approach to automatically load images and labels in the Few-Clas Regime from the full dataset, avoiding the need to duplicate data points for every additional few-class subset.
We incorporate dataset similarity as an inverse difficulty measurement in Pew-Class Arena and propose a novel Silhouette- Based Similarity Score named SimSS. By leveraging the visual feature extraction power of CLIP and DINOV2, we show that SimSS is highly cor- related with ResNet performance in the Few-Class Regime with high Pearson coefficient scores 2 0.88.
We conduct extensive experiments that comprise ten models on ten datasets and 2-1000 numbers of classes on ImageNet, totalling 1591 training and testing runs. In-depth analyses on this large body of testing reveal new insights in the Tew-Class Regime.
2 RELATED WORK
Visual Datasets and Benchmarks. To advance deep neural network research, a wealth of large- scale many-class datasets has been developed for benchmarking visual neural networks over a va- riety of tasks. Typical examples include 1000 classes in ImageNet (Deng et al., 2009) for image classification, and 80 object categories in COCO (Lin et al., 2014) for object detection. Previous benchmarks also extend vision to multimodal research such as image-text (Lee et al., 2024; Le et al., 2024; Laurençon et al., 2024; Bitton et al., 2022). While prior works often scale up the num- ber of object categories for general purpose comparison, studies (Fang al., 2024; Mayo et al., 2023) raise a concern on whether models trained on datasets with such large number of classes (e.g. ImageNet) can be reliably transferred to real world applications often with far fewer classes. A close work to ours is vision backbone comparison (Goldblum et al., 2024) whose focus is on model architectures. Our perspective differs in a focus on cases with fewer number of classes, which often better aligns with real-world scenarios.
Dataset Difficulty Measurement. Research has shown the existence of inherent dataset difficulty (Mayo al., 2023) for classification and other analytic tasks. Efficient measurement methods are proposed to characterize dataset difficulty using Silhouette Score (Rousseeuw, 1987), K-means
A detailed list of many-class datasets used in this paper can be found in the Appendix.
3

---

Published as a conference paper at ICLR 2025
Fréchet inception distance (Dowson & Landau, 1982; Heusel et al., 2017; Lucie al., 2018), and Probe nets (Scheidegger et al., 2021). Prior studies have proposed image quality metrics using statistical heuristics, including peak signal-to-noise ratio (PSNR) (Hore & Ziou, 2010), structural similarity (SSIM) Index ( (Wang et al., 2004), and visual information fidelity VIF (Sheikh & Bovik, 2006). A neuroscience-bass viewing time related to object solution time (OST) (Kar et al., 2019). Another type of difficulty mea- sure method consists of additional procedures such as c-score (Jiang et al., 2020), prediction depth (Baldock et al., 2021), and adversarial robustness (Goodfellow et al., 2014). Our work aligns with the line of research (Arun, 2012; Trick & Enns, 1998; Wolfe et al., 2010) involving similarity based difficulty measurements: similar images are harder to distinguish from each other while dissimilar images are easier. Previous studies are mainly in the image retrieval context (Zhang
image difficulty metrie (Mayo al., 2023) is defined as the minimum
Lu, 2003; Wang et al., 2014; Tudor lonescu et al., 2016). Similarity score is used in (Cao et al., 2023) with the limitation that a model serving similarity measurement has to be trained for one dataset. We push beyond this limit by leveraging large vision models that learn general visual features using CLIP (Radford et al., 2021) 1) and DINOv2 (Oquab et al., 2023). The study (Mayo et al., 2023) shows that CLIP generalizes well to both easy and hard images, making it good candidate for measuring image difficulty. Supported by the evidence that better classifiers can act as better perceptual feature extractors (Kumar et al., 2022), in later sections we show how CLIP and DINOv2 will be used as our similarity base function.
Despite the innovation of difficulty measure algorithms on many-class datasets, little attention has been paid to leveraging these methods in the Few-Class Regime. We show that, as the number of classes decreases, sub-dataset difficulty in the Few-Class Regime plays an increasingly critical role in efficient model selection. To summarize, unlike previous work on many-class benchmarks and difficulty measurements, our work takes few-class and similarity- based dataset difficulty into consideration, and in doing so we believe the work pioneers the development of visual benchmark dedicated to research in the Few-Class Regime.
FEW-CLASS ARENA (FCA)
We introduce the Few-Class Arena (FCA) benchmark in this section. In practice, we have integrated FCA into the MMPreTrain framework (Contributors, 2023), implemented in Python and Pytorch?. Our benchmark usage guidelines are detailed in A.2 of the Appendix.
3.1 FEW-CLASS DATASET PREPARATION
Pew-Class Arena provides an easy way to prepare datasets in the Few-Clas Regime. By leveraging the MMPreTrain framework, users only need to specify the parameters of few-class subsets in the configuration files, which includes the list of models, datasets, number of classes (NcL), and the number of seeds (Ns). Few-Class Arena generates the specific model and dataset configuration files for each subset, where subset classes are randomly extracted from the full set of classes, as specified by the seed number. Note that only one copy of the full, original dataset is maintained during the whole benchmarking life cycle because few-class subsets are created through the lightweight con- figurations, thus maximizing storage efficiency. We refer readers to the Appendix and the publicly released link for detailed implementations and use instructions.
3.2 MANY-CLASS FULL DATASET TRAINED BENCHMARK
We conducted large-scale experiments spanning ten popular vision models (including CNN and ViT architectures) and ten common datasets Except for ImageNetIK, where pre-trained model weights
*Code is available at ht https: // gi www.t wtt.://c////c/aae wtw......//byaaabccaa.o/. Eca, including detailed documentation and long- parg-term plans of maintenance.
3 "Models include: ResNet50 (RN50), VGGI6, ConvNeXt V2 (CNv2), Inception V3 (INCv3), EfficientNet V2 (EFv2), ShuffleNet V2 (SNv2), MobileNet V3 (MNv3), Vision Transformer base (VTTb), Swin Trans- former V2 base (SWv2b) and MobileVIT small (MVITs). Datasets include 1a (CT101), Ca1Tech256 (CT256), CIFAR 100 (CF100), CUB200 (CB200), Food101 (FD101), GTSRB43 (GT43), ImageNetl K (INIK), Indoor67 (ID67), Quickdraw345 (QD345) and Textures47 (TT47).
4

---

Table 1: Top-1 accuracy across ten models in ten datasets. Models are trained and tested on full datasets with their original number of classes (e.g. 1K from ImageNetIK) denoted in the last few digits of the abbreviation of the dataset name. The best score is highlighted in bold while the second Published as a conference paper at ICLR 2025
available, train models in other datasets from scratch. While different models' training pro- cedures may incur various levels of complexity (particularly in our case for MobileNet V3 and Swin Transformer V2 base), we have endeavored minimize changes in the existing training pipelines from MMPre'rrain. The rationale is that if a model exhibits challenges in adapting it to a dataset, then it is often not helpful choice for practitioner to select for deployment.
Results are summarized in Table 1. We observe (1) models in different datasets (in rows) yield highly variable levels of performance by Top-1 accuracy; (2) single best model (bold, in columns) exists across all datasets; and (3) model rankings vary across various datasets.
The first observations are consistent with the findings in (Scheidegger et al., 2021; Fang al., 2024). For (1), it suggests there exists underlying -ataset-specific difficulty. To capture this char- acteristic, we adopt the reference dataset classification difficulty number (DCN) (Scheidegger al., 2021) to refer to the empirically highest accuracy achieved in a dataset from finite number of mod- els shown in Table and Figure (a). For observation (3), we can examine the rankings among the ten datasets of ResNet50 and EfficientNet V2 in Figure 2 (b). ResNer50's ranking varies dramati- cally across different datasets, for instance ranking 7th on ImageNet K and 1st on Quickdraw345. This ranking variability is also observed in other models (see all models in the Appendix). How- ever, a common practice is to benchmark models even for efficiency- - on large datasets, especially ImageNetIK. The varied dataset rankings in experiments expose the limitations of such prac- tice, further supporting our new benchmark paradigm, especially in the Few-Class Regime. later sections, we leverage DCN and image similarity for further analysis.
Dataset
RN50
VGG16
CNv2
INCv3
EFv2
SNv2
MNv3
VITb
SWv2b MViTs
DCN
GT43
99.85
96.60
99.83
99.78
99.86
99.87
99.83
99.31
99.78
99.69
99.87
CF100
74.56
71.12
85.89
75.97
77.05
77.89
74.35
32.65
78.49
76.51
85,89
INIK
76.55
71.62
84.87
77.57
85.01
69.55
66.68
82.37
84.60
78,25
85.01
FD101
83.76
75.82
63.80
83,96
80.82
79.36
76.03
$2.21
84.30
82.23
84.30
CT101
77.70
74.99
77.52
77.52
77.82
84.13
80.71
59.59
78.82
80.06
84.13
CT256
65.07
59.08
73.57
66.09
62.80
68.13
62.62
44.23
67.28
65.80
73.57
QD345
69.14
19.86
62.86
68.25
68.81
67.32
66.42
19.67
66.54
68.76
69.14
CB200
45.86
21.26
27.61
45.58
44.48
53.95
53.80
23.73
54.52
58,46
58.46
ID67
53.75
26.01
33.21
45.95
43.85
54.72
51.65
30.51
48.58
54.05
54.72
TT47
30.43
12.55
6.49
14.20
21.17
43.83
40.27
31.38
33.94
24.41
43.83
best underlined for each dataset. References for all models and datasets are in the Appendix.
100
DCN
1st
RN50
Top-1 Abc, (N)
80
2nd
3rd
60
40
20
6th
7th
O741
wik
rami
CT101
CTaW
oo34
De7
merr
00145
CT256
ce2n0
FD101
crim
cri00
067
TTA7
dre
(a) Top-1 accuracy and DCN in ten full datasets.
(b) Ranking of ResNet50 (RN50) and Er-
ficientNet V2 (FFv2) across 10 datasets by
Top-1 acc.
Figure 2: Many-Class Full Dataset Benchmark.
In the next subsections, we introduce three new types of benchmarks: (1) Few-Class Full Dataset Trained Benchmark (FC-Full), which benchmarks vision models trained on the full dataset with the original number of classes; (2) Few-Class Subset Trained Benchmark (FC-Sub), which benchmarks vision models trained on subsets of fewer number of classes than the full dataset, and (3) Few-Class
5

---

Published as a conference paper at ICLR 2025
Similarity Benchmark (FC-Sim), which benchmarks image similarity methods and their correlation with model performance.
3.3 FEW-CLASS FULL DATASET TRAINED BENCHMARK (FC-FULL)
Traditionally, a large number of models are trained and compared on many-class datasets. However, such results cannot be directly transferred to the Few-Class Regime and many real-world scenar- ios. Therefore, we introduce the Few-Class Full Dataset Trained Benchmark (FC-Full), with the objective of effortlessly conducting large-scale experiments and analyses in the Few-Class Regime.
The procedure of FC-Full consists of two main stages. In the first stage, users select the models and datasets upon which they would like to conduct experiments. They can choose to download pre- trained model weights, which are usually available on popular model hubs (PyTorch Hub (Founda- tion, 2024), TensorFlow Hub (Inc., 2024). Hugging Face (Face, 2024), MMPreTrrai (Contributors, 2023) etc.). case no pre-trained weights available from public websites, users can resort to the option of training from scratch. To that end, our tool is designed and implemented to generate bash scripts for easily configurable and modifiable training through the use of configuration files.
In the second stage, users conduct benchmarking in the Few-Class Regine. By specifying the list of classes,
Arena automatically loads pre-trained weights of the chosen models and eval- uates performance of the models on the selected datasets. Note that this process is accomplished through configuration files created by the user's specifications, thus enabling hundreds of experi- ments to be launched by single command. This dramatically reduces the human effort that would otherwise be expended to run these experiments without Few- Cew-Class& Arena.
3.4 FEW-CLASS SUBSET TRAINED BENCHMARK (FC-SUB)
Our study in Figure (red lines) reveals the limits of existing pre-trained models in the Few-Class Regime. To facilitate further research and analyze the upper bound performance in the Few-Class Regime, we introduce the Few-Class Subset Trained Benchmark (FC-Sub).
FC-Sub follows a similar procedure to FC-Full, except that, when evaluating a model in a subset with a specific number of classes, that model should have been trained on that same subset. Specifically, in Stage One (described for FC-Full), users specify models, datasets and the list of number of classes in configuration files. Then Few-Class Arena generates bash scripts for model training on each subset. In Stage two, Few- Cla--Clas&aa Arena tests each model in the same subset that it was trained on.
3.5 FEW-CLASS SIMILARITY BENCHMARK (FC-SIM)
One objective of our tool is to provide the Similarity Benchmark as a platform for researchers to design custom similarity scores for efficient comparison of models and datasets.
The intrinsic image difficulty of a dataset affects a model's classification performance (and human) al., 2018: Mayo al., 2023). We show is intuitive
(Geirhos et al., 2017: Rajalingham that the more similar two images are, the more difficult is for a vision classifier to make a correct prediction. This suggests that the level of similarity of images in a dataset can be used as proxy for a dataset difficulty measure. In this section, we first adopt and provide the basic formulation of similarity, the baseline of similarity metric. Then we propose a Similarity-Base Silhouette Score to capture the characteristic of image similarity in a dataset.
We first adopt the basic similarity formulation from (Cao al., 2023). Intra-Class Similarity s(C) is defined as a scalar describing the similarity of images within a class by taking the average of all the distinct class pairs in C, while Inter-Class Similarity denotes a scalar describing the similarity among images in two different classes G1 and Cz. For a dataset D, these are defined as the mean of their similarity scores over all classes, respectively:
((D)
1
((C))
Σ
cos (Z, Zz),
(1)
T
L X P(C
JeT. ijECt in
(DD) $
Σ
CoC
Σ
cos cos(Z, Zz), (2)
[(D]
|lldd x P(C).C))
L.
d, ab iECIJEC2

---

Published as a conference paper at ICLR 2025
where [L] is the number of classes in dataset, Z, is the visual feature of an image i, |P(C) is the total number of distinct image pairs in class C, [P() is the total number of distinct class pairs, and |p(C.,Ca) is the total number of distinct image pairs excluding same-class pairs.
Averaging these similarities provides a single scalar score at the class or dataset level. However, this simplicity neglects other cluster-related information that can better reveal the underlying dataset difficulty property of a dataset. In particular, the (1) tightness of a class cluster and (2) distance to other classes of class clusters, are features that characterize the inherent class difficulty, but not captured by Sa or S3 alone.
To compensate the aforementioned drawback, we adopt the Silhouette Score (SS) (Rousseeuw, 1987; Shahapure Nicholas, 2020): SS(i) = ba where SS()) is the Silhouette Score of the data point i, a(i) is the average dissimilarity between i and other instances in the same class, and b(i) is the average dissimilarity between i and other data points in the closest different class.
-(()
Observe that the above Intra-Class Similarity ((C) already represents the tightness of the class (C), therefore a(i) can be replaced with the inverse Intra-Class Similarity a(i) (1)... For the second term b(i), we adopt the previously defined Inter-Class Similarity S/ Ca) and introduce a new similarity score as Nearest Inter-Class Similarity S/ which is scalar describing the similarity among instances between class C' and the closest class of each instance in C. The dataset- level Nearest Inter-Class Similarity So(P) is expressed as:
D 8
1 L
(n() B
Σ
Σ
cos 2,),
(3)
14 (PCC IET. VECHJEE
IEL
where C is the nearest class to instance i (Ó # C). To summarize, we introduce our novel Similarity-Bas -Based Silhouette Score SimS5 for dataset D:
1
Sa(1) S' max (S(()
SimSS(D)
|0| REC
(4)
4
EXPERIMENTAL RESULTS
4.1 RESULTS ON FC-FULL
In this section, we present the results FC-Full. A model trained on the dataset with its original number of classes (e.g. 1000 in ImageNetIK) is referred to as a full-class model. These experiments are designed to understand how full-class model performance changes when the number of classes Net. decreases from many few classes. We analyze the results of DCN-Full, shown in Figure (details of all models are presented in the Appendix), and we make two key observations when Ner. reduces the Few-Class Regime (from right to left). (1) The best performing models do not always increase its accuracy for fewer classes, as shown by the solid red lines that represent the average of DCN for each NeL- (2) The variance, depicted by the light red areas, of the best models broaden dramatically for low NCL. especially for NeL < 10.
Both observations support evidence of the limitations of using the common many-class benchmark for application model selection in the Few- Pew-Class Regime, since it is not consistent between datasets that a model can be made smaller with higher accuracy. Furthermore, the large variance in accuracy means that prediction of performance for few classes is unreliable for this approach.
4.2 RESULTS ON FC-SUB
In this section, we show how using Few-Class Arena can help reveal more insights in the Few-Class Regime to mitigate the issues of Section 4.1.
FC-Sub results are displayed in Figure 4. Recall that a sub-class model a model trained on a subset of the dataset where NeL is smaller than the original number of classes in the full dataset.
"The extended derivation is detailed in the Appendix.
7

---

Published as a conference paper at ICLR 2025
CalTech101
CalTech256
CIFAR100
CUB200
Food101
90
95.0
80
DCN-Full (%)
85
92.5
90.0
80
60
87.5
75
60
80
85.0
70
50
200
50
100
50
GTSRB43
ImageNet1K
Ind
Quickdraw345
Textures47
100.0
DCN-Fuil (96)
99.5
90
60
80
45
85
55
40
80
98.5
45
75
98.0
30
500
50
200
10
NcL
Figure DCN-Full by Top-1 Accuracy (%). NeL ranges from many to
Observe that in the Few-Class Regime (when Net decreases from 2) that: (1) DCN increases as shown by the solid blue lines, and (2) variance reduces displayed by the light blue areas.
The preceding observation for FC-Full 4.1 seems contradict the common belief that, the fewer the classes, the higher the accuracy model can achieve. Conversely, the FC-Sub results do align with this belief. We argue that full-class model needs accommodate many parameters to learn features that will enable high performance across all classes in a many-class, full dataset. With the same parameters, however, sub-class model can adapt to finer and more discriminative features that improve its performance when the number of target classes are much smaller.
CalTech101
CalTech256
CIFAR100
CUB200
Food101
100
100
100
100
97.5
95
DCN (%6
90
80
92.5
80
70
90.0
60
60
87.5
70
50
2
2
2
2
3
GTSRB43
ImageNet1K
Indoor67
Quickdraw345
Textures47
100.0
100
100
99.8
80
DCN (%6)
99.6
90
80
60
99.4
60
99.2
80
99.0
2
2
4
2
4
4
3
Ne.
Figure DCN-Sub (blue) DCN-Full (red) by Top-1 Accuracy (%). Not. ranges from to
8

---

Published as a conference paper at ICLR 2025
4.3 RESULTS ON FC-SIM
In this section, we analyze the use of SimSS (Equation 4) as proxy for few-class dataset difficulty. Experiments are conducted on ImageNet1K using the ResNet family for the lower Ncr. & 10% range of the original 1000 classes, Ncr. E (2, 3, 4, 10, 100), and the results are shown in Figure 5. Each datapoint of DCN-Full (diamond red) or DCN-Sub (square in blue) represents an experiment in subset of specific NcL. where classes are sampled from the full dataset. For reproducible results, we use seed numbers from O to 4 to generate 5 subsets for one Net by default. A similarity base function (sim()) is defined as the atomic function that takes a pair of images input and outputs a scalar that represents their image similarity.
In our experiments, (Radford et al., 2021) and DINOv2 (image) (Oquab et al., 2023) by self-supervised learning. Specif- ically, pair of images are fed into its latent space from which the the cosine score is calculated and normalized to 0 to 1. Note that we only use the Image Encoder in CLIP.
leverage the general visual feature extraction ability of CLIP (image + text)
Comparing Accuracy and Similarity To evaluate SimSS, we compute the Pearson correlation coefficient (PCC) (r) between model accuracy and SimSS. Results in Figure 5 (a) (b) show that SimSS is poorly correlated with DCN-Full (r 0.18 and r = 0.26 for CLIP and DINOv2) due to the large variance shown in Section as well as the general features learned on a full dataset, those of which can be extraneous to a much smaller set of larget sub-class features. In contrast, SimSS is highly correlated with DCN-Sub (shown in blue squares), with
0.90 and = 0.88 using CLIP (dashed) and DINOv2 (solid), respectively. We attribute the advantages of DCN-Sub to its focus on the minimal features tailored to the target sub-classes, while maintaining the same number of parameters as the DCN-Full architecture. The high PCC (Wicklin, 2024; Schober et al., 2018) demonstrates that SimSS is reliable metric estimate few-class dataset difficulty, and this can help predict the empirical upper-bound accuracy of a model in the Few-Class Regime. Comparison between SimSS and all models can be found in the Appendix. Such a high correlation suggests this offers a reliable scaling relationship to estimate model accuracy by similarity for other values of NeL without an exhaustive search. Due to the dataset specificity of the dataset difficulty property, this score is computed once and used for all times the same dataset is used. We have made available difficulty scores for many datasets the Few- New-Class Arena site.
100
100
80
suu 190
DCN- Suln (94)
DCN-FuI
70
70
D00 So0
95
95
r
60
pi-0.44
60
p- 16
90
90
50
Be
p-2.4e 10
6.05
810
035
0.2 SimSS[DINOv2]
0.1
0.05
0.10
0.15
0.1
0.2
0.3
SimSS|CLIP)
SimSS(CLIP]
SMSSSDDNDD2]]
(a) DCN-Full
(b) DCN-Full
(c) DCN-Sub
(d) DCN-Sub
Figure 5: Pearson correlation coefficient (r) between DCN and SimSS when Net E (2. 3, 4,5, 10, 100). DCN-Sub (blue squares) is more highly correlated than DCN Full (red diamonds) with SimSS using both similarity base functions of CLIP (dashed line) and DINOv2 (solid line) with 2 0.88.
4.4 COMPARISON WITH FINE-TUNED MODELS
Fine-tuning a model through transfer learning from a praitedieed model has become as common practice in many real-world scenarios. We perform experiments on CNNs (ResNet18, 50) and Trans- former architectures (Mobile Vit-smlll (Mehta & Rastegari, 2021), ViT-Base) summarized in Table 2. To fine-tune (FT) the ResNet1 8, 50 and MobileVit-small models for Net E (2.4). we first train their full models Net = {100} on CIFARI00 for 100 epochs, and subsequently fine-tune their weights for the target NC1. E {2, 4} for another 20 epochs. For ViT, we fine-tune a ViT-B model initialized with weights from the CLIP pre- trained backbone. A linear layer is added on top, and the model is trained for 10 epochs. This setup is indicated by the star symbol (+). We conclude that the fine-tuned models exhibit patterns and trends consistent with the observations presented in Fig. 1. Note our focus of this work is to leverage the proposed difficult measurement method, FC-Sim, to
9

---

Published as a conference paper at ICLR 2025
efficiently estimate the achievable model accuracy, thereby assisting in model selection in the Few- Class Regime. Sub-models can offer insights into the minimal visual features required for a specific real-world scenario as they trained exclusively trained on large full datasets whether through fully supervised manner or self-supervised - may include extraneous features that irrelevant to the target classes. We therefore prioritize sub-model study in this work.
the target classes. In contrast, weights pre-
MT
NeL
ResNet18
ResNet50
MVIT-S
VIT-B
F
100
76.11
73.71
73.83
32.54
F
4
75.10
72.20
72.35
36,15
FT
87.60
90.55
90.00
91.16*
S
90.65
90.15
89.45
85.40
F
75.00
71.30
71.80
40.80
87.90
93.70
90.50
95.20*
96.30
95.30
95.50
95.90
Table 2: Top-1 Accuracies for different configurations CIFARI00. Not E {2, 1000. MT: Model Type, F: Full model, S: Sub-model, FT: Fine-tuned model, MVīT-S: MobileViT-ssmall *: fine-tuned from the CLIP pre-trained model. Best scores are highlighted in bold. The gray bar indicates sub-models as the primary focus of this research.
5 CONCLUSION
We have proposed Few-Class Arena and dataset difficulty measurement, which together form a benchmark tool to compare and select efficient models in the Few-Clas Regime. Extensive exper- iments and analyses over 1500 experiments with ten models on ten datasets have helped identify new behaviors specific to the Few-Class Regime as compared to many-classes setting. One finding reveals a new NNLL-saling law whereby dataset difficulty must be taken into consideration for accu- racy prediction. Such benchmark will be valuable the community by providing both researchers and practitioners with a unified framework for future research and real applications.
Limitations and Future Work. The current difficulty benchmark supports image similarity while in the future and DINOv2 are trained toward general visual features, is unclear if they will be appropriate for other types of images such as sketches without textures in Quickdraw (Ha Eck, 2017). For this reason, universal similarity foundation model would be appealing that applies to any image type. Moreover, integrating image similarity with representation similarity (AntixK, 2023; Nguyen al., 2020; Cao et al., 2024) could further enhance model efficiency, leveraging complementary insights from both approaches. In summary, Few- Tew-Class Arena identifies promising new path for achieving efficiencies focused on the important and practical Few-Class Regime, establishing baseline for future work.
be expanded to other difficulty measurements (Scheidegger al., 2021). CLIP
ACKNOWLEDGMENTS
This research has been supported in part by the National Science Foundation (NSF) under Award numbers 2055520 and 2238553. We appreciate the GPU-cluster maintenance support from Rob Dinoff and Hans Woithe for running large-scale experiments, as well as insightful discussions with Zhengyu Wu and Xi Han significantly enhanced the clarity and readability our plots.
REFERENCES
P. Lejeune J. Linchant J. Théau A. Delplanque, S. Foucher. Multispecies detection and identification of african mammals in aerial imagery using convolutional neural networks. Remote Sensing in Ecology and Conservation, 166pri)):166-111 2022.
AntixK.
Pytorch-model compare..
https: /tte::///tthhtt wwm.coo//maooomm Ant ixk/
PyTorch-Mode Compare, 2023. GitHub repository.
10

---

Published as a conference paper at ICLR 2025
SP Arun. Turning visual search time on its head. Vision Research, 74:86 92, 2012.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34: 111110071188000 2021.
Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. Winogavil: Gamified association benchmark to challenge vision-and- -anion--nd-laguaeee models. Advances in Neural Information Processing Systems, 35:26549-26564, 2022.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-148 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- mining discriminative com- ponents with random forests. In European Conference on Computer Vision, 2014.
Yingfeng Cai, Tianyu Luan, Hongbo Gao, Hai Wang, Long Chen, Yicheng Li, Miguel Angel Sotelo, and Zhixiong Li. Yolov4-5d: An effective and efficient object detector for autonomous driving. IEEE Transactions on Instrumentation and Measurement, 70:1-13, 2021. 2021.3065438.
doi: 10.1109/TIM.
Bryan Bo Cao, Lawrence O'Gorman, Michael Coss, and Shubham Jain. Data-side efficiencies for lightweight convolutional neural networks. arXiv preprint arXiv:2308. 13057, 2023.
Bryan Bo Cao, Abhinav Sharma, Manavjeet Singh, Anshul Gandhi, Samir Das, and Shubham Jain. Representation similarity: A better guidance of dnn layer sharing for edge computing without training. In Proceedings the 30th Annual International Conference on Mobile Computing and Networking, pp. 2-44-2244, 2024.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng. Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 900d: 190.007155 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429) 042997 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp- 9640-9649, 2021.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and accelcration for deep neural networks. arXiv preprint arXiv:1710.09282 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
MMPreTrain Contributors. Openmmlab's pre-training toolbox and benchmark. https://
g thub. http:/cccc////ooo///oo// oppnnmmmnmmm hetmmla/mmminn abb/mmpretra campertriin. n, 2023.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: / com/ wwapn//pen-aeeee mml ab/ ammeementation. 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.

---

Published as a conference paper at ICLR 2025
Xiaohan Ding. X. Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Mak- ing vtg-style convnets great again. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13728-137377 2021. URL htt.s//aapp..mmtttcccccchhoooa
org/ org/Corpus TD: 231572790.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
Hugging Face. Hugging face models. https: / /ttess//huuugge wwwgginggfen co so/models, 2024. URL http::s://hugggggaa...e.m. wwt..oe.oooooooeeeeeeee Accessed on 2024-06-04.
Alex Fang, Simon Kornblith, and Ludwig Schmidt. Does progress on imagenet transfer to real-world datasets? Advances in Neural Information Processing Systems, 36, 2024.
The Linux Foundation. Pytorch hub. sttps://pytore //pytorch. /hub. Accessed on 2024-06-04.
org/ org/hub, hub, 2024. URL https:
Robert Geirhos. David HJ Janssen, Heiko H Sch"utt utt, Jonas Rauber, Matthias Bethge, and Felix A Wichmann. Comparing deep neural networks against humans: object recognition when the signal gets weaker. arXiv preprint arXiv: 06969, 2017.
Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the back- bones: A large-scale comparison of pretrained models across computer vision tasks. Advances in Neural Information Processing Systems, 36, 2024.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint we.iv:1412.66111 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech 256, Apr 2022.
Philipp Gysel, Jon Pimentel, Mohammad Motamedi, and Sohcil Ghiasi. Ristretto: A framework for empirical study of resource-efficien inference in convolutional neural networks. IEEE transac- tions on newral networks and learning systems, 29(1 29(11):5784-5789 2018.
David Ha and Douglas Bck. arXiv:1704.03477 2017.
neural representation of sketch drawings. arXiv preprint
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369. IEBE, 2010.
12

---

Published as a conference paper at ICLR 2025
Andrew Howard, Mark Sandler, Grace Chu, Liang- Ching Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro-
ceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Shell Xu Hu, Da Li, Jan St"uhmer, Minyoung Kim, and Timothy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
9068 0108-9977, 2022.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv:
2016.
Google Inc. Tensorflow hub. https:/ / www. tensorfoww wwwoor/wwworo////// brrg/hu, 2024, URL https: www.tensorflow...m org/ org/hub. hubb. Accessed 2024-06-04.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2 2020.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regu- larities of labeled data in overparameterized models. arXiv preprint arXiv:2002.033006 2020.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 43311):4037-4058, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6)9974-983, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin D Cubuk. Do better imagenet classifiers assess perceptual similarity better? arXiv preprint UrXiv:2203.04946. 2022.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Tiep Le. Vasudev Lal, and Phillip Howard. Coco-counterfactuals: Automatically constructed coun- terfactual examples for image-text pairs. Advances in Neural Information Processing Systems, 36, 2024.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et Holistic evaluation of text- to-image models. Advances in Neural Information Processing Systems, 36, 2024.
Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural net- work: Squeeze the last bit out with admm. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608.000000 2016.
Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, S. Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. ArXiv, abs/2206.01191, 2022b. URL https:/ petttss:/aat //api. i www.r/cchooo...rooorooooeeoe. aot.a/orrrr/ooopsss12222933933
13

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992-10002, 2021. URL https:/ /tpppppppp.p//
seeeaticcschoo ar. www.aar.orroooeeeeeoeeeeeeoo Ptrp/CorrpsTD.. 232352874.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 12009-12019, 2022a.
Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 11966-11776, 2022b. URL https:/ /ttp://appi... http:///aap..aaaiissesseeeeeeeee www.ssemmnniccoo...... org/ erg/Corpus 245837420.
Mario Lucic, Karol Kurach, Marein Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
David Mayo, Jesse Cummings, Xinyu Lin, Dan Gutfreund, Boris Katz, and Andrei Barbu. How hard are computer vision datasets? calibrating dataset difficulty to viewing time. Advances in Neural Information Processing Systems, 36:1 1008-11036, 2023.
Mehta and M Rastegari. Mobilevit: Light-weight, general-purpose purpose, and mobile-friendly vision transformer. arxiv 2021. arXiv preprint arXiv:2110.0217.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile- friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint MrXiv:2010.15327 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07133, 2023.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Mique' Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint wwxiv:2104.1033. 2021.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. 2027 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 357-366, 2021. URL
https: //ts:///pp.. wwwenticscholaa......e..c wtw.:iiscccoaaaooooooooooooooeeo www.//orr//oooo.o :
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pp. 413-420. IEEE, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Jack W Rae. Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12. 2021.
14

---

Published as a conference paper at ICLR 2025
Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large-scale, high-resolution comparison the core visual object recognition behavior of humans, monkeys, and fatte-of-the-ar deep artificial neural networks. Journal Neuroscience, 38(33): 7255-7269, 2018.
Peter Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-65, 1987.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4400 4520, 2018.
Florian Scheidegger, Luca Benini, Costas Bekas, and A Cristiano Malossi. Constrained deep neural network architecture search for iot devices accounting for hardware calibration. Advances in Neural Information Processing Systems, 32, 2019.
Florian Scheidegger, Roxana Istrate, Giovanni Mariani, Luca Benini, Costas Bekas, and Cristiano Malossi. Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy. The Visual Computer, 37(6)) 2021.
Patrick Schober, Christa Boer, and Lothar A Schwarte, Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia, 126(5):1763-176) 2018.
Ketan Rajshekhar Shahapure and Charles Nicholas. Cluster quality analysis using silhouette score. In 2020 IEEE 7th international conference on data science and advanced analytics (DSAA), pp- 747-748. IEEE, 2020.
Zhenfeng Shao, Linggang Wang, Zhongyuan Wang, Wan Du, and Wenjing Wu. Saliency-aware con- volution neural network for ship detection in surveillance video. IEEE Transactions on Circuits and Systems for Video Technology, 30(3):781-794, 2020.
Hamid R Sheikh and Alan C Bovik. Image information and visual quality. IEEE Transactions on image processing, 15(2):430-444 2006.
Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Structural pruning via latency-saliency knapsack. arXiv preprint urXiv:22/0.0660 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, htd//409.1555 2014a. URL : //l::///app. /api. swmantccscoooll........ccccccc CorpusID: 14124311:
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014b.
Debjyoti Sinha and Mohamed El-Sharkawy. Thin mobilenet: An enhanced mobilenet architecture. In 2019 IEEE 10th annual ubiquitous computing, electronics & mobile communication conference (UEMCON), pp. 0280-0285. IEEE, 2019.
Yisheng Song. Ting Wang. Puyu Cai, Subrota K. Mondal, and Jyoti Prakash Sahoo. A comprehen- sive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 55(13s): 1-40, 2023.
Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, (0):-, 2012. ISSN 0893-6080. doi: n0.10066j.neuuet.20022 222.2.2..2.26666. URL http:://ww//tt www www.:ssiieeedeieeeeccecc.. com/ sttp//ccenneeee// artiicleeel//0083330010000000
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pp. 1199-1208, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826, 2016.
15

---

Published as a conference paper at ICLR 2025
M. Tan and Q. L. Efficientnetv2: Smaller models and faster training. In International conference machine learning. pp. 101966 PMLR, 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In International conference on machine learning, pp. 6105-61 14. PMLR, 2019a.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural 2019b. URL http:::://ap..seee wtw.appssettttscscoooloooooooeee org/
networks. ArXiv, CorpusID: 167217261.
Lana M Trick and James Enns. Lifespan changes attention: The visual search task. Cognitive Development, 13(3):669-388 1998.
Radu Tudor lonescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim P Papadopoulos, and Vittorio Ferrari. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2157- 2166, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD
Dataset. Jul 2011.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine- ine-grained image similarity with deep ranking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1386-1393, 2014.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34. 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Rick Wicklin. correlation.
Weak or strong? how to interpret spearman or kendall
com/ wowtentt interpret spearman https://bloge http://caeeee///////2///000000/0 wwteee//imm/2//////0000 -icerpppetssspperrraaeeeeaaaaloe
2024. URL stttps://bloog sas.
weeeeeepreetsppparreeeeeee spearman wwwd-kk-dddlllrraeeeeee html.
Accessed on 2024-06-04.
Jeremy M Wolfe, Evan M Palmer, and Todd $ Horowitz. Reaction time distributions constrain models of visual search. Vision research, 50(14):1304-1311, 2010.
Sanghyun Woo, Shoubhik Debmath, Ronghang Hu, Xinlei Chen, Zhuang Liu, So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133- 16142, 2023.
Serena Yeung. Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detec- tion from frame glimpses in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2678-2687, 2016.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 12104-12113 12113, 2022.
Dengsheng Zhang and Guojun Lu. Evaluation of similarity measurement for image retrieval. In International conference on neural networks and signal processing, 2003. proceedings of the 2003, volume 2, pp. 928-931. IEEE, 2003.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient con- volutional neural network for mobile devices. 2018 IEFF/CVF Conference on Computer Vision and Patter Recognition, pp. 6848-6856, 2017. URL https: appi. semant icscholar. org/CorppsID:22 24:2:221:::29
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 GOALS
Generality, All vision models and existing datasets for classification should be compatible in this framework. In addition, users can extend to custom models and datasets for their needs.
Efficiency. The benchmark should be time- and space-efficient for users. The experimental setup for the few-class benchmark should be easily specified by few pppar--rrameterss (e.g. number of classes). Since the few-class regime usually includes sub-datasets extracted from the full dataset, the benchmark should be able to locate those sub-datasets without generating redundant duplicates for reasons of storage efficiency. For time-efficiency, should conduct training and testing automat- ically through use of sper-spe configuration files, without users' manual execution.
3. Large-Scale Benchmark. The tool should allow for large-scale benchmarking, including train- ing and testing of different vision models on various datasets when the number of classes varies.
A.2 BENCHMARK USAGE GUIDELINE
Users should prepare the dataset detailed in 3.1 in the "CUSTOM" format based on the MMPre (Contributors, 2023) documentation. The tool is designed for both practitioners and researchers.
Users select the target NCL and then execute the FC-Sim on the custom dataset. FC- Sim calculates the image similarity score which can be used index narrow range of potential target models, given the deployment accuracy requirement (c.g. accuracy). We provide the index table covering ten models on ten datasets for the Few-Class Regime.
Practitioner:
Researcher: Users specify the configurations for FC-Full, FC-Sub and FC-Sim, as well list of NeL. the next step, users can execute the scripts running on all benchmarks on the custom dataset. The results for all benchmarks are then used for further analysis.
A.3 EXTENDED RELATED WORK
Few-Shot Learning. There has been a large body research on Few-Short Learning (FSL) (Song al., 2023; Wang al., 2020; Hu et al., 2022; Sung al., 2018). However, the fundamental research questions differ from ours in the Few-Class Regime. The FSL framework aims to address the problem of data scarsity with the goal for model to leverage the representations from very few samples (or none, in the case of Zero-Shov Learning), or prior knowledge that can generalize effectively to other tasks or domains.
Self-Supervised Learning. To leverage the knowledge from unlabeled data, Self-Supervised Learn- ing (SSL) has emerged as an effective learning framework to learn general vision features (Jaiswal et al., 2020; Jing & Tian, 2020). This includes techniques such as Contrastive Learning, applied to single modalities (Chen et al., 2020a; Chen He, 2021; He et al., 2020; Chen et al., 2020b; 2021) or multiple modalities (Radford et al., 2021), as well as mask-and-reconstruct methods (He et al., 2022), among others.
In contrast the advancements of the aforementioned learning frameworks, Few-Class Arena fo- cuses on the research problem of selecting the most efficient model with minimal features (e.g. model parameters, model multiplies) needed for the target application deployment.
A.4 FULL MODELS ON IMAGENET
In practice, ImageNet serves as a common benchmark for vision neural networks. We list the details of ten pre-trained models from MMPreTrain (Contributors, 2023) in terms of Top-1 Accuracy and scale (#Paarmm)) in Fig. 6.
17

---

Published as a conference paper at ICLR 2025
Diameter #FLOPs
85.0
8
16
32
82.5
64
80.0
Conformer
Top-1 Acc. (%)
ConvNeXt
EfficientFormer
EfficientNet
77.5
RepVGG
Res
ShuffleNet
75.0
Swin
VGG
VIT
72.5
70.0
67.5
10
10a
# Params (M)
Figure 6: Top-1 Accuracy (%) vs. number of parameters and FLOPs (G) (size of circle) on Ima-
geNet.
Model
Ref.
Model
Ref.
Conformer
(Peng et al., 2021)
ConvNeXt
(Liu et al., 2022b)
EfficientFormer
(Li et al., 2022b)
EfficientNet
(Tan Le, 2019b)
RepVGG
(Ding et al., 2021)
ResNet
(He et al., 2016)
ShuffleNet
(Zhang et al., 2017)
Swin
(Liu et al., 2021)
VGG
(Simonyan & Zisserman, 2014a)
VIT
(Dosovitskiy et al., 2020)
Table 3: Full models pre-trained on ImageNet.
A.5 EXTENDED MANY-CLASS FULL DATASET TRAINED BENCHMARK RESULTS
A complete ranking of ten models in ten datasets is depicted in Fig. 7. Observe that the ten models' rankings differ dramatically among ten different datasets where each line changes from ImageNet K (INIK) to other datasets. This poses be a reliable indicator for a practitioner to select an efficient neural network, especially when the deployed environment changes from application application. A major variable in this process is the reduced number of classes from benchmark datasets to deployed environments in the Few-Class Regime, As such, our tool developed facilitate research the Few-Clat Regime.
questions whether rankings in existing benchmarks can
18

---

Published as a conference paper at ICLR 2025
VGG16
INCv3
SNv2
MVITs
1st
2nd
Rank Top-1 Acc.
4th
5th
7th
8th
10th
INIK
QD345
CT256
CB200
FD101
CT101
CF100
ID67
TT47
GT43
Dataset
Figure 7: Extended Details of Fig. (b) in the main paper. Full Ranking of models across datasets by Top-1 acc.
A.6 DATASETS
Dataset information is presented in Table 4.
Dalasel
Abbrey. Ref.
Homepage
Path FCA
Name
Caltech ta1
CTIDI (1.
al,
duta.caltoch cit/ccooddd///iggeee.....
toxllalnc soittttellttteeerrcmmm stcaltcch
2022g)
Caltech 256
CT256 (Griflin et al, 2022y
data, caltoch www.coorrriiiyyl1----20 15-4444
CIFAR-100
CF100 (Krizhevaky
es toronto. edul knioicifar..mml
sooiia/ecttiiattaseess.c atciffa 1000
20099
Caltech- UCSD
CB200 all,
caltech htt.://wwvviiiieeiiuSCUBB L.tml
201
Binds 200
dasa www.ecllleccccc www.ltech..duuccoooddtteeeee UBBBee 20110000001000011100000000000010
2011
Food 101
RDI01 (Bossard al., 2014)
vision on ather chidatasets en extn//ood-101
www.la/cchllittaeee sTood 101
Iugginggacc Mossttot
(Stallkamp et all, 2012)
Traffic Sign
ImageNer
(Deng 2009)
al,,
imsago-net nrg.eeeee..tteeeeeeee www.chatlenngeeeeeoooo 2/index
Indoor
ID67
(Quationi
web.mit. wdw.orralbaaaaaooooooooo findoor.hhnll
sookis/ac coolss/acliddaaaseessssi inndoooreeeesiionnnniiooo
Sceme
Tomalbu, 2009)
Recognition
Quickfraw
QD345 (Ha & Eek,, 2017)
gitlish hwwp://ccoooooeeeeiiiiiiiiiaiaie htw.r/rreeeiiiiiiiieeeeeeeeeeerr
tenserflow http://www..tttttttes/ommnmnnn www.gigggggggggggiigieooo
Describable
(Cimpoi 2014l
ac vggllaatadddfiideee...m
Textures
Dataset
Table 4: Dataset information. Note that ImageNet dataset format is used as the reference for other datasets. Therefore, the Path in FCA is not required for ImageNet.
License. We have searched available online resources and list the license of each dataset in Table For licenses not found the datasets or websites denoted as **n we assume they are non- commercial research use only.
19

---

Published as a conference paper at ICLR 2025
Dataset
License
Dataset
License
CT101
CC 4.0
CT256
CC BY 4.0
CF100
MIT
CB200
CC BY 4.0
FD101
BC-SA 4.0
GT43
GPLv2
INIK
ID67
QD345
B.0
TT47
Table 5: Licenses of ten datasets.
Train/val splits. The dataset format follows the convention of ImageNet:
imagenetkk/ meta
train.txt val.txt
train
<IMAGE_ JDUGE_ID>,,, jpeg
val
TIMAE__IDS jpeg
where a .xxt stores a pair of image id and and class number in each row the following format:
<IMAGE ID>. jpeg <CLASS_NUN
We follow the same train/val splits when the original dataset has already provided. the dataset does not have explicit splits, first assign image IDss to all images, starting from O. We then select 4 4/5 of the images as the training set and place the rest in the validation set. Specifially, if an image's moved the validation set; otherwise, is assigned
ID satisfies the condition ID % as training sample.
20

---

Published as a conference paper at ICLR 2025
A.7
MODEL TRAINING DETAILS
Model training details are presented in Table 6.
Model
Ref.
Optimizer
Weight Decay
Other Params
2016)
momemum-DD99
VGGT6
VGG16
(Simoeyan (01/bb
7issermnn,
SGD
le4
ConNNeXI
(Woo al, 2023)
ops=le-8
Base
betza-1009 0.999)
Inception V3
(Soegody 2016)
0.1
momemmmn--.9
EtticientNet
(Ton 2021)
momemmmm-D.9
Medium
clip grad:
max nomm-3.0000
ShoffleNet V2
SNv2
(Ma et al.
SGD
0.5
0.9
momenium:iie-5
MobileNet V3
MN+3
(Howwrd et 2019)
RMSprop
6.4e44
1e-5
alpha=019
Small
momenium=0,
eps-0.1116
Vision Transformer
VITN
(Doowitskly ot 20200
Adamm
36.3
0.3
Buse
Swin Transformer V2
SWv2b
(Lin al., 2022a)
AdamW
1c-4
0.08
ops-1e-8
betas=10.9,
McbikeVT
MV/T'
(Malta & Rastegari)
SGD
0.1
1e-4
momeettum-0.)
Small
Table 6: Model Training detials. LR: Learning rate. SGD: Stochastic gradient descent. Adam W: Adam with weight decay. RMSprop: Root mean square propagation.
21

---

Published as a conference paper at ICLR 2025
A.8 EXTENDED FEW-CLASS FULL DATASET TRAINED BENCHMARK (FC-FULL) RESULTS
We present the details of FC-Full results for each experiment model, including ResNet50, VGGI6, ConNeXt V2 Base, Inception V3, EfficientNet V2 Medium, ShuffleNet V2, MobileNet V3 Small, VIT Base, Swin Transformer V2 Base, MobileViT Small in Fig. 8-17, respectively.
CafTech101
CafTech256
CIFAR100
CUB200
Food101
92.5
Top-1 Acc. (%)
80
90.0
87.5
75
50
50
85.0
70
82.5
40
80.0
50
GTSRB43
ImageNelIK
Indoor67
Quickdraw345
Textures47
100.1
30
Top-1 Acc. (%)
100.0
80
20
99.9
70
30
65
99.8
10
60
60
30
500
200
Figure 8: FC-Full Top-1 Accuracy (%) for ResNet50.
Calfech101
256
CIFAR100
CUB200
Food101
65
80
40
80
Top-1 Acc. (96)
75
85
30
70
80
60
50
65
20
50
45
60
75
40
0
50
0
200
50
100
50
GTSRB43
ImageNet1K
Indoor67
Quickdraw345
Textures47
15
40
15.0
98
Top-1 Acc. (%6
10
12.5
30
96
5
10.0
65
7.5
94
60
5.0
55
-5
2.5
10
20
30
500
0
50
200
10
20
30
No.
Figure 9: FC-Full Top-1 Accuracy (%) for VGG16.
22

---

Published as a conference paper at ICLR 2025
CafTech101
TTech2256
CIFAR100
Food101
90
80
Top-1 Acc. (%)
80
90
40
80
70
70
70
60
60
80
50
20
60
50
200
0
50
0
100
50
GTSRB43
Ima
Indoor67
Quickdraw345
Textures47
100.0
90
40
Top-1 Acc. (%)
99.8
85
30
20
80
2
99.6
10
75
10
20
30
500
50
0
200
10
20
30
NeL
Figure 10: FC-Full Top-1 Accuracy (%) for ConNeXt V2 Base.
CalTech101
CaITech256
CIFAR100
CUB200
Food101
85
60
80
70
Top-1 Acc. (%)
70
80
55
90.0
87.5
60
45
85.0
50
70
35
65
50
200
50
GTSRB43
ImageNet1K
Indoor67
Quickdraw345
Textures47
50
80
99.8
75
Top-1 Acc. (%)
99.6
45
75
99.4
65
40
70
99.2
35
99.0
55
60
10
30
0
500
0
200
Ne
Figure 11: FC-Full Top-1 1 Accuracy (%) for Inception V3.
23

---

Published as a conference paper at ICLR 2025
CalTech101
CalTech256
CIFAR100
Food101
85
Top-1 Acc. (%)
80
87.5
85.0
75
50
82.5
50
70
40
77.5
0
50
0
200
0
50
0
100
50
GTSRB43
ImageNetIK
Indoor67
Quickdraw335
Textures47
50
45
Top-1 Acc. (96
25.0
99.8
40
22.5
70
70
35
20.0
99.6
65
30
17.5
60
25
99.4
10
20
30
0
500
0
50
0
200
10
20
30
NcL
Figure 12: FC-Full Top-1 Accuracy (%) for EfficientNet V2 Medium.
CalTech101
CalTech256
CIFAR100
CUB200
Food101
85
65
90
Top-1 Acc. (%))
85
70
60
80
80
55
85
60
75
75
70
50
70
45
80
65
40
200
100
GTSRB43
ImageNet1K
Indoor67
Quickdraw3345
Textures47
99.5
75
60
80
50
Top-1 Acc. (%6
99.0
70
55
75
45
98.5
65
70
98.0
60
45
65
97.5
55
60
35
97.0
10
500
200
10
20
No
Figure 13: FC-Full Top-1 Accuracy (%) for ShuffleNet V2.
24

---

Published as a conference paper at ICLR 2025
CalTech101
CafTech256
CIFAR100
CUB200
Food101
70
85
90
70
Top- Acc. (%)
80
80
60
85
75
60
80
50
70
50
75
40
60
50
200
0
50
100
0
50
GTSRB43
ImageNet1K
Indoor67
Quickdraw.345
Textures47
99.9
55
80
45
Top-1 Acc. (%)
99.8
70
50
99.7
40
70
99.6
35
99.5
50
60
30
99.4
10
20
30
500
0
50
200
10
20
30
Ne1
Figure 14: FC-Full Top-1 Accuracy (%) for MobileNet V3 Small.
CalTech101
Caflech256
CIFAR100
CUB200
Food101
70
50
Top-1 Acc. (%)
60
40
40
30
40
30
40
30
20
30
30
10
20
200
100
50
GTSR843
ImageNet1K
Indoor67
Quickdraw345
Textures47
35
99.50
30
Top-1 Acc. (%)
85
30
35
99.00
80
25
30
10
98.75
75
20
25
98.50
70
15
20
500
50
200
10
Na.
Figure 15: FC-Full Top-1
Accuracy (%) for VIT Base.
25

---

Published as a conference paper at ICLR 2025
CafTech101
CalTech256
CIFARIOD
CUB200
Food101
70
95.0
85
Top-1 Acc. (%)
80
70
92.5
80
60
70
75
87.5
60
50
85.0
70
82.5
50
0
200
0
50
100
0
50
GTSRB43
ImageNetIK
Indoor67
Quickdraw.345
Textures47
60
90
40
Top-1 Acc. (%)
50
75
35
70
99.6
40
65
30
75
99.4
30
60
70
10
20
30
0
500
0
50
200
10
20
30
No.
Figure 16: FC-Full Top-1 Accuracy (%) for Swin V2 Base.
CelTech101
CalTech256
CIFAR100
CUB200
Food101
70
70
Top-1 Acc. (%6)
80
65
80
60
60
70
85
65
80
100
GTSRB43
ImageNet1K
Indoor67
Quickdraw345
Textures47
100.2
85
27.5
Top-1 Acc. (%6
100.0
0.4
25.0
99.8
22.5
99.6
0.2
70
20.0
99.4
0.0
99.2
15.0
10
30
500
50
200
Ne
Figure 17: FC-Full Top-1 1 Accuracy (%) for MobileNetViT Small.
26

---

Published as a conference paper at ICLR 2025
A.9 EXTENDED FEW-CLASS SIMILARITY BENCHMARK (FC-SIM) DETAILS
We present the extended mathematical details of Section 3.5 Few-Class Similarity Benchmark (FC- Sim) in the main paper.
The basic similarity formulation is adopted from (Cao et al., 2023). Notations are defined as follows:
Dataset D: set of image instances in dataset.
Class C: a set of image instances a class and |C] is the number of instances within class C.
Class Label L: set of class labels in dataset and [L] is the number of classes in dataset.
Feature Z3 visual feature of an image and is the instance index.
Class Pair P(D); set of distinct class pairs in a dataset D; P(D) is the total number of distinct class pairs.
Intra-Class Image Pair P(C); set of distinct image pairs in a class C; | is the total number of distinct image pairs.
Inter-Class Image Pair ((C).C2); a set of distinct image pairs in two classes C1, C2; |p(C).Ca) is the total number of distinct image pairs. Note that this does not include same-class pairs.
Intra-Clas Similarity s(
scalar describing the similarity of images within class by taking average of all distinet class pairs
Σ
cos cos 7z)-
(5)
AJJEC
Inter-Class Similarity $ C2,, scalar describing the similarity among images in two different classes C1 and C2:
$ S(Cr.
cos cos Zj),
(6)
P(C1.C2)
iECIJECa
where C and C2 are distinct classes, and j are the image instance indices in C₁ and C2. respec- tively. C(), is the set of distinct pairs of images between C₁ and C2.
The above equations formulate class-level similarity scores. For letaset-level, Intra-Class Simi- larity and Inter-Class Similarity dataset D are defined as the mean of their similarity scores, respectively:
S(D)
((C))
Σ IEL
Σ
cos (Z, Zz),
= L
a
IL| X P(C))
IEL
ideC (
((D))
Σ
s
S(C,,CC))
Σ
Σ
cOs (z, Zz). (8)
P
|iDi x C2)
L:
a,bEL; aæb iEC,,JEC2
Averaging these similarities can provide a summary of score in class or dataset levels by a single scalar. However, this simplicity neglects other cluster-related information that better reveal the underlying dataset difficulty property of dataset. In particular, the (1) tightness of a class cluster and (2) distance other classes of class clusters, are features that characterize the inherent class difficulty. but are not captured by S,, or Sg alone.
To compensate the aforementioned drawback, we adopt the Silhouette Score (SS) (also called Sil- houette Coefficient in the literature) (Rousseeuw, 1987; Shahapure & Nicholas, 2020):
b(1) - a(i)
SS(1) max max(a b(1))
where SS(1) is the Silhouette Score of the data point i, a(i) is the average dissimilarity between and other instances in the same class, and b(i) is the average dissimilarity between and other data
27

---

Published as a conference paper at ICLR 2025
points in the closest different class. Intuitively, this metric summarizes the quality of clusters by jointly considering the distance between instances of the same class and distinct clusters, normalized by the longest distance of a(t) and b(2). By this definition, -1 indicates a dataset is poorly clustered (data points with different classes are scattered around) while 1 represents well-clustered dataset.
that -1 SS(i) S 1 where
Euclidean Distance is commonly used to measure two data points' differences; in contrast, we incor- porate the inverse of similarity (dissimilarity) as data points' differences into the existing Silhouette Score. Observe that the above Intra-Class Similarity s(C) already represents the tightness of the class (C), therefore a(i) can be replaced with the inverse of Intra-Class Similarity a(i) (t). For the second term b(i), we adopt the previously defined Inter-Class Similarity ((Ca and intro- duce a new similarity score as follows:
Nearest Inter-Class Similarity S((0 scalar describing the similarity among instances between class C' and the closest class of each instance in C:
5o 8
Σ iEC.EEČ
cos (os Zz),
(10)
|P|CO
where Ò' is the nearest class to instance (Ò + C). To determine Ć, we first iterate over all other classes C" that are different from C. For each C², we compute the average similarity between and all samples in C". The class C' with the highest average similarity score is then chosen C.
Consequently, the dataset-level Nearest Inter-Class Similarity Sy(D) is expressed as:
3
S S
1
Σ
cos Zz).
(11)
3
B
x (PCC,.) LEL iECr,jčc,
IEL
The second term of SS(() can be written as b(i) = -S' si).
Replacing a(i) and b(i) from equation 9 with these similarity terms, we introduce our novel similar- ity metric:
Similarity-Ban Silhouette Score SimSS:
Sa) - (i)
SimSS(i)- marr (1), Si
for instance
(12)
Sa S' s()
SimSS SimSS(i)
Σ
for class C
(13)
C
max
( S. (i),
iSC
SEC
SimSS(D)
1 Σ SimSs(C))
Σ
SimSS(i)
L
[1] x Ci
LEL
LEL.EC;
(14)
Sa(i) - (' (1)
for dataset D.
[.| x |C|
(ax(Sa (1),
IEC
28

---

Published as a conference paper at ICLR 2025
A.10 EXTENDED FEW-CLASS SIMILARITY BENCHMARK (FC-SIM) RESULTS
We present the relationship of similarity scores using our proposed SimSS and number of classes Ner. in datasets. CLIP and DINOv2 are used as similarity base functions SimSS shown in Fig. 18 (a) and (b), respectively.
Overall, key observation is that the general trend among all ten datasets unveils the inverse rela- tionship between similarity and the number of classes. Specifically, image similarities, which act as proxy of inverse subset difficulty score increases as the number of classes Net decreases. This reveals that similarity plays more important role in the Few-Class Regime than for datasets with more classes. Therefore, for real applications with few classes, simply downscaling a model blindly without considering class similarity may yield model selection with sub-optimal efficiency. We propose, therefore, that image similarity must be taken into consideration for existing scaling laws (Kaplan et al., 2020; Rae al., 2021; Zhai et al., 2022). To that end, Few-Class Arena is developed to facilitate future research in this direction.
IN1K
CT256
FD101
CF100
TT47
QD345
CB200
CT101
- ID67
GT43
0.14
0.12
0.10
SimSS[CLIP]
0.08
0.06
0.04
0.02
0.00
Na
(a) SimSS using CLIP as similarity base function vs Nez curve.
IN1K
CT256
FD101
CF100
TT47
QD345
CB200
CT101
- ID67
GT43
0.32
0.28
0.24
SimSS(DINOv2)
0.20
0.16
0.12
0.08
0.00
30
75
105
Na
(b) SimSS using DINOv2 as similarity base function vs Ner curve.
Figure 18: Relation SimSS[CLIPDINO02] and NcL-
Note that both CLIP and DINOv2 are trained on images from the Internet similar to ImageNet. Therefore what extent they capture image similarity in different types is open research
29

---

Published as a conference paper at ICLR 2025
question. Examples include drawings without textures in the Quickdraw dataset (QD345), textures without shapes in the Describable Textures Dataset (TT47), etc. We mentioned this limitation also in the main manuscript.
Effeet of ResNet Scales on Similarity. We present the details of FC-Sim results of the ResNet family in different scales in the Few-Class Regime of ImageNet1K, specifically (Nez (2,3,4 4, 5, 10, 100)) shown in Fig. 19. In particular, we analyze the relationship between each full and sub-model's Top-I accuracy and SimSS by Pearson correlation coefficient (PCC) denoted in the plots. The ResNet family scales from ResNer18 to ResNet152. We experiment both CLIP (dash line the 1st and 3rd rows) and DINOv2 (solid line in the 2nd and 4th rows) as similarity base functions.
Top- Acc. (9)
Top-1 Acc. (%)
80
Top-1 Acc. (%)
Top-1 Acc. 196
80
Top-1 Acc. (%)
80
60
70
pm0
r= 0010
60
1=0.10
p=00049
r.0.09
pm0.63
60
p-0.31
0.35
0.05 0.10 0.15 SimSS[CLIP]
SimSS[CLIP]
SimSS[CLIP]
SimSS[CLIP]
SimSS[CLIP]
(a) ResNet18
(b) ResNet34
(c) ResNet50
ResNet101
(c) ResNet152
Top-1 Acc. (%6)
Top-1 Acc. 196
8o
Top-1 Acc. (36)
80
Top-1 Acc. (80
80
Top-1 Acc. (56)
80
-0
D=0.19
r=0.21
F=0.20
8 8
r=0.17
pm0.37
80
0.17
p-0.31
pm0.28
0.1
0.2
0.3
0.1
0.2
0.3
0,2 SImSS[DINOv2]
0.1
0.3
0.1
0.2
0.3
0.1
0.2
0.3
SimSS(DINOv21
SImSS|DINOv2)
SimSS|DINOv2]
SImSS(DINOv2]
(I) ResNec18
(g) ResNet34
(h) ResNer50
ResNet101
ResNet152
100
Top Acc. (89
100
Top-1 Acc. (8))
Top-1 Ace (%)
Top-1 Acc. (%)
Top-1 Acc.
85
95
95
95
80
r-0 88
90
r- 90
90
f=0.8
KG0 88
1= 20.82
85
p- 1.1e-10
85
p- 1 30-11
6.7-88
p=1 1e-07
3e30-08
0.05 0.
0.05 0.10 0.15
0.15
0.30 0.15
0.10 0.15
SimSS[CLIP]
SimSS[CLIP]
SimSS[CLPP
SimSS[CLIP]
SimSS(CLiP)
(k) ResNet18
()) ResNet34
(m) ResNet50
(n) ResNet101
(0) ResNet152
Top-1 Acc. (%)
lop-1 Acc. (%)
100
Top-1 Acc. (%)
lop Acc. (%)
1 Acc. 1X
95
95
95
95
90
0 69
90
r=0
0.88
30
1 80
90
0 82
4
85
2m2 202266em 6
p 1.3e-10
P-9 9.9--0
p 6e-08
pm5 5.708
0.3
0.2
0.3
0.1
0.2
0.3
0.1
0.2 0.
0.1
0.2
0.3
0.1
022 03
SimSSiDINOv]]
Simis/DDINv2]
SimsS(DiNOv2]
Simss(DINov2]
SimSSiDINOv2]
(p) ResNer18
(g) ResNet34
(r) ResNet50
(s) ResNet101
(t) ResNet152
Figure 19: Top-1 Accuracy of different ResNet scales vs SimSS on ImageNetIK. NeL E
(2.3, 4,5, 10, 1003.
In general, ResNet presents high correlation (r 2 0.80) between sub-models' performances and SimSS (blue in the last two rows), compared to full models' performances and SimSS (r < 0.26, red in the first two rows). This high correlation indicates that SimSS can be used as a reliable tool to estimate upper bound accuracies of ResNet sub-models. Comparing CLIP (r in the 1st row) with DINOv2 (r in the 2nd row) as similarity base functions, observe that PCC is slightly higher for DINOv2 on full models than CLIP, while these differences are subtle for sub-mu models (r in the 3rd row vs 4th row). Regarding sub-models of ResNet in different scales, the smallest models'
30

---

Published as a conference paper at ICLR 2025
accuracies (ResNet18 and ResNet34) have higher correlation with SimSS (r 0.88), compared to larger models with (ResNet50, ResNet101 and ResNet152) 2 0.80. We opens a new direction of novel scaling law considering image similarity for efficient models in Few-Class Regime.
A.11 COMPARING VARIOUS ARCHITECTURES
We perform experiments on CNNs (ResNet18, 50) and Vision Transformer- Base (ViT-B). Results summarized in Table 7. The overall trend of ViT-B exhibits behavior consistent with the obser- vations for ResNet described in Fig. I.
ResNet18
ResNet50 Top-1 Acc.
VIT-B
MT
NCL
Top-1 Acc.t
STDEV
STDEV
Top-1 Acc.t
STDEV
F
1000
69.90
76.55
82.37
F
10
66.68
4,372
73.76
3.675
79.52
2.704
F
5
65,68
9.680
72.16
9.302
79.12
8,146
F
2
62.80
14.18
70.60
14.67
78.00
14.47
S
10
91.88
1.640
91.48
2.265
82.
3.508
S
5
93.68
1.213
92.96
3.170
89.60
2.227
S
1.304
94.80
2.168
95.60
1.949
Table 7: Performance results for different models and configurations on ImageNetIK, NOL E (2,5, 10, 1000), MT: Model Type, F: Full model, Sub-model.
A.12 EXTENSION TO OBJECT DETECTION AND SEGMENTATION
To assess the generalization of the few-class properties to Object Detection (OD) and Segmentation (Seg), we conducted validation experiments using YOLOv8. The procedure is consistent with the method outlined in Section 3.3 and 3.4. Specifically, for specific NeL (2 in this example), we randomly sample the Nct. classes from the full dataset of COCO, where each consists of five subsets with seed numbers from 0 to 4. We performed experiments with 2, 5, 80. The YOLOv8-nano model was chosen since we focus on efficiency. Image size of 320x320 was used. Model performance was evaluated using the standard metric, mean average precision at IoU threshold of 0.5 (mAP@50). Table 8 summarizes our results:
OD
Seg
MT
NeL
MAP@501
STDEV]
mAP@501
STDEV
F
80
0.405
0.195
0.378
0.200
F
0.456
0.090
0.435
0.098
FT
0.488
0.069
0.465
0.082
S
0.503
0.069
0.474
0.084
F
2
0.488
0.161
0.475
0.180
FT
0.505
0.127
0.457
0.159
0.538
0.106
0.482
0.152
Table 8: Performance results for different tasks and configurations on COCO. Nct E (2, 80), MT: Model Type, OD: Object Detection, Seg: Segmentation, F; Full model, S: Sub-model, FT: Fine-tuned model. Best scores are highlighted in bold. The gray bar indicates sub-models as the primary focus of this research.
We also include a comparison with fine-tuned pre- are-trained models (denoted as F). These models are initialized with weights from the pre-trained TOLOv8-nand subsequently ine-ttune for a limited number of epochs (five in this example). We conclude that OD and Seg task exhibit similar observations to those make in Fig. 1.
the full dataset with 80 classes, and
31

---

Published as a conference paper at ICLR 2025
A.13 EXPERIMENTS COMPUTE RESOURCES
Experiments are conducted on two internal clusters with the following hardware specifications: (1) NVIDIA RTX A5000 GPUs (24GB), an AMD 7513 32-Core Processor, and 882GB of RAM; and (2) NVIDIA TITAN Xp GPUs (12GB), an Intel(R) Xeo CPU E5-2650 v4 @2.20GHz, and 126GB of RAM. When GPUs in two clusters are fully utilized, training ten models in nine datasets takes two weeks; obtaining a single experiment result for FC-Full usually takes less than one minute since it only involves inference without training: getting one FC-Sub experiment result takes approximately two days on average depending on the size of subset and model, which includes both training and testing: computing the SimSS in the Pew-Claas Regime for ten datasets takes around three weeks. Additional experiments are conducted on gaming desktop with the following hardware specifications: 2 NVIDIA RTX 3090 GPUs (24GB), an AMD Ryzen 5 3600 6-Core Processor, and 78.5GB of RAM.
32


Reviews:
[{"review_id": "sSnbk0hmtU", "reviewer": "Reviewer_trsf", "summary": "This paper tackles the problem of choosing image classifiers for tasks with only a small number of categories (\"Few-Class\"). To do so, they introduce a new benchmark, termed \"Few Class Arena\" (FCA) on which they train and evaluate a range of models on subsets of various full datasets (e.g wit so-called sub-models trained on between 2 and all 1k ImageNet categories). The FCA benchmark is open-sourced with code available on GitHub. The paper provides detailed discussion of how the open-source package can be used for model selection in the few-class setting.\n\nOverall, the authors show that models trained on specific sub-classes (sub-models) are better than models trained on the full dataset and evaluated on the same sub-classes, across model sizes. They further show that there is no single best model architecture for a given dataset, and that training models on different datasets result in different rankings of architecture. The authors also propose a \"dataset difficulty\" metric which can be computed without training a model, and correlates well with the few-class performance of a model on a dataset.", "questions": "* I may have missed something, but I cannot understand exactly how the proposed difficulty metric is intended to be used?\n* The proposed difficulty metric seems expensive to compute, with pairwise similarity scores required for large subsets of the data. How does this compare to the cost of conducting a single model training run?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Tab 1 / Fig 2b shows the results when training different base architectures on different classification datasets from scratch. Interestingly, not all results are correlated with the trend on ImageNet-1K, indicating the optimal architecture choice depends on the dataset.\n- The code is open sourced and well documented. It seems that it would be simple for a researcher to reproduce the authors claim with limited effort (though I have not run the code myself).\n- It is interesting that sub-models consistently outperform full models on ImageNet. The fact that full models have seen more training datapoints in total may have compensated for fewer classes, which makes the result not totally intuitive.", "weaknesses": "My main issue with this paper is in overall utility. The high level goal of the paper is to provide a tool with which practitioners can select a model (dominantly through the lens of model *architecture*) for a few-class classification task. The tool basically allows authors to train a model (with most results presented from scratch) on subsets of a given dataset. However, this does not align with the practical problem to me, where practitioners might take a model pretrained on a large amount of *data* (e.g DINOv2 or CLIP) is finetuned for a given task (note that lightweight variants of these models are also open-source). \n\nGiven that this paper is predominantly an empirical examination which proposes a practical open-source library, I feel that the lack of experiments with pretrained models prevents acceptance. \n\nOther issues:\n\n- The citation format makes the main text quite difficult to parse\n- L52: Main text does not seem to describe Figure 1 accurately?\n- There is no discussion of few-shot literature, which is at least tangentially related to this problem", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3sWmCwedNg", "reviewer": "Reviewer_6NMw", "summary": "The paper proposes a benchmark tool called Few-Class Arena to benchmark models on different datasets with smaller number of classes (e.g. < 10) and propose a similarity metric called SimSS to measure dataset difficulty measure. They show that ResNet family models trained on full ImageNet 1K classes show reduced performance when tested only for few ImageNet classes (< 10 classes). On the other hand, the same models when trained on smaller number of ImageNet classes from scratch show higher performance on these classes when compared to models trained on all classes of ImageNet 1k. They show that the proposed SimSS metric can serve as a proxy to estimate the upper bound accuracy of model performance on few-class datasets.", "questions": "1. For FC-Full, when N_{CL} decreases, how to make sure that model predicts only few classes? Are the logits of those few classes are selected to get the prediction and discard logits of all other classes?\n\n2. If a user has a custom dataset with few classes and want to find a model that works better on this custom dataset, it would be helpful to have an explanation on how this benchmark can assist the user in this case.\n\n---------------------------------------------------------------------------------------\nFinal review: I appreciate authors for their detailed responses to the comments from all the reviewers. Authors comprehensive responses, clarifications, and additional experiments have addressed the key weaknesses. I now vote towards borderline acceptance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tAddressed an important problem by proposing the benchmarking tool.\n\n•\tThe tool is designed to be user friendly and allows to run wide range of experiments by setting few hyper-parameters. The tool allows to benchmark on custom models and datasets.\n\n•\tProvided a behavioral understanding between models trained on large number of classes vs smaller number of classes in the few-class regime.\n\n•\tThe proposed similarity metric shown to be linearly corelated with the model performance on small number of classes. Such proxy helps to save computation cost and time of conducting various experiments.", "weaknesses": "Despite focusing on an interesting problem setting, the analysis shown in the paper has limited scope. Authors shown experiments on models evaluated or trained on smaller number of classes, however there are no details discussed on how these few classes have been selected and how semantically close these few classes to each other? Would the analysis presented differ by choosing the those few classes differently?\n\nThe aspect of transfer learning has not been discussed. It is a common practice to finetune ImageNet pretrained models like ResNet50 or ViT, or recent foundation models like CLIP and DINOv2 to different downstream tasks that include adapting or finetuning them on few classes. The analysis presented in the paper is missing this exploration. Is it better to train the models from scratch on the few classes or finetuning these models work better for few classes? Does SimSS score also align on the finetuned models?\n\nTo compute SimSS, a score called Nearest Inter-Class Similarity requires a nearest class (C_hat) to the target class (C), it is not clear how this C_hat is acquired.\n\nOverall, I appreciate the motive and tool for benchmarking few-class regime, however the analysis presented in the paper is incomplete, and I suggest authors to extend their analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIOy9Soakw", "reviewer": "Reviewer_2XhF", "summary": "This paper presents a new benchmark for the “few-class” problem, which  is a classification problem with very few classes. Most of the scientific literature focuses on datasets with many classes while practitioners often encounter the few-class scenario. The benchmark consist of several selected datasets and several settings, such as training on large set of classes and evaluating on a smaller set, and popular vision models are evaluated and compared. Finally, an analysis of what happens in few-class regimes is proposed.", "questions": "- What original research do you expect will use this benchmark and what do you hope it will achieve or unlock ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The benchmark is well executed and will be useful for “few-class” adaptation research. There are many models evaluated with many datasets and the analysis is thorough. In particular, the author study in depth the evolution of the performance of models trained on large set of classes compared to specialized models, as a function of the number of classes, and show the importance, and therefore propose a metric for evaluating this adaptation.\n\n- The similarity benchmark is a nice addition. It correlates well with the performance while being easy to evaluate and with a modest cost.\n\n- The presentation and writing are very clear. The figures are very informative.", "weaknesses": "- The motivation behind few-class evaluations is not fully convincing. In practice, one will take a large model and fine-tune it (without the classification layer) to a target set of classes, hence obtaining a specialized model. Evaluating the capabilities of a full model on few-class is only interesting when there are too many subsets to consider ? When does that happen in practice, and could you just not use small adaptation layers on top of frozen backbone for each of the subsets ?\n\n- One thing that is missing from the paper is a recommendation for practitioners on which vision model to use for someone interested in the few-class problem. Basically discussing in more details the results from Table 1 and providing comparison between models in Section 4.2 and 4.3. One interesting question is, do models that perform really well on the many classes setup are the same that also perform well on the few-class setup ?\n\n- Some of the findings in the paper are fully expected. The fact that a model specifically trained on the target subset of classes perform better that a larger model trained on a superset is not very surprising or novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iec0TXiXtx", "reviewer": "Reviewer_o3dH", "summary": "The paper introduces a benchmark designed to evaluate and select efficient image classification models in scenarios with a limited number of classes. This setting, common in real-world applications (e.g., 2-10 classes), contrasts with widely used benchmarks like ImageNet and COCO, which involve hundreds or thousands of classes. The paper presents FCA as a tool to help researchers and practitioners efficiently select models for few-class tasks. The paper coins the term ``few-class regime'' and presents some interesting non-intuitive insights regarding the performance of models that are pre-trained with many class datasets and then applied in few-class settings.\nIn addition, they introduce a dataset difficulty metric by inverting image similarity measured via CLIP and DINOv2 features.", "questions": "1. (Related to W2) The finding that the scaling law w.r.t. model size is violated for submodes is interesting. I would be curious if this only applies to supervised training or also to self-supervised pre-training combined with minimal fine-tuning or linear probing. Did you conduct any experiments into this direction or do you have an intuition on that?\n2. You mention that you conducted experiments on different architectures such as ResNet’s and ViT’s. However, the results are presented in an aggregated way. Did you find any significant differences between model architectures? Do the findings of Fig 1 equally apply for both architectures?\n3. I agree with your description and ad-hoc interpretation of Fig. 5. However, I am missing a discussion on why we see the low correlation for DCN-Full. Do you have any interpretation of this?\n\n\n**Minor comments:**\n\n- The correct use of author-year citations would improve readability. I.e.: Author (2024) for in-text citations and (Author, 2024) elsewise.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The necessity of a few-class benchmark is well-motivated by a strong finding that models pre-trained on many-class datasets perform worse than expected on few-class datasets which is an issue not addressed in literature thus far.\n2. The paper presents some interesting insights that contradict expected model behavior based on intuition.\n3. The authors provide a ready to use code base integrated with other frameworks and libraries ensuring usability.", "weaknesses": "1. The scope of the study only covers image classification, while it is in principle also applicable to dense prediction tasks where object classes are present (e.g., object detection, semantic segmentation). It would be insightful if the same findings hold for these tasks. The authors could add a discussion or experiments (if available) for other vision tasks.\n2. The experiments cover classification tasks based on supervised (pre-)training. However, there is an increasing trend that classification models are fine-tuning based on self-supervised pre-trained models. This paradigm is not covered in this study, and therefore, the findings are limited to the more traditional fully supervised paradigm. The authors could discuss these aspects or add experimentation with self-supervised pre-trained models (if available).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)