Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement

## Abstract

Abstract We propose Few-Class Arena ( FCA ), as a unified benchmark with focus on testing efficient image classification models for few classes. A wide variety of benchmark datasets with many classes (80-1000) have been created to assist Computer Vision architectural evolution. An increasing number of vision models are evaluated with these many-class datasets. However, real-world applications often involve substantially fewer classes of interest (2-10). This gap between many and few classes makes it difficult to predict performance of the few-class applications using models trained on the available many-class datasets. To date, little has been offered to evaluate models in this Few-Class Regime . We conduct a systematic evaluation of the ResNet family trained on ImageNet subsets from 2 to 1000 classes, and test a wide spectrum of Convolutional Neural Networks and Transformer architectures over ten datasets by using our newly proposed FCA tool. Furthermore, to aid an up-front assessment of dataset difficulty and a more efficient selection of models, we incorporate a difficulty measure as a function of class similarity. FCA offers a new tool for efficient machine learning in the Few-Class Regime , with goals ranging from a new efficient class similarity proposal, to lightweight model architecture design, to a new scaling law. FCA is user-friendly and can be easily extended to new models and datasets, facilitating future research work. Our benchmark is available at https://github.com/fewclassarena/fca .

## 1 Introduction

The de-facto benchmarks for evaluating efficient vision models are large scale with many classes (e.g. 1000 in ImageNet  [], 80 in COCO  [], etc.). Such benchmarks have expedited the advance of vision neural networks toward efficiency [, , , , , , , ] with the hope of reducing the financial and environmental cost of vision models [, ]. More efficient computation is facilitated by using quantization  [, , ], pruning  [, , , ], and data saliency []. Despite efficiency improvements such as these, many-class datasets are still the standard of model evaluation.

Real-world applications, however, typically comprise only a few number of classes (e.g, less than 10) [, , ] which we termed Few-Class Regime. To deploy a vision model pre-trained on large datasets in a specific environment, it requires the re-evaluation of published models or even retraining to find an optimal model in an expensive architectural search space [].

Figure: (a) Accuracies for sub-models (blue) and full models (red).
Refer to caption: x1.png

One major finding is that, apart from scaling down model and architectural design for efficiency, dataset difficulty also plays a vital role in model selection []  (described in Section 4.3).

Figure 1 summarizes several key findings under the Few-Class Regime. On the left graph in red are accuracy results for a range of number of classes $N_{CL}$ for what we call the “full model”, that is ResNet models pre-trained on the full 1000 classes of ImageNet (generally available from many websites). On the right are accuracy results for what we call “sub-models”, each of which is trained and tested on the same $N_{CL}$, where this number of classes is sampled from the full dataset down to the Few-Class Regime. Findings include the following. (a) Sub-models attain higher upper-bound accuracy than full models. (b) The range of accuracy widens for full models at few-classes, which increases the uncertainty of a practitioner selecting a model for few classes. In contrast, sub-models narrow the range. (c) Full models follow the scaling law []  in the dimension of model size - larger models (darker red) have higher accuracy from many to few classes. (d) Surprisingly, the scaling law is violated for sub-models in the Few-Class Regime (see the zoomed-in subplot) where larger models (darker blue) do not necessarily perform better than smaller ones (lighter blue). From these plots, our key insight is that, instead of using full models, researchers and practitioners in the Few-Class Regime should use sub-models for selection of more efficient models.

However, obtaining sub-models involves computationally expensive training and testing cycles since they need to be converged on each of the few-class subsets. By carefully studying and comparing the experiment and evaluation setup of these works in the literature, we observe that, how models scale down to Few-Class Regime is rarely studied. The lack of comprehensive benchmarks for few-class research impedes both researchers and practitioners from quickly finding models that are the most efficient for their dataset size. To fill this need, we propose a new benchmark, Few-Class Arena (FCA), with the goal of benchmarking vision models under few-class scenarios. To our best knowledge, FCA is the first benchmark for such a purpose.

We formally define Few-Class Regime as a scenario where the dataset has a limited number of classes. Real-world applications often comprise only a few number of classes (e.g. $N_{CL}<10$ or $10\%$ classes of a dataset). Consequently, Few-Class Arena refers to a benchmark to conduct research experiments to compare models in the Few-Class Regime. This paper focuses on the image classification task, although Few-Class Regime can generalize to object detection and other visual tasks.

Statement of Contributions. Four contributions are listed below:

- •
To be best of our knowledge, we are the first to explore the problems in the Few-Class Regime and develop a benchmark tool Few-Class Arena (FCA) to facilitate scientific research, analysis, and discovery for this range of classes.
- •
We introduce a scalable few-class data loading approach to automatically load images and labels in the Few-Class Regime from the full dataset, avoiding the need to duplicate data points for every additional few-class subset.
- •
We incorporate dataset similarity as an inverse difficulty measurement in Few-Class Arena and propose a novel Silhouette-based similarity score named SimSS. By leveraging the visual feature extraction power of CLIP and DINOv2, we show that SimSS is highly correlated with ResNet performance in the Few-Class Regime with Pearson coefficient scores $\geq 0.88$.
- •
We conduct extensive experiments that comprise ten models on ten datasets and 2-1000 numbers of classes on ImageNet, totalling 1591 training and testing runs. In-depth analyses on this large body of testing reveal new insights in the Few-Class Regime.

## 2 Related Work

Visual Datasets and Benchmarks.
To advance deep neural network research, a wealth of large-scale many-class datasets has been developed for benchmarking visual neural networks over a variety of tasks. Typical examples (^1^11A detailed list of many-class datasets used in this paper can be found in the Appendix.) include 1000 classes in ImageNet [] for image classification, and 80 object categories in COCO [] for object detection. Previous benchmarks also extend vision to multimodal research such as image-text [, , , ].
While prior works often scale up the number of object categories for general purpose comparison, studies [, ] raise a concern on whether models trained on datasets with such a large number of classes (e.g. ImageNet) can be reliably transferred to real world applications often with far fewer classes.
A close work to ours is vision backbone comparison [] whose focus is on model architectures. Our perspective differs in a focus on cases with fewer number of classes, which often better aligns with real-world scenarios.

Dataset Difficulty Measurement. Research has shown the existence of inherent dataset difficulty [] for classification and other analytic tasks. Efficient measurement methods are proposed to characterize dataset difficulty using Silhouette Score [], K-means Fréchet inception distance [, , ], and Probe nets []. Prior studies have proposed image quality metrics using statistical heuristics, including peak signal-to-noise ratio (PSNR) [], structural similarity (SSIM) Index [], and visual information fidelity VIF []. A neuroscience-based image difficulty metric [] is defined as the minimum viewing time related to object solution time (OST) []. Another type of difficulty measure method consists of additional procedures such as c-score [] , prediction depth [] , and adversarial robustness [] . Our work aligns with the line of research [, , ] involving similarity-based difficulty measurements: similar images are harder to distinguish from each other while dissimilar images are easier. Previous studies are mainly in the image retrieval context [, , ]. Similarity score is used in [] with the limitation that a model serving similarity measurement has to be trained for one dataset. We push beyond this limit by leveraging large vision models that learn general visual features using CLIP [] and DINOv2 []. The study [] shows that CLIP generalizes well to both easy and hard images, making it a good candidate for measuring image difficulty. Supported by the evidence that better classifiers can act as better perceptual feature extractors [] , in later sections we show how CLIP and DINOv2 will be used as our similarity base function.

Despite the innovation of difficulty measure algorithms on many-class datasets, little attention has been paid to leveraging these methods in the Few-Class Regime. We show that, as the number of classes decreases, sub-dataset difficulty in the Few-Class Regime plays a more critical role in efficient model selection. To summarize, unlike previous work on many-class benchmarks and difficulty measurements, our work takes few-class and similarity-based dataset difficulty into consideration, and in doing so we believe the work pioneers the development of visual benchmark dedicated to research in the Few-Class Regime.

## 3 Few-Class Arena (FCA)

We introduce the Few-Class Arena (FCA) benchmark in this section. In practice, we have integrated FCA into the MMPreTrain framework [], implemented in Python3 and Pytorch(^2^22Code is available at [https://github.com/fewclassarena/fca](https://github.com/fewclassarena/fca), including detailed documentation and long-term plans of maintenance.).

### 3.1 Goals

1. Generality. All vision models and existing datasets for classification should be compatible in this framework. In addition, users can extend to custom models and datasets for their needs.

2. Efficiency. The benchmark should be time- and space-efficient for users. The experimental setup for the few-class benchmark should be easily specified by a few hyper-parameters (e.g. number of classes). Since the few-class regime usually includes sub-datasets extracted from the full dataset, the benchmark should be able to locate those sub-datasets without generating redundant duplicates for reasons of storage efficiency. For time-efficiency, it should conduct training and testing automatically through use of user-specified configuration files, without users’ manual execution.

3. Large-Scale Benchmark. The tool should allow for large-scale benchmarking, including training and testing of different vision models on various datasets when the number of classes varies.

### 3.2 Few-Class Dataset Preparation

Few-Class Arena provides an easy way to prepare datasets in the Few-Class Regime. By leveraging the MMPreTrain framework, users only need to specify the parameters of few-class subsets in the configuration files, which includes the list of models, datasets, number of classes ($N_{CL}$), and the number of seeds ($N_{S}$). Few-Class Arena generates the specific model and dataset configuration files for each subset, where subset classes are randomly extracted from the full set of classes, as specified by the seed number. Note that only one copy of the full, original dataset is maintained during the whole benchmarking life cycle because few-class subsets are created through the lightweight configurations, thus maximizing storage efficiency. We refer readers to the Appendix and the publicly released link for detailed implementations and use instructions.

### 3.3 Many-Class Full Dataset Trained Benchmark

We conducted large-scale experiments spanning ten popular vision models (including CNN and ViT architectures) and ten common datasets (^3^33Models include: ResNet50 (RN50), VGG16, ConvNeXt V2 (CNv2), Inception V3 (INCv3), EfficientNet V2 (EFv2), ShuffleNet V2 (SNv2), MobileNet V3 (MNv3), Vision Transformer base (ViTb), Swin Transformer V2 base (SWv2b) and MobileViT small (MViTs). Datasets include CalTech101 (CT101), CalTech256 (CT256), CIFAR100 (CF100), CUB200 (CB200), Food101 (FD101), GTSRB43, (GT43), ImageNet1K (IN1K), Indoor67 (ID67), Quickdraw345 (QD345) and Textures47 (TT47).). Except for ImageNet1K, where pre-trained model weights are available, we train models in other datasets from scratch. While different models’ training procedures may incur various levels of complexity (particularly in our case for MobileNet V3 and Swin Transformer V2 base), we have endeavored to minimize changes in the existing training pipelines from MMPreTrain. The rationale is that if a model exhibits challenges in adapting it to a dataset, then it is often not a helpful choice for a practitioner to select for deployment.

Results are summarized in Table 1. We make several key observations: (1) models in different datasets (in rows) yield highly variable levels of performance by Top-1 accuracy;
(2) no single best model (bold, in columns) exists across all datasets; and (3) model rankings vary across various datasets.

The first two observations are consistent with the findings in [, ]. For (1), it suggests there exists underlying dataset-specific difficulty. To capture this characteristic, we adopt the reference dataset classification difficulty number (DCN) [] to refer to the empirically highest accuracy achieved in a dataset from a finite number of models shown in Table 1 and Figure 2 (a). For observation (3), we can examine the rankings among the ten models of ResNet50 and EfficientNet V2 in Figure 2 (b). ResNet50’s ranking varies dramatically for the different datasets, for instance ranking 7th on ImageNet1K and 1st on Quickdraw345. This ranking variability is also observed in other models (see all models in the Appendix). However, a common practice is to benchmark models – even for efficiency – on large datasets, especially ImageNet1K. The varied dataset rankings in our experiments expose the limitations of such a practice, further supporting our new benchmark paradigm, especially in the Few-Class Regime. In later sections, we leverage DCN and image similarity for further analysis.

**Table 1: Top-1 accuracy across ten models in ten datasets. Models are trained and tested on full datasets with their original number of classes (e.g. 1K from ImageNet1K); this is denoted in the last few digits of the abbreviation of the dataset name. The best score is highlighted in bold while the second best is underlined for each dataset. References for all models and datasets can be found in the Appendix.**
| Dataset | RN50 | VGG16 | CNv2 | INCv3 | EFv2 | SNv2 | MNv3 | ViTb | SWv2b | MViTs | DCN |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GT43 | 99.85 | 96.60 | 99.83 | 99.78 | 99.86 | 99.87 | 99.83 | 99.31 | 99.78 | 99.69 | 99.87 |
| CF100 | 74.56 | 71.12 | 85.89 | 75.97 | 77.05 | 77.89 | 74.35 | 32.65 | 78.49 | 76.51 | 85.89 |
| IN1K | 76.55 | 71.62 | 84.87 | 77.57 | 85.01 | 69.55 | 66.68 | 82.37 | 84.6 | 78.25 | 85.01 |
| FD101 | 83.76 | 75.82 | 63.80 | 83.96 | 80.82 | 79.36 | 76.03 | 52.21 | 84.30 | 82.23 | 84.30 |
| CT101 | 77.70 | 74.99 | 77.52 | 77.52 | 77.82 | 84.13 | 80.71 | 59.59 | 78.82 | 80.06 | 84.13 |
| CT256 | 65.07 | 59.08 | 73.57 | 66.09 | 62.80 | 68.13 | 62.62 | 44.23 | 67.28 | 65.80 | 73.57 |
| QD345 | 69.14 | 19.86 | 62.86 | 68.25 | 68.81 | 67.32 | 66.42 | 19.67 | 66.54 | 68.76 | 69.14 |
| CB200 | 45.86 | 21.26 | 27.61 | 45.58 | 44.48 | 53.95 | 53.80 | 23.73 | 54.52 | 58.46 | 58.46 |
| ID67 | 53.75 | 26.01 | 33.21 | 45.95 | 43.85 | 54.72 | 51.65 | 30.51 | 48.58 | 54.05 | 54.72 |
| TT47 | 30.43 | 12.55 | 6.49 | 14.20 | 21.17 | 43.83 | 40.27 | 31.38 | 33.94 | 24.41 | 43.83 |

Figure: (a) Top-1 accuracy and DCN in ten full datasets.
Refer to caption: x5.png

In the next subsections, we introduce three new types of benchmarks: (1) Few-Class, Full Dataset Trained Benchmark (FC-Full), which benchmarks vision models trained on the full dataset with the original number of classes; (2) Few-Class, Subset Trained Benchmark (FC-Sub), which benchmarks vision models trained on subsets of a fewer number of classes than the full dataset, and (3) Few-Class Similarity Benchmark (FC-Sim), which benchmarks image similarity methods and their correlation with model performance.

### 3.4 Few-Class Full Dataset Trained Benchmark (FC-Full)

Traditionally, a large number of models are trained and compared on many-class datasets. However, results for such benchmarks are not directly useful to the Few-Class Regime and many real-world scenarios. Therefore, we introduce the Few-Class Full Dataset Trained Benchmark (FC-Full), with the objective of effortlessly conducting large-scale experiments and analyses in the Few-Class Regime.

The procedure of FC-Full consists of two main stages. In the first stage, users select the models and datasets upon which they would like to conduct experiments. They can choose to download pre-trained model weights, which are usually available on popular model hubs (PyTorch Hub [], TensorFlow Hub [], Hugging Face [], MMPreTrain [] etc.). In case of no pre-trained weights available from public websites, users can resort to the option of training from scratch. To that end, our tool is designed and implemented to generate bash scripts for easily configurable and modifiable training through the use of configuration files.

In the second stage, users conduct benchmarking in the Few-Class Regime. By specifying the list of classes, Few-Class Arena automatically loads pre-trained weights of the chosen models and evaluates performance of the models on the selected datasets. Note that this process is accomplished through configuration files created by the user’s specifications, thus enabling hundreds of experiments to be launched by a single command. This dramatically reduces human effort that would otherwise be expended to run these experiments without Few-Class Arena.

### 3.5 Few-Class Subset Trained Benchmark (FC-Sub)

Our study in Figure 1  (red lines) reveals the limits of existing pre-trained models in the Few-Class Regime. To facilitate further research and analyze the upper bound performance in the Few-Class Regime, we introduce the Few-Class Subset Trained Benchmark (FC-Sub).

FC-Sub follows a similar procedure to FC-Full, except that, when evaluating a model in a subset with a specific number of classes, that model should have been trained on that same subset. Specifically, in Stage One (described for FC-Full), users specify models, datasets and the list of number of classes in configuration files. Then Few-Class Arena generates bash scripts for model training on each subset. In Stage two, Few-Class Arena tests each model in the same subset that it was trained on.

### 3.6 Few-Class Similarity Benchmark (FC-Sim)

One objective of our tool is to provide the Similarity Benchmark as a platform for researchers to design custom similarity scores for efficient comparison of models and datasets.

The intrinsic image difficulty of a dataset affects a model’s classification performance (and human) [, , ].
We show – as is intuitive – that the more similar two images are, the more difficult it is for a vision classifier to make a correct prediction.
This suggests that the level of similarity of images in a dataset can be used as a proxy for a dataset difficulty measure. In this section, we first adopt and provide the basic formulation of similarity, the baseline of a similarity metric. Then we propose a Similarity-Based Silhouette Score to capture the characteristic of image similarity in a dataset.

We first adopt the basic similarity formulation from []. Intra-Class Similarity $S_{\alpha}^{(C)}$ is defined as a scalar describing the similarity of images within a class by taking the average of all the distinct class pairs in $C$, while Inter-Class Similarity denotes a scalar describing the similarity among images in two different classes ${C_{1}}$ and $C_{2}$. For a dataset $D$, these are defined as the mean of their similarity scores over all classes, respectively:

$$ $S_{\alpha}^{(D)}=\frac{1}{|L|}\sum_{l\in L}S_{\alpha}^{(C_{l})}=\frac{1}{|L| \times|P^{(C_{l})}|}\sum_{l\in L}\sum_{i,j\in C_{l};i\neq j}\cos(\textbf{Z}_{i },\textbf{Z}_{j}),$ (1) $$

$$ $S_{\beta}^{(D)}=\frac{1}{|P^{(D)}|}\sum_{a,b\in L;a\neq b}S_{\beta}^{(C_{a},C_ {b})}=\frac{1}{|P^{(D)}|\times|P^{(C_{1},C_{2})}|}\sum_{a,b\in L;a\neq b}\sum_ {i\in C_{1},j\in C_{2}}\cos(\textbf{Z}_{i},\textbf{Z}_{j}).$ (2) $$

where $|L|$ is the number of classes in a dataset, $Z_{i}$ is the visual feature of an image $i$, $|P^{(C)}|$ is the total number of distinct image pairs in class $C$, $|P^{(D)}|$ is the total number of distinct class pairs, and $|P^{(C_{1},C_{2})}|$ is the total number of distinct image pairs excluding same-class pairs.

Averaging these similarities provides a single scalar score at the class or dataset level. However, this simplicity neglects other cluster-related information that can better reveal the underlying dataset difficulty property of a dataset. In particular, the (1) tightness of a class cluster and (2) distance to other classes of class clusters, are features that characterize the inherent class difficulty, but are not captured by $S_{\alpha}$ or $S_{\beta}$ alone.

To compensate the aforementioned drawback, we adopt the Silhouette Score (SS) [, ]: $SS(i)=\frac{b(i)-a(i)}{max({a(i),b(i)})}$,
where $SS(i)$ is the Silhouette Score of the data point $i$, $a(i)$ is the average dissimilarity between $i$ and other instances in the same class, and $b(i)$ is the average dissimilarity between $i$ and other data points in the closest different class.

Observe that the above Intra-Class Similarity $S_{\alpha}^{(C)}$ already represents the tightness of the class $(C)$, therefore $a(i)$ can be replaced with the inverse of Intra-Class Similarity $a(i)=-S_{\alpha}(i)$. For the second term $b(i)$, we adopt the previously defined Inter-Class Similarity $S_{\beta}^{(C_{1},C_{2})}$ and introduce a new similarity score as Nearest Inter-Class Similarity $S^{\prime(C)}_{\beta}$, which is a scalar describing the similarity among instances between class $C$ and the closest class of each instance in $C$. The dataset-level Nearest Inter-Class Similarity $S^{\prime(D)}_{\beta}$ is expressed as:

$$ $S^{\prime(D)}_{\beta}=\frac{1}{|L|}\sum_{l\in L}S^{\prime(C_{l},\hat{C_{l}})}_ {\beta}=\frac{1}{|L|\times|P^{(C_{l},\hat{C_{l}})}|}\sum_{l\in L}\sum_{i\in C_ {l},j\in\hat{C_{l}}}\cos(\textbf{Z}_{i},\textbf{Z}_{j}).$ (3) $$

where $\hat{C}$ is the set of the nearest class to $C$ ($\hat{C}\neq C$).
To summarize, we introduce our novel Similarity-Based Silhouette Score $SimSS$(^4^44The extended derivation is detailed in the Appendix.):

$$ $\begin{split}SimSS^{(D)}&=\frac{1}{|L|\times|C_{l}|}\sum_{i\in C_{l}}\frac{S_{ \alpha}(i)-S^{\prime}_{\beta}(i)}{max(S_{\alpha}(i),S^{\prime}_{\beta}(i))}. \end{split}$ (4) $$

## 4 Experimental Results

### 4.1 Results on FC-Full

In this section, we present the results of FC-Full. A model trained on the dataset with its original number of classes (e.g. $1000$ in ImageNet1K) is referred to as a full-class model. These experiments are designed to understand how full-class model performance changes when the number of classes $N_{CL}$ decreases from many to few classes. We analyze the results of DCN-Full, shown in Figure 3 (details of all models are presented in the Appendix), and we make two key observations when $N_{CL}$ reduces to the Few-Class Regime (from right to left). (1) The best performing models do not always increase its accuracy for fewer classes, as shown by the solid red lines that represent the average of DCN for each $N_{CL}$. (2) The variance, depicted by the light red areas, of the best models broaden dramatically for low $N_{CL}$, especially for $N_{CL}<10$.

Both observations support evidence of the limitations of using the common many-class benchmark for application model selection in the Few-Class Regime, since it is not consistent between datasets that a model can be made smaller with higher accuracy. Furthermore, the large variance in accuracy means that prediction of performance for few classes is unreliable for this approach.

Figure: Figure 3: DCN-Full by Top-1 Accuracy ($\%$). $N_{CL}$ ranges from many to $2$.
Refer to caption: x7.png

### 4.2 Results on FC-Sub

In this section, we show how using Few-Class Arena can help reveal more insights in the Few-Class Regime to mitigate the issues of Section 4.1.

FC-Sub results are displayed in Figure 4. Recall that a sub-class model is a model trained on a subset of the dataset where $N_{CL}$ is smaller than the original number of classes in the full dataset. Observe that in the Few-Class Regime (when $N_{CL}$ decreases from 4 to 2) that: (1) DCN increases as shown by the solid blue lines, and (2) variance reduces as displayed by the light blue areas.

The preceding observation for FC-Full 4.1 seems to contradict the common belief that, the fewer the classes, the higher is the accuracy that a model can achieve. Conversely, the FC-Sub results do align with this belief.
We argue that a full-class model needs to accommodate many parameters to learn features that will enable high performance across all classes in a many-class, full dataset. With the same parameters, however, a sub-class model can adapt to finer and more discriminative features that improve its performance when the number of target classes are much smaller.

Figure: Figure 4: DCN-Sub (red) and DCN-Full (blue) by Top-1 Accuracy ($\%$). $N_{CL}$ ranges from $2$ to $4$.
Refer to caption: x8.png

### 4.3 Results on FC-Sim

In this section, we analyze the use of SimSS (Equation 14) as proxy for few-class dataset difficulty. Experiments are conducted on ImageNet1K using the ResNet family for the lower $N_{CL}\leq 10\%$ range of the original $1000$ classes, $N_{CL}\in\{2,3,4,5,10,100\}$, and the results are shown in Figure 5. Each datapoint of DCN-Full (diamond in red) or DCN-Sub (square in blue) represents an experiment in a subset of a specific $N_{CL}$, where classes are sampled from the full dataset. For reproducible results, we use seed numbers from 0 to 4 to generate 5 subsets for one $N_{CL}$ by default. A similarity base function ($sim()$) is defined as the atomic function that takes a pair of images as input and outputs a scalar that represents their image similarity.

In our experiments, we leverage the general visual feature extraction ability of CLIP (image + text) [] and DINOv2 (image) [] by self-supervised learning. Specifically, a pair of images are fed into its latent space from which the the cosine score is calculated and normalized to 0 0 to $1$. Note that we only use the Image Encoder in CLIP.

Comparing Accuracy and Similarity
To evaluate SimSS, we compute the Pearson correlation coefficient (PCC) ($r$) between model accuracy and SimSS. Results in Figure 5 (a) (b) show that SimSS is poorly correlated with DCN-Full ($r=0.18$ and $r=0.26$ for CLIP and DINOv2) due to the large variance shown in Section 4.1. In contrast, SimSS is highly correlated with DCN-Sub (shown in blue squares), with $r=0.90$ and $r=0.88$ using CLIP (dashed) and DINOv2 (solid), respectively. The high PCC [, ] demonstrates that SimSS is a reliable metric to estimate few-class dataset difficulty, and this can help predict the empirical upper-bound accuracy of a model in the Few-Class Regime. Comparison between SimSS and all models can be found in the Appendix. Such a high correlation suggests this offers a reliable scaling relationship to estimate model accuracy by similarity for other values of $N_{CL}$ without an exhaustive search. Due to the dataset specificity of the dataset difficulty property, this score is computed once and used for all times the same dataset is used. We have made available difficulty scores for many datasets at the Few-Class Arena site.

Figure: (a) DCN-Full
Refer to caption: x9.png

## 5 Conclusion

We have proposed Few-Class Arena and a dataset difficulty measurement, which together form a benchmark tool to compare and select efficient models in the Few-Class Regime. Extensive experiments and analyses over 1500 experiments with 10 models on 10 datasets have helped identify new behavior that is specific to the Few-Class Regime as compared to for many-classes. One finding reveals a new $N_{CL}$-scaling law whereby dataset difficulty must be taken into consideration for accuracy prediction. Such a benchmark will be valuable to the community by providing both researchers and practitioners with a unified framework for future research and real applications.

Limitations and Future Work. We note that the convergence of sub-models is contingent on various factors in a training scheduler, such as learning rate. A careful tuning of training procedure may increase a model’s performance, but it shouldn’t change the classification difficulty number drastically since this represents a dataset’s intrinsic difficulty property. The current difficulty benchmark supports image similarity while in the future it can be expanded to other difficulty measurements [, ]. As CLIP and DINOv2 are trained toward general visual features, it is unclear if they will be appropriate for other types of images such as sketches without textures in Quickdraw [] . For this reason, a universal similarity foundation model would be appealing that applies to any image type. In summary, Few-Class Arena identifies a promising new path to achieve efficiencies that are focused on the important and practical Few-Class Regime, establishing this as a baseline for future work.

## Appendix A Appendix

### A.1 Full Models on ImageNet

In practice, ImageNet serves as a common benchmark for vision neural networks. We list the details of 10 pre-trained models from MMPreTrain [] in terms of Top-1 Accuracy and scale ($\#Params$) in Fig. 6.

Figure: Figure 6: Top-1 Accuracy ($\%$) vs. number of parameters and FLOPs (G) (size of circle) on ImageNet.
Refer to caption: x13.png

**Table 2: Full models pre-trained on ImageNet.**
| Model | Ref. | Model | Ref. |
| --- | --- | --- | --- |
| Conformer | [] | ConvNeXt | [] |
| EfficientFormer | [] | EfficientNet | [] |
| RepVGG | [] | ResNet | [] |
| ShuffleNet | [] | Swin | [] |
| VGG | [] | ViT | [] |

### A.2 Extended Many-Class Full Dataset Trained Benchmark Results

A complete ranking of 10 models in 10 datasets is depicted in Fig. 7. Observe that the 10 models’ rankings differ dramatically among 10 different datasets where each line changes from ImageNet1K (IN1K) to other datasets. This poses some questions whether rankings in existing benchmarks can be a reliable indicator for a practitioner to select an efficient neural network, especially when the deployed environment changes from application to application. A major variable in this process is the reduced number of classes from benchmark datasets to deployed environments in the Few-Class Regime. As such, our tool is developed to facilitate research in the Few-Class Regime.

Figure: Figure 7: Extended Details of Fig. 2 (b) in the main paper. Full Ranking of 10 models across 10 datasets by Top-1 acc.
Refer to caption: x14.png

### A.3 Datasets

Dataset information is presented in Table 3.

**Table 3: Dataset information. * Note that ImageNet dataset format is used as the reference for other datasets. Therefore Path in FCA for ImageNet is not required.**
| Dataset Name | Abbrev. | Ref. | Homepage | Path in FCA |
| --- | --- | --- | --- | --- |
| Caltech 101 | CT101 | [] | [data.caltech.edu/records/mzrjq-6wc02](data.caltech.edu/records/mzrjq-6wc02) | [tools/ncls/datasets/caltech101.py](tools/ncls/datasets/caltech101.py) |
| Caltech 256 | CT256 | [] | [data.caltech.edu/records/nyy15-4j048](data.caltech.edu/records/nyy15-4j048) | [tools/ncls/datasets/caltech256.py](tools/ncls/datasets/caltech256.py) |
| CIFAR-100 | CF100 | [] | [cs.toronto.edu/ kriz/cifar.html](cs.toronto.edu/~kriz/cifar.html) | [tools/ncls/datasets/cifar100.py](tools/ncls/datasets/cifar100.py) |
|  |  |  | [github.com/knjcode/cifar2png](github.com/knjcode/cifar2png) |  |
| Caltech-UCSD | CB200 | [] | [vision.caltech.edu/visipedia/CUB-200-2011.html](vision.caltech.edu/visipedia/CUB-200-2011.html) | [tools/ncls/datasets/cub200.py](tools/ncls/datasets/cub200.py) |
| Birds-200-2011 |  |  | [data.caltech.edu/records/65de6-vp158/files/CUB_200_2011.tgz](data.caltech.edu/records/65de6-vp158/files/CUB_200_2011.tgz) |  |
| Food 101 | FD101 | [] | [vision.ee.ethz.ch/datasets_extra/food-101/](vision.ee.ethz.ch/datasets_extra/food-101/) | [tools/ncls/datasets/food101.py](tools/ncls/datasets/food101.py) |
|  |  |  | [huggingface.co/datasets/food101](huggingface.co/datasets/food101) |  |
| German Traffic Sign | GT43 | [] | [benchmark.ini.rub.de/](benchmark.ini.rub.de/) | [tools/ncls/datasets/gtsrb43.py](tools/ncls/datasets/gtsrb43.py) |
| ImageNet | IN1K | [] | [image-net.org/challenges/LSVRC/2012/index.php](image-net.org/challenges/LSVRC/2012/index.php) | * |
| Indoor Scene Recognition | ID67 | [] | [web.mit.edu/torralba/www/indoor.html](web.mit.edu/torralba/www/indoor.html) | [tools/ncls/datasets/indoor67.py](tools/ncls/datasets/indoor67.py) |
| Quickdraw | QD345 | [] | [github.com/googlecreativelab/quickdraw-dataset](github.com/googlecreativelab/quickdraw-dataset) | [tools/ncls/datasets/quickdraw345.py](tools/ncls/datasets/quickdraw345.py) |
|  |  |  | [tensorflow.org/datasets/community_catalog/huggingface/quickdraw](tensorflow.org/datasets/community_catalog/huggingface/quickdraw) |  |
| Describable Textures Dataset | TT47 | [] | [robots.ox.ac.uk/ vgg/data/dtd/index.html](robots.ox.ac.uk/~vgg/data/dtd/index.html) | [tools/ncls/datasets/textures47.py](tools/ncls/datasets/textures47.py) |

License. We have searched available online resources and list the license of each dataset in Table 4. For licenses not found in the datasets or websites denoted as “*”, we assume they are non-commercial research use only.

**Table 4: Licenses of ten datasets.**
| Dataset | License | Dataset | License |
| --- | --- | --- | --- |
| CT101 | CC BY 4.0 | CT256 | CC BY 4.0 |
| CF100 | * | CB200 | * |
| FD101 | * | GT43 | * |
| IN1K | * | ID67 | (DbCL) v1.0 |
| QD345 | CC BY 4.0 | TT47 | * |

Train/val splits.
The dataset format follows the convention of ImageNet:

where a .txt file stores a pair of image id and and class number in each row in the following format

We follow the same train/val splits when the original dataset has already provided. If the dataset does not have explicit splits, we first assign image IDs to all images, starting from 0, and select $4/5$ of all images as training set and put the rest in the validation set. Specifially, when an image whose ID satisfies the condition $ID$ $\%$ $5==0$, it will be moved to the validation set. Otherwise, it will be assigned as a training sample.

### A.4 Model Training Details

Model training details are presented in Table 5.

**Table 5: Model Training information. LR: Learning rate. SGD: Stochastic gradient descent. AdamW: Adam with weight decay. RMSprop: Root mean square propagation.**
| Model | Abbrev. | Ref. | Optimizer | LR | Weight | Other Params |
| --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | Decay |  |
| ResNet50 | RN50 | [] | SGD | 0.1 | 1e-4 | momentum=0.9 |
| VGG16 | VGG16 | [] | SGD | 0.01 | 1e-4 | momentum=0.9 |
| ConvNeXt V2 | CNv2 | [] | AdamW | 2.5e-3 | 0.05 | eps=1e-8 |
| Base |  |  |  |  |  | betas=(0.9, 0.999) |
| Inception V3 | INCv3 | [] | SGD | 0.1 | 1e-4 | momentum=0.9 |
| EfficientNet V2 | EFv2 | [] | SGD | 4e-3 | 0.1 | momentum=0.9 |
| Medium |  |  |  |  |  | clip_grad: max_norm=5.0 |
| ShuffleNet V2 | SNv2 | [] | SGD | 0.5 | 0.9 | momentum=4e-5 |
| MobileNet V3 | MNv3 | [] | RMSprop | 6.4e-4 | 1e-5 | alpha=0.9 |
| Small |  |  |  |  |  | momentum=0.9 |
|  |  |  |  |  |  | eps=0.0316 |
| Vision Transformer | ViTb | [] | AdamW | 3e-3 | 0.3 | - |
| Base |  |  |  |  |  |  |
| Swin Transformer V2 | SWv2b | [] | AdamW | 1e-4 | 0.05 | eps=1e-8 |
| Base |  |  |  |  |  | betas=(0.9, 0.999) |
| MobileViT | MViTs | [] | SGD | 0.1 | 1e-4 | momentum=0.9 |
| Small |  |  |  |  |  |  |

### A.5 Extended Few-Class Full Dataset Trained Benchmark (FC-Full) Results

We present the details of FC-Full results for each experiment model, including ResNet50, VGG16, ConNeXt V2 Base, Inception V3, EfficientNet V2 Medium, ShuffleNet V2, MobileNet V3 Small, ViT Base, Swin Transformer V2 Base, MobileViT Small in Fig. 8-17, respectively.

Figure: Figure 8: FC-Full Top-1 Accuracy ($\%$) for ResNet50.
Refer to caption: x15.png

Figure: Figure 9: FC-Full Top-1 Accuracy ($\%$) for VGG16.
Refer to caption: x16.png

Figure: Figure 10: FC-Full Top-1 Accuracy ($\%$) for ConNeXt V2 Base.
Refer to caption: x17.png

Figure: Figure 11: FC-Full Top-1 Accuracy ($\%$) for Inception V3.
Refer to caption: x18.png

Figure: Figure 12: FC-Full Top-1 Accuracy ($\%$) for EfficientNet V2 Medium.
Refer to caption: x19.png

Figure: Figure 13: FC-Full Top-1 Accuracy ($\%$) for ShuffleNet V2.
Refer to caption: x20.png

Figure: Figure 14: FC-Full Top-1 Accuracy ($\%$) for MobileNet V3 Small.
Refer to caption: x21.png

Figure: Figure 15: FC-Full Top-1 Accuracy ($\%$) for ViT Base.
Refer to caption: x22.png

Figure: Figure 16: FC-Full Top-1 Accuracy ($\%$) for Swin V2 Base.
Refer to caption: x23.png

Figure: Figure 17: FC-Full Top-1 Accuracy ($\%$) for MobileNetViT Small.
Refer to caption: x24.png

### A.6 Extended Few-Class Similarity Benchmark (FC-Sim) Details

We present the extended mathematical details of 3.6 Few-Class Similarity Benchmark (FC-Sim) in the main paper.

The basic similarity formulation is adopted from []. Notations are defined as follows:

Dataset $D$: a set of image instances in a dataset.

Class $C$: a set of image instances in a class and $|C|$ is the number of instances within class $C$.

Class Label $L$: a set of class labels in a dataset and $|L|$ is the number of classes in a dataset.

Feature $Z_{i}$: visual feature of an image and $i$ is the instance index.

Class Pair $P^{(D)}$: a set of distinct class pairs in a dataset $D$; $|P^{(D)}|$ is the total number of distinct class pairs.

Intra-Class Image Pair $P^{(C)}$: a set of distinct image pairs in a class $C$; $|P^{(C)}|$ is the total number of distinct image pairs.

Inter-Class Image Pair $P^{(C_{1},C_{2})}$: a set of distinct image pairs in two classes $C_{1},C_{2}$; $|P^{(C_{1},C_{2})}|$ is the total number of distinct image pairs. Note that this does not include same-class pairs.

Intra-Class Similarity $S_{\alpha}^{(C)}$: a scalar describing the similarity of images within a class by taking the average of all the distinct class pairs in $C$:

$$ $S_{\alpha}^{(C)}=\frac{1}{|P^{(C)}|}\sum_{i,j\in C;i\neq j}\cos(\textbf{Z}_{i} ,\textbf{Z}_{j}).$ (5) $$

Inter-Class Similarity $S_{\beta}^{(C_{1},C_{2})}$: a scalar describing the similarity among images in two different classes ${C_{1}}$ and $C_{2}$:

$$ $S_{\beta}^{(C_{1},C_{2})}=\frac{1}{|P^{(C_{1},C_{2})}|}\sum_{i\in C_{1},j\in C _{2}}\cos(\textbf{Z}_{i},\textbf{Z}_{j}),$ (6) $$

where $C_{1}$ and $C_{2}$ are distinct classes, $i$ and $j$ are the image instance indices in $C_{1}$ and $C_{2}$, respectively. $P^{(C_{1},C_{2})}$ is the set of distinct pairs of images between $C_{1}$ and $C_{2}$.

The above equations formulate class-level similarity scores. For dataset-level, Intra-Class Similarity and Inter-Class Similarity of a dataset $D$ are defined as the mean of their similarity scores, respectively:

$$ $S_{\alpha}^{(D)}=\frac{1}{|L|}\sum_{l\in L}S_{\alpha}^{(C_{l})}=\frac{1}{|L| \times|P^{(C_{l})}|}\sum_{l\in L}\sum_{i,j\in C_{l};i\neq j}\cos(\textbf{Z}_{i },\textbf{Z}_{j}),$ (7) $$

$$ $S_{\beta}^{(D)}=\frac{1}{|P^{(D)}|}\sum_{a,b\in L;a\neq b}S_{\beta}^{(C_{a},C_ {b})}=\frac{1}{|P^{(D)}|\times|P^{(C_{1},C_{2})}|}\sum_{a,b\in L;a\neq b}\sum_ {i\in C_{1},j\in C_{2}}\cos(\textbf{Z}_{i},\textbf{Z}_{j}).$ (8) $$

Averaging these similarities can provide a summary of score in class or dataset levels by a single scalar. However, this simplicity neglects other cluster-related information that can better reveal the underlying dataset difficulty property of a dataset. In particular, the (1) tightness of a class cluster and (2) distance to other classes of class clusters, are features that characterize the inherent class difficulty, but are not captured by $S_{\alpha}$ or $S_{\beta}$ alone.

To compensate the aforementioned drawback, we adopt the Silhouette Score (SS) (also called Silhouette Coefficient in the literature) [, ]:

$$ $SS(i)=\frac{b(i)-a(i)}{max({a(i),b(i)})},$ (9) $$

where $SS(i)$ is the Silhouette Score of the data point $i$, $a(i)$ is the average dissimilarity between $i$ and other instances in the same class, and $b(i)$ is the average dissimilarity between $i$ and other data points in the closest different class. Intuitively, this metric summarizes the quality of clusters jointly by the degree of instances of the same class and distinct clusters, normalized by the longest distance of $a(i)$ and $b(i)$. By this definition, we can see that $-1\leq SS(i)\leq 1$ where $-1$ indicates a dataset is poorly clustered (data points with different classes are scattered around) while $1$ represents a well-clustered dataset.

Euclidean Distance is commonly used to measure two data points’ differences; in contrast, we incorporate the inverse of similarity (dissimilarity) as data point’s difference into the existing Silhouette Score. Observe that the above Intra-Class Similarity $S_{\alpha}^{(C)}$ already represents the tightness of the class $(C)$, therefore $a(i)$ can be replaced with the inverse of Intra-Class Similarity $a(i)=-S_{\alpha}(i)$. For the second term $b(i)$, we adopt the previously defined Inter-Class Similarity $S_{\beta}^{(C_{1},C_{2})}$ and introduce a new similarity score as follows:

Nearest Inter-Class Similarity $S^{\prime(C)}_{\beta}$: a scalar describing the similarity among instances between class $C$ and the closest class of each instance in $C$:

$$ $S^{\prime(C)}_{\beta}=\frac{1}{|P^{(C,\hat{C})}|}\sum_{i\in C,j\in\hat{C}}\cos (\textbf{Z}_{i},\textbf{Z}_{j}),$ (10) $$

where $\hat{C}$ is the set of the nearest class to $C$ ($\hat{C}\neq C$).

Consequently, the dataset-level Nearest Inter-Class Similarity $S^{\prime(D)}_{\beta}$ is expressed as:

$$ $S^{\prime(D)}_{\beta}=\frac{1}{|L|}\sum_{l\in L}S^{\prime(C_{l},\hat{C_{l}})}_ {\beta}=\frac{1}{|L|\times|P^{(C_{l},\hat{C_{l}})}|}\sum_{l\in L}\sum_{i\in C_ {l},j\in\hat{C_{l}}}\cos(\textbf{Z}_{i},\textbf{Z}_{j}).$ (11) $$

The second term of $SS(i)$ can be written as $b(i)=-S^{\prime}_{\beta}(i)$.

Replacing $a(i)$ and $b(i)$ from equation 9 with these similarity terms, we introduce our novel similarity metric:

Similarity-Based Silhouette Score $SimSS$:

$$ $SimSS(i)=\frac{S_{\alpha}(i)-S^{\prime}_{\beta}(i)}{max(S_{\alpha}(i),S^{ \prime}_{\beta}(i))},\hskip 10.0pt\text{for instance}\hskip 5.0pti$ (12) $$

$$ $SimSS^{(C)}=\frac{1}{|C|}\sum_{i\in C}SimSS(i)=\frac{1}{|C|}\sum_{i\in C}\frac {S_{\alpha}(i)-S^{\prime}_{\beta}(i)}{max(S_{\alpha}(i),S^{\prime}_{\beta}(i)) },\hskip 10.0pt\text{for class}\hskip 5.0ptC$ (13) $$

$$ $\begin{split}SimSS^{(D)}&=\frac{1}{|L|}\sum_{l\in L}SimSS^{(C_{l})}=\frac{1}{| L|\times|C_{l}|}\sum_{l\in L,i\in C_{l}}SimSS(i)\\ &=\frac{1}{|L|\times|C_{l}|}\sum_{i\in C_{l}}\frac{S_{\alpha}(i)-S^{\prime}_{ \beta}(i)}{max(S_{\alpha}(i),S^{\prime}_{\beta}(i))},\hskip 10.0pt\text{for dataset}\hskip 5.0ptD.\end{split}$ (14) $$

### A.7 Extended Few-Class Similarity Benchmark (FC-Sim) Results

We present the relationship of similarity scores using our proposed SimSS and number of classes $N_{CL}$ in ten datasets. CLIP and DINOv2 are used as similarity base functions of SimSS shown in Fig. 18 (a) and (b), respectively.

Overall, a key observation is that the general trend among all ten datasets unveils the inverse relationship between similarity and the number of classes. Specifically, image similarities, which act as proxy of inverse subset difficulty score increases as the number of classes $N_{CL}$ decreases. This reveals that similarity plays a more important role in the Few-Class Regime than for datasets with more classes. Therefore, for real applications with few classes, simply downscaling a model blindly without considering class similarity may yield a model selection with sub-optimal efficiency. We propose, therefore, that image similarity must be taken into consideration for existing scaling laws [, , ]. To that end, Few-Class Arenas developed to facilitate future research in this direction.

Figure: (a) SimSS using CLIP as similarity base function vs $N_{CL}$ curve.
Refer to caption: x25.png

Note that both CLIP and DINOv2 are trained on images from the Internet similar to ImageNet. Therefore to what extent they can capture image similarity in different types is an open research question. Examples include drawings without textures in the Quickdraw dataset (QD345), textures without shapes in the Describable Textures Dataset (TT47), etc. We mentioned this limitation also in the main manuscript.

Effect of ResNet Scales on Similarity. We present the details of FCA-Sim results of the ResNet family in different scales in the Few-Class Regime of ImageNet1K, specifically ($N_{CL}\in\{2,3,4,5,10,100\}$) shown in Fig. 19. In particular, we analyze the relationship between each full and sub-Model’s Top-1 accuracy and SimSS by Pearson correlation coefficient (PCC) denoted as $r$ in the plots. The ResNet family scales from ResNet18 to ResNet152. We experiment both CLIP (red in the first two rows) and DINOv2 (blue in the last two rows) as similarity base functions.

Figure: (a) ResNet18
Refer to caption: x27.png

In general, ResNet presents high correlation ($r\geq 0.80$) between sub-models’ performances and SimSS (blue in the last two rows), compared to full models’ performances and SimSS ($r\leq 0.26$, red in the first two rows). This high correlation indicates that SimSS can be used as a reliable tool to estimate upper bound accuracies of ResNet sub-models. Comparing CLIP ($r$ in the 1st row) with DINOv2 ($r$ in the 2nd row) as similarity base functions, observe that PCC is slightly higher for DINOv2 on full models than CLIP, while these differences are subtle for sub-models ($r$ in the 3rd row vs 4th row). Regarding sub-models of ResNet in different scales, the two smallest models’ accuracies (ResNet18 and ResNet34) have higher correlation with SimSS ($r\geq 0.88$), compared to larger models with (ResNet50, ResNet101 and ResNet152) $r\geq 0.80$. We opens a new direction of novel scaling law considering image similarity for efficient models in Few-Class Regime.

Transformer vs Similarity. We present the details of Vision Transformer (ViT) [] Base performance and SimSS using CLIP and DINOv2 in the subsets of $N_{CL}\in\{2,3,4,5,10,100\}$ in ImageNet1K, shown in Fig. 20. An overview of the results indicates low correlation between Top-1 accuracy and SimSS ($r\leq 0.20$). Our tool Few-Class Arena helps discover the new challenge of using the vanilla ViT in the Few-Class Regime, as ViTs are originally designed to scale up []  when large training datasets in the order of billion samples are available. Few-Class Arena also identifies the importance of scaling down considering similarity.

Figure: (a) Full
Refer to caption: x47.png

### A.8 Experiments Compute Resources

Experiments are conducted in two internal clusters with the following specifications: (1) 8 NVIDIA RTX A5000 24GB, AMD EPYC 7513 32-Core Processor 882GB RAM and (2) 8 NVIDIA TITAN Xp 12GB, Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, 126GB RAM. When GPUs in two clusters are fully utilized, training 10 models in 9 datasets takes 2 weeks; obtaining one experiment result of FC-Full usually takes less than 1 minute since it only involves inference without training; obtaining one experiment result of FC-Sub takes around 2 days on average depending on the size of subset and model, which includes both training and testing; computing the SimSS in the Few-Class Regime for ten datasets takes around three weeks.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * I may have missed something, but I cannot understand exactly how the proposed difficulty metric is intended to be used?
* The proposed difficulty metric seems expensive to compute, with pairwise similarity scores required for large subsets of the data. How does this compare to the cost of conducting a single model training run?
soundness: 3
strengths: - Tab 1 / Fig 2b shows the results when training different base architectures on different classification datasets from scratch. Interestingly, not all results are correlated with the trend on ImageNet-1K, indicating the optimal architecture choice depends on the dataset.
- The code is open sourced and well documented. It seems that it would be simple for a researcher to reproduce the authors claim with limited effort (though I have not run the code myself).
- It is interesting that sub-models consistently outperform full models on ImageNet. The fact that full models have seen more training datapoints in total may have compensated for fewer classes, which makes the result not totally intuitive.
summary: This paper tackles the problem of choosing image classifiers for tasks with only a small number of categories ("Few-Class"). To do so, they introduce a new benchmark, termed "Few Class Arena" (FCA) on which they train and evaluate a range of models on subsets of various full datasets (e.g wit so-called sub-models trained on between 2 and all 1k ImageNet categories). The FCA benchmark is open-sourced with code available on GitHub. The paper provides detailed discussion of how the open-source package can be used for model selection in the few-class setting.

Overall, the authors show that models trained on specific sub-classes (sub-models) are better than models trained on the full dataset and evaluated on the same sub-classes, across model sizes. They further show that there is no single best model architecture for a given dataset, and that training models on different datasets result in different rankings of architecture. The authors also propose a "dataset difficulty" metric which can be computed without training a model, and correlates well with the few-class performance of a model on a dataset.
weaknesses: My main issue with this paper is in overall utility. The high level goal of the paper is to provide a tool with which practitioners can select a model (dominantly through the lens of model *architecture*) for a few-class classification task. The tool basically allows authors to train a model (with most results presented from scratch) on subsets of a given dataset. However, this does not align with the practical problem to me, where practitioners might take a model pretrained on a large amount of *data* (e.g DINOv2 or CLIP) is finetuned for a given task (note that lightweight variants of these models are also open-source). 

Given that this paper is predominantly an empirical examination which proposes a practical open-source library, I feel that the lack of experiments with pretrained models prevents acceptance. 

Other issues:

- The citation format makes the main text quite difficult to parse
- L52: Main text does not seem to describe Figure 1 accurately?
- There is no discussion of few-shot literature, which is at least tangentially related to this problem

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. For FC-Full, when N_{CL} decreases, how to make sure that model predicts only few classes? Are the logits of those few classes are selected to get the prediction and discard logits of all other classes?

2. If a user has a custom dataset with few classes and want to find a model that works better on this custom dataset, it would be helpful to have an explanation on how this benchmark can assist the user in this case.

---------------------------------------------------------------------------------------
Final review: I appreciate authors for their detailed responses to the comments from all the reviewers. Authors comprehensive responses, clarifications, and additional experiments have addressed the key weaknesses. I now vote towards borderline acceptance.
soundness: 2
strengths: •	Addressed an important problem by proposing the benchmarking tool.

•	The tool is designed to be user friendly and allows to run wide range of experiments by setting few hyper-parameters. The tool allows to benchmark on custom models and datasets.

•	Provided a behavioral understanding between models trained on large number of classes vs smaller number of classes in the few-class regime.

•	The proposed similarity metric shown to be linearly corelated with the model performance on small number of classes. Such proxy helps to save computation cost and time of conducting various experiments.
summary: The paper proposes a benchmark tool called Few-Class Arena to benchmark models on different datasets with smaller number of classes (e.g. < 10) and propose a similarity metric called SimSS to measure dataset difficulty measure. They show that ResNet family models trained on full ImageNet 1K classes show reduced performance when tested only for few ImageNet classes (< 10 classes). On the other hand, the same models when trained on smaller number of ImageNet classes from scratch show higher performance on these classes when compared to models trained on all classes of ImageNet 1k. They show that the proposed SimSS metric can serve as a proxy to estimate the upper bound accuracy of model performance on few-class datasets.
weaknesses: Despite focusing on an interesting problem setting, the analysis shown in the paper has limited scope. Authors shown experiments on models evaluated or trained on smaller number of classes, however there are no details discussed on how these few classes have been selected and how semantically close these few classes to each other? Would the analysis presented differ by choosing the those few classes differently?

The aspect of transfer learning has not been discussed. It is a common practice to finetune ImageNet pretrained models like ResNet50 or ViT, or recent foundation models like CLIP and DINOv2 to different downstream tasks that include adapting or finetuning them on few classes. The analysis presented in the paper is missing this exploration. Is it better to train the models from scratch on the few classes or finetuning these models work better for few classes? Does SimSS score also align on the finetuned models?

To compute SimSS, a score called Nearest Inter-Class Similarity requires a nearest class (C_hat) to the target class (C), it is not clear how this C_hat is acquired.

Overall, I appreciate the motive and tool for benchmarking few-class regime, however the analysis presented in the paper is incomplete, and I suggest authors to extend their analysis.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What original research do you expect will use this benchmark and what do you hope it will achieve or unlock ?
soundness: 3
strengths: - The benchmark is well executed and will be useful for “few-class” adaptation research. There are many models evaluated with many datasets and the analysis is thorough. In particular, the author study in depth the evolution of the performance of models trained on large set of classes compared to specialized models, as a function of the number of classes, and show the importance, and therefore propose a metric for evaluating this adaptation.

- The similarity benchmark is a nice addition. It correlates well with the performance while being easy to evaluate and with a modest cost.

- The presentation and writing are very clear. The figures are very informative.
summary: This paper presents a new benchmark for the “few-class” problem, which  is a classification problem with very few classes. Most of the scientific literature focuses on datasets with many classes while practitioners often encounter the few-class scenario. The benchmark consist of several selected datasets and several settings, such as training on large set of classes and evaluating on a smaller set, and popular vision models are evaluated and compared. Finally, an analysis of what happens in few-class regimes is proposed.
weaknesses: - The motivation behind few-class evaluations is not fully convincing. In practice, one will take a large model and fine-tune it (without the classification layer) to a target set of classes, hence obtaining a specialized model. Evaluating the capabilities of a full model on few-class is only interesting when there are too many subsets to consider ? When does that happen in practice, and could you just not use small adaptation layers on top of frozen backbone for each of the subsets ?

- One thing that is missing from the paper is a recommendation for practitioners on which vision model to use for someone interested in the few-class problem. Basically discussing in more details the results from Table 1 and providing comparison between models in Section 4.2 and 4.3. One interesting question is, do models that perform really well on the many classes setup are the same that also perform well on the few-class setup ?

- Some of the findings in the paper are fully expected. The fact that a model specifically trained on the target subset of classes perform better that a larger model trained on a superset is not very surprising or novel.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. (Related to W2) The finding that the scaling law w.r.t. model size is violated for submodes is interesting. I would be curious if this only applies to supervised training or also to self-supervised pre-training combined with minimal fine-tuning or linear probing. Did you conduct any experiments into this direction or do you have an intuition on that?
2. You mention that you conducted experiments on different architectures such as ResNet’s and ViT’s. However, the results are presented in an aggregated way. Did you find any significant differences between model architectures? Do the findings of Fig 1 equally apply for both architectures?
3. I agree with your description and ad-hoc interpretation of Fig. 5. However, I am missing a discussion on why we see the low correlation for DCN-Full. Do you have any interpretation of this?


**Minor comments:**

- The correct use of author-year citations would improve readability. I.e.: Author (2024) for in-text citations and (Author, 2024) elsewise.
soundness: 3
strengths: 1. The necessity of a few-class benchmark is well-motivated by a strong finding that models pre-trained on many-class datasets perform worse than expected on few-class datasets which is an issue not addressed in literature thus far.
2. The paper presents some interesting insights that contradict expected model behavior based on intuition.
3. The authors provide a ready to use code base integrated with other frameworks and libraries ensuring usability.
summary: The paper introduces a benchmark designed to evaluate and select efficient image classification models in scenarios with a limited number of classes. This setting, common in real-world applications (e.g., 2-10 classes), contrasts with widely used benchmarks like ImageNet and COCO, which involve hundreds or thousands of classes. The paper presents FCA as a tool to help researchers and practitioners efficiently select models for few-class tasks. The paper coins the term ``few-class regime'' and presents some interesting non-intuitive insights regarding the performance of models that are pre-trained with many class datasets and then applied in few-class settings.
In addition, they introduce a dataset difficulty metric by inverting image similarity measured via CLIP and DINOv2 features.
weaknesses: 1. The scope of the study only covers image classification, while it is in principle also applicable to dense prediction tasks where object classes are present (e.g., object detection, semantic segmentation). It would be insightful if the same findings hold for these tasks. The authors could add a discussion or experiments (if available) for other vision tasks.
2. The experiments cover classification tasks based on supervised (pre-)training. However, there is an increasing trend that classification models are fine-tuning based on self-supervised pre-trained models. This paradigm is not covered in this study, and therefore, the findings are limited to the more traditional fully supervised paradigm. The authors could discuss these aspects or add experimentation with self-supervised pre-trained models (if available).

Meta-review: This work aims to benchmark image classification in the few-class regime as a proxy for performance on real-world tasks of this size. Such a focus is complementary to existing large-scale benchmarks such as ImageNet, COCO, etc. with 10s of classes or 1000 classes. The contributions include experiments over many subsets of classes on ImageNet, evaluations of popular model architectures like ResNets and ViTs, and a metric that incorporates class similarity into a difficulty score. The application of these contributions is to guide model selection, to identify the most efficient architecture for a given set of classes, and provide scaling predictions in this regime, and these applications are shown by this work. The benchmark is released as an open-source project for reproduction and future work.

Strengths: The topic is complementary to the majority of work focusing either on (A) the large-scale regime of many classes and a lot of data or (B) the few-shot regime of scarce data. The experiments are thorough in covering different model architectures, numbers of classes, and learning with and without transfer. The proposed difficulty metric can save experimentation time (6NMw).

Weaknesses: The evaluation of the full-class models is fair but incomplete in not conditioning on the subclasses at all: the full models could predict the argmax of the logits for the classes intersecting with a given subset. There is doubt about the utility of the contributed benchmark and analysis to the community, because transfer learning and prompting are nearly ubiquitous (trsf, 6NMw).

Decision: This is a borderline submission, and reviewers questioned the motivation and the justification of its experimental scope. However, the revision and additional results have proven the use of the benchmark, even though it is has a somewhat narrow scope of minimal/efficient architectures that are sufficient for few-class accuracy, and reviewers were convinced to maintain or improve their ratings. The proposed difficulty metric and how it might guide model and dataset scaling brings another lens to scaling questions that are now faced in research and practice alike. Given these uses, and the full open-source release of the benchmark, the meta-reviewer sides with acceptance. Congratulations!

Decision: Accept (Poster)