Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Authors: Zhijing Wan
Zhixiang Wang
Zheng Wang
Xin Xu
Shin'ichi Satoh
Venue: ICML 2025 oral (2025)
Keywords: one-shot subset selection
foundation models
data-efficient learning
Primary area: general_machine_learning
TL;DR: This paper investigates the effectiveness of using foundation models (FMs) as information extractor for one-shot subset selection on a set of image datasets, and proposes a novel multi-foundation-model subset selection method called RAM-APL.

Abstract: One-shot subset selection serves as an effective tool to reduce deep learning training costs by identifying an informative data subset based on the information extracted by an information extractor (IE). Traditional IEs, typically pre-trained on the target dataset, are inherently dataset-dependent. Foundation models (FMs) offer a promising alternative, potentially mitigating this limitation. This work investigates two key questions: (1) Can FM-based subset selection outperform traditional IE-based methods across diverse datasets? (2) Do all FMs perform equally well as IEs for subset selection? Extensive experiments uncovered surprising insights: FMs consistently outperform traditional IEs on fine-grained datasets, whereas their advantage diminishes on coarse-grained datasets with noisy labels. Motivated by these finding, we propose RAM-APL (RAnking Mean-Accuracy of Pseudo-class Labels), a method tailored for fine-grained image datasets. RAM-APL leverages multiple FMs to enhance subset selection by exploiting their complementary strengths. Our approach achieves state-of-the-art performance on fine-grained datasets, including Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011.

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Zhijing Wan 12
Zhixiang Wang 34 Zheng Wang'
Xin Xu
Shin'ichi Satoh 43
Abstract
b
Training set D
Pre-trained
One-shot subset selection serves as an effective tool reduce deep learning training costs by iden- tifying informative data subset based on the information extracted by information extractor (IE). Traditional IEs, typically pre-trained on the target dataset, are inherently Cataset-dependens Foundation models (FMs) offer a promising al- ternative, potentially mitigating this limitation. This work investigates two key questions: (1) Can FM-based subset selection outperform traditional IE- [E-based methods across diverse datasets? (2) Do all FMs perform equally well as IEs for subset se- lection? Extensive experiments uncovered surpris- ing insights: FMs consistently outperform tradi- tional IEs on fine-gra line-graieed datasets, whereas their advantage diminishes on poarse-graines datasets with noisy labels. Motivated by these finding, we propose RAM-APL (RAnking Mean-Accuracy of Pscudo-class Labels), method tailored for fine-graines image datasets. RAM-APL lever- ages multiple FMs to enhance subset selection by exploiting their complementary strengths. Our ap- proach achieves state-of-the-art performance on fine- innegneined datasets, including Oxford-IIIT Pet. Food-101, and Caltech-UCSD Birds-200--2011.
Information Extractor (IE)
O0 DINOV2 SigUIP
$ CUIP
CNN
FM pool
EVA-CLIP
Subset selection
Subset
selection
Subset
CIFAR-10 CIFAR-10N CIFAR-101 101
Pet
Pet-N
Model-TD
Model-TIN
Single
10
Number of times the best IE
Figure I. Comparison of pipelines for one-shot subset selection. Traditional pipeline (He al., 2024): Relies model pre- trained the full training set of the target task extract data information. but this leads to strong dataset dependency and ad- ditional pre-training cost. (h) Pipeline with single foundation model (Xie al., 2023): Replaces the small pre-trained model with single FM, potentially mitigating dataset dependency. As shown (c), on fine-grained datasets, using a single FM as an IE significantly and consistently superior to using traditional IE, and improves the performance of subset selection at different sampling rates.
1. Introduction
Subset selection, also known coreset selection (Zheng et al., 2023; Wan et al., 2024b), has become an effective approach to improve model training efficiency by identi-
fying small, representative subset of training data with- out significantly compromising model performance. This task is particularly important in scenarios involving large- scale datasets (Wan et al., 2024a; Wang et al., 2025; Jia et al., 2025), where full dataset training is computationally prohibitive. Subset selection methods can broadly cate- gorized into one-shot (Xia et al., 2024; Yang et al., 2024) and adaptive approaches (Karanam et al., 2022; Killamsetty et al., 2022). In this work, we focus on one-shot subset selection, which identifies subsets in single pass, offer- ing computational advantages over adaptive methods that require iterative selection during model training.
National Engineering Research Center for Multimedia Soft- ware, Institute of Artificial Intelligence, School Computer Sci- ence, Wuhan University, China "Hubei Key Laboratory of Multi- media and Network Communication Engineering 'The University of Tokyo, Japan 'National Institute of Informatics, Japan SScooo of Computer Science and Technology, Wuhan University of Sci- ence and Technology, Wuhan 430065 China. Correspondence to: Zheng Wang <wangzwhu @ wewhu.edu.o....
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Traditional one-shot subset selection methods typically rely

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
on pre-trained model an information extractor (IB) to derive data characteristies such features, gradients, or uncertainty scores. These characteristics are then used to identify the most representative subset. While numerous strategies-suuh as feature-based (Agarwal et al., 2020; Sener & Savarese, 2017), uncertainty-boet 2019; Wu et al., 2024), and gradient matching- -baed ap- proaches (Mirzasoleiman et al.. 2020) have been proposed, these methods fundamentally depend on pre-trained mod- els obtained by training on the full dataset of the target task, as shown in Figure 1 (a). This inherently introduces significant dataset dependency, which limits their applica- bility, particularly in large-scale data scenarios. Efforts to reduce this dependency, such as employing lightweight proxy models (Coleman et al., 2019) or minimizing pre- training epochs (Guo al., 2022), only partially mitigate the computational burden without fundamentally addressing the dataset dependency issue.
10 (Krizhevsky et al., 2009), CIFAR-1ON-worse (CIFAR-
10N) (Wei et al., 2022), CIFAR-1O-imbalane (CIFAR-
10I) (Cui et al., 2019), Oxford-IIIT Pet (Pet) (Parkhi et al.,
2012)) and Oxford -II' Pet-N (Pet-N), revealed surpris- ing findings: (1) FMs consistently outperform traditional IEs on both clean and noisy fine-grained datasets; and (2) FMs demonstrate limited advantages for subset selection on coarse- coarse-grained datasets with noisy labels.
(Coleman et al.,
While FMs are well-suited for fine ine-grained datasets, the op- timal choice of FM as feature extractor for subset selection remains an open question. Moreover, existing feature-based methods fail to comprehensively analyze feature distribu- tions from both intra- and inter-class perspectives, result- ing in suboptimal selection performance. To address these limitations, we introduce a novel subset selection pipeline that leverages multiple FMs with unknown selection per- formance to enhance fine-grained dataset selection. Our proposed RAM-APL method integrates diverse FMs (i.e., DINOv2 and CLIP) and quantifies data importance through a systematic analysis of feature distributions across both intra- and inter-class levels, achieving state-of-the- per- formance on three fine-grained image datasets.
Recent advancements in foundation models (FMs), such as pre-trained vision models (Caron et al., 2021; Oquab et al., 2023) and languag models (Radford et al., 2021; Zhai et al., 2023; Sun et al., 2023), offer a promising al- ternative. A natural alternative subset-based methods is fine-tuning or adapting FMs to the target dataset (Ding et al., 2023). While these approaches leverage pre-trained knowl- edge, they still require full-dataset access during fine-tuning, which undermines the computational efficiency that subset selection seeks to achieve. Moreover, these methods often face challenges such as overfitting on noisy datasets (Feng et al., 2024) and scalability issues on large datasets. In contrast, subset-based methods decouple the data selection process from task-specific training, enabling efficient leam- ing without full-dataset reliance. With their robust general- ization capabilities, FMs can serve as direct alternatives to traditional IEs, enabling dataset-agnostic subset selection pipelines, as illustrated in Figure (b). Unlike traditional pipelines that rely on task-specific pre-training, FM-based pipelines eliminate the need for task-specific pre-training, making them well-suited for large and diverse datasets. De- spite their potential, the advantages of FM-based pipelines over traditional methods remain under-explored. While some studies (Xie et al., 2023; Killamsetty et al., 2023) have investigated this approach, prior work (Xie et al., 2023) has revealed that simply using FMs for subset selection does not consistently lead to superior performance. This highlights critical open questions: Can FMs truly replace task-specific IEs in subset selection? If so, under what conditions?
The contributions of our work are three-fold:
An in-depth study on the strengths and limitations of foundation models compared to traditional information extractors for subset selection reveals that foundation models consistently outperform traditional IEs on fine- grained datasets, whereas their advantage diminishes on coarse-grained datasets with noisy labels.
A novel subset selection pipeline employing multi- ple foundation models with unknown selection per- formance as IEs is proposed for fine- ane-gnaieed image datasets. RAM-APL, an effective subset selection method, is designed based on the novel pipeline.
Extensive experiments verify the superiority of RAM- APL on three fineggrinedd image datasets. Specifi- cally on the Maltech-UCSSI Birds-200-2011 dataset, RAM-API. achieves an average improvement of 6.4% in prediction accuracy over Random method across all sampling rates.
2. Related Works
Current one-shot subset selection methods typically fol- low traditional selection pipeline, which consists of an information extractor, measurer, and selector. Various measures have been proposed to leverage the information provided by the extractor to assess data importance, includ- ing feature- (Agarwal et al., 2020; Sener & Savarese, 2017), gradient- gradient-based (Kothawade et al., 2022; Killamsetty et al., 20)1a), training dynamic- (Toneva et al., 2018:
In this paper, we conduct extensive experiments to inves- tigate the strengths and limitations of using FMs as IEs for subset selection. Detailed experimental statistics and analysis can be found in Single Model Study section. Our experiments on subset selection using three kinds of models as IEs on five different types of image datasets, i.e., CIFAR-
2

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Swayamdipta et al., 2020; He et al., 2024; Zhang et al., 2024) and other weighting strategies (Zhou al., 2020; Coleman et al., 2019; Zheng et al., 2022). Regardless of the above methods, their extractors are usually trained to converge on the full training set of the target task, rendering the pre-trained extractor data-dependent and limiting the applicability of subset selection to new large-scale datasets. For example, TDDS (Zhang et al., 2024) required 90 epochs of extractor training ImageNet-1K 1 to gather training dy- namics, surpassing the 60 epochs needed for training the target model on the coreset. To solve this problem, Coleman et al. (Coleman et al., 2019) designed a small proxy model to perform data selection, achieving significantly faster pre- training. Guo ef al. (Guo et al., 2022) proposed to pre-train model for small number of epochs. However, they do not break free from dataset dependency. Recently, some stud- ies (Xie et al., 2023; Killamsetty et al., 2023) have explored using foundation models (FMs) as IEs for data selection, showing promise in addressing dataset dependency. Nev- ertheless, neither study has conclusively demonstrated that FMs outperform traditionally trained IEs. Specifically, (Xie et al., 2023) found that simply utilizing an FM does not guarantee superior data selection performance, raising ques- tions about the viability of FMs substitutes for traditional IEs. Our comprehensive investigation reveals that FMs uni- versally dominate traditional IEs on fine-grate -eeeggriied datasets (both clean and noisy), while their advantage diminishes on coarse- gaasee-rained datasets with noisy labels. Furthermore, the contribution of an FM to subset selection varies across datasets. To maximize the potential of FMs for fine- ane-grained subset selection, we propose strategically combining multi- ple FMs with complementary capabilities.
3. Preliminary: Subset Selection
In downstream tasks such as image classification and recognition, we consider a large-scale training set D [II....,AN] with a dataset size N, where each sample I, (zi, 96) consists of input data 2; and its corresoondigg class label yi E (1....,C) .J. specified budget p. subset selection is used to identify a subset S of 'D that contains the most informative data for the target downstream task. It is expected that the model @ trained on can perform on par with the model de trained on D. The performance of subset selection is eval- uated by the performance of model 8 the test set of the target downstream task. The subset S = [-1,..... Ia has a size M, where M < N, and the sampling rate for subset selection is defined as p = M/N. In the practical study, p is pre-specified, and the subset S is selected with the expectation of maximizing the target model's accuracy while adhering to the budget constraint.
In scenarios where there's s a
Subset selection relies on an Information Extractor (IE) to extract information from each sample, which is then used to assess the importance of the sample and select the most informative data. Traditionally, the IE is a model pre- trained on the full training set, which inherently introduces dataset dependency, limiting the applicability of this ap- proach across different datasets. To address this limitation, more flexible and generalizable approach is necessary, and it is therefore crucial to explore alternatives that reduce or eliminate dataset dependency.
4. Single-Model Study
Foundation Models (FMs) have recently emerged as a promising alternative to traditional information extractors (IEs) for subset selection. However, the advantages of FM- based selection over conventional methods remain largely unexplored. In this section, we investigate whether a single foundation model can effectively replace traditional IEs and address the following two key questions: Question 1: In which cases are foundation models most effective, and in which cases are they not? Question 2: Do all FMs per- form equally? Our extensive experiments reveal several key findings:
Since only features can be obtained from each FM, how to ef- fectively use the unaligned features extracted from multiple FMs to measure and select data is the key problem. Existing feature-based subset selection methods can be classified into two main categories: geometry-based methods (Welling, 2009; Sener & Savarese, 2017: Xia et al., 2023) and deci- sion Moundary-based methods (Ducoffe & Precioso, 2018; Margatina et al., 2021). For geometry- bassed methods, stud- ies (Welling, 2009; Sener & Savarese, 2017) selected sam- ples whose distributions are not close to each other in fea- ture space so that subsets do not have redundant information. These subsets usually make the model a good generaliza- tion. However, they treat samples whose distributions are not close to each other with equal importance, making sub- set selection for fine-graines datasets disregard inter-class distribution differences. Decision boundary-based meth- ods select data close to the decision boundary, which is a time-consuming and biased selection process that is not beneficial for model generalization. Taking the best of both types of methods, we propose the subset selection method RAM-APL for iineggraiee datasets.
FMs demonstrate limited advantages for subset selection on noisy, coarse-grained datasets.
Observation 1:
Observation 2: Conversely, FMs significantly and con- sistently outperform traditional IEs for subset selection on fine-grained datasets (both clean and noisy).
Observation 3: Different FMs perform differently as information extractors for subset selection.
Inspired by Observations 2 and 3, we propose a FM-based
3

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
algorithm for superior fine-grained subset selection, which is elaborated in Section 5. In subsequent paragraphs, we
DINOK2
CLIP
SigLIP
EWVOLIP
Porfer Purtorarcs of FM on Pot
96
17
Q
0
provide detailed explanations for these observations.
94
0
Aocuutty (3))
92
Experimental Setting. To assess the applicability of foun- dation models as information extractors (IEs), we conducted
90
88
subset selection experiments using a single model as the IE across five distinct image datasets: CIFAR- 10 (Krizhevsky et al., 2009), CIFAR-ION- (CIFAR-ION) (Wei al., 2022), CIFAR-IO-imballe 10-imbalance (CIFAR- 101) (Cui et al., 2019). Oxford-IIIT Pet (Pet) (Parkhi et al., 2012)) and Oxford-IIIT Pet with 20% symmetric label noise (Oxford-IIIT Pet-N, abbreviated as Pet-N). We apply three kinds of models for feature extraction in subset selection respectively. Three kinds of models are: (1) models pre-trained on the target training dataset for ten epochs (Guo al., 2022), referred to as model-TD. Once the target task changes, the model needs to be pre-trained again; (2) models pre-trained on Tiny- ImageNet (TIN) (Krizhevsky et al., 2012) for ten epochs, referred to as model-TIN. Tiny ImageNet is a larger classifi- cation dataset, models pre-trained on it possess a stronger representation ability compared to those pre-trained on tar- get datasets. Given this, we think that model-TIN has the po- tential to serve as alternative to traditional IEs without re- training when the target task changes; and (3) single foun- dation model (i.e. DINOv2, CLIP, SigLIP, or EVA-CLIP). To explore the impact of the above three kinds of models as IEs on selection algorithms, we implement four classical algorithms, i.e., MIN, K-center Greedy (KCG) (Sener & Savarese, 2017), Graph Cut (GC) (Iyer et al., 2021) and Moderate_DS (MDS) (Xia et al., 2023) over the extracted features. Besides, we use each selection algorithm to se- lect training samples with various sampling rates (i.e., 10%, 30%, and 50%). and train target models over the selected subsets. We provide the detailed experimental setup and results in Appendix A.
88
44
48
45
44
46
44
Performance MIN
Performance KCG Accumity (%)
Performanie of Accurasy
Performanos MDS Accurasy
Accurasy
Figure 2. Relationship between foundation model performance on the target task and subset selection performance using that FM as IE. Superior target task accuracy does not necessarily lead to better subset selection performance across different foundation models and selection methods.
out of 12 experimental setups, but the best result at each sampling rate was achieved using model-TIN as the IE. In the case of CIFAR-10I, the FM was optimal in 8 out of 12 experimental setups, but at a low sampling rate of 1, model-TD yielded the best results. Encouragingly, the single FM performed best in 9 out of 12 experimental setups on the Pet and Pet-N datasets and achieved the best results across all sampling rates. Thus, the FM presents a viable alternative to traditionally trained IEs for fine-grained image datasets. The Single-Model Study on more coarse- and fine- grained tasks shows the same conclusions, as summarized in Appendix A.2.
(Observation) Different FMs perform differently as in- formation extractors for subset selection, and the su- perior performance of FMs on downstream tasks does not guarantee better subset selection effects. Various FMs are available, including DINOv2, CLIP, SigLIP, and EVA-CLIP. If the method is be designed for fine-grained datasets according to the subset selection pipeline (b), an optimal FM needs to be identified first as the IE. An intu- itive idea is to identify the optimal FM by testing each on the target task, with the best-performing FM chosen as the IE. However, we observe that superior performance on the downstream task does not guarantee better subset selection. As shown in Figure 2, although EVA-CLIP has strong zero- shot classification on Pet, it is not optimal for any selection method. Furthermore, our experiments indicate that the optimal FM as the IE varies depending on factors such as target datasets, selection methods, and sampling rates. For instance, Figure 2 demonstrates that for selecting 50% of the Pet dataset, DINOv2 performs best as the IE for the MIN method, while CLIP excels for the KCG method, Additional analysis of optimal FMs across sampling rates is presented in Appendix A.3. Therefore, Pipeline (b) requires an ad- ditional step to identify the best FM to achieve the most effective performance across different scenarios. This un- doubtedly introduces an optimization detour, diverting focus from the primary goals of data measurement and selection.
We analyzed which of the three single models served as the most effective IE across four subset selection methods and three sampling rates. The frequency of each type of single model being the optimal IE under 12 settings on each dataset is presented in Figure (c). Surprisingly, we found that directly using features extracted from the FM for subset selection does not consistently outperform features extracted from traditional pre-trained models.
(Observation) FMs demonstrate limited advantages for subset selection on noisy, coarse-grained datasets. In contrast, FMs consistently outperform traditional IEs for subset selection on both clean and noisy fine-grained datasets. In the case of selecting CIFAR-1ON, the FM only emerged as the optimal IE out of 12 experimental setups. Conversely, the FM performed well on the other four datasets, especially on the Pet and Pet-N, For subset selection on 10, the FM was the optimal IE in 6
4

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
While FMs are well-suited for fine-grained datasets, the optimal choice of FM an feature extractor for FM-based subset selection remains an open question. Moreover, exist- ing feature-based methods fail to comprehensively analyze feature distributions from both intra- and inter-class perspec- tives, resulting in suboptimal selection performance. To ad- dress these limitations, we explore a novel subset selection pipeline that directly employs multiple FMs with unknown individual contributions IEs. Building on our pipeline, we propose the RAM-APL method, achieving state-of-the-art performance on multiple fine-grained datasets.
1th model. Since FMs may produce features of varying di- mensions, their representations are necessarily aligned.
RAnking Mean (RAM) RAM maps features extracted by different foundation models from their unaligned feature spaces into a distance ranking space (an aligned space), facilitating the evaluation of data importance based on intra- class distribution.
After acquiring the feature set F, we map the fea- tures extracted by each foundation model to a distance- ranking space. Specifically, given the feature set F [F .F...... FjJ from foundation model My, we define the central feature of class as the mean feature vector:
5. Proposed Method: RAM-APL
We are the first to investigate selection with multiple founda- tion models. In this section, we mainly propose a baseline method with multiple models as feature extractors. We in- troduce the problem formulation in Section 5.1. The subset selection method is then explained in detail in Section 5.2, which includes two metrics, namely ranking mean and ac- curacy of pleudo-class labels.
F
F
(I)
S
JES
where S represents the set of indices belonging to class c. The Euclidean distance between sample Fj and its class center F serves as a measure of representativennss, with smaller distances indicating higher representativeness (Xia et al., 2023):
5.1. Problem Formulation
Multiple foundation models M3 are used to extract in- formation of training data in our method, where M₁= = ,M MP} J. Foundation models can be directly used as feature extractors, but features of the same samples ex- tracted by different models are not aligned. Therefore, the two key challenges in our method design are effectively fus- ing features and accurately measuring sample importance based on the fused representations.
(F))
(F) F:
= Fj
Fill2,
(2)
where ||-112 denotes the Euclidean norm. Samples are ranked within each class according to their computed distances, producing ranked values R = [ri..... ris]] for model Mf. where rj E Zz and smaller values indicate closer distances. This process is repeated for all m foundation models, mapping unaligned features into unified distance- ranking space. The final ranking mean of class c is denoted as:
5.2. Method
The primary challenge in learning from fine-graines image datasets lies in their large intra-class differences and small inter-class differences. Existing subset selection methods either emphasize intra-class distribution while overlook- ing inter-class similarities or focus on boundary samples while neglecting samples from other distributions within the class. To address these limitations, we propose RAM-APL, selection method that quantifies data impor- tance by jointly considering both intra-class and inter-class distributions.
R₂ [ Fi.....F s|,
(3)
where F, m=|ST -i=11 rjj [0, 1] represents the normal- ized ranking mean for sample Ij. A smaller normalized rank- ing mean indicates greater alignment with class prototypes across foundation models. Visual analysis in Appendix B.4 further reveals that samples with lower normalized rank- ing means tend to exhibit more distinct target objects and simpler backgrounds.
Accuracy of Pseudo-class Labels (APL) APL maps fea- tures extracted by various foundation models from their un- aligned feature space into a pseudo-class confidence score based on the unified inter-class distance ranking.
Feature Extraction Given a fine-grained image dataset D, we extract features using multiple FMs Mf. where i E [1..... m). The extracted feature set denoted as F F F",, where F [F F& represents the feature representations of D obtained from the tt foun- dation model M Each feature vector Fj E IR for data sample I is defined as: F [3, -..0 "....... -1] E
After obtaining the feature set F, we assign pseudo-class labels to features extracted from each foundation model separately. Specifically, given the feature set F Fi...., F from foundation model Mi. we first com- pute the central features of all C' classes using Equation (I), collectively denoted as F = FF....., Fx ((c-n)l- Next, we
RRK, where K; represents the feature dimensionality of the
5

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
calculate the Euclidean distances between each sample Fj and all central features following Equation (2). These dis- tances are represented as: ((F)) = [d) (.......,, ................ 11ll where di,- represents the distance between Fj and the cen- tral feature F The pseudo-class label for sample 1 in the feature space of M is then assigned based on the nearest central feature, computed as:
6. Experiments
6.1. Experimental Settings
Datasets. We evaluate our method on three classical fine-grained image classification datasets: Oxford-IIIT Pet (Per) (Parkhi al., 2012), Food-101 (Bossard et al., 2014), and Caltech-UCSSD Birds-200-2011 (CUB-220-2011) (Wah et al., 2011). The Oxford-IIIT Pet comprises 7,349 images of 37 different breeds of cats and dogs. Food- 101 has 101 classes, each with 750 training images and 250 test im- ages. CUB-200 CUB-200-2011 1 consists of 11,788 images of 200 bird subcategories. Detailed dataset statistics are provided in Appedix A.1.
(4) If the assigned pseudo-class label matches the ground-truth label, i.e., 0/3 = y/y then the sample is considered correctly classified in the feature space of f. and we assign a score
5j = arg min D(F)).
of vp) 1. Otherwise, we set vp} 0.
Foundation Mudels as Feature Extractor. We adopt two FMs as feature extractors for fine-grained image datasets, i.e. CLIP-VITI14 (Radford et al., 2021) and DINOv2- VITs14 (Oquab et al., 2023). The visual encoder of CLIP- VITI14 is used to extract image features, while the final layer [CLS] token embedding output of DINOv2- serves as the feature representation. We emphasize that these FMs were not fine-tuned on the target datasets and were used solely as feature extractors for subset selection. The impact of varying the number of foundation models on selection performance is discussed in Section 6.4.
By repeating this process across all vn foundation models, we obtain set of classification scores for each sample across different feature spaces, The average pseudo-class label accuracy for sample I, is then computed as:
1
(5)
A lower value of f, indicates that sample I is more fre- quently misclassified across different feature spaces, sug- gesting higher degree of similarity to other classes and thus greater difficulty in distinguishing it within the feature distribution. Finally, we represent the overall pseudo-class label accuracy for the entire dataset D as:
Target Model Architecture & Training Parameters. For Pet and Food-101 datasets, we use the 18-layer residual net- work (ResNet-18) (He et al., 2016) as the model backbone, initializing it randomly for training. For the CUB-200-2011 dataset, we adopt ResNet-50 as the model backbone, ini- tialized with weights pre-trained on ImageNet (Deng et al., 2009). We follow the experimental setup from (Guo et al., 2022). Specifically, we use SGD as the optimizer with batch size 128, initial learning rate 0.1, Cosine decay scheduler, momentum 0.9, weight decay 5 x 10-1, and 200 training epochs. For data augmentation, we employ a random resized crop to 224 224 resolution, followed by random horizon- tal flipping on training images. The code of our study is available at: https//github..om//ijjjn Saape/////iiinaa/////AAaaaa Wan/R
P P x.l.
(6)
Subset Selection The importance of data samples in fine- grained learning is quantified through a linear combination
of RAnking Mean and the Accuracy of Pseudo-class Labels (RAM-APL), formulated as:
Score W1 R W2 .)
Here, W1 and W2 control the contributions of intra- class and inter-class distributions, respectively. Inspired by (Swayamdipta et al., 2020), which highlights that easier samples facilitate optimization, we prioritize high intra-class similarity at lower sampling rates p. gradually incorporat- ing harder samples as p increases. Thus, W1 and W2 are dependent on the sampling rate p. The weight functions are defined as:
Evaluation Metric. Prediction accuracy of well-trained target model on the test set is used as the evaluation metric.
Comparison Methods. Multiple subset selection methods act as baselines for comparison. Specifically, we compare with (1) Random, which uniformly selects samples as the subset; (2) Herding (Welling, 2009); (3) K-Center Greedy
Wi a (1 (1-a) a) + g 0.5)
(8)
(KCG) (Sener & Savarese, 2017); (4) Contextual Diversity
W2 = 1 1 W1
(CD) (Agarwal et al., 2020); (5) Margin (Coleman et al., 2019); (6) Forgetting (Toneva et al., 2018); (7) GraNd (Paul et al., 2021); (8) Cal (Margatina et al., 2021); (9) Glis- ter (Killamsetty et al., 2021b); (10) Graph Cut(GC) (lyer et al., 2021); (11) Moderate DS (MDS) (Xia et al., 2023); (12) MINimum distance (MIN), which selects samples with the minimum distance from the central feature of its class. Details of baselines are in Appendix B.I.
Samples with the smallest scores are selected into S up the predefined budget. The hyper-parameters a and /3 regulate the balance between intra-class and inter- class information across different sampling rates. Exper- imental results demonstrate that the best selection perfor- mance on fine-gras -reigrdi datasets is achieved using My= = (CLIP, DINOv2).
6

---

Foundation Model Insights and Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
60
-F- Random
80
70
MIN
Herding
270
times
40
Margin
CD
660
50
Forgeting
50
840 40
30
Glister
400
GC
30
20
ours
MDS
30
Accurscy
20
20
100
10
10
10
30
50
70
10
30
50
70
0
10
30
50
70
Sampling Rotest%)
Sampling Rates/%)
Sempling Rates(%)
(a) Oxford-II IIII Pet
(b) Food-101 1ood-101
(c) CUB-200-2011 200-2011
Figure 3. Comparison of our method with baselines on three classical fine-grained image datasets. Reported values correspond to mcan accuracy.
We implemented each selection method based on the one- shot subset selection pipeline using code in the DeepCore library'. The information extractors used in baselines (2)- (12) were obtained using the traditional method, i.e., training model with the same backbone the target model on the target training set for 10 epochs to ensure fair comparison.
Table I. Ablation study on the Pet dataset. Model-TD refers to model pre- trained on Pet for 10 epochs.
Sampling rates
Method
IE
1%
50%
70%
Model-TD
5.6:0..7
40.3+2.6
55.2+2.77
MIN
CLIP
5.6+00.2
45.9+
56.3+00.7
6.2. Comparison with Baselines
DINOV2
6,2:10.1
46.8+2.0
60.5
The results comparing the accuracy between the different subset selection methods on each fine-grained dataset are shown in Figure 3. Given each sampling rate, class-balanced sampling is performed. The experiments of each method on Pet were performed five times with different random seeds, while the experiments on Food- 101 and CUB-200-201 performed three times with different random seeds due to the high computational effort. We adopt 0.2 and 3 = for our method across all datasets.
RAM RAM-APL
5.9
17.1
56.5
CLIP+DINOv2
6.5.10.4
47.5
58.712.2
of each design the Pet dataset. Firstly, the RAM is designed primarily to effectively fuse the features extracted from multiple foundation models in terms of intra-class distribution, enabling the subset selection performance to not inferior to that of any individual foundation model. As shown in Table when using RAM fuse the features extracted from CLIP and DINOv2 and selecting the samples with the minimum ranking mean, the performance of "RAM" is better than that of " "MIN" with Model-TDD or CLII the IE at each sampling rate. This validates the effectiveness of the RAM strategy. By combining APL and RAM to assess data importance for subset selection, our method outperforms the "MIN" bascline with DINOv2 as the IE both I % and 50% sampling rates. These results highlight the effectiveness of the joint RAM-APL strategy in fine-graines subset selection. Further analysis in Appendix B.5 shows that RAM-APL selects more diverse samples, enhancing overall coverage of the feature space.
As shown in Figure 3, our method outperforms all baselines at each sampling rate. We compute the average performance gain of each method over Random across all sampling rates. On Pet, our method achieves a 3.7 3.74% average improvement, substantially outperforming the sub-optimal GC method, which shows 1.52% average improvement. On Food-101, 101, our gain reaches 4.44% compared to GC's 3.04%. On CUB- 200-2011, 1, our method shows 6.40% average improvement versus GC's 28%. Detailed performance and additional cross-architecture generalization results are provided Ap- pendix B.
6.3. Ablation Study
Our method mainly consists of two novel designs: two feature measure metrics for multiple foundation models (i.e., "RAM" and "API."). We evaluate the effectiveness
6.4. Analysis and Discussion
Parameter analysis. The hyper-parameters a and 3 are used to set the joint weights W, and W2 according to For- mula We study the impact of them in Figure 4, testing five different values for a and (3. In particular, we compared
7

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
.9
Table 2. Performance comparison our method using differ- ent numbers of foundation models as information extractors. Here, "D", "C", "S" and "E" represent DINOv2, CLIP, SigLIP. and EVA-CLIP, respectively.
Aounstt (eee of
58.5
58.5
58
57.5
57.5
57
IE
Sampling rates
56.5
56
56
D C S E
1%
10%
30%
50%
70%
(a. (F=1)
(a=0.2. 2, B)
5.910.33
15,411.1
31.612.3
47.71111
57.914.11
5.740.4
15.0++022
27.9+1.2
43.6+1.9
57.0+0.4
Figure 4. Parameter analysis when sampling 70% of the Pet. It shows that our method achieves the best performance when 0.2 and B= = 1. The grey dotted line indicates the selection method with score computed as Score = R + (1 p). i.e., direct assignment of W,- W2 1 without applying Formula 7.
6.6+0.3
14.1#1.0
28.8=1.1
43.9+1.7
55.1
5.4+0.3
15.0+0.6
30.2+2.5
44.4+2.3
1.
6.5:10.4
15.2
32.41299
47.5+19
58.71:2.2
5.94:0.3
16.24...1
31.443.2
45.0+
58.6:11.2
6.0+0.6
16.0+0.9
35.8+2.9
46.5+1188
54,9
6.41:02
15.110.4
29.811.6
4..9
56.212.7
the basic weight-setting strategy for fusion, i.e., the equal- weighted fusion strategy, where W1 = and W2 = 1. As illustrated in Figure 4, the best performance was achieved with a = 0.2 and B 1, outperforming the equal-weighted fusion strategy. When a = 0.2 and | = 1. the fusion weights (W1, W2) corresponding to I 1% 10%, 30%, 50%, and 70% sampling rates were (0.696, 0.304), (0.679, 0.321), (0.640, 0.360), (0.600, 0.400), (0.560, 0.44), respectively. As the sampling rate increases, W2 increases while W, re- mains greater than W2. This observation suggests that focus- ing more on inter-class feature distributions as the sampling rate increases helps to select better ine-grained subsets, but it is crucial to ensure that the intra-class assessment scores continue to dominate.
5.910.3
15.5+0.7
31.4
44.2#2.2
5.
6.7.10.4
16.2.1.6
34.7.0.3
45.710..8
56.6.12.44
6.240.8
15.6+0.5
33.2+1.4
48.3+1111
57.6+0.1
6.0:0.4
17.5+1
35.21188
4..911.5
55.612.1
6.110.33
16.810.6
34.412.1
47.0.2.0
55.111.6
6.110.2
16.110.3
33,911.4
46.811.5
55.110.5
6.5+0.2 16.8+1.1 34.0+2.7
46.3++0.5 56.9+1.1
Table 3. Comparison of feature fusion strategies.
Fusion strategy
Sampling rates
1%
10%
30%
50%
70%
Concatenate 5.9:0.4
16.3:0.4
31.7
47.7=3.0 57,8+1.2
Ours
6.5
1..2+1.2 32.412.9
47.5=1.9 58.7+2.2
Performance impact of the number of different FMs used for IE. There exists a diverse set of FMs capable of extracting visual features, including DINOv2 (Oquab et al., 2023), CLIP (Radford et al., 2021), SigCLIP (Zhai et al., (Sun al., 2023). These models
subset selection baselines across three fine-grained datasets.
Feature fusion strategy. Features extracted from differ- ent foundation models often exhibit misalignment due to architectural and training discrepancies. In RAM-APL, a simple fusion baseline is to concatenate features from dif- ferent foundation models, referred to as "Concatenate." " As shown in Table 3, our proposed fusion strategy outperforms simple concatenation, particularly under higher sampling ratios, which are critical in real-world deployment scenarios.
2023) and differ in their architectures, training strategies, and training datasets, leading to distinct knowledge and representation capabilities (as demonstrated in Appendix B.6). This raises a natural question: Does incorporating more FMs as IEs enhance our method's performance?
To explore this, we evaluate different combinations of DINOv2- V-TTs1, CLIP-VITLI4, SigLIP-base-patchi 224, and EVA-CLIP-SB3 on the Pet dataset. As shown in Table 2, using multiple FMs yields better overall per- formance than any single model. DINOv2+CLIP achieves the best trade-off between efficiency and accuracy, while adding EVA-CLIP yields further overall gains when compu- tational resources permit. These findings support the benefit of multi-mode consensus in our framework.
7. Conclusion
This work is the first to explore in-depth foundation models as information extractors (IEs) for one-shot subset selection. Our analysis revealed surprising insights: FMs consistently outperform traditional IEs on fine-grained datasets, whereas their advantage diminishes noisy labels. Motivated by these findings. we developed the RAM-APL method, which integrates multiple FMs to enhance subset selection for fine-grat -reegga datasets. This pioneering integration paves the way for exploring vast frontiers of subset selection in the era of big data.
coarse-graine datasets with
In the main experiments, we adopt DINOv2 and CLIP as our default IE pair, which yields consistent improvements over
We use the hast----bae-pacch16-22 and EVA-CLIP-8B from HuggingFace (Wolf et al., 2019)
8

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Impact Statement
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
RAM-APL improves data efficiency in machine learning by leveraging multiple foundation models for one-shot sub- set selection, particularly benefiting fine-grann ane-grained classifica- tion scenarios. By reducing reliance large, fully labeled datasets, RAM-APL has the potential to lower the barriers to deploying high-performing models in low-resource or un- derrepresented domains. However, as with all data selection techniques, careful consideration is needed to avoid rein- foreing dataset biases or unintentionally omitting critical minority samples. Our method does not involve synthetic data generation human subjects and poses no direct ethi- cal risks.
Ding, N., Qin, Y., Yang, G., Wei, F., Yang. Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220- 235, 2023.
Ducoffe, M. and Precioso, F. Adversarial active learning for deep networks: margin based approach. arXiv preprint arXiv: / 882.09841, 2018.
Feng, C., Tzimiropoulos, G., and Patras, I. Clipcleaner: Cleaning noisy labels with clip. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 876-885, 2024.
Acknowledgment
We thank the anonymous reviewers for their valuable feed- back. This work partly supported by ISPS KAKENHI Grant Number JP23K24876, JST ASPIRE Program Grant Number JPMJAP2303 and the National Natural Science Foundation of China under Grant 62376201.
Guo, C.,, Zhao, B., and Bai, Y. Deepcore: A comprehen- sive library for coreset selection in deep learning. arXiv preprint arXiv:2204.088 08499, 2044992,, 2022.
Work was done during Zhijing Wan's internship at the Na- tional Institute of Informatics, Japan.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
References
Agarwal, S., Arora, H., Anand, S., and Arora, C. Contextual diversity for active learning. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 137-153. Springer, 2020.
He, M., Yang, S., Huang. T., and Zhao, B. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7713-7722, 2024.
Tyer, R., Khargoankar, N., Bilmes, J., and Asanani, H. Sub- modular combinatorial information measures with appli- cations in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101. mining discriminative components with random forests. In Computer vision-ECCV 2014: 13th European con- ference, zurich, Switzerland, September 6-12, 2014, pro- ceedings. part VI 13, pp. 446-461. Springer, 2014.
Jia, X., Du, J., Wei, H., Xue, R., Wang. Z., Zhu, H., and Chen, J. Balancing privacy and performance: many-in- one approach for image anonymization. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pp. 17608-17616, 2025.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 185009660, 2021.
Karanam, A., Killamsetty, K., Kokel, H., and Iyer, R. Orient: Submodular mutual information measures for data subset selection under distribution shift. Advances in Neural Information Processing Systems, 35:3 35:31796-3180S 2022.
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang. P., Leskovec, J., and Zaharia, M. Selec- tion via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 118006.18829. 1829, 2019.
Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data sub- set selection for efficient deep model training. Interna- tional Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongic, S. Class- balanced loss based on effective number of samples. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, pp. 926899289998 9207,927,2222 2019.
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
selection for efficient and robust learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8765 PMLR, 2021.
Killamsetty, K., Abhishek, G. S., Lnu, A., Ramakrishnan, G., Evfimievski, A., Popa, L., and Iyer, R. Automata: Gradient based data subset selection for compute-efficiens hyper-paramets tuning. Advances in Neural Information Processing Systems, 35:28721-28733, 2022.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:/70e 00489, 2017.
Killamsetty, K., Evfimievski, A. V., Pedapati, T., Kate, K., Popa, L., and Iyer, R. Milo: Model-agnostic subset se- lection framework for efficient model training and tuning. arXiv preprint arXiv:2301. 13287, 2023.
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.1538, 2023.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi, H., Smith, N. A., and Choi, Y. Dataset cartog- raphy: Mapping and diagnosing datasets with training dynamies. arXiv preprint ar09v:2009. 10795, 2020.
Kothawade, S., Kaushal, V. Ramakrishnan, G., Bilmes, J., and Iyer, R. Prism: rich class of parameterized submodular information measures for guided data sub- set selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10238-10246, 2022.
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. An empirical study of example forgetting during deep neural network learning, In International Conference Learning Representations, 2018
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Tech- nology, 2011.
Margatina, K., Vernikos, G., Barrault, L.,, and Aletras, N. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.0377 2021.
Wan, Z., Wang, Z., Chung, C., and Wang, Z. A survey of dataset refinement for problems in computer vision datasets. ACM computing surveys, 56(7):1-34, 2024a.
Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Wan, Z., Wang, Z., Wang, Y., Wang, Z., Zhu, H., and Satoh, S. Contributing dimension structure of deep feature for coreset selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 9080-9088, 2024b.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 204.0744499 193, 2023.
Wang. S., Xu, X., Chen, H., Jiang, K., and Wang. Z. The parables of the mustard seed and the yeast: Extremely low-budget, eigh-performance nighttime semantic seg- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7853-7861, 2025.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2012.
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: study using real- world human annotations. In International Conference Learning Representations, 2022. URL https:// openreview net forummiiii dww.iidddtwwwPPP...
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Welling, M. Herding dynamical weights to learn. Interna- tional Conference on Machine Learning, pp. 1121-1128. 2009.
Pearson, K. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,
559-572, 1901.
10

---

Foundation Model Insights and a Multi-Model Approach for Superior
One-shot Subset Selection
et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint 2019.
1. I.cquisition of the Information Extractor
II. Subset selection
Pre- Train
Training set
Information Extractor
Subset 5
Training sel
CNN
Subset selection
Wu, J., Xu, D., Liu, W., Zhou, J., Ong, Y., Hu, S., Zhu, H., and Wang, Z. Assess and guide: Multi-modal fake news detection via decision uncertainty. In Proceedings of the 1st ACM Multimedia Workshop on Multi-modal Misinformation Governance in the Era of Foundation Models, pp. 37-44,20 2024.
Pre-Train
Subset 5
Target Model
TinylmageNet
CNN
3
Single FM
III. Target model training
Figure 5. Framework of the Single-Model Study.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. LESS: Selecting influential data for targeted instruction tuning. In Proceedings of the s International Confer- ence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 54104-54132. PMLR, 21-27 Jul 2024.
A. More Details on the Single-Model Study
To investigate whether foundation models (FMs) can serve as alternatives to traditional information extractors (IEs), we conducted a comprehensive subset selection study us- ing a single model as the IE across nine image classifica- tion datasets, uncovering surprising insights. The findings were derived primarily based on results from five differ- ent types of datasets (i.e. CIFAR-10 (Krizhevsky al., 2009), CIFAR-ION-we imbalance (Cui al., 2019), Oxford-IIIT Pet (Parkhi et al., 2012) and Oxford-IIIT Pet with 20% symmetric label noise), with results from four additional datasets (i.e. CIFAR- 100 (Krizhevsky et al., 2009), Caltech-UCSD Birds-200 2011 (CUB-200-2011 (Wah et al., 2011), CIFAR- 10 with 20% symmetrie label noise, and Oxford-IIIT Pet with 40% symmetric label noise) serving
Xia, X., Liu, J., Yu, J., Shen, X., Han, B., and Liu, T. Mod- erate coreset: A universal method of data selection for real-world data-efficient deep learning. In International Conference on Learning Representations, pp. 1-20, 2023.
Xie, Y., Ding, M., Tomizuka, M., and Zhan, W. Towards free data selection with general- purpose models. Advances in Neural Information Processing Systems, 36:1309-1325, 2023.
(Wei et al., 2022), CIFAR-10-
Yang, S., Cao, Z., Guo, S., Zhang, R., Luo, P., Zhang, S., and Nie, L. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Proceedings of the 41st International Conference on Machine Learn- ing, volume 235 of Proceedings of Machine Learning Research, pp. 55948-55960 PMLR, 21-27 Jul 2024.
strengthened proof of the findings to ensure rigor. To adequately compare with tra- ditional IEs, we evaluated various FMs, including a vision foundation model (i.e., DINOV2 (Oquab et al., 2023)) and vision-and lnn-lag foundation models (i.e., CLIP (Rad- ford et al., 2021), SigLIP (Zhai et al., 2023) and EVA- CLIP (Sun et al., 2023)). Besides, we implement four clas- sical subset selection methods, i.e., MIN, K-center Greedy (KCG) (Sener & Savarese, 2017), Graph Cut (GC) (Iyer et al., 2021), and Moderate DS (MDS) (Xia et al., 2023) over the extracted features. Our study follows the frame- work shown in Figure 5. Details and results of the study are provided below.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.. Sig- moid loss for language image pre-training, 2023.
Zhang, X., Du, J., Li, Y., Xie, W., and Zhou, J. T. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26223-26232, 2024.
Zheng, H., Liu, R., Lai, F., and Prakash, A. Coverage- centric coreset selection for high pruning rates. arXiv preprint arXiv:2210. 15809, 2022.
A.1. Experimental Settings
Dataset Details As summarized in Table 4, We adopted nine datasets with different characteristics for the Single- Model Study:
Zheng, H., Liu, R., Lai, F., and Prakash, A. Coverage- centric coreset selection for high pruning rates. In In- ternational Conference on Learning Representations, forum?
2023. URL id Id=OWXvL6wC8Y1.
(1) CIFAR-10: Contains 60,000 images of 32 32 3 size in 10 classes, with 6,000 images per class. The dataset is split into 50,000 training images and 10,000 test images;
Zhou, T., Wang, S., and Bilmes, J. Curriculum learning by dynamic instance hardness. Advances in Neural Informa- tion Processing Systems, 33:8602-8613 -613, 2020.
(2) CIFAR-10 with 20% % symmetric label noise (CIFAR-
10N 0.225mmm: Adds 20% symmetric label noise to the orig- inal CIFAR-10 dataset;

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Table Dataset statisties. "Task" indicates whether the dataset was used in the Model Study (Single) or the based Subset Selection task (Multi).
Table 5. Characteristics of image datasets.
Datasets
Grained Level
Noise label
Imbalance
CIFAR-IR
Coarse
x
Dataset
Train
Test
Classes
Naise
Task
CIFAR-ION
Coarse
V
x
CIFAR-10
50,000
10,000
10
Single
CIFAR-101
Coarse
X
V
y5m
50,000
10,000
10
20%
Single
CIFAR-ION
50,000
10,000
40.21%
Single
CIFAR-100
Coarse
X
X
CIFAR-10I
20,431
10,000
10
Single
Oxford-IIIT Pet
Fine
X
CIFAR-100 100
50,000
10,000
100
Single
Oxford-IIIT Pet-N
Fine
Pet
3,680
3,669
37
Mungi
CUB-200-2011
Fine
x
Pet-N
3,680
3,669
20%
Single
Pet-N-0.4Sym
3,680
3.669
40%
Single
CUB-200 20B-200-2011
5.994
5,794
200
Mingie&Mult
Food-101
75,750
25,250
101
Multi
tractor was evaluated by the prediction accuracy of models trained on the selected subsets.
(3) LIFAR-ION-WOrE (CIFAR-ION): Maintains the same structure as CIFAR-IO but includes 40.21% human- annotated real-world noisy labels;
Subset Selection Methods. We employ multiple subset se- lection methods utilizing feature information for the Single- Model Study:
(4) CIFAR-10-imbalanee (CIFAR-10T): Constructed us- ing the method in (Cui et al., 2019) to create long-tailed distribution. The number of samples in class k was set to ng x n², where ng is the original count. The imbalance ratio p is defined as p ming [mx] In our experiments, we set p 10. The training sample counts for classes 0 through 9 are 5,000, 3,871, 2,997, 2,320, 1,796, 1,391, 1,077, 834, 645, and 500, respectively. The original test set is used for evaluation;
(1) MINimum distance (MIN): Selects samples closest to the central feature of their class;
(2) K-center Greedy (KCG) (Sener & Savarese, 2017): Selects a budget-sized subset S to minimize the largest distance between any sample in D and its closest sample in S:
(3) Moderate DS (MDS) (Xia et al., 2023): Selects sam- ples with scores near the median of all samples, where scores represent distances to their class central feature;
(5) CIFAR-100: Contains 60,000 images of 32 x 32 x 3 size in 100 classes, with 500 training images and 100 test images per class;
(4) Graph Cut (GC) ayyer et al., 2021): Greedily selects samples that maximize the Graph Cut function constructed using their features.
(6) Oxford-IIIT Pet (Pet): Includes 7,349 images in 37 classes, with 3,680 training images and 3,669 test images. The dataset is imbalanced;
Target Model Architecture & Training Parameters. For datasets (1)-(8,, we use an 18-layer residual network (ResNet-18) (He et al., 2016) as the model backbone, ini- tializing it randomly for pre-training. For the CUB-200- 2011 dataset, we adopt ResNet-50, initialized with weights pre-trained on ImageNet. We use SGD as the optimizer with batch size 128, initial learning rate 0.1, Cosine decay scheduler, momentum 0.9, weight decay x 10-4, and 200 training epochs. For data augmentation, we apply the ran- dom crop with 4-pixel padding and random flipping on the 32 x 32 training images of datasets (1)-(5). For datasets (6)-(9), employ random resized crop to resolution of 224 x 224, followed by random horizontal flipping. Pre- diction accuracy is used as the evaluation metric, and the results of each method are averaged over three independent selections with different random seeds.
(7) Oxford-IIT Pet with 20% symmetric label noise (Pet-N): Adds 20% symmetric label noise to the original Pet dataset;
(8) Oxford-IIIT Pet with 40% symmetric label noise (Pet- N-0.4Sym): Adds 40% symmetric label noise to the original Pet dataset;
(9) Caltech-UCSD Birds-200-2011 (CUB-200-201)): Con- tains 11,788 images in 200 classes, with approximately 30 images per class for training.
Under symmetric noise, true lahels are randomly reassigned to other classes. Table 5 shows characteristics of each kind of dataset.
Single Foundation Model. Our experiments employed MINOV2-VITs14 CIIP-VITI14, SigLIP- -egsl--aae-p 16-22, and EVA- -LLI--B as the information extractor, respec- tively. Notably, these FMs were not fine-tuned on the target datasets and were solely used for feature extraction during subset selection. The selection performance of each ex-
A.2. When Are FMs Most Effective, and When Are They Not?
To answer this question, we conducted a comprehensive Single-Model Study and derived two key insights, primarily
12

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Model-TD
Model TIN
DINOY2
CUP
SiJUPP
EVA-CLI
Min for CIFAR1O
KCG for CIFAR1O
MDS for CIFAR10
BC for CIFARID
89
Accuracy
79
69
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
Sampling Rates
Sampling Rates
Sampling Rates
Sampling Rates
(a) Subset selection
CIFAR-10
Min for CIFARTON
KOG for CIFAR1ON
MDS CIFAR1ON
GC for CIFAR1ON
Accuracy (%)
60
40
20
0
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
Sampling Rates
Samplnng Rates
Sampling Rates
Samping Rates
(b) Subset selection on CIFAR- -ON--OON---- Worse (CIFAR-ION)
Min for CIFAR101
KOG for CIFAR1OI
MDS CIFARIOI
for CIFARIDI
BestDon
Accuracy (%)
70
60
50
10%
30%
80%
10%
30%
50%
10%
30%
50%
10%
30%
60%
Sampling Rates
Sampling Rates
Sampling Ratos
Sampling Rates
(c) Subset selection on CIFAR-I0-imbalan (CIFAR-10T)
Min Pet
KCG for Pet
MDS
GC
Dos to
Accuracy (3))
39
29
19
10
10%
30%
60%
10%
30%
50%
10%
30%
50%
10%
30%
50%
Sampiing
Sampling Rates
Samplnng
Sampling Rates
(d) Subset selection on Oxford IITT Pet (Pet)
Min for Pac-N
KGG for Pei-N
MDS for Pat-N
GC for Pat-N
Accurcc (%)
30
20
10
10%
30%
60%
10%
30%
50%
10%
30%
60%
10%
30%
50%
Sampling Rases
Sampling Rates
Sampling Rates
Sampling Rates
(e) Subset selection Oxford-IIIT Pet with 20% symmetric label noise
Figure 6. Single-mode, model study on five Target Datasets (TDs). Best viewed in color.
13

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
based on Figure 6. After analyzing which single model performs best as the IE when sampling datasets with dif- ferent rates and subset selection methods, we surprisingly found that directly using features extracted from the FM
minimize the distance between the subset center and the full training dataset center in the feature space;
(3) K-Center Greedy (KCG) (Sener & Savarese, 2017): Selects a budget-sized subset S to minimize the largest distance between any sample in D),S and its closest sample in S:
for subset selection does not always outperform features ex- tracted from traditional pre-trained models. Specifically, (1) FMs consistently outperform traditional IEs on both clean and noisy fine-grand greigra datasets; and (2) FMs demonstrate limited advantages for subset selection on coarse- grained datasets with noisy labels. Additionally, while Model-TIN shows potential as an alternative to traditional IEs on CIFAR- 10, its utility does not generalize to other scenarios.
(4) Contextual Diversity (CD) (Agarwal et al., 2020): Se- lects samples with diverse contexts based on predicted prob- abilities;
(5) Margin (Coleman et al., 2019): Selects samples with the smallest predictive probability difference between the most and second most likely labels;
These insights are further validated by additional results on clean datasets (Figure 7) and datasets with noisy labels (Figure 8), reinforcing the rigor of our findings. Specifically, as shown in Figure 7, the FM serves the optimal IE in 7 out of 12 setups for CIFAR-100, but the best result at 10% and 50% sampling rates are achieved using model-TI as
(6) Forgetting (Toneva et al., 2018): Selects samples with more forgetting events (where a sample changes from cor- rectly to incorrectly predicted more times), identifying them as hard samples;
the IE. In comparison, the FM is optimal in 9 out of 12 setups for
(7) GraNd (Paul et al., 2021): Selects samples according to their GraNd scores, which quantify their contribution to the decline of training loss during early epochs;
consistently achieving the best results across all sampling rates. As shown in Figure 8, while FMs are the optimal IE in 9 out of 12 setups on CUFAR-1ON-0.2Sy)
(8) Cal (Margatina al., 2021): Selects samples based on the KL divergence between the prediction of itself and that of its neighbors:
model-TD achieves the best result at the 10% sampling rate. In comparison, on Pet-N-0.4Sym, despite 40% symmetric label noise, FMs are the optimal IE in 9 out of 12 setups. More importantly, FMs achieve the best performance across all sampling rates, significantly outperforming results obtained with model-TD.
(9) Glister (Killamsetty et al., 2021b): Regards subset selec- tion as a bilevel optimization problem and selects samples that satisfy optimization constraints;
(10) Graph Cut(GC) (lyer et al., 2021): Greedily selects samples to maximize the Graph Cut function based on data gradients. Note that this implementation of GC uses gra- dients for data importance, unlike the version described in Section A. which uses features;
A.3. Do All FMs Perform Equally?
The unexpected findings motivated our design of a more effective subset selection method for fine- ine-grained image datasets, leveraging the strengths of FMs as IEs. Given the availability of multiple FMs, designing single-FM-basee subset selection pipeline requires identifying the optimal FM for the task. However, as illustrated in Figure 6(d), the performance of FMs as IEs varies significantly depending on the sampling rate and subset selection algorithm. For example, under the KCG method, SigLIP achieves the best performance at the 30% sampling rate, while CLIP opti- mal when sampling 50% of Pet. In contrast, when using the MIN method at 50% of Pet, DINOv2 emerges the best IE, whereas GC favors SigLIP. These observations highlight that FMs do not perform equally across different scenarios.
(11) Moderate DS (MDS) (Xia et al., 2023): Selects sam- ples with scores near the median of all samples, where scores represent distances to their class central feature;
(12) MINimum distance (MIN): Selects samples closest to the central feature of their class.
B.2. Performance of Baselines
In the main paper, we presented comparisons between the proposed method, RAM-APL, and several state-of- -f-th-rr methods on fine-grained datasets through illustrative analy- sis. Here, we provide the exact numerical results for refer- ence and verification in Table 8, Table 9, and Table 10. To evaluate the overall effectiveness of each method across all sampling rates, we introduce metric that computes the av- erage performance improvement of each method relative to the baseline Random method, where Random is the most ba- sic subset selection method. The results clearly demonstrate that RAM-APL achieves sate-of-thh-ane performance, de-
B. More Details on Multi-Model Selection
B.1. Comparison Methods
We compare with 12 subset selection methods:
(1) Random:
Uniformly selects samples to form the subset;
(2) Herding (Welling, 2009); Greedily selects samples to
14

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Model-TD
Model-TIN
DINOY2
SigUP
EVA-CLIP
Min for CIFAR100 S0089776
KCG CIFAR100
MDS for CIFAR100
GC for CIFAR10D
75
85
Accuracy (%)
66
45
35
25
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
Sumpling Rases
Sempling Rates
Sampling Rates
Sampling Rates
(a) Subset selection on CIFAR-100
Min for CUB-200-2011
KCG for CUB-200-2011
MDS for CUB-200-2011
GC for CUB-200-2011
Beat50ng
60
Accuracy (%)
Besp109
20
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30% 50% Sampling Rates
Sampling Rates
Sampling Rates
Sampling Rales
(b) Subset selection on CUB-200-2011
Figure 7. Single-model study on the clean, coarse-grained CIFAR- 100 dataset and the clean, fine-grained CUB-200-2011 dataset, which reinforces the insight that FMs are well-suited for (clean) fine- grained image datasets. Best viewed in color.
Mo
Modal-TIN
DINOK2
CLIP
SigLIP
EVA-CLIP
Min for CIFAR10044 25mm
KCG for CIFARICN-02255
MR5 for CIFARION- 25ym
CaC for 21FAR110-0225mm
100
80
Accuracy (%)
60
40
20
10%
30%
50%
10%
30% 50%
10% 30% 50%
10%
30%
50%
Sampling Rates
Sampling Rates
Sampling Rates
Sampling Rates
(a) Subset selection on CIFAR- 10 with 20% symmetric label noise
Pet-N-0.4Sym
KCG Pot-N-O.4Sym
MDS for Pet-N-0.44ym
Po.-5-0.SSy
Bawa
40
Accurapy (%)
20
DeestoNS
10
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
Sampling Rates
Sampling Rates
Sampling Rates
Sampling Rates
(b) Subset selection on Oxford-IIIT Pet with 40% symmetric label noise
Figure 8. Single-model study on the coarse-grained CIFAR-10 dataset with label noise and the fine-grained Oxford-IIIT Pet dataset with label noise, which reinforces the insight that FMs are well-suited for (noisy) fine-grained image datasets. Best viewed in color.
livering the highest average improvement over the Random method at all sampling rates.
15

---

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
RAmking Mean
Table Crost-architecture generalization of RAM-API.. The
Small
Large
get model architecture is MobileNet-V3 (MBV3).
Sampling rates
Method
IE Target Model
(Ragdol)
10%
30%
50%
Random
MBV3 MBV3
10.9+111
42.1+:3.6
61.6+119
Forgetiing
MBV3 - MBV3
13,3+08
4200+20
61.0+2.3
MBV3
12,4+1.7
40,4+0.2
MDS
MBV3 - MBV3
11.920.7
39.81200
62.1
Food101 (French toast)
MBV3
11,9+1.8
38,4+0.8
61.0+14
RAM-APL (Ours)
(CLIP+DINOWD>=MMBBV MBV3
13.610.3
45.7.00.9
62.31:1.4
Figure 9. Visualisation of samples with RAM metric.
1.0
Table 7. Average cosine distance of data pairs in the 70% subset of Pet.
CLI
9.76:06
2.66.05
-1.1.005
Class
0.8
Method
Whole
2
3
SIGLIP
MIN RAM RAM-APL
CLIP
0.1617
0.1795
0.1176
0.1509
0.1327
0.2680
9.7e-06
0.00012
-4e06
-066
CLIP-DINOw2
0.1695
0.1919
0.1250
0.1611
0.1392
0.2767
CLIR-DINOV2
0,1659
0,1986
0,1317
0.1597
0.1399
0,2787
EVACUIP
2.6e-05
0.00012
-1.6e-05
-0.4
DINDV2
B.3. Cross-architecture Generalization
-1.1e-05
-4e-06
-1.6e-05
To evaluate whether our selected subsets remain effective across model architectures beyond ResNet, we conducted additional experiments on the Pet dataset using MobileNet- V3 as the target model. The results, presented in the Table 6, compare RAM-APL against five strong baselines that main- tain identical architectures for their IEs and target models. RAM-APL consistently outperforms all baselines across dif- ferent sampling rates, indicating its strong cross-architecture generalization ability.
CLIP
SIGUP
EVACLIP
DINOv2
0.0
Figure 10. Cosine similarity matrix.
B.6. Relationships Between Features Extracted by Different FMs
To explore the relationships between features extracted by different foundation models (FMs), we employed the cosine similarity metric. Specifically, we extracted features for the Pet dataset using DINOv2- 11Ts14, CLIP- LITII1 SigLIP- base base-patch sate-patch16-222 16- and EVA-CLIP-SS CLIIP-SB. To ensure uniformity, we first determined the minimum dimension d among the feature matrices and applied the PCA algorithm (Pearson, 1901) to reduce all feature matrices to this dimension. Next, we computed the average cosine similarity between pairs of feature matrices and visualized the results in a heat map, as shown in Figure 10. The heat map reveals that the cosine similarity between features extracted by different founda- tion models is close to This indicates that the feature distributions are largely independent in high-dimensional space, showing diverse data representation by the models.
B.4. Visual Analysis of RAM Metric
As shown in Figure 9, we visualized samples from the Pet and Food101 datasets, arranging images within the same class from left to right based on their RAnking Mean (RAM) values, sorted from smallest to largest. The visualization reveals that images with smaller RAM values tend to exhibit clearer target objects and fewer background distractions. This indicates that RAM effectively distinguishes between easy and hard samples, highlighting its utility in subset selection.
B.5. Influence of RAM and APL
To evaluate the influence of RAM and APL on subset diver- sity, we quantified the average cosine distance between data pairs. The results, summarized in Table 7, show that both RAM and RAM-APL yield subsets with greater average cosine distance compared to the MIN baseline, indicating an increase in both intra-class and overall diversity. For brevity, the table presents the results for first 5 classes out of 37 classes from the Pet dataset, along with the metric for the whole subset.
16

---

Foundation Model Insights and Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Table Comparison with haselines on the Pet dataset. "IE" denotes the Information Extractor. "Traditional" stands for models pre-trained on the Pet dataset for ten epochs. The means and variances are reported across five independent experiments with different random seeds. The best performance values are highlighted bold, while second best results are marked blue.
Sampling rates
Average Improvement over Random
Methods
IE
1%
10%
30%
50%
70%
Random
5.4.10.6
12.2
2..4
42.6+13.0
55.0+:2.9
Herding
4.7+
10.8+0.7
22.5++2.7
38.2+11.7
53.2++2.7
-2.44
KCG
5.4
10.8+0.9
24.1=3.0
43.0=2.99
56.7+2.2
-0.32
CD
5.4=0.6
10.2#0.9
23.4=3.3
43.5=2.0
57.5+2.5
-0.32
Margin
5.0
11.6=0.4
25.3=1.5
42.0=2.6
14.8+115
-0.58
Forgetting
6.5
14.0#1.6
26.7=11.5
41.9=
54.1+3.0
+0.32
GraNd
Traditional
4.910.7
11.7:0.6
28.0+3..6
45.1
58.643.11
+1.34
Cal
5,920.6
13.920.8
27.1=1.4
41.5=112
54.3+3.7
+0.22
Glister
5.1+0.7
12.2+0.4
26.7+114
42.7=009
56.7+2.6
+0.36
GC
5.420.3
14.3+0.6
28.6=1.8
43.6=2.0
57.3+2.4
+1.52
MDS
4.7+00.4
11.9+1.6
24.5+2..
42.04:22
55.44:3.4
-0.62
MIN
5.620..7
14.6:0.55
26.421.6
40.3+2.6
55.242.7
+0.10
Ours
CLIP+DINOv2
6.5.1.44
15.2
32.4+2.9
47.5
58.7+2.2
+3.74
Table Comparison with baselines the Food-101 dataset. "Traditional" stands for models pre-trained the Food-101 dataset for ten epochs. The mcans and variances are reported across three independent experiments with different random seeds. The best performance values are highlighted in bold, while the second-bes best results marked in blue.
Sampling rates
Average Improvement over Random
Methods
IE
1%
10%
30%
50%
70%
Random
6.8
45.5++0.5
69.5++0.2
0.3
79.
Herding
6.6:10.5
42.6+1.2
64.6+10.2
74.3:10.2
78.7+00.3
-2.16
KCG
4.7
35.0.10.7
67.2.0.2
75.910.2
79.6+10.2
-3.04
CD
3.910.1
27.7-0.6
66.210.4
75.7+0.2
80.0+0.1
-4.82
Margin
5.0+0.77
36.8+2.7
68.411..
76.410.1
79.8
-2.24
Forgetting
9.5+0.5
53.4+0.5
71.7-0.1
77.3+0.1
79.910.1
+2.84
GraNd
Traditional
2.9+0.2
18.4+0.44
59.7+0.5
74.7+00.2
80.0+-0.1
-8.38
Cal
10.340.4
51.8-10.3
69.510.5
76.010.1
79.5+0.2
+1.90
Glister
6.910.3
38.8
67.410.4
75.910.4
80.1
-1.70
GC
10.840.4
54.4-10.3
71.340.4
76.810.2
79.5
+3.04
MDS
6.1+0.55
45.1++0.4
69.740.3
76.2+0.2
79.6+0.1
-0.18
MIN
9.4-00.2
52.110.6
70.710.7
76.740.1
79.8
+2.22
Ours
CLIP+DINOv2
12.
56.1-.01
7..
7..
81.2
+4.44
17

---

Foundation Model Insights and Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
Table 10. Comparison with baselines the CUB-200-201 dataset. "Traditional" stands for models pre-trained on the CUB-200-2011 dataset for ten epochs. The means and variances are reported across three independent experiments with different random seeds. The best performance values highlighted in bold, while the second-best results marked in bluc.
Sampling rates
Average Improvement over Random
Methods
IE
10%
30%
50%
70%
Random
21.6++0.9
27.5
59.
67.0
Herding
1.6
17.5
55.1
65.7.10.1
-5.75
KCG
18.612.6
17.7+4.3
58.010.9
66.210.3
-3.78
CD
16.4+1.0
26.0.18.3
56.7+00.7
66.2+0.2
-2.58
Margin
16.0+1..0
19.644.88
56.5+0.44
66.240.7
-4.33
Forgetting
17.3+0.7
22.7#3.1
59.1
68.1
-2.10
GraNd
Traditional
13.2+1..
20.6:19.1
55.0+0..6
66.210.8
-5.15
Cal
22.0=1.7
32.7#4.55
60.7+0.4
68.0+0..4
+1.95
Glister
15.5+0.4
21.8+
56.6+0.77
65.9+0.5
-3.95
GC
22.410.1
36.1
61.1
67.1
+2.78
MDS
19.1
7.6+2.9
60.0
67.
-5.40
MIN
24.1
32.849.4
61.0+10.7
67.7+0.1
+2.50
Ours
CLIP+DINOv2
25.110.4
42.613.1
63.5:10.3
70.0
+6.40
18


Reviews:
[{"review_id": "PkcuJiBwyu", "reviewer": "Reviewer_aGx6", "summary": "This paper investigates one-shot subset selection using Foundation Models (FMs) to reduce deep learning training costs by improving efficiency. Traditional Information Extractors (IEs) rely on models pre-trained on the target dataset, introducing dataset dependency. The paper addresses two key questions: (1) Can FM-based subset selection outperform traditional IE-based methods across diverse datasets? (2) Do all FMs perform equally well for subset selection? Experimental results show that FMs excel on fine-grained datasets but underperform on coarse-grained datasets with noisy labels. Based on these findings, the authors propose RAM-APL (RAnking Mean-Accuracy of Pseudo-class Labels), a novel method that leverages multiple FMs to enhance subset selection performance on fine-grained datasets. Extensive experiments validate the superiority of RAM-APL on three fine-grained datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback! We greatly appreciate your insightful questions, which have deepened our analysis of the findings and inspired further exploration for future work. Below, we address your questions in sequence.\n\n___\n\n**W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.**\n\nA1: We sincerely appreciate the reviewer's insightful comments regarding the theoretical understanding of foundation models (FMs) on noisy datasets. We have conducted extensive additional analyses to explain this phenomenon, with key findings visualized in Figures 1-8 (https://anonymous.4open.science/r/RAM-APL-DED5/README.md).\n\nOur empirical investigation reveals:\n\n**In coarse-grained datasets (CIFAR-10N-worse, Figures 1-4)**\n- FM-extracted features exhibit:\n  - Weak inter-class separation for visually similar categories (e.g., dog/cat in CIFAR-10N-worse);\n  - Substantial overlap between clean and noisy samples' feature distributions.\n\nThis explains FMs' limited effectiveness as information extractors for coarse-grained noisy data.\n\n**By contrast, in fine-grained datasets (Oxford-IIIT Pet with 40% symmetric label noise, Figures 5-8):**\n- FM-extracted features exhibit:\n  - Compact clustering of correctly-labeled samples;\n  - Strong inter-class separation for visually similar categories;\n  - Smaller overlap between clean and noisy samples in feature space.\n- Features from models trained on full noisy set show:\n  - Loose clustering of correctly-labeled samples;\n  - Significant overlap between clean and noisy samples in feature space.\n\nThis leads to the selection of more noise samples (visible in dark red points) and substantially weaker performance of traditional information extractors (IEs) compared to FMs.\n\n**Key Inference:**\n\nThe comparative analysis reveals that FMs serve as superior information extractors when their features demonstrate:\n- Tighter clustering of correctly-labeled samples;\n- Reduced overlap between clean and noisy distributions.\n\nWe will incorporate these analyses in the supplementary material to strengthen our empirical analysis and contributions.\n\n___\n\n**W2&Q2: Theoretical Analysis of Multi-FM Complementarity in RAM-APL.**\n\nA2: We thank the reviewer for this important question. RAM-APL's effectiveness in leveraging multiple FMs stems from two fundamental principles:  \n\n1.Feature Space Orthogonality:\n\nOur analysis reveals that different FMs learn nearly orthogonal feature representations (Figure 6 in the Suppl.), with:  \n\n$$\n\\text{cossim}⟨M_i(x), M_j(x)⟩ ≈ 0 \\quad \\forall i \\neq j\n$$\n\nThis orthogonality demonstrates that each FM (e.g., $ M_i, M_j $) captures distinct, complementary aspects of the data $x$.  \n\n2.Bias Reduction via Ensemble Consensus:\n\nThe ensemble mechanism could mitigate individual FM biases and preserve robust cross-model agreements. Table 2 in the manuscript demonstrates RAM-APL's performance gains when combining CLIP and DINOv2 versus individual FMs, confirming the benefits of multi-FM integration.  \n\n___\n\n**W3: Expanded Discussion of FM Literature.**\n\nA3: We sincerely appreciate the reviewer for this constructive suggestion. We will significantly strengthen our discussion of FM literature by incorporating CLIPCleaner (Chen et al., ACM MM 2024). The key insights are:\n\nBoth our work and CLIPCleaner leverage CLIP's zero-shot capabilities. Differently, CLIPCleaner, a single-FM method, focuses on noisy label cleaning via prediction probabilities. Our RAM-APL, a Multi-FM approach, specializes in subset selection for clean and noisy fine- rained data using visual deep features.\n\nWe'll discuss CLIPCleaner in Section 2 of the revised manuscript.\n\n___\n\n**Q3: Extensions to Few-shot Learning and Noisy Data.**\n\nA4: We sincerely appreciate the reviewer's insightful question regarding the broader applicability of RAM-APL. By leveraging the strong feature discriminability of foundation models and mitigating biases through ensemble consensus, RAM-APL shows a strong ability to distinguish noisy datasets. While our current work focuses on standard subset selection for fine-grained datasets, its theoretical framework and algorithmic design naturally extend to:\n\n- Noisy Few-shot Learning: \nEnhances robustness in few-shot scenarios by effectively identifying label-feature mismatches in small support sets.\n\n- Noisy Label Scenarios:\nParticularly effective for fine-grained noisy data (as demonstrated in our experiments). Moving forward, we plan to develop more effective denoising strategies tailored to such datasets within the RAM-APL framework.\n\nWe will include this extended analysis in the supplemental materials to better position RAM-APL's broader applicability.", "claims_and_evidence": "The main claims of the paper are supported by experimental data. For instance, the superiority of FMs as IEs on fine-grained datasets and the effectiveness of the RAM-APL method are validated through experiments. However, some conclusions (e.g., the limitations of FMs on coarse-grained datasets) lack deeper explanations.", "methods_and_evaluation_criteria": "The proposed RAM-APL method significantly improves subset selection performance on fine-grained datasets by leveraging the feature extraction capabilities of multiple FMs. The method is well-designed, and the evaluation criteria (e.g., prediction accuracy) are appropriate for the subset selection task. The experimental results demonstrate that RAM-APL outperforms SOTA methods on multiple datasets, validating its effectiveness.", "theoretical_claims": "The paper does not provide rigorous theoretical proofs but validates the effectiveness of FMs for subset selection through experiments. The experimental design is sound, and the results support the advantages of FMs on fine-grained datasets. However, the paper lacks a deeper theoretical analysis of the RAM-APL method, such as why it effectively leverages the complementary strengths of multiple FMs. The authors are encouraged to supplement the paper with relevant theoretical analysis to enhance the credibility of the method.", "experimental_designs_or_analyses": "The experimental design of Single-model Study is comprehensive, covering multiple datasets (e.g., CIFAR-10, CIFAR-10N, Oxford-IIIT Pet) and different FMs (e.g., DINOv2, CLIP). The results demonstrate that FMs perform well on fine-grained datasets but struggle on coarse-grained datasets with noisy labels. The RAM-APL method significantly improves performance on fine-grained datasets by combining the feature extraction capabilities of multiple FMs.\n\nA limitation of the experimental design is the lack of in-depth analysis of why FMs underperform on coarse-grained datasets. For example, is this due to the feature distribution or noise levels in the datasets?", "relation_to_broader_scientific_literature": "The paper clearly situates itself within the existing literature. Traditional subset selection methods rely on IEs pre-trained on the target dataset, which introduces dataset dependency. By introducing FMs, the paper proposes a dataset-agnostic subset selection method, expanding the scope of subset selection research. However, the paper does not sufficiently discuss the relationship with existing FM-related work, such as FM applications to few-shot learning or noisy datasets.", "essential_references_not_discussed": "The paper cites a wide range of related literature but omits some key works. For example, FM applications on noisy datasets (e.g., “CLIPCleaner: Cleaning Noisy Labels with CLIP” by Chen Feng et al., 2024) are highly relevant. The authors are encouraged to include relevant references and discuss the implications.", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe findings of the effectiveness of FMs for subset selection are interesting.\n2.\tThe proposed RAM-APL method significantly improves subset selection performance on fine-grained datasets.\n3.\tThe experimental design is comprehensive, covering multiple datasets and FM combinations.\n4.\tThe supplementary material provides detailed experimental explanations and results, enhancing the paper's credibility.\n\nWeaknesses:\n1.\tThe paper lacks a theoretical explanation for the underperformance of FMs on coarse-grained datasets with noisy labels.\n2.\tThe paper lacks a theoretical analysis of the RAM-APL method, explaining why it effectively leverages the complementary strengths of multiple FMs.\n3.\tThe discussion of related FM literature is insufficient.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1.\tWhy do FMs underperform on coarse-grained datasets with noisy labels? Is this related to feature distribution or noise levels in the datasets?\n2.\tCan the authors provide a theoretical analysis of the RAM-APL method, explaining why it effectively leverages the complementary strengths of multiple FMs?\n3.\tHave the authors considered applying the RAM-APL method to other tasks, such as few-shot learning or various noisy datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFCJZmw6v4", "reviewer": "Reviewer_g2WV", "summary": "To investigate whether foundation models (FMs) can truly replace task-specific information extractors (IEs) in subset selection, this paper examines the effectiveness of FMs as IEs for one-shot subset selection. Through extensive experiments across a set of image datasets, this paper identifies the strengths and limitations of FMs as IEs: they excel on fine-grained image datasets but underperform on coarse-grained datasets with noisy labels. To capitalize on the complementary strengths of multiple FMs and overcome limitations in existing feature-based selection methods, this paper introduces RAM-APL, which maps misaligned features from multiple FMs into a unified distance ranking space, considering intra-class and inter-class distributions. The selection methods are evaluated on three fine-grained classification datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedbacks! We address your questions in the following responses.\n\n___\n\n**W1&Q1: Cross-architecture generalization of RAM-APL.**\n\nA1: We sincerely appreciate the reviewer’s insightful question regarding the cross-architecture generalization of RAM-APL. We acknowledge the importance of evaluating whether our selected subsets remain effective across different model architectures beyond ResNet. \n\nTo address this concerns, we conducted additional experiments on the Oxford-IIIT Pets dataset (Pets) using MobileNet-V3 as the target model. The results, presented in the table below, compare RAM-APL against five strong baselines that maintain identical architectures for their information extractors (IE) and target models.\n\nMobileNet-V3 (MBV3)\n| Method\t| IE→Target Model | 10%\t| 30% | 50% |\n|----------------------|--------------------------|-------------|-------------|-------------|\n| Random\t| MBV3 → MBV3| 10.9±1.1 | 42.1±3.6 | 61.6±1.9 |\n| Forgetting\t| MBV3 → MBV3| 13.3±0.8 | 42.0±2.0 | 61.0±2.3 |\n| GC\t\t| MBV3 → MBV3| 12.4±1.7    | 40.4±0.2    | 61.3±1.5 |\n| MDS\t\t| MBV3 → MBV3| 11.9±0.7    | 39.8±2.0    | 62.1±3.3 |\n| MIN\t\t| MBV3 → MBV3| 11.9±1.8    | 38.4±0.8    | 61.0±1.4 |\n| **RAM-APL (Ours)** | (CLIP+DINOv2)→ MBV3 | **13.6±0.3** | **45.7±0.9** | **62.3±1.4** |\n\nWe observe that RAM-APL consistently outperforms all baselines across different sampling rates, indicating its strong cross-architecture generalization ability.\n\nYour suggestion has been highly valuable, inspiring us to further explore multi-model subset selection in broader cross-architecture settings in future work. The above experimental results and discussion will be included in the supplementary material.\n\n___\n\n**W2&Q2: A deeper discussion of why FMs struggle with coarse-grained datasets containing noisy labels but perform well on fine-grained image datasets.**\n\nA2: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's \"W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.\"\n\n___\n\n**W3&Q3: How do RAM and APL influence the distribution of representations in selected subsets?**\n\nA3: We sincerely appreciate the reviewer’s insightful question regarding the influence of RAM and APL strategies on the distribution of representations in the selected subsets. We acknowledge the importance of analyzing how these strategies shape the feature space and their impact on sample diversity and representativeness.\n\nTo address this concern, we conducted additional experiments and analyzed the feature distributions of different selection strategies. Specifically, we examined the average cosine distance between data pairs within the selected subsets, which provides insights into intra-class and overall diversity. The results are summarized in the table below:\n\nTable. Average cosine distance of data pairs in the subset\n| Method    | IE          | Class 0   | Class 1   | Class 2   | Class 3   | Class 4   | Whole subset |\n|-----------|-------------|-----------|-----------|-----------|-----------|-----------|--------------|\n| Min       | CLIP        | 0.1617    | 0.1795    | 0.1176    | 0.1509    | 0.1327    | 0.2680       |\n| RAM       | CLIP+DINOv2 | 0.1695    | 0.1919    | 0.1259    | 0.1611    | 0.1392    | 0.2767       |\n| RAM-APL   | CLIP+DINOv2 | 0.1659    | 0.1986    | 0.1317    | 0.1597    | 0.1399    | **0.2787**   |\n\nFrom these results, we observe that RAM and RAM-APL lead to a more diverse feature distribution in the selected subset compared to Min-based selection. The whole-subset average cosine distance is highest under RAM-APL (0.2787), indicating that it selects more diverse samples overall, improving coverage of the feature space. Moreover, the per-class distances suggest that RAM-APL encourages a balance between inter-class and intra-class diversity, with slightly higher values in harder-to-distinguish classes.\n\nFurthermore, the t-SNE visualizations in Figures 9-11  (https://anonymous.4open.science/r/RAM-APL-DED5/README.md) further confirm these findings. Compared to Min-based selection, which tends to concentrate samples within certain regions of the feature space, RAM and RAM-APL distribute samples more broadly across the space, ensuring better representational coverage. This suggests that our approach enhances model performance by capturing a more comprehensive representation of the dataset.\n\nYour suggestion has been highly valuable in strengthening our analysis. The above results and discussions will be included in the supplementary material to provide a clearer understanding of the impact of our proposed selection strategies.", "claims_and_evidence": "Yes. The claims in this paper are generally supported by clear experimental results, particularly in demonstrating that RAM-APL improves subset selection on fine-grained image datasets.", "methods_and_evaluation_criteria": "Yes. The methodology is well-motivated for subset selection, and the benchmark datasets are appropriate.", "theoretical_claims": "Yes. Since this paper is largely data-driven, there are no formal proofs for the theoretical claims, but the empirical justification is sound.", "experimental_designs_or_analyses": "Yes.  The experimental design is rigorous and well-structured:\n1. The evaluation considers three fine-grained image datasets (CUB-200-2011, Oxford-IIIT Pets, Food-101), making the conclusions well-supported in the targeted domain.\n2. A range of baselines is compared, including random selection, single-FM approaches (DINOv2, CLIP, et al.), and other subset selection methods.\n3. The ablation study analyzes hyperparameters (\\alpha, \\beta) and the effect of different FM combinations, showing that DINOv2 + CLIP provides the best results.\n\nHowever, a few concerns:\n1. This paper does not assess whether the selected subsets generalize across different model architectures. A key question is whether subsets selected by RAM-APL would maintain their effectiveness when applied to architectures beyond ResNet.\n2. This paper claims FMs are ineffective on coarse-grained datasets with noisy labels but does not analyze why in depth. A more detailed study (e.g., feature visualizations or error analysis) would help substantiate this finding.", "relation_to_broader_scientific_literature": "This paper relates to subset selection approaches but differs by leveraging multiple FMs to form a unified ranking space.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. This paper structure is reasonable, with each component of the proposed method clearly explained, making it easy to understand and implement.\n2. This paper introduces a well-motivated and innovative approach to subset selection. RAM-APL effectively harnesses the complementary advantages of multiple foundation models, addressing the variability in FM performance across datasets and selection methods. The empirical evaluations are thorough and provide strong evidence supporting the effectiveness of the proposed method. \n\nWeaknesses:\n1.This paper does not assess whether the selected subsets generalize across different model architectures. Would the subsets selected by RAM-APL retain their effectiveness when applied to architectures beyond ResNet?\n2. While this paper argues that FMs perform poorly on coarse-grained datasets with noisy labels, it lacks an in-depth analysis of the underlying reasons. Incorporating feature visualizations or error analysis could provide stronger empirical justification for this claim.\n3. The analysis is somewhat limited, primarily focusing on accuracy. Additional insights, such as diversity or difficulty analysis of the selected subsets, would enhance the evaluation.", "other_comments_or_suggestions": "See the Weakness part.", "questions_for_authors": "1. Would the subsets selected by RAM-APL retain their effectiveness when applied to architectures beyond ResNet?\n2.Why do FMs struggle with coarse-grained datasets containing noisy labels but perform well on fine-grained image datasets?\n3. How do RAM and APL strategies influence the distribution of representations in selected subsets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cx5qQ5ibKB", "reviewer": "Reviewer_35yt", "summary": "The paper makes comparisons between traditional information extractors (IEs) and a single foundation model (FM) on a series of datasets to explore scenarios in which a single FM would be advantageous as an IE. It reveals that a single FM performs poorly on coarse-grained image datasets with noisy labels and performs well on fine-grained image datasets with clean and noisy labels. The paper introduces a one-shot subset selection approach (called RAM-APL) tailed for fine-grained datasets, which ingeniously maps the misaligned features extracted by an ensemble of FMs into a unified distance ranking space, considering both intra-class distribution and inter-class distribution of samples. Experiments demonstrate the SOTA selection performance of RAM-APL on three fine-grained image datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedbacks! We address your questions in the following responses.\n\n___\n\n**W1: Evaluation on fine-grained image datasets with noisy labels.**\n\nA1: We sincerely appreciate the reviewer’s insightful suggestion. We acknowledge the importance of evaluating the effectiveness of our approach with other selection methods on fine-grained image datasets with noisy labels.\n\nTo address this concerns, we conducted additional experiments (as detailed in the tables below) on the Oxford-IIIT Pets dataset with 20% symmetric label noise and with 40% symmetric label noise. Subsets are sampled following the same experimental setup described in the manuscript. \n\nDataset: Oxford-IIIT Pets dataset with 20% symmetric label noise\n| Method     | IE | 1%       | 10%      | 30%       | 50%       | 70%       | 100%     |\n|------------|---------------|----------|----------|-----------|-----------|-----------|----------|\n| Random     | -             | 4.9±0.7  | 10.0±1.0 | 16.7±0.6  | 25.3±0.4  | 33.4±2.6  | 42.7±1.8 |\n| Harding    | Model-TD   | 5.0±0.1  | 8.1±1.8  | 15.3±1.8  | 20.7±0.5  | 33.1±0.4  | 42.7±1.8 |\n| KCG        | Model-TD   | 5.3±1.4  | 7.8±0.8  | 15.4±1.1  | 22.3±1.7  | 32.2±1.5  | 42.7±1.8 |\n| CD         | Model-TD   | 5.2±0.4  | 6.6±0.8  | 13.7±0.2  | 22.4±1.6  | 32.1±1.4  | 42.7±1.8 |\n| Margin     | Model-TD   | 4.7±0.1  | 9.0±0.7  | 16.3±0.5  | 23.9±0.6  | 33.5±1.2  | 42.7±1.8 |\n| Forgetting | Model-TD   | 5.9±0.6  | 11.5±0.9 | 18.7±1.3  | 29.5±0.9  | 36.9±0.4  | 42.7±1.8 |\n| GraNd      | Model-TD   | 4.3±0.2  | 7.8±0.8  | 15.6±0.7  | 22.9±1.5  | 32.4±2.1  | 42.7±1.8 |\n| Cal        | Model-TD   | 6.2±0.8  | 12.2±0.6 | 22.2±2.6  | 29.4±1.3  | 38.7±0.8  | 42.7±1.8 |\n| Glister    | Model-TD   | 4.8±0.2  | 10.5±1.1 | 17.2±1.0  | 26.1±2.6  | 34.7±2.2  | 42.7±1.8 |\n| GC         | Model-TD   | 5.2±0.6  | 12.8±1.5 | 20.3±1.3  | 27.0±0.6  | 32.9±0.9  | 42.7±1.8 |\n| MDS        | Model-TD   | 3.8±0.4  | 9.8±0.3  | 17.1±0.6  | 24.3±1.7  | 30.7±3.1  | 42.7±1.8 |\n| MIN        | Model-TD   | 5.6±0.2  | 11.6±0.4 | 19.8±1.4  | 28.0±2.2  | 35.3±2.5  | 42.7±1.8 |\n| **Ours**   | **CLIP+DINOv2** | **6.7±0.3** | **16.7±0.3** | **32.5±1.8** | **46.0±1.6** | **56.7±0.7** | 42.7±1.8 |\n\nDataset: Oxford-IIIT Pets dataset with 40% symmetric label noise\n| Method     | IE | 1%       | 10%      | 30%       | 50%       | 70%       | 100%     |\n|------------|---------------|----------|----------|-----------|-----------|-----------|----------|\n| Random     | - | 5.1±0.5  | 8.0±0.6  | 12.6±0.6  | 15.0±0.3  | 19.1±0.5  | 23.0±0.6 |\n| Harding    | Model-TD | 4.4±0.2  | 6.3±0.5  | 11.1±0.9  | 13.1±0.6  | 18.2±1.3  | 23.0±0.6 |\n| KCG        | Model-TD | 4.9±0.8  | 6.3±0.5  | 9.9±1.2   | 14.3±1.1  | 18.1±0.9  | 23.0±0.6 |\n| CD         | Model-TD | 4.8±0.8  | 6.3±0.5  | 10.3±0.8  | 14.0±0.3  | 17.7±1.2  | 23.0±0.6 |\n| Margin     | Model-TD | 4.1±0.3  | 7.0±0.8  | 11.1±0.9  | 14.3±0.9  | 19.0±0.8  | 23.0±0.6 |\n| Forgetting | Model-TD | 5.4±0.8  | 10.2±1.6 | 12.9±0.8  | 17.2±0.4  | 21.4±0.9  | 23.0±0.6 |\n| GraNd      | Model-TD | 4.4±0.9  | 6.7±1.0  | 10.2±0.5  | 14.5±1.6  | 18.8±1.2  | 23.0±0.6 |\n| Cal        | Model-TD | 5.4±0.3  | 10.6±1.1 | 14.9±1.1  | 18.9±1.0  | 22.2±1.2  | 23.0±0.6 |\n| Glister    | Model-TD | 5.2±0.3  | 7.6±1.1  | 12.4±0.8  | 18.3±0.8  | 21.8±1.6  | 23.0±0.6 |\n| GC         | Model-TD | 4.9±0.7  | 9.7±1.1  | 12.8±0.7  | 15.4±0.8  | 20.5±1.7  | 23.0±0.6 |\n| MDS        | Model-TD | 3.9±0.2  | 7.2±0.3  | 12.0±0.2  | 15.0±1.5  | 18.5±0.8  | 23.0±0.6 |\n| MIN        | Model-TD | 5.3±0.4  | 9.4±0.7  | 14.3±0.7  | 18.3±0.6  | 20.9±0.6  | 23.0±0.6 |\n| **Ours**   | **CLIP+DINOv2** | **6.1±0.3** | **15.0±1.2** | **30.4±1.7** | **44.8±0.1** | **42.6±0.8** | 23.0±0.6 |\n\n(\"IE\" means information extractor, \"Model-TD\" denotes the model trained on the full set for 10 epochs.)\n\nWe observe that RAM-APL consistently outperforms all baselines across different sampling rates on each noisy fine-grained  dataset, demonstrating its effectiveness.\n\nYour suggestion has been highly valuable. Through experimental analysis, we have identified the significant advantages of designing selection algorithms based on foundation models for noisy datasets, which motivates us to explore more effective foundation model-based denoising approaches in future work. The above experimental results and discussions will be included in the supplementary material.\n\n___\n\n**W2: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.**\n\nA2: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's \"W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.\"\n\n___\n\n**S1: Code release.**\n\nA3: We thank the reviewer for this suggestion. We will release the full implementation code in a public repository upon paper acceptance to ensure reproducibility.", "claims_and_evidence": "The claims regarding the foundation model insights and the effectiveness of RAM-APL in the paper are supported by strong empirical evidence. However, additional analysis on fine-grained datasets with noisy labels could further strengthen the claims.", "methods_and_evaluation_criteria": "The proposed approach is reasonable, but the evaluation is limited to fine-grained datasets. A broader range of datasets would help confirm RAM-APL’s robustness.", "theoretical_claims": "The paper does not present formal theoretical proof, as it is largely focused on empirical evaluation. The conceptual framework of the RAM-APL method is well-explained, and the reliance on empirical analysis is justified. No significant issues were found in the presentation of the algorithmic ideas.", "experimental_designs_or_analyses": "To explore scenarios in which a single FM would be advantageous as an IE, the paper employs systematic and rigorous experiments to analyze the impact of various factors (such as coarse-grained and fine-grained, labels that are clean or noisy, and balanced or unbalanced class distributions) and provides in-depth discussions of the results. Besides, the experimental design is robust in validating the effectiveness of RAM-APL, and the chosen datasets (Oxford-IIIT Pet, Food-101, and CUB-200-2011) are well suited for evaluating subset selection methods in the context of fine-grained image classification. The paper compares RAM-APL against multiple baseline methods, showing clear improvements. \n\nHowever, it would be beneficial to include additional experiments to explicitly examine the performance of subset selection methods on fine-grained datasets with noisy labels. The paper highlights the strengths of FMs as IEs on both fine-grained datasets with clean and noisy labels, so more explicit comparisons between RAM-APL and other methods on fine-grained datasets with noisy labels would further strengthen the effectiveness of RAM-APL.", "relation_to_broader_scientific_literature": "The paper effectively situates itself within the subset selection literature, particularly focusing on feature-based subset selection. The use of FMs for subset selection is both relevant and timely, especially as research increasingly relies on large pre-trained models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The paper is well-written and clear and presents a novel contribution to subset selection in fine-grained image classification.\n- The paper systematically compares FMs and traditional IEs through rigorous experiments, offering practical insights into their relative performance across diverse image datasets.\n- The paper is highly original in combining multiple FMs for subset selection, a novel approach that significantly improves performance in fine-grained datasets.\n- The paper provides comprehensive empirical results, strengthening its practical impact.\n\nWeaknesses\n1. The paper provides convincing results on three classical image fine-grained datasets. However, the paper does not compare the performance of RAM-APL with other methods on fine-grained image datasets with noisy labels. Experiments on fine-grained image datasets with noisy labels are important to further demonstrate the effectiveness of RAM-APL.\n2. The paper lacks a deeper discussion of why FM as IE performs well on fine-grained datasets with noisy labels and poorly on coarse-grained datasets with noisy labels. For example, which specific types of images or classes does FM as IE perform better on, and which classes does it not perform well on? Understanding these nuances would help in understanding the advantages of FM as an IE and would help in adapting RAM-APL to other domains.", "other_comments_or_suggestions": "- Consider including a code release or a link to a public repository for reproducibility purposes.", "questions_for_authors": "Please see [Weaknesses] 1-2 in the Other Strengths And Weaknesses. If authors address them, reviewer would like to change the rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "DD9lWQT0xb", "reviewer": "Reviewer_g7Wk", "summary": "This paper explores the use of foundation models (FMs) for one-shot subset selection, focusing on fine-grained image datasets. The authors find that FMs outperform traditional information extractors (IEs) in fine-grained tasks but struggle with noisy, coarse-grained datasets. To address this, they propose RAM-APL, a multi-FM framework that combines intra-class (via Ranking Mean, RAM) and inter-class (via Accuracy of Pseudo-class Labels, APL) feature analysis for improved subset selection. The method is evaluated on fine-grained datasets like Oxford-IIIT Pet and CUB-200-2011, demonstrating its effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedbacks! We address your questions in the following responses.\n\n___\n\n**W1: A detailed analysis of why FM as IE underperform on coarse-grained datasets with noisy labels.**\n\nA1: We sincerely appreciate the reviewer's insightful comments. Due to the character limit in the rebuttal, we kindly refer the reviewer to our response to Reviewer aGx6's \"W1&Q1: A deeper discussion of why FM as IE performs poorly on coarse-grained datasets with noisy labels.\"\n\n___\n\n**W2: Limited sampling rate evaluation in ablation studies.**\n\nA2: We sincerely appreciate the reviewer’s insightful feedback and acknowledge the importance of conducting ablation studies across diverse sampling rates.\n\nTo address this concerns, we have conducted additional Ablation Studies at 10% and 30% sampling rates. The revised Tables 1, 2, and 3 are provided below.\n\nTable 1. Ablation study based on Pet.\n| Method    | Information Extractor (IE) | 1%       | 10%      | 30%       | 50%       | 70%       |\n|-----------|---------------------------|----------|----------|-----------|-----------|-----------|\n| MIN |Model-TD| 5.6±0.7  | 14.6±0.5 | 26.4±1.6  | 40.3±2.6  | 55.2±2.7  |\n| MIN | CLIP | 5.6±0.2  | 15.4±1.0 | 29.3±2.4  | 45.9±1.8  | 56.3±0.7  |\n|  MIN | DINOv2 | 6.2±0.1 | **15.5±0.7** | 32.0±1.4 | 46.8±2.0 | **60.5±2.9** |\n| RAM | CLIP+DINOv2 | 5.9±0.3  | 15.1±0.5 | **33.1±2.3**  | 47.1±1.4  | 56.5±2.7  |\n| RAM-APL | CLIP+DINOv2 | **6.5±0.4**  | 15.2±1.2 | 32.4±2.9  | **47.5±1.9**  | 58.7±2.2  |\n\nAcross various sampling rates, RAM consistently outperforms MIN (CLIP as IE), while RAM-APL further improves performance, reaching levels comparable to DINOv2. Though RAM-APL (CLIP+DINOv2) demonstrates overall superior performance, its effectiveness at 70% sampling can be improved. In future work, we aim to enhance our method’s effectiveness at high sampling rates to further improve its practical utility.\n\nTable 2. Comparison of the performance of our method using different numbers of foundation models as information extractors. \n| DINOv2 | CLIP | SigLIP | EVA-CLIP | 1%       | 10%      | 30%       | 50%       | 70%       |\n|----------|------------|------------|-----------|----------|----------|-----------|-----------|-----------|\n| ✓        |            |            |           | 5.9±0.3  | 15.4±1.1 | 31.6±2.3  | 47.7±1.1  | 57.9±4.1  |\n|          | ✓          |            |           | 5.7±0.4  | 15.0±0.2 | 27.9±1.2  | 43.6±1.9  | 57.0±0.4  |\n|          |            | ✓          |           | 6.6±0.3  | 14.1±1.0 | 28.8±1.1  | 43.9±1.7  | 55.1±2.6  |\n|          |            |            | ✓         | 5.4±0.3  | 15.0±0.6 | 30.2±2.5  | 44.4±2.3  | 56.6±1.8  |\n| ✓        | ✓          |            |           | 6.5±0.4  | 15.2±1.2 | 32.4±2.9  | 47.5±1.9  | **58.7±2.2**  |\n| ✓        |            | ✓          |           | 5.9±0.3  | 16.2±0.1 | 31.4±3.2  | 45.0±1.3  | 58.6±1.2  |\n| ✓        |            |            | ✓         | 6.0±0.6  | 16.0±0.9 | **35.8±2.9**  | 46.5±1.8  | 54.9±3.5  | \n|          | ✓          | ✓          |           | 6.4±0.2  | 15.1±0.4 | 29.8±1.6  | 45.9±1.3  | 56.2±2.7  |\n|          | ✓          |            | ✓         | 5.9±0.3  | 15.5±0.7 | 31.4±1.7  | 44.2±2.2  | 55.9±1.8  |\n|          |            | ✓          | ✓         | **6.7±0.4**  | 16.2±0.6 | 34.7±0.3  | 45.7±0.8  | 56.6±2.4  |\n| ✓        | ✓          | ✓          |           | 6.2±0.8  | 15.6±0.5 | 33.2±1.4  | **48.3±1.1**  | 57.6±0.1  |\n| ✓        | ✓          |            | ✓         | 6.0±0.4  | **17.5±1.0** | 35.2±1.8 | 47.9±1.5  | 55.6±2.1  |\n| ✓        |            | ✓          | ✓         | 6.1±0.3  | 16.8±0.6 | 34.4±2.1  | 47.0±2.0  | 55.1±1.6  |\n|          | ✓          | ✓          | ✓         | 6.1±0.2  | 16.1±0.3 | 33.9±1.4  | 46.8±1.5  | 55.1±0.5  |\n| ✓        | ✓          | ✓          | ✓         | 6.5±0.2  | 16.8±1.1 | 34.0±2.7  | 46.3±0.5  | 56.9±1.1  |\n\nWe observe that leveraging multiple foundation models outperforms using a single model. The optimal balance of computational efficiency, memory usage, and performance is achieved with DINOv2 + CLIP. For the highest overall accuracy, DINOv2+CLIP+EVA-CLIP is recommended. These findings validate the benefits of multi-model selection, and the results will be included in the supplementary material.\n\nTabel 3. Comparison of feature fusion strategies.\n| Fusion Method  | 1%       | 10%      | 30%       | 50%       | 70%       |\n|----------------|----------|----------|-----------|-----------|-----------|\n| Concatenate    | 5.9±0.4  | 16.3±0.4 | 31.7±1.3  | 47.7±3.0  | 57.8±1.2  |\n| Ours| 6.5±0.4 | 15.2±1.2| 32.4±2.9 | 47.5±1.9 |58.7±2.2|\n\nWe observe that Our strategy outperforms Concatenate, especially at higher sampling rates, which are crucial for practical applications. To maximize the performance of multi-model method at high sampling rates, we adopt the Ours fusion strategy. The findings and new results will be included in the supplementary material.", "claims_and_evidence": "The authors demonstrate RAM-APL's superiority on fine-grained datasets like Oxford-IIIT Pet and CUB-200-2011, with ablation studies validating its components. However, the claim that FMs underperform on noisy, coarse-grained datasets could benefit from broader experiments. Overall, the evidence is solid but could be further reinforced.", "methods_and_evaluation_criteria": "The proposed RAM-APL method addresses fine-grained subset selection by leveraging multiple foundation models and combining intra-class and inter-class feature analysis, which appears reasonable for the task.  The evaluation criteria, including accuracy on datasets like Oxford-IIIT Pet and CUB-200-2011, are appropriate for assessing performance.  However, the evaluation could be expanded to include more diverse datasets, particularly those with noisy or coarse-grained characteristics, to better validate the method's robustness and generalizability.  Overall, the methods and evaluation criteria are suitable, though broader validation could strengthen the findings.", "theoretical_claims": "The paper does not present any explicit theoretical claims or proofs, which focuses on the empirical evaluation of the proposed RAM-APL method.", "experimental_designs_or_analyses": "The experimental design is well-structured, with appropriate benchmark datasets (e.g., Oxford-IIIT Pet, CUB-200-2011) and a comprehensive set of baseline methods for comparison.  The inclusion of ablation studies and parameter analyses is good, as they provide valuable insights into the contributions of different components of the proposed RAM-APL method.  However, the ablation experiments in Tables 1, 2, and 3 are limited to sampling rates of 1%, 50%, and 70%, which raises some concerns.  The 1% sampling rate seems unconventional and may not reflect practical scenarios, while the 70% rate sometimes underperforms, as seen in the results.  A more balanced evaluation, including the 10% and 30% sampling rates used in Figure 3, would provide a clearer understanding of the method's performance across a wider range of realistic settings.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to the broader literature on subset selection and foundation models.", "essential_references_not_discussed": "From my perspective, this article does not omit any essential references.", "other_strengths_and_weaknesses": "There are no other strengths and weaknesses. Here I summarize the strengths and weaknesses I answered in the previous parts.\n\nStrengths:\n\n1. Clarity of Methodology: The paper presents the proposed method, RAM-APL, in a straightforward and concise manner, making it easy to understand and follow.\n2. Comprehensive Experimental Setup: The experiments are well-designed and cover a range of datasets and scenarios, providing a thorough evaluation of the method's effectiveness.\n\nWeaknesses:\n\n1. Lack of Analysis on FM Performance in Coarse-Grained Tasks: While the paper highlights that foundation models (FMs) underperform in coarse-grained, noisy datasets, it does not provide a detailed analysis or explanation for this behavior. Additional experiments or theoretical insights could help clarify why FMs struggle in these scenarios.\n2. Limited Sampling Rate Evaluation in Ablation Studies: Tables 1, 2, and 3 only compare results at 1%, 50%, and 70% sampling rates. The 1% rate is unconventional and may not reflect practical use cases, while the 70% rate sometimes underperforms. It would be more informative to include results for 10% and 30% sampling rates, as shown in Figure 3, to provide a more balanced and realistic evaluation of the method's performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weaknesses of the ``Other Strengths And Weaknesses'' part, where there is a summary.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors explored using foundation models for the one-shot subset selection problem. Based on their findings, they further propose a method that leverages multiple foundation models to enhance subset selection by exploiting their complementary strengths. Extensive experimental results on fine-grained datasets, including Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011, are provided to show the effectiveness of the proposed method. 

This paper received consistent positive ratings after the rebuttal period. All four expert reviewers decided to accept this paper. Three of them confirmed that their concerns have been addressed during the author-reviewer discussions. Therefore, it is a clear acceptance.
Decision: Accept (oral)