Towards Free Data Selection with General-Purpose Models
Authors: Yichen Xie
Mingyu Ding
Masayoshi Tomizuka
Wei Zhan
Venue: NeurIPS 2023 poster (2023)
Keywords: data selection
unsupervised learning
TL;DR: We exploit a general-purpose pretrained model to freely select data samples for annotation with negligible extra time cost.

Abstract: A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. However, current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. In this paper, we challenge this status quo by designing a distinct data selection pipeline that utilizes existing general-purpose models to select data from various datasets with a single-pass inference without the need for additional training or supervision. A novel free data selection (FreeSel) method is proposed following this new pipeline. Specifically, we define semantic patterns extracted from inter-mediate features of the general-purpose model to capture subtle local information in each image. We then enable the selection of all data samples in a single pass through distance-based sampling at the fine-grained semantic pattern level. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods. Extensive experiments verify the effectiveness of FreeSel on various computer vision tasks.

Towards Free Data Selection with General-Purpose Models
Yichen Xie, Mingyu Ding', Masayoshi Tomizuka, Wei Zhan
UC Berkeley
(yichen_xio) xie, myding, tomizuka, wyhan))berkeeley..o edu
Abstract
A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility limited annotation budgets. However, current approaches, represented by active learning methods, typically follow cumbersome pipeline that iterates the conseminn model training and batch data selection repeatedly. In this paper, challenge this status quo by designing distinet data selection pipeline that utilizes existing general-purpos models to select data from various datasets with single-pass inference without the need for additional training or supervision. A novel free data selection (FreeSel) method is proposed following this new pipeline. Specifically, we define semantic patterns extracted from inter- mediate features of the general-purpose model to capture subtle local information each image. We then enable the selection of all data samples in a single pass through distance- based sampling at the fine- fine-grained semantic pattern level. FreeSel bypasses the heavy batch selection process, achieving significant improvement efficiency and being 530x faster than existing active learning methods. Extensive experiments verify the effectiveness of FreeSel on various computer vision tasks. Our code is available at
Introduction
Deep Neural Network (DNN) models have achieved remarkable progress various tasks, benefiting from abundant training samples and labels. Unfortunately, data labeling tends to be time-consuming and costly, especially for dense prediction tasks such as object detection and semantic segmentation, where experts may spend up to 90 minutes per image [33]. As such, effectively exploiting the limited annotation budget has become a long-standing problem in the advancement of computer vision.
Many methods have been proposed to identify the most suitable samples for annotation, where the mainstream follows the active learning [43, 45] or subset selection [42] pipelines. However, both kinds of methods rely on task-specific models. As the most popular data selection strategy, consuming and computationally expensive batch selection strategy [44], as shown in Fig. la. Specifically, a task-specific model is first trained using small initial set of labeled samples. Then, the model is utilized to select images within a specified batch budget size. These selected images are annotated and added to the labeled pool, after which the model is retrained or fine-tuned using the labeled samples. This iterative process is repeated multiple times for large unlabeled data pool. Since the selection of data is tightly coupled with the lusk-specific model, the entire pipeline needs to be restarted from scratch and repeated when working on different tasks or datasets. In many cases, even requires up to several days to select sufficient samples from a medium sized data pool (e.g., Core-Set [44] in Tab. 1).
active learning algorithms employ
" CCorresponiing author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Labelel Pool
Label
Deep Model Fraining
Uniabeled Pool
General Madil
Seletall semple mnn-hhl
Remove
from
Batch Data
Selection
Sugle Inferunc
Unlobeled
Rejeat for mulippo Herniions
(a) Active Learning Pipeline: It follows the batch- selection strategy with iterative training.
(b) Our Free Data Selection Pipeline: Samples are selected a single-pass without extra model training.
Figure 1: Comparisons between active learning pipeline and our proposed free selection pipeline.
In this paper, we challenge this status quo by introducing an efficient data selection pipeline that enables the selection of data within a single pass (as illustrated in Fig. 1b), therefore achieving comparable efficiency to random selection. We identify the efficiency bottleneck of data selection methods as the training of the task-specific model. Building upon insights from recent research on unsupervised learning [2, 64], we recognize that pretrained models [5, 63] possess the ability to encode the semantic information of images a fine-grained level. This observation inspires us to integrate pretrained models into the data selection process, thereby decoupling data selection from task-specific models and leveraging the inherent diversity captured by pretrained models. By leveraging publicly available pretrained models, pipeline incurs no additional training costs. To provide a concrete foundation for our design, we consider the following three guiding principles.
Generality: We strive for decoupling data selection from task-specifie models. It is desired that a general model works on the data selection of multiple tasks or datasets.
Efficiency: The batch selection setting of active learning (Fig. Ia) is known to be time- consuming due to its iterative nature. It is expected to be replaced with a single-pass model inference on unlabeled data pools.
Non-supervision: Annotators may not always respond in time, and the entire data selection progress may be delayed by frequent requests for labels. It is preferred that annotations are not required until the end of data selection.
In view of the above principles, we propose the first free data selection (FreeSel) method, to the best of knowledge, satisfying all the above principles simultaneously. FreeSel selects data samples based on the diversity of local features. The features are extracted by a publicly available pretrained vision transformer [12], which is generic enough to facilitate data selection for different networks, datasets, and tasks after pretraining on large-scale datasets [11] in an unsupervised manner, e,g., DINO [5]. We extract our newly defined semantic patterns by clustering the intermediate local features after an attention filter. The images are selected following distance-based sampling strategy at the level of semantic patterns. In pursuit of efficiency, this whole process is finished within a single-pas, model inference without any extra training. The data selection process is indeed unsupervised, which relieves the troubles of assigning responsive annotators.
As a result, our method pursues a free data selection using public pretrained models with a time efficiency close random selection. We conduct extensive experiments different tasks, datasets, and networks. When compared with existing active learning methods, our algorithm can achieve comparable performance with significantly advantageous efficiency.
Our contributions are three-fold. 1) We for the first time, introduce a new free data selection pipeline that adheres to three important principles of generality, efficiency, and non-supervision with negligible time costs. 2) We propose FreeSel, novel method following our proposed pipeline. It can fill the annotation budget in single pass based on the inherent diversity of semantic patterns captured by pretrained models. 3) Extensive experiments on image classification, object detection, and semantic segmentation demonstrate the effectiveness of our pipeline.
2
Related Work
Active Learning. Active learning aims to choose the most suitable samples for annotation so that model performance can be optimized with limited annotation budget. Most existing work in this

---

Table 1: Principles of Data Selection Methods: Task Model refers to the coupling between data selection and task-specific model. Batch Selection shows whether the method repeats the data selection batch multiple times. Multi-time Labeling denotes whether it requests ground-truth labels in the data selection process. Time estimates the approximate time to select 8000 images from PASCAL VOC datasets (Sec. 5.5).
Task Model
Batch Selection
Multi-time Labeling
Methods
Time
Core-Set [44]
42 hours
Learn-Loss [57]
/
+
CDAL [1]
/
label query
FreeSel (ours)
x
285 8 (~530x faster)
field [47, 44, 18, 59, 60] follows a pool-based protocol, selecting samples based on the ranking methods i.e.
of the whole dataset. There exist two mainstream sampling strategies for
uncertainty and diversity. Uncertainty inside the model prediction reflects the difficulty of data samples, estimated by different heuristics such probabilistic models [16, 13], entropy [26, 36), ensembles[3, 32], and loss function [57, 24]. Some other algorithms try to find the diverse subset which well represents the entire data pool. They measure the diversity with the Euclidean distance between global features [44], adversarial loss [47], or KL-divergence between local representations [1]. However, all these methods couple the data selection with task model and require repetitive model training in the batch selection pipeline, resulting in inefficiency. Differently, our proposed pipeline selects samples through single-pass model inference each unlabeled pool.
Subset Selection. As another category of data selection algorithms, subset selection methods often select all the required samples in single pass with the model trained on labeled seed set. The subset is usually selected based on some criterion of uncertainty [27], diversity [6, 4], or their combination [42]. In contrast, our proposed pipeline needs neither extra training on the target dataset nor knowledge about the label space.
Unsupervised Learning. Both contrastive methods [17, 20, 61, 53, 25, 48, 39] and generative models [52, 19, 49] have achieved great success in unsupervised representation learning. Contrastive methods discriminate different images without using any explicit categories. In contrast, generative methods directly predict masked visual information inside images. We exploit general pretrained model [5] to represent imput images for task-agnostic data selection. As result, we do not train models specific each task like the traditional active learning pipeline.
Data Selection with Pretrained Models. There are some attempts to combine unsupervised pretrain- ing and data selection. [56] selects data samples by the loss of pretext tasks, but requires different pretext tasks for different downstream tasks. [37] formulates active learning as an integer program- ming problem in the feature space, handling low-budget cases. [5]] and [54] select samples based on the diversity of global features, targeted for semi-supervised learning and model finetuning settings respectively. Active labeling proposed in [23] is the most similar our paper, but their method considers selective partial labeling in each sample instead of sample selection and is limited to 3D tasks with the same networks for pretraining and downstream tasks.
3 Preliminary Study: Off-the-Shelf Features for Data Selection
Active learning work [44, 1] often selects repre- sentative samples based on the features extracted by task-specific models trained separately for each task. A straightforward alternative is to use off- the-shelf features instead, which are extracted by ceneral-purposs models pretrained on a large- scale dataset. If it performs well, we trivially improve the efficiency by eliminating the training step on each dataset.
70
60
555
45
randems selection
unsupreviedd pretrained features
40
supeevised pretrained features
We conduct this preliminary study on the object detection task over the PASCAL VOC dataset [14].
4000
2000
Image Number
Figure 2: Core-Set over Off-the-Shelf Features
3

---

Per-Image peratioon
Last Patch Fratures
Festar
siteel
Didance Sampling
Selected Semantic Patters Poel
stglo-pan ovee dataset
[CL Tokom tlretion
Task Model
Tinage
Selected
Our method uses a general pretrained vision transformer to extract features from images. Semantic patterns are derived from the intermediate features. Afterwards, we perform distance-based sampling algorithm to select semantic patterns as well as the associated images. These selected images are labeled for downstream task model training-
Figure 3: Overview of Our Proposed FreeSel:
Consistent with our following experiments, we apply DeiT-S [50] for feature extraction in data selection. The model is pretrained in either supervised or unsupervised (with DINO framework [5]) manners on ImageNet [11]. For data selection, we implement the classical Core-Set algorithm [44] over the extracted global features, Le. the [CLS] token feature in the last layer. We use Core-Set with these features to select various numbers of training samples, and train object detection models (SSD-300 [34]) over the selected subsets.
Fig. 2 shows results in comparison with random selection. Unfortunately, we find that this naive com- bination of off-the-shelf features and Core-Set algorithms degrades the object detection performance, especially with relatively low sampling ratios. We consider two potential reasons for this failure: 1) Complex scenes are hard to represent globally. Images may contain multiple objects including some very small ones. difficult for global feature represent all useful details in the image. 2) K-Center selects corner cases, In the feature space. in order to cover all the data samples with small radius, the K K-Center algorithm of Core-Set tends to select all the corner cases.
The above two concerns motivate our design in Sec. 4. We represent each image with dense semantic patterns maintain useful local information. Images are sampled based on some probability related to the distance between local semantic patterns to relieve the harmful preference for corner cases.
4 Methodology
We detail our new data selection method FreeSel, formulated in Sec. 4.1. We define concept called semantic pattern in Sec. 4.2. Afterward, the sample selection strategy is explained in Sec. 4.3. An overview our approach is illustrated in Fig. 3.
4.1 Formulation
We aim to select a diverse subset from the unlabeled data pool for annotation, which covers as much discriminative regional information in the original pool as possible. The regional information inside an image / is reflected by the local features f' = = H, W are the height and width of the feature map. The r-th region feature f R& in the feature map mainly describes the r-th region of the image [62, 41]. The discriminative power of all regional features fr can be represented by countable knowledge points [31]. f is considered as a knowledge point w.r.t. pseudo-category e if it is similar enough to the corresponding direction vector Pe-
1, HW) } of a pretrained DNN.
To D..PMM (f (fflc))
p(c) -
> (ffee PeMr(flc) = Ca(n.)) exp(x. cos(f!.uu))
(1)
Ee Te
is pseudo-category describing some specific visual patterns, egg. an object part, which is repre- sented by vector μle in the feature space. Fe is the prior probability of pseudo-category c, is concentration parameter, and Ca(s.) is normalizing constant.
Inversely, given knowledge points inside an image 1, they can be clustered to estimate the K pseudo- categories inside the image as f j 1,2,....
........ ... K. We define the estimation semantic patterns in
We follow the name of networks in [50] in paper. DeiT-S is also called ViT-small in [S].

---

STOP
Figure 4: Visualization of Semantic Patterns: Every two images are considered as a group. Left: The filtered local features (dots) of each image grouped into semantic patterns (arrows). Gray features are eliminated in Eq. 2. Dimensions are reduced by PCA for visualization. Right: Regions inside images can be associated with corresponding local features and then semantic patterns.
Algorithm I: Semantic Pattern Extraction
Input:
Pretrained transformer y, unlabeled image pool I, attention ratio r, centroid number K Output: Semantic patterns ji = T
for I E do ca' -
last layer [CLS] and patch token attention patch-wise features.
Sort eat, pa. fL 1,2,..... HW in the decreasing order of attention ca..
F(F'= ( /* filter important regions based on the sorted attention (Eq. 2).
E/= caj T Σ}= ca
I Derive local similarity pii with Eq.
/* ignore attention between faraway regions.
pa ppa,lks11,.....
/* only consider filtered regions.
[ 1* divide t regions into K clusters with spectral clustering algorithm.
11=1 SpectralClusterppit K)
j = |Cl Ereo, ft. j j=1,2..., K
calculate the representation each semantic pattern.
Sec. 4.2. To ensure the diversity of our selection, algorithm desires to find subset of images Sz in Sec. 4.3, whose semantic patterns Ules, (2g J=1 can be representative in the unlabeled pool.
4.2 Per-Image Semantic Patterns Extraction
To estimate the pseudo-categories, we define a novel notion called semantic patterns, which are extracted from each image separately. Given a pretrained vision transformer [12], we consider its last layer features for image 7 as f [ where each patch corresponds to region r.
According Eq. 1, only few regional features may be considered meaningful knowledge points, while other regions are useless or even distracting. However, it is non-trivial to distill these knowledge points without any information about the pseudo-categories. To this end, resort the [CLS] token self-attention map of the transformer, which serves as a natural filter for regional importance even without the supervision of category information [5].
Attention Filter. For image I, the last layer [CLS] token attention map (average of multi heads) is denoted as ca' = {cof E | 1, 2,.. HW), }, > caf = 1. We can filter the important regional features that jointly represent the most useful information in the entire image with Eq.
1+1
F(r²) = ( = 1,2, I, Σ <T<
Σ ca
(2)
j=1
where regions 1,,,..., HW are sorted in the decreasing order of ca., and T (0. 1) is hyper-parameter, meaning the maintenance ratio of information represented by the filtered important features. The filtered features F(F!) are considered the knowledge points inside the images.
Feature Clustering. To estimate the feature vectors pseudo-categories, we perform clustering over the filtered t knowledge points inside each image separately. Since K-Means is unreli-
5

---

able in the high-din dimensioonn feature space (details in supplementary materials), we adopt spectral clustering instead. The self-attention map provides strong cues about the region-wise similarity inside each image. We denote the last layer attention map between patch tokens for image / as HWW pal = 1, Vi. It is more likely for nearby regions to interact with each other, so we only consider the attention between nearby patches [22].
pa' pals R 1,j,,,...... Ej=1
paly
d(i,j) S do d(i,j) > do
0
where j) is the Chebyshev distance between regions i.j in the feature map. We empirically set the threshold dg 2 in our experiments. Besides, only consider the regions after the filter in Eq. 2. In this case, we denote the new similarity matrix between patches as pa p(a) 2...,
With this above x similarity matrix, we utilize spectral clustering algorithms [38, 55] to divide the remaining regions after filtering (Eq. 2) into K clusters Cj.j 1, ..... K, each corresponding to a aseudo-categony where K is sypee-parammter The details of the spectral clustering algorithm are our supplementary materials. We average the corresponding feature fr, = 1,2,..., ......, of each region r belonging to each cluster Cj as follows.
Σ
J 1
|cyl
rEC,
where f. E F(ff),r Cg are local features of image I grouped into cluster j through spectral clustering. jul = {af)) represents semantic patterns inside the image I. Fig. 4 visualizes some examples of jny. The whole process of semantic pattern extraction is shown Alg.
4.3 Sample Selection with Semantic Patterns
Our main target of data selection is to make the distributions of selected samples diverse and representative in the level of local semantic patterns instead of the global feature level. This fine- grained strategy guarantees that our selected subset can cover rich local visual patterns represented by different pseudo-categories, which are crucial for detection and segmentation tasks.
To this end, we adopt distance-based sampling strategy at the semantic pattern level. The detailed algorithm is shown in Alg. 2. Given unlabeled image pool I, this process starts from randomly selecting an initial image Io i.e. selecting all semantic patterns a inside it. Then, we choose the next semantic pattern nz inside image / with probability in proportion to its squared distances from the nearest already selected semantic pattern (Eq. 5).
p(a)) min [D(a). p)] , IIj.j 1.2 K
(5)
AES
where Sx: the pool of all the already selected semantic patterns. When we choose a semantic pattern ūz. all the semantic patterns je' inside the image / that contains jú are put into the selected pool Sx. We use cosine distance for D(.,.) as analyzed in the supplementary materials. This process continues until enough images have been selected. The selection only requires semantic patterns constructed from intermediate features offline beforehand. Consequently, only single-pas model inference without any training or supervision is required in the entire data selection pipeline.
Experiments
We evaluate FreeSel on object detection (Sec. 5.2), semantic segmentation (Sec. 5.3), and image classification (Sec. 5.4). The results of FreeSel are averaged over three independent selections with different random seeds. Features are extracted by the same general pretrained model for all the tasks (Sec. 5.1). We make some analysis of our proposed pipeline and method in Sec. 5.5. Finally, we examine the roles of different modules inside FreeSel in Sec. 5.6. We refer readers to supplementary materials for more implementation details, results, and ablation studies.
5.1 General Model for Feature Extraction
We adopt DeiT-S [50] (path size 16x 16) pretrained with the unsupervised framework DINO [5] on ImageNet [11] to extract features for data selection. The same pretrained model is used for all tasks.

---

Algorithm 2: Distance-based Selection
Input: all semantic patterns ÿı' = for each image I, total annotation budget size Output: selected image pool Sz
Initialize Sr = {1o)
/* initialize with random image
2 Initialize Sx = (ng):j :j= = m.j==,..... ,,...,
/* initialize selected semantic pattern pool with all semantic patterns in To
repeat
Sample A/ with probability p(a)) & mingesx [D]]
/+ sample next semantic pattern ag with the distance-based probability.
SI = SI {1}
/a add image I containing sampled of to selected image pool
Sx Sx U {il, l...., K)
/+ add all semantic patterns in image I to selected semantic pattern pool
7
until |5 = b;
58
70
mAP
65
randam
544
Loara-Loss
Core-et
Entropy
50
CDAL-CS
Core-Set
SRAAL
CDAL-RL
VAAL
TOD
50
46
1009 2000 3000 4009 5008 6090 7089 X009 9000 10000
10
15
20
25
30
35
40
Image Number
Sampling Ratio (%)
Figure 6: Results on Semantic Segmentation: The mloU on 100% training data is 62.95. FreeSel can also fit other frameworks well, shown in supplementary materials. We emphasize that this pretrained DeiT-S model only applied to the data selection process. For the downstream tasks, still train the convolutional task models from scratch in accordance with prior work.
Figure 5: Results Object Detection: The mAP on 100% training data is 77.43.
5.2 Object Detection
Dataset and Task Model. We carry out experiments on PASCAL VOC [14]. In line with prior work 57], we combine the training and validation sets of PASCAL VOC 2007 and 2012 as the training data pool with 16, 551 images. The performance of task model is evaluated
PASCAL VOC 2007 test set using mAP metric. We follow previous work [57, 1] to train an SSD-300 model [34] with VGG-16 backbone [46] on the selected samples. It reaches 77.43 mAP with 100% training data.
Results and Comparison. We compare our performance with existing active learning methods (Fig. 5) for multiple sampling ratios. For faimess, we only include task-agnosti methods instead of those designed specifically for object detection [59, 8] which should naturally perform better. Results show that FreeSel outperforms most traditional pipeline methods and remains competitive with the best ones. Besides, all these previous methods require repetitive model training and batch selection on each target dataset separately, while FreeSel can efficiently select all samples in single pass. Sec. 5.6 also shows that FreeSel can outperform other alternative general- general-purpose purpose model basclines.
Semantic Segmentation
Dataset and Task Model. We use Cityscapes [9] dataset for semantic segmentation. This dataset composed of 3,475 frames with pixel-level annotation of 19 object classes. We report the result using mloU metric. We follow previous active learning research to apply DRN [58] model for this task. reaches 62.95 mloU with 100% training data.

---

90
Top-1 Accuracy
30
70
70
randem
68
60
((BOI+DeT-S)
Learn-Loss
PreeSel (MoCoV3-DeiT-S)
58-
ISAL
Frudd (DINO- DeiT-Bi (DINO-IeIT-S)
FreeSd
55
2000
4000
6000
8009
10000
1000 2000 3800 4008 5000 6460 7000 5000 9080 10000
Image Number
Image Number
Figure 7: Results Image Classification: The accuracy on 100% training data is 93.02%.
Figure 8: Effect of Pretraining Methods: Ex- periments are conducted on PASCAL VOC.
Results. We compare the performance FreeSel with traditional active learning methods Fig. Given the domain gap between the pretraining dataset (i.e. ImageNet) and Cityscapes, it is quite challenging to utilize the general pretrained model for data selection. However, with much higher efficiency, FreeSel still beats most traditional pipeline methods in performance.
5.4 Image Classification
Dataset and Task Model. We use CIFAR-IO [29] datasets and ResNet-18 [21] model in line with prior work [35, 57J. CIFAR-10 contains 60,000 images with size 32x32 (50,000 for training and 10,000 for test) belonging to 10 categories. We report the results using Top-I Accuracy metric. The model reaches 93.02% Top-1 Accuracy with 100% training data CIFAR-10.
Results. We demonstrate the results of data selection methods in Fig. Our performance is compared with traditional active learning methods as well. Since image classification focuses on global information, the advantage of semantic patterns cannot be fully demonstrated. However, with most sampling ratios, FreeSel still beats all its counterparts.
5.5 Analysis
Time Efficiency Analysis. Time efficiency of data selection is crucial for its practical use. Tab. 1 shows the comparison between FreeSel and other existing counterparts. The estimation conducted on PASCAL VOC to choose 8, 000 samples. We follow prior papers [1, 44, 57] to use SSD [34] as task model (same as Sec. 5.2). The time estimated on a single NVIDIA TITAN RTX GPU. Since FreeSel directly utilizes the publicly available pretrained model instead of training models separately for each dataset, only the feature extraction, semantic pattern construction, and data selection time should be considered, i.e. 285 seconds in total. In contrast, for other active learning methods, networks are trained repetitively on each dataset. We follow [57, 1] to set their initial set size and batch selection budget both 1k, so their model should be trained for 7 times
subsets of size 11 ~ 7k to select 8,000 samples. These previous methods have similar time efficiency, requiring about 42 hours in total. They also need to wait for the oracle for ground-truie labels after selecting each batch of data. Based on the above information, our method can be 530x faster than prior work.
Single-Pass Data Selection. Unlike prior active learning methods, FreeSel follows the new pipeline select all the data samples single pass. This allows for great practical use. Firstly, makes our method free of random initial set. For one thing, FreeSel can bring performance gain in the lowest sampling ratio. This is beneficial in practice when the annotation budget is extremely low. For another thing, FreeSel would not suffer from the imbalanced initial set. As discussed in [47], low-quality initial sets would hurt the performance of prior active learning work significantly. Secondly, FreeSel simplifies the active learning pipeline from the iterative model training-+bule data selection-<batch annotation ...... to a single-pass pagge--passaae data selection -data annotation, which saves notable efforts in the management, communication, and coordination of traditional sequential steps.
Introduction of Pretrained Model. Our proposed pipeline introduces a pretrained model (Fig. 1b) to satisfy the three principles of our new pipeline. Since the pretraining is not designed specifically for the data selection, directly using publicly available model would not lead extra time-cost expense. According to Sec. 3, it is non-trivial to improve the efficiency of active learning with

---

pretrained model. We further show that our great performance does not come from the pretrained model in Sec. 5.6.
Effect of Different Pretraining Algorithms. In this part, we pay attention to the effect of pretraining
on the final performance of FreeSel. In addition to DeïT-S [50] pretrained with DINO framework [5] in Sec. 5.1, we also adopt two alternative pretraining frameworks MoCoV3 [7] and iBOT [63] as well as larger DeiT-B model [50]. Those different pretrained models are applied the data selection on PASCAL VOC dataset [14]. Same as Sec. 5.2, we train an SSD-300 model [34] on the selected samples for the object detection task. Fig. 8 demonstrates that FreeSel with different pretrained models for data selection only has marginal differences in the performance of the downstream object detection task. This result verifies that FreeSel can widely fit different pretraining algorithms. The great performance of data selection comes from our carefully designed modules in FreeSel instead of the strong representative ability of some specific pretrained models.
5.6 Ablation Study
We delve into different parts of our method. Firstly, we analyze the contribution of each module inside FreeSel to the final performance. Then, the role of the pretrained DeiT-S model is also discussed.
Each Module Contribution. Starting from the failure case in Fig. 2. modules of FreeSel are added to it one by one. Tab. 2 demonstrates the step-by-step performance improvement. This experiment is conducted on PASCAL VOC in the same setting as Sec. 5.2. The following three modules analyzed. More quantitative analysis of syper-parametter is available in the supplementary materials.
Feature Extraction Manner: In Sec. 3, the global feature of [CLS] token is directly used. We replace it with the proposed semantic patterns defined in Eq. 4.
Attention Filter: We apply the attention filter in Eq. 2 to filter local features.
Selection Strategy: Apart from the distance- based sampling in Eq. 5, we consider the (FDS) w.r.f. semantic patterns, which is theoretically
alternative perthes-ddittnnc--aamppii justified in [44] as an approximation of K-Centers. It chooses the next semantic pattern farthest from the nearest selected one as jnj = arg maxp! minjesse d(i P))
The failure case of Core-Set off- the-shelf features is shown in the first line of Tab. 2. Then, we extract fea- tures by constructing semantic pat- terns (K 5) without applying the at- tention filter in the second line. It im- proves notably compared with the first line because the semantic patterns can represent useful local information im- portant for object detection. However, is only slightly better than random
Table Module Contribution: We discuss the contribution of each module inside FreeSel. SP means semantic pattern. Experiments are conducted on PASCAL VOC.
Image Number
Feature
Filter
Select
3k
7k
global SP
X
FDS
60.59
66.65
70.30
x
FDS
64.15
68.22
70.42
SP
7
0.5
FDS
64.45
68.49
71.35
SP
0.5
Prob.
65.66
69.24
71.79
random sampling
64.21
67.53
69.32
selection since the semantic patterns are dominated by local noisy information in this stage. We apply attention ratio r = 0.5 (Eq. 2) the third line of the table, and the performance gets improved again. Finally, the FDS selection strategy replaced by the distance-basee based probability sampling in Eq. 5. provides extra performance gain because it would select more representative samples with fewer corner cases.
Role of Pretrained Model. There is a concern that the performance gain of FreeSel comes from the great representation ability of the pretrained vision transformer for data selection instead of our designed method. About this, we conduct an ablation study on CIFAR-IO in the same setting as Sec. 5.4. We equip Core-Set [44] and Learn-Lose [57] with the same pretrained network for data selection, i.e. DeiT-S [50] pretrained with DINO [5]. During the data selection period, pretrained DeiT-S is finetuned supervisedly to select data samples with Core-Set and Learn-Loss algorithms. After selection, we still train ResNet-18 [21] over the selected samples from scratch. In Fig. 9, this pretrained DeiT-S damages the performance of Core-Set and Learn-Loss. A potential explanation comes from the differences in the representation spaces of pretrained DeiT' and from-scratch ResNet. The samples selected by DeiT-S with Core-Set and Learn-Loss algorithms may not be suitable for the

---

from-scratch training of the ResNet-18 model. This reflects that our performance gain does not come from the use of pretrained DeiT-S. Instead, the proposed FreeSel method plays an important role.
General-Purpose Model
Baselines. To further dis- entangle the roles of the general-purpose model and our designed FreeSel frame- work, we compare FreeSel with the following baselines which can also select a sub- set from the data pool using the general- purpose models. 1) K-Means: We perform the K-Means algorithm on the global features extracted by the pretrained DeïT-S model [50, 5J, choosing the sample closest to each cluster center. 2) Inconsistency: We select the most difficult samples based on the inconsistency of multiple-time model predictions. To measure the inconsistency, we perform data augmentations (RandAugment [10]) generate 10 different augmented copies for each image and calculate the average pairwise distances of global features between these copies extracted by the pretrained DeiT-S model [50, 5]. We select data samples by the order of inconsistency. 3) Entropy: We select the most ambiguous samples based on the classification uncertainty of the pretrained model. Since the classification score is required, the DeïT-S model [50] pretrained on ImageNet in a supervised manner and measure the uncertainty with the entropy of classification scores. We select data samples by the order of entropy. Experiments are conducted on object detection task in the same settings as Sec. 5.2. Tab. 3 shows that all the above baselines perform notably worse than FreeSel, especially with low sampling ratios. This reflects the importance of our proposed FreeSel algorithm. Trivial utilization of general-purpose model would not lead to great performance of data selection.
Table 3: Baselines Using General- Model: We compare FreeSel with other baselines using the general-purpose model. Experi- ments are conducted on PASCAL VOC object detection task.
Image Number
Methods
Pretrained Model
34
5k
7k
K-Means
DeïT-S (DINO)
64.85
68.05
71.50
Inconsistency Entropy
DeīT-S (DINO)
63,29
67.65
71.35
peneral-purpose
DeÎT-S (supervised)
56.33
66.03
69.72
FreeSel
DeïT-S (DINO)
65.66
69.24
71.79
adopt
Conclusion and Limitations
The main goal of this paper is to enable free data
selection pipeline by proposing novel pipeline with three key principles: generality, efficiency, and con-supervision. We verify its feasibility by designing the first method FreeSel following this new pipeline. Through single-pass model infer- ence, the semantic patterns are constructed based on the intermediate features of a general pretrained model, over which distance- baatance-bbeedddee selection strat- egy is performed to find the most diverse and in- formative data samples. Our method outperforms most existing counterparts with remarkably supe- rior efficiency on different tasks including detec- tion, segmentation, and classification.
90-
Top-1 Top--AAccua Accuracy
80
random
Cure-Sel
60
Sere-Set (INO)
Learn-Loss
Loar (DeIT-DINO)
FreeSel
4000
6000
10000
Image Number
Figure 9: Effeec of Pretraining Methods: Ex- periments are conducted on PASCAL VOC.
We realize that FreeSel cannot beat all the other data selection methods in current stage due to the absence of training the target datasets. Never- theless, this direction matters in boosting the training of downstream models without any extra time and cost the shoulders of existing general pretrained models. It gains more significance given the current landscape dominated by large foundation models pretrained on multi-modality data [40, 15), which we believe can help extend our method wide range of domains and modalities.
Acknowledgement. This work partially supported by Berkeley DeepDrive.
10

---

References
[1] S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In European Conference Computer Vision, pages 137-153. Springer, 2020. 3, 7, 8, 17
[2] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05884, 2(3)4, 2021. 2
[3] W. Beluch, Genewein, A. Nümberger, and J. M. Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9368-9377, 2018.
[4] V. Birodkar, H. Mobahi, and S. Bengio. Semantic redundancies in image-clasiffictioon datasets: The 10% you don't 't need. arXiv preprint arXiv:1901.1 1409, 2019.
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in sele-super vision transformers. arXiv preprint arXiv: 104. 1e..114..44499 14294, 2021. 2, 4, 5, 6, 9, 10, 16
[6] E. Chang, X. Shen, H.-S. Ych, and V. Demberg. On training instance selection for few-shot neural text generation. arXiv preprint arXiv:2107.03176 2021. 3
[7] X. Chen, S. Xie, and K. He. An empirical study of training self-supervise vision transformers. arXiv preprint arXiv:2 104.0004777 2021.
[8] J. Choi, I. Elezi. H.-J Lee, C. Farabet, and J. M. Alvarez. Active learning for deep object detection via probabilistic modeling. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 10264-10273, 2021. 7
[9] M. Cordts, M. Omran, Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016. 15
[10] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020. 10
[11] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009. 2, 4, 6, 15
[12] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020. 5,5
[13] Ebrahimi, M. Elhoseiny, Darrell, and M. Rohrbach. pnncertainty-guidee continual learning with bayesian neural networks. arXiv preprint arXiv: 1906.0225, 2019.
[14] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010. 7,
[15) R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 15180-15190, 2023. 10
[16] M. Gorriz, A. Carlier, E. Faure, and X. Giro-i Nieto. Cost-effective active learning for melanoma segmentation. arXiv preprint arXiv:1711.0916. 2017. 3
[17] J.BB. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning, arXiv preprint arXiv:2006.07733. 2020. 3

---

[18] E. Haussmann, M. Fenzi, K. Chitta, J. Ivanecky, H. Xu, D. Roy, A. Mittel, N. Koumchatzky, C. Farabet, and M. Alvarez. Scalable active learning for object detection. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1430-1435. IEEE, 2020. 3
[19] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVE Conference Computer Vision and Patlern Recognition, pages 16000-16009, 2022. 3
[20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE//CV Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. 3
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778, 2016. 8, 17
[22] Y. He, W. Liang, D. Zhao, H.-Y. Zhou, W. Ge. Y. Yu, and W. Zhang. Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9119 9129, 2022.
[23] J. Hou, B. Graham, M. Nießner, and S. Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15587-15525 2021.
[24] S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou. Semi-supervised active learning with temporal output discrepancy. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021. 3, 17
[25] S. Huang, Y. Xie, S.-C. Zhu, and Y. Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545, 2021.
[26] A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classifica- tion. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pages 2372-2379. IEEE, 2009. 3
[27] V. Kaushal, A. Sahoo, K. Doctor, N. Raju, S. Shetty, P. Singh, Iyer, and G. Ramakrishnan. Learning from less data: Diversified subset selection and active learning in image classification tasks. arXiv preprint 1.5iv:1805.1119900 2018.
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014. 17
[29] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 105, 2012. 16
[31] M. Li, S. Wang, and Q. Zhang. Visualizing the emergence of intermediate visual patterns in dnns. arXiv preprint arXiv:2111.033 2021. 4
[32] X. Li and Guo. Adaptive active learning for image classification. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pages 859-866, 2013.
[33] H. Lin, Upehurch, and K. Bala. Block annotation: Better image annotation with sub-image decomposition. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 5290-5300, 2019.
[34] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. European conference on computer vision, pages 21-37. Springer, 2016. 4, 7, 8, 9, 16, 17
12

---

[35] Z. Liu, H. Ding, H. Zhong, W. J. J. Dai, and C. He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9274-9283,
2021. 8, 17
[36] D. J. MacKay. Information-basee objective functions for active data selection. Neural computa- tion, 4(4):590-66)) 604, 1992. 3
[37] R. Mahmood, S. Fidler, and M. T. Law. Low budget active learning via wasserstein distance:
An integer programming approach. arXiv preprint arXiv:2106.02900 2021. 3
[38] A. Ng, M. Jordan, and Weiss. On spectral clustering: Analysis and an algorithm. Advances
in neural information processing systems, 14, 2001. 6, 15
[39] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint Stti:200030429966 2020. 3
[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Míshkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 10
[41] M. Raghu, T. Unterthiner, S. Komblith, C. Zhang, and A. Dosovitskiy. Do vision transformers like convolutional neural networks? In Thirty-Fiithh Conference on Neural Information Processing Systems, 2021. 4
[42] S. Ramalingam, D. Glasner, K. Patel, R. Vemulapalli, S. Jayasumana, and S. Kumar. Less is more: Selecting informative and diverse subsets with balancing constraints. arXiv preprint 100::2204.12835, 2021. 3,3
[43] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, X. Chen, and X. Wang. A survey of deep active learning. arXiv preprint arXiv:2009. 00236, 2020.
[44] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1708 00489, 2017. 1, 4, 8,
170.00048,
[45] B. Settles. Active learning literature survey. 2009.
[46] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 7, 17
[47] S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019. 3, 8, 17
[48] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005. 10243, 2020. 3
[49] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-1000000 2022. 3
[50] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021. 4, 6, 9, 10, 16, 17
[51] X. Wang, L. Lian, and S. X. Yu. Unsupervised selective labeling for more effective semi- supervised learning. In European Conference on Computer Vision, pages 427-445. Springer, 2022.
[52] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668-14678, 2022.
[53] T. Xiao, X. Wang, A. Efros, and T. Darrell. What should not be contrastive in contrastive learning, arXiv preprint 1100::000800053050 2020. 3
13

---

[54] Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining gretraning-fiettninng paradigm. arXiv preprint 1r342 2023. 3
[55] D. Yan, L. Huang, and M. Jordan. Fast approximate spectral clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 907-916, 2009. 6, 15
[56] S. K. Yi, M. Seo, J. Park, and D.-G. Choi. Pt4al: Using supe-super pretext tasks for active learning. In European Conference on Computer Vision, pages 596-612. Springer, 2022. 3
[57] D. Yoo and I. S. Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 93-102, 2019. 3, 7, 8, 9, 17
[58] Yu, Koltun, and T. Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472-480, 2017. 7, 16, 17
[59] T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5330-5339, 2021. 3,7
[60] B. Zhang, L. Li, S. Yang, S. Wang, Z.-J. Zha, and Q. Huang. State-relabeling adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8756-8765, 2020. 3
[61] Z. Zhang, R. Girdhar, A. Joulin, and I. Misra. Self-supervised pretraining of 3d features on any point-cloud. arXiv preprint arXiv:2 200.0299,,,,,911,,, 2021. 3
[62] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. arXiv preprint
2014. 4
[63] Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXi::2111.07832 2021. 2.9
[64] Zhu, Qi. M. Ding, X. Chen, P. Luo, X. Wang. W. Liu, L. Wang, and Wang. Under- standing self-supervised pretraining with part-aware representation learning. arXiv preprint a30iv:2301. 1915, 2023. 2
14

---

In this supplementary material, we first explain the details of spectral clustering algorithm (Sec. 4.2) in Sec. A. We then analyze the sensitivity of FreeSel to the values of hyperparameters in Sec. B. Finally, implementation details of our experiments are explained in Sec. C.
A Spectral Clustering Algorithm
In this section, explain the spectral clustering algorithm [38, 55] in the semantic pattern extraction process for each image I (Sec. 4.2 and Alg. 1). The detailed spectral clustering algorithm is shown in Alg. 3. This spectral clustering algorithm should be inserted into line 7 of Alg. 1.
To justify the use of spectral clustering algorithm for semantic pattern extraction, we also try another alternative which directly performs K-Means w.r.f. the local features f 1, 2, ...,. , to divide the t
regions of image into K clusters without using the patch token attention to achieve the same feature clustering goal in Sec. 4.2. Tab. 4 shows the comparison between spectral clustering and K-Means. Interestingly. these two feature clustering strategies lead to similar data selection performance on PASCAL VOC [14] object detection task. However, spectral clustering is stably superior when selecting data samples for Cityscapes [9] semantic segmentation task. We attribute this difference the large domain gap between Cityscapes dataset and ImageNet dataset [11]. The DeiT-S model pretrained on ImageNet may extract local features with weaker discriminative ability from images inside Cityscapes dataset. Since spectral clustering algorithm depends less on the feature quality, can bring better performance than direct K-Means over intermediate local features on Cityscapes.
Algorithm 3: Spectral Clustering
Input: Similarity matrix between patches pa
[pag]] [ i..=1.2....! semantic pattern number Output: Clusters Cj j = 1,2, L,2,...,K, K, where each region r 1, 2, ..,.. ,t of image belongs to unique Cj
Derive the symmetric adjacent matrix A from pa
A pa
pa
1/2,
A E
Derive the diagonal degree matrix D:
Dij = E Au i=
DER
i i
Calculate the normalized Laplacian matrix L:
L= L A)D
L L
Obtain the K eigenvectors vr 01,/= - 1,,,=11,,.... K corresponding to the K smallest eigenvalues 01, ..,, K of matrix L. 5 Compose the matrix V based on the K eigenvectors
V [11, U2 U2.,.., .., UK] V R
Denote u. the i-th row of V, = 1,2.....
7 Normalize each row of V: û = /1/ Σ u
8 Perform K-Means to divide ù, i = 1,2,..., .,, into clusters CJ.j 1.2,...,KK
(C) KMeans ((.
B Sensitivity to Hyperparameters
In this part, we analyze the sensitivity of our FreeSel to some hyperparameters including the mainte- nance ratio 7 in the attention filter (Eq.. 2), the semantic pattern number (Fq. 4), the neighborhood threshold do (Eq. 3), the distance function ((..-) (Eq. 5), and pretraining manner for the general
15

---

Table Effect of Feature Clustering Strategies: We compare spectral clustering and K-Means for feature clustering. Experiments are conducted on PASCAL VOC object detection task and Cityscapes semantic segmentation task.
(a) Performance PASCAL VOC Object Detec- tion Task: The task model is SSD-300 [34].
(b) Performance Cityscapes Semantic Segmen- tation Task: The task model DRN [58].
Image Number
Sampling Ratio
Feature Clustering
Feature Clustering
3k
5k
7k
15%
25%
35%
K-Means
65.35
69.43
71.76
K-Means
51.43
54.84
57.96
Spectral Clustering
65.66
69.24
71.79
Spectral Clustering
51.77
55.72
58.58
Table Sensitivity to Hyperparameters: K, do, D). separately denote the maintenance ratio, semantic pattern number, neighborhood threshold, and distance function. Experiments are conducted on the PASCAL VOC object detection task.
Image Number
K
do
D(.-))
Pretraining
3k
5k
7k
0.3
65.22
69.00
70.69
0.5
COs.
unsupervised
65.66
69.24
71.79
0.7
64.77
69.33
71.64
64.90
69.01
71.02
0.5
COS.
wnsupervised
65.21
69.13
71.50
65. .488
68.73
71.39
0.5
5
COS.
unsupervised
65.37
69.41
71.74
0.5
euc.
unsupervised
64.77
69.42
71.31
0.5
5
COS.
supervised
64.40
68.82
71.43
model. Experiments are conducted on the object detection task, where samples are selected from PASCAL VOC dataset and SSD-300 is the downstream task model same settings as Sec, 5.2. Results are shown in Tab. 5.
Maintenance Ratio (Eq. 2) Maintenance ratio notably affects the final performance FreeSel. Too low ratios lead to the ignorance of some crucial local visual patterns, while too high ratios introduce some harmful noisy information to the semantic patterns. Thus, moderate attention ratio plays important role in the high performance of FreeSel.
Semantic Pattern Number K (Eq. 4) When K 1, the performance is hurt since semantic patterns degrade to global features in this case. When K = 10, a slight performance drop may be witnessed in comparison with K 5.
Neighborhood Threshold dp (Eq. 3) When dg = 1, the neighborhood too small to represent the relationship between nearby regions. When dg 3, the performance is little worse than do 2. We think each region feature mainly interacts with nearby regions with distance d < 2.
Distance Function D(--) (Eq. 5)
We find the cosine distance can lead to better performance than Euclidean distance. This result shows that the directions of local feature vectors are important to reflect the diversity of local visual patterns.
Pretraining Manner Instead of using the unsupervised pretraining framework DINO [5], we
also try the DeiT-S model [50] pretrained in supervised manner on ImageNet [30]. Results show a performance drop with supervised pretraining. We think this because supervised pretraining introduces some biases of categories to the pretrained model.
16

---

C
Implementation Details
C.1 Object Detection Implementation
C.1.1 Implementation of FreeSel
We set attention ratio = 0.5 and semantic pattern number K = 5. The input images are resized to 224 x 224 when fed into the pretrained DeiT-S [50] model in the data selection process.
C.1.2 Implementation of Task Model
The implementation of task model is same as previous active learning research [57. 1]. The SSD-300 model [34] with VGG-16 [46] backbone adopted for this experiment. The model implemented based on imdetectioon We follow [57, 1] to train the model for 300 epochs with batch size 32 using SGD optimizer (momentum 0.9). The initial learning rate 0.001, which decays 0.0001 after 240 epochs.
C.2 Semantic Segmentation Implementation
C.2.1 Implementation of FreeSel
The input images resized 448 224 in line with their original aspect ratios when fed into the pretrained DeiT-S [50] model in the data selection process. Same as object detection, we set attention ratio т 0.5 and semantic pattern number is doubled K 10 in line with the doubled input size compared to object detection task.
C.2.2 Implementation of Task Model
We follow prior active learning work [47, 24] apply DRN [58] model for semantic segmentation task. The model is trained for 50 epochs with batch size and learning rate 5e-4 using Adam optimizer [28].
C.3 Image Classification Implementation
C.3.1 Implementation of FreeSel
We follow previous tasks to set attention ratio r = 0.5. Since image classification depends less on local information, we directly set the semantic pattern number K The input images resized to 224 x 224 when fed into the pretrained DeiT-S [50] model in the data selection process.
C.3.2 Implementation of Task Model
We follow [57, 35] use ResNet-18 [21] classification model in this task, which is implemented based on mmclassification The model is trained for 200 epochs with batch size 128 using an SGD optimizer (momentum 0.9, weight decay 5e-4). The initial learning rate is 0.1, which decays to 0.01 after 160 epochs. We apply standard data augmentation to the training including 32x32 size random crop from 36x36 zero- pero-padded images and random horizontal flip.
Phups://github. >httss//gttub..oo/ppen--- com/ooonnmmmmmmmmmeetrccii "narp://githuh.com//yuun
17


Reviews:
[{"review_id": "xegm1NCUmj", "reviewer": "Reviewer_xhjx", "summary": "This paper proposed a diversity-based method for active learning. Given an image, features are extracted from a pretrained visual transformer model (DeiT-S), features with [CLS] token attention above a threshold are kept and clustered into K clusters. Each cluster is represented by the average of features belonging to the cluster. The cluster center, named semantic pattern in the paper, is used in a distance-based probabilistic sampling framework (similar to kmeans++) for sample selection. The proposed method is evaluated in various visual tasks including object detection, semantic segmentation and image classification and obtained promising results.", "questions": "The proposed method is essentially doing class-wise aggregation to generate feature representation for an image, which has been explored in CDAL. Apart from the obvious difference that CDAL can obtain pseudo label directly from the target classifier while this paper need to employ clustering to generate pseudo labels, what are other differences and advantages of the proposed method compared to CDAL? ", "limitations": "The authors discuss the limitation of the proposed method (i.e., cannot beat all state-of-the-art methods) in Section 6 and leave it for future work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method utilized pretrained model for feature extraction and sample selection, eliminating the need to train a model at each AL cycle and thus is much more time efficient than traditional AL methods.\n2. The method is evaluated on various tasks and datasets and shown promising results.", "weaknesses": "1. The effectiveness of the proposed method is questionable. \n(1) From Fig.5 and Fig.7, it can be seen that the proposed method is always outperformed by competing methods at later stage, suggesting  that the pretrained features become less effective in selecting informative samples as the training progresses. Performance at later stage is more important as active learning is only useful when your method can achieve comparable performance (e.g., 95%) of the fully-supervised baseline. \n(2) It is not clear how the method performs when the target domain has large domain shift to the pretrained domain. Currently the evaluation datasets are all natural images that are similar to ImageNet. I would expect that the effectiveness of the proposed method would be undermined when the domain shift becomes large.\n(3) Benchmarking is not comprehensive, missing BADGE [1] for image classification, EnmsDivproto [2] for object detection.\n\n2. Technical contribution is limited. The proposed method combines existing techniques in a straightforward manner, lacking novelty and technical contribution. \n\n\n[1] Ash, Jordan T., et al. \"Deep batch active learning by diverse, uncertain gradient lower bounds.\" ICLR2020.\n[2] Wu, Jiaxi, Jiaxin Chen, and Di Huang. \"Entropy-based active learning for object detection with progressive diversity constraint.\" CVPR2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for your efforts to improve the quality of our papers. We thank you for pointing out your concerns, which we believe have been carefully clarified and addressed, as follows.\n\n**Q1: [Outperformed by some active learning methods in late stage.]**\n\nGood question! This observation is reasonable, but our method still has wide applicability, such as scenarios where resources are severely constrained.\n- Existing active learning methods train their models on the selected samples after each batch selection step. As the sample number increases, their models have been trained on a large subset of the target dataset, so it should naturally fit the target domain well. In contrast, our method utilizes a general pretrained without any domain-specific knowledge regarding the target dataset. It is understandable that existing methods can fit the domain of the targeted dataset better after seeing many training samples in a later stage.\n- We respectfully push the comment back that data selection is only useful in the late stage, as in the following points.\n    - The budgets in different cases vary significantly. Sometimes the budget can only handle very low sampling ratios such as in medical imaging where labeling requires human experts. \n    - Data is usually cheap and abundant. For example, a vehicle can capture hundreds of images every minute, but we only need to keep a very small part. An efficient data selection algorithm to sample a small percentage of data is really desirable.\n    - Our method can be complementary to SOTA techniques. Since we aim to get a free data selection without any extra training efforts, our method can be easily combined with other techniques like replacing the initial random seed sets for other active learning or subset selection methods.\n\n**Q2: [Performance under large domain shift to the pretrained domain.]**\n\n- We conduct an experiment on RealPizza dataset (multi-label classification with mAP as the metric) with a large domain shift to ImageNet. FreeSel still achieves notable performance gain in comparison to random selection: 10% 35.0->37.5, 15%: 37.8->39.2, 20%: 39.1->40.7, showing its robustness to different domains.\n\n- Besides, we already conduct experiments on datasets from different domains in the paper (PASCAL VOC: natural images, CIFAR: low-resolution images, Cityscapes: driving scenarios). Results show that our method is general enough to most data domains.\n\n**Q3: [Missing comparison with BADGE for classification, EnmsDivproto for detection.]**\n\n+ Thanks for pointing out. In Line 213, we explain that we do not compare with active learning methods specifically designed for one task which should naturally perform better than general methods. We will include these related works and analyses in our revision.\n\n+ Besides, we assert that performance isn't the sole metric of consideration. We aim to develop a \"free\" data selection algorithm. We can directly exploit public pretrained model and only need negligible extra efforts for data selection, reducing the data selection time from days to minutes as well as simplifying the data selection pipeline greatly (Line 249-257). In many scenarios, such as selecting data from an extremely large data pool, we believe this advantage of \"free\" far outweighs minor disparities in performance.\n\n**Q4: [The proposed method combines existing techniques in a straightforward manner, lacking novelty and technical contribution.]**\n\n- Our main novelty lies in the brand-new data selection pipeline. Existing active learning methods need repetitive model training and batch data selection, while other subset selection algorithm also requires to train the model on a small initial subset of the target dataset. The problem is that training models is costly, and the size of subset for training directly affects their performance of data selection. In contrast, our paper designs a novel pipeline to directly exploit general pretrained model to select all the samples from a large data pool in one go. In this case, no training on the target dataset is needed, and data selection time goes down from days to minutes, almost negligible, which means large-scale data selection becomes doable.\n- Although our method is notably straightforward without intricate modules, we believe it's unnecessary to inject excessively complex designs to artificially make it \"looks novel\". In contrast, it is always desirable that a simple method can finish the task well, especially given that our main novelty lies in the pipeline design. This is verified by our experiment results. If a simple method can rival well-designed existing algorithms in performance, it unquestionably becomes a compelling choice.\n\n**Q5: [Difference with CDAL.]**\n\nWe share some common insights into the importance of regional information with CDAL. However, our work is still totally different with CDAL in both pipeline and algorithm. \n1) The key difference lies in the pipeline. As shown by Reviewer 7NQc, our problem setting and pipeline are quite different with traditional active learning. We exploit public pretrained model to achieve a free data selection with negligible extra efforts. In contrast, CDAL follows the traditional active learning pipeline and requires computationally intensive multi-step training and batch selection. \n2) Our algorithm is also quite different with CDAL. CDAL relies on a known label space to get the softmax probability for each region, but our algorithm computes the diversity based on the feature clusters which is more friendly to the open set. CDAL aggregates the image-level confusion from region-level softmax probability, but our method directly selects images based on the region-level diversity.\n\nWe wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments!"}, {"review_id": "hr4W5jcXid", "reviewer": "Reviewer_SbPr", "summary": "This paper aims to introduce a unique method called FreeSel for data selection. This method is accompanied by a newly designed pipeline that incorporates existing general-purpose models, requiring minimal additional time. By leveraging a publicly available pre-trained model, FreeSel can select data from diverse datasets through a single-pass inference, eliminating the need for extra training or supervision. Specifically, the method involves extracting semantic patterns from intermediate features of the general-purpose model to capture nuanced local information within each image. Furthermore, it enables the selection of all data samples in a single pass using distance-based sampling at a fine-grained semantic pattern level.", "questions": "Please answer all the concerns raised in the weaknesses section.", "limitations": "The authors have mentioned the limitations of the proposed approach in the conclusion section. The authors have also presented an ablation study to discriminate the contribution of pre-trained vision transformers (LLM) and other components. It is essential to understand that the performance of the proposed model is not only due to the capability of pre-trained LLM models.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- For the first time, this paper presents a novel model that introduces a data selection pipeline. This pipeline incorporates crucial principles of a robust feature extractor, including generality, efficiency, and non-supervision, while incurring minimal time costs. This innovative idea holds potential for application in active learning scenarios.\n\n2- This approach leverages unsupervised extraction of intermediate features by harnessing the benefits of publicly available pre-trained vision transformer (LLM) models. These features are then utilized to define semantic patterns.\n\n3- In contrast to existing active learning approaches, this method selects data examples for a given budget in a single pass, focusing on the diversity found in the local patterns extracted by the pre-trained model. This approach is considerably more time-effective compared to existing methods, as it avoids the need for multiple passes or iterations.\n\n4- To demonstrate the generalizability of the proposed approach, experiments were conducted across various types of tasks, including image classification, object detection, and semantic segmentation. The results of these experiments reveal significant performance improvements, underscoring the effectiveness of the approach across different domains and applications.", "weaknesses": "1- We have noticed a decrease in the performance of the proposed model as the number of samples increases, which is in contrast to several active learning approaches. We require a justification for this observation. Is the model sensitive to the size of the datasets?\n\n2- In the paper, the variable \"c\" represents the number of pseudo categories, which defines the number of visual parts in an image. However, it is unclear how \"c\" is determined. Is it a fixed value for every image, or does it vary from image to image?\n\n3- The proposed method relies solely on local features to identify semantic patterns, disregarding global features which can be crucial in certain cases. Completely disregarding global features may not be the most desirable approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for the positive comments and insightful suggestions. The concerns are answered as follows.\n\n**Q1: [Performance decreases as the sample number increases. Sensitivity to the dataset size.]**\n\n- Existing active learning methods train their models on the selected samples after each batch selection step. As a result, as the sample number increases, their models have been trained on a large subset of the target dataset, so it should naturally fit the target domain well. In contrast, our method makes use of a general pretrained without any domain-specific knowledge regarding the target dataset. It is understandable that existing methods can fit the domain of the targeted dataset better after seeing many training samples.\n- Our method is not sensitive to the dataset size. \n    - We conduct experiments on two extra classification datasets with different sizes: CalTech 256 dataset (~30k), CIFAR100 dataset (50k). We do not experiment on ImageNet since it is our pretraining dataset. \n        - For CalTech dataset, we can improve the random selection from 70.2 to 71.4 (15% data) and from 73.7 to 75.8 (25% data). \n        - For CIFAR100, it improves the random selection fron 49.8 to 56.3 (10k samples) and 62.3 to 67.8 (20k samples). \n    - Our experiments in the paper also include PASCAL VOC (16k) and Cityscapes (3k).\n    - These results show that our method is not sensitive to the dataset size.\n\n\n**Q2: [How to determine the visual parts number in each image.]**\n\nWe currently set a fixed number for each image. There is an ablation study about this number in the **supplementary materials Tab. 2**, where we find a moderate visual parts number can lead to the best performance.\n\n\n**Q3: [The method disregards the global features.]**\n\nOur method pays more attention to the local features instead of global features since global features are hard to represent complex images (the 1st and 2nd row of Tab. 2 in our paper). However, our method does not totally disregard the global features. Since the pretraining task is image-level contrastive learning, the patch-token features already implicitly encode some useful global information. Besides, we also filter the regions based on the [CLS] token attention map, so the regions unimportant to the image would be discarded.\n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window."}, {"review_id": "1BtDTGszhj", "reviewer": "Reviewer_7NQc", "summary": "This work introduces a method named FreeSel, which is designed to efficiently select information-rich data points from an unlabeled pool. These selected data points can then be annotated and utilized for training deep learning models, thereby mitigating the costs and resources needed to annotate an entire dataset. The method primarily consists of two key components: feature extraction using a large-scale pretrained model (DINO), and the construction of semantic patterns based on intermediate features. Subsequently, a distance-based selection strategy is employed to identify the most diverse and informative data samples.\n\nThe authors assert that the proposed approach addresses three critical aspects: Generality, by decoupling data selection from task-specific models; Efficiency, by enabling the selection of samples in a single pass; and Non-Supervision, by eliminating the need for annotations until the data selection process is complete.\n\nWhile the results on multiple datasets and tasks indicate that the proposed methodology holds promise, the paper seems to lack some important comparisons. Additionally, characterizing this work as an alternative to traditional active learning may not be entirely accurate.", "questions": "1. What is the rationale for choosing CoreSet in Figure, there are much more recent and useful active learning strategies that are specifically proposed for the problem of object detection? \n2. No comparisons were made with the works listed in `Data Selection with Pretrained Models` paragraph of related work though they seem very relevant, is there any specific reason for this? Including comparisons with these pertinent works could enhance the comprehensiveness and depth of the evaluation.", "limitations": "All the limitations are clearly discussed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of employing a large-scale pre-trained model for data selection is interesting. This is especially pertinent in today's environment, where there is a proliferation of large-scale foundational models. Harnessing these models to select data samples for training of downstream models could prove to be highly advantageous for addressing a wide range of problems.\n2. The paper is well-structured and easy to understand\n3. The inclusion of experiments across multiple tasks and datasets provides a comprehensive view of the empirical performance of the proposed approach", "weaknesses": "1. The primary concern regarding this work pertains to its positioning. The paper seems to be more aligned with the domain of subset selection rather than active learning (AL).\n    * AL methods, as the name suggests, actively learn which samples to select in a manner that is specific to the model at hand. While I concur with the authors' perspective that this iterative method is resource-intensive, drawing a direct comparison between the proposed \"passive\" selection method and traditional active learning in terms of training time may not be entirely appropriate.\n\n2. There is a need for additional baselines to more effectively evaluate the merits of the proposed approach.\n    * For instance, in the absence of any iterative active learning, if a random subset of data is selected and a model is trained on it, how does the accuracy of this model compare to one trained on samples selected using FreeSel?\n\n3. There appears to be some conflation between the method (FreeSel) and the problem setting (data selection using large-scale pre-trained models). For example, the principle of Non-supervision (Line 48) seems more pertinent to the methodology than to the problem setting.\n4. To gain a clearer understanding of the novelty and contributions of this work, it would be beneficial to compare the method with existing subset selection methods.Such comparisons do not necessarily need to be empirical but could include theoretical or conceptual comparisons.\n5. It would be beneficial for the authors to clarify the positioning of FreeSel in relation to traditional active learning and to provide additional comparisons to strengthen the evaluation.\n\nReferences for consideration: \n\n1. [Kaushal et al., Learning From Less Data: Diversified Subset Selection and Active  Learning in Image Classification Tasks](https://arxiv.org/pdf/1805.11191)\n2. [Chang et al., On Training Instance Selection for Few-Shot Neural Text Generation](https://arxiv.org/pdf/2107.03176)\n3. [Birodkar et al., Semantic Redundancies in Image-Classification Datasets: The 10% You  Don't Need](https://arxiv.org/pdf/1901.11409)\n4. [Ramalingam, et al., Less is more: Selecting informative and diverse subsets with balancing  constraints](https://arxiv.org/pdf/2104.12835)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and insightful suggestions. The concerns are answered as follows.\n\n**Q1: [Our method is more like subset selection instead of active learning.]**\n\n+ We totally agree with your points. Our method should be considered as subset selection more than active learning. That is actually why we name this method \"free selection\" instead of \"XXX active learning\". We compare with active learning methods instead of subset selection methods because active learning is the mainstream kind of data selection algorithms. \n+ For subset selection algorithms, although they do not require the repetitive batch selection strategy, most of them still need to train a model on a labeled seed subset of the target dataset. Therefore, our method is still much more efficient than theirs since we do not need any extra training on the target dataset. We will make more discussion about the positioning in the final version.\n\n\n**Q2: [Additional baselines for subset selection.]**\n\nIn our Fig. 5,6,7, we also provide the random selection baseline (denoted in blue) under different sampling ratios. In the following table, We also give more heuristic subset selection baselines without extra training for SSD object detection model on PASCAL VOC. 1) Perform K-Means on image features extracted by PCA. 2) Perform K-Means on image features extracted by DINO pretrained DeiT. 3）For image features extracted by DINO pretrained DeiT, we calculate the average pairwise feature distances among 10 augmented copies to find hard samples. 4) Using ImageNet supervised pretrained model, we calculate the entropy of prediction to find uncertain samples.\n\n\n| Methods | Num. 3k | Num. 5k | Num. 7k |\n| -------- | -------- | -------- |  -------- |\n| PCA+Kmeans     | 63.87     | 67.77     |  71.02 |\n| DINO+Kmeans     | 64.85     | 68.05     |  71.50 |\n| Inconsistency     | 63.29    | 67.65    |  71.35 |\n| Entropy     | 56.33    | 66.03    |  69.72 |\n| FreeSel     | **65.66**    | **69.24**    |  **71.79** |\n\n\n**Q3: [Some conflation between the method (FreeSel) and the problem setting.]**\n\nThanks for pointing out. Our problem setting is to select samples with the pretrained model. In this process, we select all the samples in a single run, so we do not have any knowledge about the label space in this process. As a result, this process is naturally unsupervised whatever the detailed selection algorithm is. Therefore, we consider non-supervision as the problem setting instead of the method. We will make more clarification between the problem setting and concrete method in the final version. \n\n\n**Q4: [Compare the method with existing subset selection methods.]**\n\nThanks for giving those references. They are different from our paper in both problem settings and concrete algorithms. [R1] relies on a small initial seed set to train the model for subset selection, and their subset selection is based on uncertainty instead of diversity. [R2] focuses on the few-shot learning and finetuning of the pretrained model. Their algorithm is based on K-Means, but our answer in Q2 shows that K-Means does not work in our setting. [R3] also trains the model on the target dataset first for the further analysis. Their algorithm is based on the Agglomerative Clustering of global features. Our ablation studies also show that global features cannot solve our problem. [R4] also needs a small initial seed set for model training. Their selection algorithm is a combination of uncertainty and diversity. In contrast to all of them, our method does not need training on the target dataset or knowledge about the label space. We will include more discussion of these methods in the final version.\n\n\n**Q5: [Relation to traditional active learning and additional comparisons.]**\n\nThanks for your suggestion. We include more explanation of the positioning of our work and make some comparisons with existing subset selection methods in Q1 and Q4. We will also add more information in the final version.  \n\n\n**Q6: [Rationale for choosing CoreSet for analysis.]**\n\nAs you mentioned in Q1, our problem setting is different from active learning. Thus, we have to adapt those active learning methods to our problem setting for analysis. CoreSet is fully based on the feature diversity and does not require the knowledge of label space, so it can be trivially transplanted to our problem setting while some other methods such as LearnLoss and CDAL cannot be easily adapted. Alternatively, in Fig. 8, we also give results of trivially combining the pretrained model with active learning methods, which does not improve the performance.\n\n\n**Q7: [No comparisons with the works listed in `Data Selection with Pretrained Models` paragraph.]**\n\n+ Most papers mentioned in this paragraph are working on other problems different from ours. [28] focuses on the extremely low budget (less than 1%). [39] applies to semi-supervised learning. [42] is targeted for model finetuning with low budgets (mostly less than 5%). Thus, it is hard to compare our method with them directly. \n+ Among them, only [44] works on general active learning following the traditional batch selection pipeline. They do not use pretrained models but just borrow some ideas of unsupervised learning for data selection. Their performance is extremely sensitive to the selection of unsupervised learning algorithms, and different unsupervised learning fits different tasks. On CIFAR10, FreeSel performs competitively with PT4AL (rotation) and outperforms PTAL (jigsaw, colorization, SimSiam). However, for segmentation, PT4AL (rotation) performs worst. Therefore, [44] cannot find a general unsupervised learning algorithm that can be used for active learning for all tasks.\n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.\n"}, {"review_id": "BCwuOsLGYN", "reviewer": "Reviewer_yLeE", "summary": "This paper presents a data selection methods without training, so saving lots of time than previous methods. By a publicly available pretrained model, FreeSel can select data from datasets with a single-pass inference, without need for additional training. This method uses comparison between semantic patterns of samples with distance-based sampling. Experiments on various tasks including classification, segmentation and detection verify the motivation.", "questions": "Please see weaknesses. If authors address them, reviewer would like to change the rating.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method requires no training to select useful examples, saving lots of time.\n2. The experimental performance shows the priority about this method.", "weaknesses": "1. The method is plain, to compare the similarities among input sample features. Though saving time, it cannot outperform other methods. \n2. The downstream experiments are still using detection and segmentation models and backbones 5 or 6 years ago, the reviewer would like to see results on COCO, ADE20k and backbone like ViT, or DETR detectors.\n3. The active learning which select samples from already annotated datasets is meaningless, more experiments about using FreeSel to select data from web or unannotated images are much more significant to show the effectiveness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to improve the quality of our papers. We thank you for pointing out your concerns, which we believe have been carefully clarified and addressed, as follows.\n\n\n**Q1: [Our method is plain and cannot outperform all active learning algorithms.]**\n\nThank you for the question. We answer from two aspects: novelty and performance.\n\n> Regarding our novelty:\n- The main novelty of our work lies in the brand-new data selection pipeline. Existing active learning methods need repetitive model training and batch data selection, while other subset selection algorithm also requires to train the model on a small initial subset of the target dataset. The problem is that training models is costly, and the size of subset for training directly affects their performance of data selection. In contrast, our paper designs a novel pipeline to directly exploit general pretrained models to select all the samples from a large data pool in one go. In this case, no training on the target dataset is needed, and data selection time goes down from days to minutes, almost negligible, which means large-scale data selection becomes doable.\n- Although our method is notably straightforward without intricate modules, we believe it's unnecessary to inject excessively complex designs to artificially make it \"looks novel\". In contrast, it is always desirable that a simple method can finish the task well, especially given that our main novelty lies in the pipeline design. This is verified by our experiment results. If a simple method can rival well-designed existing algorithms in performance, it unquestionably becomes a compelling choice.\n\n> Regarding our performance:\n- In comparison with SOTA active learning algorithms, we can outperform most of them but underperform some of them in the late stage. Existing active learning methods train their models on the selected samples after each batch selection step. As a result, as the sample number increases, their models have been trained on a large subset of the target dataset, so it should naturally fit the target domain well. In contrast, our method makes use of general pretrained knowledge without any domain-specific training regarding the target dataset. That's why existing methods can fit the domain of the targeted dataset better after seeing many training samples in a later stage. This is reasonable, and our model therefore has wider applicability, such as scenarios where data is scarce or resources are severely constrained. We believe that the early stage is also critical as training budgets and data vary.\n- Nonetheless, we assert that performance isn't the sole metric of consideration. We aim to develop a \"free\" data selection algorithm. We can directly exploit public pretrained model and only need negligible extra efforts for data selection, reducing the data selection time from days to minutes as well as simplifying the data selection pipeline greatly (Line 249-257). In many scenarios, such as selecting data from an extremely large data pool, we believe this advantage of \"free\" far outweighs minor disparities in performance.\n\n**Q2: [Results on more datasets and backbones.]**\n\nThe networks and datasets for our experiments follow the popular settings of existing active learning papers so that we can make fair comparisons with them. We also conduct extra experiments, demonstrating the effectiveness of our work: \n- On MS-COCO dataset with RetinaNet for object detection, FreeSel still outperforms random selection: 7.0 mAP -> 9.0 mAP (2% data) and 14.3 mAP -> 15.3 mAP (4% data). \n- On ADE20k dataset with Segmenter (ViT backbone) for semantic segmentation, FreeSel outperforms random selection as well: 27.1 mIoU -> 28.2 mIoU (5% data).\n\n\n**Q3: [Select data from web or unannotated data.]**\n\nWe agree with the importance of performing data selection algorithms on the web or annotated data. However, as an academic research team, we do not have enough budget to annotate the selected samples to evaluate our method. Besides, most existing active learning algorithms experiment on these standard datasets, so we follow their settings to make a fair comparison. We will consider it as our future work, thanks!\n\n\\\nWe wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments! "}]

Author rebuttal:

We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our contributions including the novelty of our proposed pipeline (Reviewers SbPr,7NQc), the efficiency of our method (Reviewers xhjx, SbPr, yLeE), detailed experiments on different tasks (Reviewers xhjx, SbPr, 7NQc, yLeE), and our well-written paper (Reviewer 7NQc).


In response to reviewer comments, we have added several new experiments and results to strengthen the paper.

**Summary of new experiments.**
1. New experiments on RealPizza dataset to evaluate the performance under large domain shift. [Reviewer xhjx Q2]
2. New experiments on CalTech256 and CIFAR100 to show the method's ability on datasets of different sizes. [Reviewer SbPR Q1]
3. Extra subset selection baselines. [Reviewer 7NQc Q2]
4. New experiments on ADE20k and MSCOCO to show the performance on modern datasets and networks. [Reviewer yLeE Q2]


We believe these new additions help address reviewer concerns and issues. We thank the reviewers' time and feedback in improving the quality of our work and we hope the revisions further highlight the contributions made. Before answering the point-wise questions raised by reviewers, we first provide the responses to some common concerns, which we believe help reviewers to better position our work and understand our novelty and strengths.


**Response to common concerns.**

**Q1: [Concerns about the novelty and method simplicity.]**

- The main novelty of our work lies in the brand-new data selection pipeline. Existing active learning methods need repetitive model traning and batch data selection, while other subset selection algorithm also requires to train the model on a small initial subset of the target dataset. The problem is that training models is costly, and the size of subset for training directly affects their performance of data selection. In contrast, our paper designs a novel pipeline to directly exploit general pretrained models to select all the samples from a large data pool in one go. In this case, no training on the target dataset is needed, and data selection time goes down from days to minutes, almost negligible, which means large-scale data selection becomes doable.
- Although our method is notably straightforward without intricate modules, we believe it's unnecessary to inject excessively complex designs to artificially make it "looks novel". In contrast, it is always desirable that a simple method can finish the task well, especially given that our main novelty lies in the pipeline design. This is verified by our experiment results. If a simple method can rival well-designed existing algorithms in performance, it unquestionably becomes a compelling choice.



**Q2: [Concerns about the performance and comparison with SOTA active learning methods.]**
- In comparison with SOTA active learning algorithms, we can outperform most of them but underperform some of them in the late stage. Existing active learning methods train their models on the selected samples after each batch selection step. As a result, as the sample number increases, their models have been trained on a large subset of the target dataset, so it should naturally fit the target domain well. In contrast, our method makes use of general pretrained knowledge without any domain-specific training regarding the target dataset. That's why existing methods can fit the domain of the targeted dataset better after seeing many training samples in a later stage. This is reasonable, and our model therefore has wider applicability, such as scenarios where data is too scarce to do batch selection or resources are severely constrained. We believe that the early stage is also critical as training budgets and data vary.
- Nonetheless, we assert that performance isn't the sole metric of consideration. We aim to develop a "free" data selection algorithm. We can directly exploit public pretrained model and only need negligible extra efforts for data selection, reducing the data selection time from days to minutes as well as simplifying the data selection pipeline greatly (Line 249-257). In many scenarios, such as selecting data from an extremely large data pool, we believe this advantage of "free" far outweighs minor disparities in performance.


Please let us know if any clarification or additional experiments would further strengthen the paper. We would be happy to incorporate all these suggestions in the final version. Thank you again for your time and efforts!

Meta-review: The paper titled "Towards Free Data Selection with General-Purpose Models" proposes a method called FreeSel for data selection in active learning using a pretrained model. The method extracts features from a pretrained visual transformer model, clusters the features into semantic patterns, and uses a distance-based sampling framework for selecting informative samples. The paper highlights the efficiency of FreeSel, showcasing significant time savings compared to traditional active learning methods. The proposed method is evaluated across various visual tasks and datasets. 

Reviewer xhjx acknowledges the time-saving advantage of the proposed method. However, the reviewer raises concerns about the effectiveness of the method, pointing out that it is outperformed by competing methods in later stages of active learning. Additionally, the reviewer questions the method's performance under large domain shifts and notes the absence of benchmarking against certain sota techniques. In response, the authors emphasizes on the applicability in scenarios with tight resource constraints. They argue that the effectiveness of data selection is not limited to the late stage and can be valuable for cases where budgets vary significantly or data is abundant. The authors also highlight that their main innovation lies in the pipeline design, enabling large-scale data selection with minimal extra effort. Reviewer xhjx was not convinced and maintains their initial rating as they deem the contribution and significance is not enough.

Reviewer SbPr provides a positive rating (6). They observe a decrease in the method's performance as the number of samples increases, and they inquire whether the model is sensitive to dataset size. They also inquire about the determination of the variable "c," which represents the number of pseudo categories, and its role in the method. Additionally, the reviewer questions the method's disregard for global features and how this approach might affect performance. In response, the authors explain that the performance decrease and demonstrate that their method is not sensitive to dataset size. Regarding the use of local features, the authors mention that their method still considers global features when filtering regions based on attention maps. 

Reviewer 7NQc also provide a final positive rating (6). The primary concern raised is regarding the positioning of the paper. The reviewer suggests that the paper aligns more with subset selection than with traditional active learning (AL), and direct comparisons with AL methods might not be suitable. In response, the authors address the concern by including more details about their ablation studies, showcasing the limitations of other subset selection methods in their setting. The authors also clarify the conflation between the method and the problem setting, indicating that the principle of non-supervision applies to the problem setting. Reviewer recommends that the authors further refine the paper by adding more comparisons with relevant approaches. The reviewer also suggests that the authors explicitly position their work within the field's broader context and consider new baselines and comparisons to strengthen the paper's contributions and distinction.

Finally, reviewer yLeE suggest including results on more recent datasets and using modern backbones for evaluation. The reviewer points out that applying FreeSel to select data from web or unannotated sources would provide a stronger demonstration of effectiveness. In response, the authors provide additional experimental results on MS-COCO and ADE20k datasets with different models and backbones. They explain the limitations of performing experiments on web or unannotated data due to resource constraints but express interest in considering this for future work. Reviewers yLeE kept their negative score but did not provide any further justification.

Taking into account the contrasting reviews, the ACs conducted a thorough examination of the paper and concluded that the problem setting itself holds merit, warranting a recommendation for acceptance. However, it is evident that the paper requires significant refinement, particularly in terms of providing a better positioning and a broader context. On a final note, the ACs were reminded of a paper authored by Hou et al titled "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts." This work employs a strategy similar to the proposed framework, termed "active labeling," indicating that the proposed general framework could potentially find widespread utility across diverse domains. This observation gains even more significance given the current landscape dominated by large foundational models. In this context, the authors are encouraged to amplify the prominence of this message through improved writing.
Decision: Accept (poster)