Title: Towards Free Data Selection with General-Purpose Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: data selection

unsupervised learning
Abstract: A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. However, current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. In this paper, we challenge this status quo by designing a distinct data selection pipeline that utilizes existing general-purpose models to select data from various datasets with a single-pass inference without the need for additional training or supervision. A novel free data selection (FreeSel) method is proposed following this new pipeline. Specifically, we define semantic patterns extracted from inter-mediate features of the general-purpose model to capture subtle local information in each image. We then enable the selection of all data samples in a single pass through distance-based sampling at the fine-grained semantic pattern level. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods. Extensive experiments verify the effectiveness of FreeSel on various computer vision tasks.

Towards Free Data Selection with General-Purpose Models

Abstract

A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. However, current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. In this paper, we challenge this status quo by designing a distinct data selection pipeline that utilizes existing general-purpose models to select data from various datasets with a single-pass inference without the need for additional training or supervision. A novel free data selection (FreeSel) method is proposed following this new pipeline. Specifically, we define semantic patterns extracted from intermediate features of the general-purpose model to capture subtle local information in each image. We then enable the selection of all data samples in a single pass through distance-based sampling at the fine-grained semantic pattern level. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530 × faster than existing active learning methods. Extensive experiments verify the effectiveness of FreeSel on various computer vision tasks. Our code is available at https://github.com/yichen928/FreeSel.

1 Introduction

Deep Neural Network (DNN) models have achieved remarkable progress in various tasks, benefiting from abundant training samples and labels. Unfortunately, data labeling tends to be time-consuming and costly, especially for dense prediction tasks such as object detection and semantic segmentation, where experts may spend up to 90 minutes per image [33]. As such, effectively exploiting the limited annotation budget has become a long-standing problem in the advancement of computer vision.

Many methods have been proposed to identify the most suitable samples for annotation, where the mainstream follows the active learning [43, 45] or subset selection [42] pipelines. However, both kinds of methods rely on task-specific models. As the most popular data selection strategy, active learning algorithms employ a time-consuming and computationally expensive batch selection strategy [44], as shown in Fig. 1a. Specifically, a task-specific model is first trained using a small initial set of labeled samples. Then, the model is utilized to select images within a specified batch budget size. These selected images are annotated and added to the labeled pool, after which the model is retrained or fine-tuned using all the labeled samples. This iterative process is repeated multiple times for a large unlabeled data pool. Since the selection of data is tightly coupled with the task-specific model, the entire pipeline needs to be restarted from scratch and repeated when working on different tasks or datasets. In many cases, it even requires up to several days to select sufficient samples from a medium-sized data pool ( e.g. , Core-Set [44] in Tab. 1).

Training

Selection

Repeat for multiple iterations

Label

Remove

from

Training

Unlabeled Pool

Batch Data

Selection

Repeat for multiple iterations

Label

Remove

from

General

Model

Single-Pass

Inference

General

Model

Single-Pass

Inference

Unlabeled Pool

Select all samples in one-shot

Select all samples over one-pass

In this paper, we challenge this status quo by introducing an efficient data selection pipeline that enables the selection of data within a single pass (as illustrated in Fig. 1b), therefore achieving comparable efficiency to random selection. We identify the efficiency bottleneck of data selection methods as the training of the task-specific model. Building upon insights from recent research on unsupervised learning [2, 64], we recognize that pretrained models [5, 63] possess the ability to encode the semantic information of images in a fine-grained level. This observation inspires us to integrate pretrained models into the data selection process, thereby decoupling data selection from task-specific models and leveraging the inherent diversity captured by pretrained models. By leveraging publicly available pretrained models, our pipeline incurs no additional training costs. To provide a concrete foundation for our design, we consider the following three guiding principles.

Generality: We strive for decoupling data selection from task-specific models. It is desired that a general model works on the data selection of multiple tasks or datasets.

Efficiency: The batch selection setting of active learning (Fig. 1a) is known to be timeconsuming due to its iterative nature. It is expected to be replaced with a single-pass model inference on unlabeled data pools.

Non-supervision: Annotators may not always respond in time, and the entire data selection progress may be delayed by frequent requests for labels. It is preferred that annotations are not required until the end of data selection.

In view of the above principles, we propose the first free data selection (FreeSel) method, to the best of our knowledge, satisfying all the above principles simultaneously. FreeSel selects data samples based on the diversity of local features. The features are extracted by a publicly available pretrained vision transformer [12], which is generic enough to facilitate data selection for different networks, datasets, and tasks after pretraining on large-scale datasets [11] in an unsupervised manner, e.g. , DINO [5]. We extract our newly defined semantic patterns by clustering the intermediate local features after an attention filter. The images are selected following a distance-based sampling strategy at the level of semantic patterns. In pursuit of efficiency, this whole process is finished within a single-pass model inference without any extra training. The data selection process is indeed unsupervised, which relieves the troubles of assigning responsive annotators.

As a result, our method pursues a free data selection using public pretrained models with a time efficiency close to random selection. We conduct extensive experiments on different tasks, datasets, and networks. When compared with existing active learning methods, our algorithm can achieve comparable performance with significantly advantageous efficiency.

Our contributions are three-fold. 1) We for the first time, introduce a new free data selection pipeline that adheres to three important principles of generality , efficiency , and non-supervision with negligible time costs. 2) We propose FreeSel, a novel method following our proposed pipeline. It can fill in the annotation budget in a single pass based on the inherent diversity of semantic patterns captured by pretrained models. 3) Extensive experiments on image classification, object detection, and semantic segmentation demonstrate the effectiveness of our pipeline.

2 Related Work

Active Learning. Active learning aims to choose the most suitable samples for annotation so that model performance can be optimized with a limited annotation budget. Most existing work in this

Unlabeled Pool

field [47, 57, 44, 18, 59, 60] follows a pool-based protocol, selecting samples based on the ranking of the whole dataset. There exist two mainstream sampling strategies for pool-based methods i.e. uncertainty and diversity. Uncertainty inside the model prediction reflects the difficulty of data samples, estimated by different heuristics such as probabilistic models [16, 13], entropy [26, 36], ensembles[3, 32], and loss function [57, 24]. Some other algorithms try to find the diverse subset which well represents the entire data pool. They measure the diversity with the Euclidean distance between global features [44], adversarial loss [47], or KL-divergence between local representations [1]. However, all these methods couple the data selection with a task model and require repetitive model training in the batch selection pipeline, resulting in inefficiency. Differently, our proposed pipeline selects samples through a single-pass model inference on each unlabeled pool.

Subset Selection. As another category of data selection algorithms, subset selection methods often select all the required samples in a single pass with the model trained on a labeled seed set. The subset is usually selected based on some criterion of uncertainty [27], diversity [6, 4], or their combination [42]. In contrast, our proposed pipeline needs neither extra training on the target dataset nor knowledge about the label space.

Unsupervised Learning. Both contrastive methods [17, 20, 61, 53, 25, 48, 39] and generative models [52, 19, 49] have achieved great success in unsupervised representation learning. Contrastive methods discriminate different images without using any explicit categories. In contrast, generative methods directly predict masked visual information inside images. We exploit a general pretrained model [5] to represent input images for task-agnostic data selection. As a result, we do not train models specific to each task like the traditional active learning pipeline.

Data Selection with Pretrained Models. There are some attempts to combine unsupervised pretraining and data selection. [56] selects data samples by the loss of pretext tasks, but requires different pretext tasks for different downstream tasks. [37] formulates active learning as an integer programming problem in the feature space, handling low-budget cases. [51] and [54] select samples based on the diversity of global features, targeted for semi-supervised learning and model finetuning settings respectively. Active labeling proposed in [23] is the most similar to our paper, but their method considers selective partial labeling in each sample instead of sample selection and is limited to 3D tasks with the same networks for pretraining and downstream tasks.

3 Preliminary Study: Off-the-Shelf Features for Data Selection

Active learning work [44, 1] often selects representative samples based on the features extracted by task-specific models trained separately for each task. A straightforward alternative is to use offthe-shelf features instead, which are extracted by general-purpose models pretrained on a largescale dataset. If it performs well, we can trivially improve the efficiency by eliminating the training step on each dataset.

We conduct this preliminary study on the object detection task over the PASCAL VOC dataset [14].









































Last Layer Patch Features

Pretrained

Transformer

Feature

Clustering

maintenance

ratio τ

[CLS] Token Attention Map

attention filter

Semantic Patterns

Semantic Pattern Pool

Distance-based Sampling

Selected Semantic Pattern Pool

Selected Image Pool

Task Model

single-pass over dataset

Unlabeled Image Pool

Per-Image

Operation

Consistent with our following experiments, we apply DeiT-S 2 [50] for feature extraction in data selection. The model is pretrained in either supervised or unsupervised (with DINO framework [5]) manners on ImageNet [11]. For data selection, we implement the classical Core-Set algorithm [44] over the extracted global features, i.e. the [CLS] token feature in the last layer. We use Core-Set with these features to select various numbers of training samples, and train object detection models (SSD-300 [34]) over the selected subsets.

Fig. 2 shows results in comparison with random selection. Unfortunately, we find that this naive combination of off-the-shelf features and Core-Set algorithms degrades the object detection performance, especially with relatively low sampling ratios. We consider two potential reasons for this failure: 1) Complex scenes are hard to represent globally. Images may contain multiple objects including some very small ones. It is difficult for a global feature to represent all useful details in the image. 2) K-Center selects corner cases. In the feature space, in order to cover all the data samples with a small radius, the K-Center algorithm of Core-Set tends to select all the corner cases.

The above two concerns motivate our design in Sec. 4. We represent each image with dense semantic patterns to maintain useful local information. Images are sampled based on some probability related to the distance between local semantic patterns to relieve the harmful preference for corner cases.

4 Methodology

We detail our new data selection method FreeSel, formulated in Sec. 4.1. We define a concept called semantic pattern in Sec. 4.2. Afterward, the sample selection strategy is explained in Sec. 4.3. An overview of our approach is illustrated in Fig. 3.

4.1 Formulation

We aim to select a diverse subset from the unlabeled data pool for annotation, which covers as much discriminative regional information in the original pool as possible. The regional information inside an image I is reflected by the local features f I = { f I r | r = 1 , 2 , . . . , HW } of a pretrained DNN. H,W are the height and width of the feature map. The r -th region feature f I r ∈ R K in the feature map mainly describes the r -th region of the image [62, 41]. The discriminative power of all regional features f I can be represented by countable knowledge points [31]. f I r is considered as a knowledge point w.r.t. a pseudo-category c if it is similar enough to the corresponding direction vector µ c .

c is a pseudo-category describing some specific visual patterns, e.g. an object part, which is represented by a vector µ c in the feature space. π c is the prior probability of pseudo-category c , κ c is a concentration parameter, and C d ( κ c ) is a normalizing constant.

Inversely, given knowledge points inside an image I , they can be clustered to estimate the K pseudocategories inside the image as ˆ µ I j , j = 1 , 2 , . . . , K . We define the estimation as semantic patterns in

STOP

中

Algorithm 1: Semantic Pattern Extraction

4.2 Per-Image Semantic Patterns Extraction

To estimate the pseudo-categories, we define a novel notion called semantic patterns , which are extracted from each image separately . Given a pretrained vision transformer [12], we consider its last layer features for image I as f I = { f I r } r HW =1 , where each patch corresponds to a region r .

According to Eq. 1, only a few regional features may be considered as meaningful knowledge points, while other regions are useless or even distracting. However, it is non-trivial to distill these knowledge points without any information about the pseudo-categories. To this end, we resort to the [CLS] token self-attention map of the transformer, which serves as a natural filter for regional importance even without the supervision of category information [5].

Attention Filter. For image I , the last layer [CLS] token attention map (average of multi-heads) is denoted as ca I = { ca I r ∈ R + | r = 1 , 2 , . . . , HW } , ∑ HW r =1 ca I r = 1 . We can filter the important regional features that jointly represent the most useful information in the entire image with Eq. 2.

where regions r = 1 , 2 , . . . , HW are sorted in the decreasing order of ca I r , and τ ∈ (0 , 1) is a hyper-parameter, meaning the maintenance ratio of information represented by the filtered important features. The filtered features F ( f I ) are considered as the knowledge points inside the images.

Feature Clustering. To estimate the feature vectors for pseudo-categories, we perform clustering over the filtered t knowledge points inside each image separately . Since K-Means is unreli-

able in the high-dimensional feature space (details in supplementary materials), we adopt spectral clustering instead. The self-attention map provides strong cues about the region-wise similarity inside each image. We denote the last layer attention map between patch tokens for image I as pa I = [ pa I ij ∈ R ] i,j =1 , 2 ,...,HW , ∑ HW j =1 pa I ij = 1 , ∀ i . It is more likely for nearby regions to interact with each other, so we only consider the attention between nearby patches [22].

where d ( i, j ) is the Chebyshev distance between regions i, j in the feature map. We empirically set the threshold d 0 = 2 in our experiments. Besides, we only consider the t regions after the filter in Eq. 2. In this case, we denote the new similarity matrix between patches as ̂ pa I = [ ̂ pa I ij ] i,j =1 , 2 ,...,t .

With this above t × t similarity matrix, we utilize spectral clustering algorithms [38, 55] to divide the remaining t regions after filtering (Eq. 2) into K clusters C j , j = 1 , 2 , . . . , K , each corresponding to a pseudo-category, where K is a hyper-parameter. The details of the spectral clustering algorithm are in our supplementary materials. We average the corresponding feature f r , r = 1 , 2 , . . . , t of each region r belonging to each cluster C j as follows.

where f I r ∈ F ( f I ) , r ∈ C j are local features of image I grouped into cluster j through spectral clustering. ˆ µ I = { ˆ µ I j } represents semantic patterns inside the image I . Fig. 4 visualizes some examples of ˆ µ I j . The whole process of semantic pattern extraction is shown in Alg. 1

4.3 Sample Selection with Semantic Patterns

Our main target of data selection is to make the distributions of selected samples diverse and representative in the level of local semantic patterns instead of the global feature level. This finegrained strategy guarantees that our selected subset can cover rich local visual patterns represented by different pseudo-categories, which are crucial for detection and segmentation tasks.

To this end, we adopt a distance-based sampling strategy at the semantic pattern level. The detailed algorithm is shown in Alg. 2. Given an unlabeled image pool I , this process starts from randomly selecting an initial image I 0 i.e. selecting all semantic patterns ˆ µ I 0 inside it. Then, we choose the next semantic pattern ˆ µ I j inside image I with probability in proportion to its squared distances from the nearest already selected semantic pattern (Eq. 5).

where S K is the pool of all the already selected semantic patterns. When we choose a semantic pattern ˆ µ I j , all the semantic patterns ˆ µ I inside the image I that contains ˆ µ I j are put into the selected pool S K . We use cosine distance for D ( · , · ) as analyzed in the supplementary materials. This process continues until enough images have been selected. The selection only requires semantic patterns constructed from intermediate features offline beforehand. Consequently, only a single-pass model inference without any training or supervision is required in the entire data selection pipeline.

5 Experiments

We evaluate FreeSel on object detection (Sec. 5.2), semantic segmentation (Sec. 5.3), and image classification (Sec. 5.4). The results of FreeSel are averaged over three independent selections with different random seeds . Features are extracted by the same general pretrained model for all the tasks (Sec. 5.1). We make some analysis of our proposed pipeline and method in Sec. 5.5. Finally, we examine the roles of different modules inside FreeSel in Sec. 5.6. We refer readers to supplementary materials for more implementation details, results, and ablation studies.

5.1 General Model for Feature Extraction

We adopt DeiT-S [50] (path size 16 × 16) pretrained with the unsupervised framework DINO [5] on ImageNet [11] to extract features for data selection. The same pretrained model is used for all tasks.

Algorithm 2: Distance-based Selection

Input: all semantic patterns ˆ µ = { ˆ µ j } for each image I , total annotation budget size b Output: selected image pool S

        

















































































FreeSel can also fit other frameworks as well, as shown in supplementary materials. We emphasize that this pretrained DeiT-S model is only applied to the data selection process. For the downstream tasks, we still train the convolutional task models from scratch in accordance with prior work.

5.2 Object Detection

Dataset and Task Model. We carry out experiments on PASCAL VOC [14]. In line with prior work [1, 57], we combine the training and validation sets of PASCAL VOC 2007 and 2012 as the training data pool with 16 , 551 images. The performance of task model is evaluated on PASCAL VOC 2007 test set using mAP metric. We follow previous work [57, 1] to train an SSD-300 model [34] with VGG-16 backbone [46] on the selected samples. It reaches 77 . 43 mAP with 100% training data.

Results and Comparison. We compare our performance with existing active learning methods (Fig. 5) for multiple sampling ratios. For fairness, we only include task-agnostic methods instead of those designed specifically for object detection [59, 8] which should naturally perform better. Results show that FreeSel outperforms most traditional pipeline methods and remains competitive with the best ones. Besides, all these previous methods require repetitive model training and batch selection on each target dataset separately, while FreeSel can efficiently select all samples in a single pass. Sec. 5.6 also shows that FreeSel can outperform other alternative general-purpose model baselines.

5.3 Semantic Segmentation

Dataset and Task Model. We use Cityscapes [9] dataset for semantic segmentation. This dataset is composed of 3,475 frames with pixel-level annotation of 19 object classes. We report the result using mIoU metric. We follow previous active learning research to apply DRN [58] model for this task. It reaches 62 . 95 mIoU with 100% training data.



































        























Results. We compare the performance of FreeSel with traditional active learning methods in Fig. 6. Given the domain gap between the pretraining dataset ( i.e. ImageNet) and Cityscapes, it is quite challenging to utilize the general pretrained model for data selection. However, with much higher efficiency, FreeSel still beats most traditional pipeline methods in performance.

5.4 Image Classification

Dataset and Task Model. We use CIFAR-10 [29] datasets and ResNet-18 [21] model in line with prior work [35, 57]. CIFAR-10 contains 60,000 images with size 32 × 32 (50,000 for training and 10,000 for test) belonging to 10 categories. We report the results using Top-1 Accuracy metric. The model reaches 93 . 02% Top-1 Accuracy with 100% training data on CIFAR-10.

Results. We demonstrate the results of data selection methods in Fig. 7. Our performance is compared with traditional active learning methods as well. Since image classification focuses on global information, the advantage of semantic patterns cannot be fully demonstrated. However, with most sampling ratios, FreeSel still beats all its counterparts.

5.5 Analysis

Time Efficiency Analysis. Time efficiency of data selection is crucial for its practical use. Tab. 1 shows the comparison between FreeSel and other existing counterparts. The estimation is conducted on PASCAL VOC to choose 8 , 000 samples. We follow prior papers [1, 44, 57] to use SSD [34] as the task model (same as Sec. 5.2). The time is estimated on a single NVIDIA TITAN RTX GPU. Since FreeSel directly utilizes the publicly available pretrained model instead of training models separately for each dataset, only the feature extraction, semantic pattern construction, and data selection time should be considered, i.e. 285 seconds in total. In contrast, for other active learning methods, networks are trained repetitively on each dataset. We follow [57, 1] to set their initial set size and batch selection budget both as 1 k , so their model should be trained for 7 times over subsets of size 1 k ∼ 7 k to select 8,000 samples. These previous methods have similar time efficiency, requiring about 42 hours in total. They also need to wait for the oracle for ground-truth labels after selecting each batch of data. Based on the above information, our method can be 530x faster than prior work.

Single-Pass Data Selection. Unlike prior active learning methods, FreeSel follows the new pipeline to select all the data samples in a single pass. This allows for great practical use. Firstly, it makes our method free of a random initial set. For one thing, FreeSel can bring performance gain in the lowest sampling ratio. This is beneficial in practice when the annotation budget is extremely low. For another thing, FreeSel would not suffer from the imbalanced initial set. As discussed in [47], low-quality initial sets would hurt the performance of prior active learning work significantly. Secondly, FreeSel simplifies the active learning pipeline from the iterative model training → batch data selection → batch annotation →······ to a single-pass data selection → data annotation , which saves notable efforts in the management, communication, and coordination of traditional sequential steps.

Introduction of Pretrained Model. Our proposed pipeline introduces a pretrained model (Fig. 1b) to satisfy the three principles of our new pipeline. Since the pretraining is not designed specifically for the data selection, directly using a publicly available model would not lead to extra time-cost or expense. According to Sec. 3, it is non-trivial to improve the efficiency of active learning with a

pretrained model. We further show that our great performance does not come from the pretrained model in Sec. 5.6.

Effect of Different Pretraining Algorithms. In this part, we pay attention to the effect of pretraining on the final performance of FreeSel. In addition to DeiT-S [50] pretrained with DINO framework [5] in Sec. 5.1, we also adopt two alternative pretraining frameworks MoCoV3 [7] and iBOT [63] as well as a larger DeiT-B model [50]. Those different pretrained models are applied to the data selection on PASCAL VOC dataset [14]. Same as Sec. 5.2, we train an SSD-300 model [34] on the selected samples for the object detection task. Fig. 8 demonstrates that FreeSel with different pretrained models for data selection only has marginal differences in the performance of the downstream object detection task. This result verifies that FreeSel can widely fit different pretraining algorithms. The great performance of data selection comes from our carefully designed modules in FreeSel instead of the strong representative ability of some specific pretrained models.

5.6 Ablation Study

We delve into different parts of our method. Firstly, we analyze the contribution of each module inside FreeSel to the final performance. Then, the role of the pretrained DeiT-S model is also discussed.

Each Module Contribution. Starting from the failure case in Fig. 2, modules of FreeSel are added to it one by one. Tab. 2 demonstrates the step-by-step performance improvement. This experiment is conducted on PASCAL VOC in the same setting as Sec. 5.2. The following three modules are analyzed. More quantitative analysis of hyper-parameters is available in the supplementary materials.

Feature Extraction Manner: In Sec. 3, the global feature of [CLS] token is directly used. We replace it with the proposed semantic patterns defined in Eq. 4.

Attention Filter: We apply the attention filter in Eq. 2 to filter local features.

Selection Strategy: Apart from the distance-based sampling in Eq. 5, we consider the alternative farthest-distance-sampling (FDS) w.r.t. semantic patterns, which is theoretically justified in [44] as an approximation of K-Centers. It chooses the next semantic pattern farthest from the nearest selected one as ˆ µ I j = arg max ˆ µ I j min ˆ µ ∈ s K d (ˆ µ I j , ˆ µ ) .

The failure case of Core-Set on offthe-shelf features is shown in the first line of Tab. 2. Then, we extract features by constructing semantic patterns ( K = 5 ) without applying the attention filter in the second line. It improves notably compared with the first line because the semantic patterns can represent useful local information important for object detection. However, it is only slightly better than random selection since the semantic patterns

are dominated by local noisy information in this stage. We apply attention ratio τ = 0 . 5 (Eq. 2) in the third line of the table, and the performance gets improved again. Finally, the FDS selection strategy is replaced by the distance-based probability sampling in Eq. 5. It provides extra performance gain because it would select more representative samples with fewer corner cases.

Role of Pretrained Model. There is a concern that the performance gain of FreeSel comes from the great representation ability of the pretrained vision transformer for data selection instead of our designed method. About this, we conduct an ablation study on CIFAR-10 in the same setting as Sec. 5.4. We equip Core-Set [44] and Learn-Loss [57] with the same pretrained network for data selection, i.e. DeiT-S [50] pretrained with DINO [5]. During the data selection period, pretrained DeiT-S is finetuned supervisedly to select data samples with Core-Set and Learn-Loss algorithms. After selection, we still train ResNet-18 [21] over the selected samples from scratch. In Fig. 9, this pretrained DeiT-S damages the performance of Core-Set and Learn-Loss. A potential explanation comes from the differences in the representation spaces of pretrained DeiT and from-scratch ResNet. The samples selected by DeiT-S with Core-Set and Learn-Loss algorithms may not be suitable for the

from-scratch training of the ResNet-18 model. This reflects that our performance gain does not come from the use of pretrained DeiT-S. Instead, the proposed FreeSel method plays an important role.

General-Purpose Model Baselines. To further disentangle the roles of the general-purpose model and our designed FreeSel framework, we compare FreeSel with the following baselines which can also select a subset from the data pool using the general-purpose models. 1) K-Means: We perform the K-Means algorithm on the global features extracted by the pretrained DeiT-S

model [50, 5], choosing the sample closest to each cluster center. 2) Inconsistency: We select the most difficult samples based on the inconsistency of multiple-time model predictions. To measure the inconsistency, we perform data augmentations (RandAugment [10]) to generate 10 different augmented copies for each image and calculate the average pairwise distances of global features between these copies extracted by the pretrained DeiT-S model [50, 5]. We select data samples by the order of inconsistency. 3) Entropy: We select the most ambiguous samples based on the classification uncertainty of the pretrained model. Since the classification score is required, we adopt the DeiT-S model [50] pretrained on ImageNet in a supervised manner and measure the uncertainty with the entropy of classification scores. We select data samples by the order of entropy. Experiments are conducted on object detection task in the same settings as Sec. 5.2. Tab. 3 shows that all the above baselines perform notably worse than FreeSel, especially with low sampling ratios. This reflects the importance of our proposed FreeSel algorithm. Trivial utilization of a general-purpose model would not lead to great performance of data selection.

6 Conclusion and Limitations

The main goal of this paper is to enable a free data selection pipeline by proposing a novel pipeline with three key principles: generality, efficiency, and non-supervision. We verify its feasibility by designing the first method FreeSel following this new pipeline. Through a single-pass model inference, the semantic patterns are constructed based on the intermediate features of a general pretrained model, over which a distance-based selection strategy is performed to find the most diverse and informative data samples. Our method outperforms most existing counterparts with remarkably superior efficiency on different tasks including detection, segmentation, and classification.

We realize that FreeSel cannot beat all the other data selection methods in current stage due to the absence of training on the target datasets. Never-





































theless, this direction matters in boosting the training of downstream models without any extra time and cost on the shoulders of existing general pretrained models. It gains more significance given the current landscape dominated by large foundation models pretrained on multi-modality data [40, 15], which we believe can help to extend our method to a wide range of domains and modalities.

Acknowledgement. This work is partially supported by Berkeley DeepDrive.

References

S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In European Conference on Computer Vision , pages 137-153. Springer, 2020. 3, 7, 8, 17

S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021. 2

W. H. Beluch, T. Genewein, A. Nürnberger, and J. M. Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9368-9377, 2018. 3

V. Birodkar, H. Mobahi, and S. Bengio. Semantic redundancies in image-classification datasets: The 10% you don't need. arXiv preprint arXiv:1901.11409 , 2019. 3

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294 , 2021. 2, 3, 4, 5, 6, 9, 10, 16

E. Chang, X. Shen, H.-S. Yeh, and V. Demberg. On training instance selection for few-shot neural text generation. arXiv preprint arXiv:2107.03176 , 2021. 3

X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 , 2021. 9

J. Choi, I. Elezi, H.-J. Lee, C. Farabet, and J. M. Alvarez. Active learning for deep object detection via probabilistic modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10264-10273, 2021. 7

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016. 7, 15

E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020. 10

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 2, 4, 6, 15

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 2, 5

S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. arXiv preprint arXiv:1906.02425 , 2019. 3

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010. 3, 7, 9, 15

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15180-15190, 2023. 10

M. Gorriz, A. Carlier, E. Faure, and X. Giro-i Nieto. Cost-effective active learning for melanoma segmentation. arXiv preprint arXiv:1711.09168 , 2017. 3

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 , 2020. 3

E. Haussmann, M. Fenzi, K. Chitta, J. Ivanecky, H. Xu, D. Roy, A. Mittel, N. Koumchatzky, C. Farabet, and J. M. Alvarez. Scalable active learning for object detection. In 2020 IEEE Intelligent Vehicles Symposium (IV) , pages 1430-1435. IEEE, 2020. 3

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022. 3

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020. 3

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016. 8, 9, 17

Y. He, W. Liang, D. Zhao, H.-Y. Zhou, W. Ge, Y. Yu, and W. Zhang. Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9119-9129, 2022. 6

J. Hou, B. Graham, M. Nießner, and S. Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15587-15597, 2021. 3

S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou. Semi-supervised active learning with temporal output discrepancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3, 17

S. Huang, Y. Xie, S.-C. Zhu, and Y. Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6535-6545, 2021. 3

A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 2372-2379. IEEE, 2009. 3

V. Kaushal, A. Sahoo, K. Doctor, N. Raju, S. Shetty, P. Singh, R. Iyer, and G. Ramakrishnan. Learning from less data: Diversified subset selection and active learning in image classification tasks. arXiv preprint arXiv:1805.11191 , 2018. 3

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 17

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. 8

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25:1097-1105, 2012. 16

M. Li, S. Wang, and Q. Zhang. Visualizing the emergence of intermediate visual patterns in dnns. arXiv preprint arXiv:2111.03505 , 2021. 4

X. Li and Y. Guo. Adaptive active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 859-866, 2013. 3

H. Lin, P. Upchurch, and K. Bala. Block annotation: Better image annotation with sub-image decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5290-5300, 2019. 1

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In European conference on computer vision , pages 21-37. Springer, 2016. 4, 7, 8, 9, 16, 17

Z. Liu, H. Ding, H. Zhong, W. Li, J. Dai, and C. He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9274-9283, 2021. 8, 17

D. J. MacKay. Information-based objective functions for active data selection. Neural computation , 4(4):590-604, 1992. 3

R. Mahmood, S. Fidler, and M. T. Law. Low budget active learning via wasserstein distance: An integer programming approach. arXiv preprint arXiv:2106.02968 , 2021. 3

A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems , 14, 2001. 6, 15

M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.04298 , 2020. 3

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 10

M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? In Thirty-Fifth Conference on Neural Information Processing Systems , 2021. 4

S. Ramalingam, D. Glasner, K. Patel, R. Vemulapalli, S. Jayasumana, and S. Kumar. Less is more: Selecting informative and diverse subsets with balancing constraints. arXiv preprint arXiv:2104.12835 , 2021. 1, 3

P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, X. Chen, and X. Wang. A survey of deep active learning. arXiv preprint arXiv:2009.00236 , 2020. 1

O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017. 1, 3, 4, 8, 9

B. Settles. Active learning literature survey. 2009. 1

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 7, 17

S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5972-5981, 2019. 3, 8, 17

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005.10243 , 2020. 3

Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems , 35:10078-10093, 2022. 3

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , pages 10347-10357. PMLR, 2021. 4, 6, 9, 10, 16, 17

X. Wang, L. Lian, and S. X. Yu. Unsupervised selective labeling for more effective semisupervised learning. In European Conference on Computer Vision , pages 427-445. Springer, 2022. 3

C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14668-14678, 2022. 3

T. Xiao, X. Wang, A. A. Efros, and T. Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659 , 2020. 3

Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. arXiv preprint arXiv:2303.14382 , 2023. 3

D. Yan, L. Huang, and M. I. Jordan. Fast approximate spectral clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 907-916, 2009. 6, 15

J. S. K. Yi, M. Seo, J. Park, and D.-G. Choi. Pt4al: Using self-supervised pretext tasks for active learning. In European Conference on Computer Vision , pages 596-612. Springer, 2022. 3

D. Yoo and I. S. Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 93-102, 2019. 3, 7, 8, 9, 17

F. Yu, V. Koltun, and T. Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 472-480, 2017. 7, 16, 17

T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5330-5339, 2021. 3, 7

B. Zhang, L. Li, S. Yang, S. Wang, Z.-J. Zha, and Q. Huang. State-relabeling adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8756-8765, 2020. 3

Z. Zhang, R. Girdhar, A. Joulin, and I. Misra. Self-supervised pretraining of 3d features on any point-cloud. arXiv preprint arXiv:2101.02691 , 2021. 3

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:1412.6856 , 2014. 4

J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021. 2, 9

J. Zhu, J. Qi, M. Ding, X. Chen, P. Luo, X. Wang, W. Liu, L. Wang, and J. Wang. Understanding self-supervised pretraining with part-aware representation learning. arXiv preprint arXiv:2301.11915 , 2023. 2

In this supplementary material, we first explain the details of spectral clustering algorithm (Sec. 4.2) in Sec. A. We then analyze the sensitivity of FreeSel to the values of hyperparameters in Sec. B. Finally, implementation details of our experiments are explained in Sec. C.

A Spectral Clustering Algorithm

In this section, we explain the spectral clustering algorithm [38, 55] in the semantic pattern extraction process for each image I (Sec. 4.2 and Alg. 1). The detailed spectral clustering algorithm is shown in Alg. 3. This spectral clustering algorithm should be inserted into line 7 of Alg. 1.

To justify the use of spectral clustering algorithm for semantic pattern extraction, we also try another alternative which directly performs K-Means w.r.t. the local features f I r , r = 1 , 2 , . . . , t to divide the t regions of image I into K clusters without using the patch token attention to achieve the same feature clustering goal in Sec. 4.2. Tab. 4 shows the comparison between spectral clustering and K-Means. Interestingly, these two feature clustering strategies lead to similar data selection performance on PASCAL VOC [14] object detection task. However, spectral clustering is stably superior when selecting data samples for Cityscapes [9] semantic segmentation task. We attribute this difference to the large domain gap between Cityscapes dataset and ImageNet dataset [11]. The DeiT-S model pretrained on ImageNet may extract local features with weaker discriminative ability from images inside Cityscapes dataset. Since spectral clustering algorithm depends less on the feature quality, it can bring better performance than direct K-Means over intermediate local features on Cityscapes.

Algorithm 3: Spectral Clustering

Input: Similarity matrix between patches ̂ pa = [ ̂ pa ij ] i,j =1 , 2 ,...,t , semantic pattern number K Output: Clusters C I j , j = 1 , 2 , . . . , K , where each region r = 1 , 2 , . . . , t of image I belongs to a unique C I j .

I

1 Derive the symmetric adjacent matrix A from ̂ pa I :

2 Derive the diagonal degree matrix D :

3 Calculate the normalized Laplacian matrix L :

4 Obtain the K eigenvectors v l , l = 1 , 2 , . . . , K corresponding to the K smallest eigenvalues σ l , l = 1 , 2 , . . . , K of matrix L .

5 Compose the matrix V based on the K eigenvectors

6 Denote u T i as the i -th row of V , i = 1 , 2 , . . . , t

7 Normalize each row of V : ˆ u i = u i / √ ∑ K j =1 u 2 i,j

8 Perform K-Means to divide ˆ u i , i = 1 , 2 , . . . , t into K clusters C I j , j = 1 , 2 , . . . , K :

B Sensitivity to Hyperparameters

In this part, we analyze the sensitivity of our FreeSel to some hyperparameters including the maintenance ratio τ in the attention filter (Eq. 2), the semantic pattern number K (Eq. 4), the neighborhood threshold d 0 (Eq. 3), the distance function D ( · , · ) (Eq. 5), and pretraining manner for the general

̸

(a) Performance on PASCAL VOC Object Detection Task: The task model is SSD-300 [34].

(b) Performance on Cityscapes Semantic Segmentation Task: The task model is DRN [58].

model. Experiments are conducted on the object detection task, where samples are selected from PASCAL VOC dataset and SSD-300 is the downstream task model in the same settings as Sec. 5.2. Results are shown in Tab. 5.

Maintenance Ratio τ (Eq. 2) Maintenance ratio τ notably affects the final performance of FreeSel. Too low ratios lead to the ignorance of some crucial local visual patterns, while too high ratios introduce some harmful noisy information to the semantic patterns. Thus, a moderate attention ratio plays an important role in the high performance of FreeSel.

Semantic Pattern Number K (Eq. 4) When K = 1 , the performance is hurt since semantic patterns degrade to global features in this case. When K = 10 , a slight performance drop may be witnessed in comparison with K = 5 .

Neighborhood Threshold d 0 (Eq. 3) When d 0 = 1 , the neighborhood is too small to represent the relationship between nearby regions. When d 0 = 3 , the performance is a little worse than d 0 = 2 . We think each region feature mainly interacts with nearby regions with distance d ≤ 2 .

Distance Function D ( · , · ) (Eq. 5) We find the cosine distance can lead to better performance than Euclidean distance. This result shows that the directions of local feature vectors are important to reflect the diversity of local visual patterns.

Pretraining Manner Instead of using the unsupervised pretraining framework DINO [5], we also try the DeiT-S model [50] pretrained in a supervised manner on ImageNet [30]. Results show a performance drop with supervised pretraining. We think this is because supervised pretraining introduces some biases of categories to the pretrained model.

C Implementation Details

C.1 Object Detection Implementation

C.1.1 Implementation of FreeSel

We set attention ratio τ = 0 . 5 and semantic pattern number K = 5 . The input images are resized to 224 × 224 when fed into the pretrained DeiT-S [50] model in the data selection process.

C.1.2 Implementation of Task Model

The implementation of task model is same as previous active learning research [57, 1]. The SSD-300 model [34] with VGG-16 [46] backbone is adopted for this experiment. The model is implemented based on mmdetection 3 . We follow [57, 1] to train the model for 300 epochs with batch size 32 using SGD optimizer (momentum 0 . 9 ). The initial learning rate is 0 . 001 , which decays to 0 . 0001 after 240 epochs.

C.2 Semantic Segmentation Implementation

C.2.1 Implementation of FreeSel

The input images are resized to 448 × 224 in line with their original aspect ratios when fed into the pretrained DeiT-S [50] model in the data selection process. Same as object detection, we set attention ratio τ = 0 . 5 and semantic pattern number is doubled to K = 10 in line with the doubled input size compared to object detection task.

C.2.2 Implementation of Task Model

We follow prior active learning work [47, 24] to apply DRN [58] model 4 for semantic segmentation task. The model is trained for 50 epochs with batch size 8 and learning rate 5e-4 using Adam optimizer [28].

C.3 Image Classification Implementation

C.3.1 Implementation of FreeSel

We follow previous tasks to set attention ratio τ = 0 . 5 . Since image classification depends less on local information, we directly set the semantic pattern number K = 1 . The input images are resized to 224 × 224 when fed into the pretrained DeiT-S [50] model in the data selection process.

C.3.2 Implementation of Task Model

We follow [57, 35] to use ResNet-18 [21] classification model in this task, which is implemented based on mmclassification 5 . The model is trained for 200 epochs with batch size 128 using an SGD optimizer (momentum 0.9, weight decay 5e-4). The initial learning rate is 0.1, which decays to 0.01 after 160 epochs. We apply standard data augmentation to the training including 32×32 size random crop from 36×36 zero-padded images and random horizontal flip.

Peer reviews:
comments: SUMMARY:
This paper presents a data selection methods without training, so saving lots of time than previous methods. By a publicly available pretrained model, FreeSel can select data from datasets with a single-pass inference, without need for additional training. This method uses comparison between semantic patterns of samples with distance-based sampling. Experiments on various tasks including classification, segmentation and detection verify the motivation.

STRENGTHS:
1. The proposed method requires no training to select useful examples, saving lots of time.
2. The experimental performance shows the priority about this method.

WEAKNESSES:
1. The method is plain, to compare the similarities among input sample features. Though saving time, it cannot outperform other methods. 
2. The downstream experiments are still using detection and segmentation models and backbones 5 or 6 years ago, the reviewer would like to see results on COCO, ADE20k and backbone like ViT, or DETR detectors.
3. The active learning which select samples from already annotated datasets is meaningless, more experiments about using FreeSel to select data from web or unannotated images are much more significant to show the effectiveness of this method.

QUESTIONS:
Please see weaknesses. If authors address them, reviewer would like to change the rating.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a method named FreeSel, which is designed to efficiently select information-rich data points from an unlabeled pool. These selected data points can then be annotated and utilized for training deep learning models, thereby mitigating the costs and resources needed to annotate an entire dataset. The method primarily consists of two key components: feature extraction using a large-scale pretrained model (DINO), and the construction of semantic patterns based on intermediate features. Subsequently, a distance-based selection strategy is employed to identify the most diverse and informative data samples.\n\nThe authors assert that the proposed approach addresses three critical aspects: Generality, by decoupling data selection from task-specific models; Efficiency, by enabling the selection of samples in a single pass; and Non-Supervision, by eliminating the need for annotations until the data selection process is complete.\n\nWhile the results on multiple datasets and tasks indicate that the proposed methodology holds promise, the paper seems to lack some important comparisons. Additionally, characterizing this work as an alternative to traditional active learning may not be entirely accurate.\n\nSTRENGTHS:\n1. The idea of employing a large-scale pre-trained model for data selection is interesting. This is especially pertinent in today's environment, where there is a proliferation of large-scale foundational models. Harnessing these models to select data samples for training of downstream models could prove to be highly advantageous for addressing a wide range of problems.\n2. The paper is well-structured and easy to understand\n3. The inclusion of experiments across multiple tasks and datasets provides a comprehensive view of the empirical performance of the proposed approach\n\nWEAKNESSES:\n1. The primary concern regarding this work pertains to its positioning. The paper seems to be more aligned with the domain of subset selection rather than active learning (AL).\n    * AL methods, as the name suggests, actively learn which samples to select in a manner that is specific to the model at hand. While I concur with the authors' perspective that this iterative method is resource-intensive, drawing a direct comparison between the proposed \"passive\" selection method and traditional active learning in terms of training time may not be entirely appropriate.\n\n2. There is a need for additional baselines to more effectively evaluate the merits of the proposed approach.\n    * For instance, in the absence of any iterative active learning, if a random subset of data is selected and a model is trained on it, how does the accuracy of this model compare to one trained on samples selected using FreeSel?\n\n3. There appears to be some conflation between the method (FreeSel) and the problem setting (data selection using large-scale pre-trained models). For example, the principle of Non-supervision (Line 48) seems more pertinent to the methodology than to the problem setting.\n4. To gain a clearer understanding of the novelty and contributions of this work, it would be beneficial to compare the method with existing subset selection methods.Such comparisons do not necessarily need to be empirical but could include theoretical or conceptual comparisons.\n5. It would be beneficial for the authors to clarify the positioning of FreeSel in relation to traditional active learning and to provide additional comparisons to strengthen the evaluation.\n\nReferences for consideration: \n\n1. [Kaushal et al., Learning From Less Data: Diversified Subset Selection and Active  Learning in Image Classification Tasks](https://arxiv.org/pdf/1805.11191)\n2. [Chang et al., On Training Instance Selection for Few-Shot Neural Text Generation](https://arxiv.org/pdf/2107.03176)\n3. [Birodkar et al., Semantic Redundancies in Image-Classification Datasets: The 10% You  Don't Need](https://arxiv.org/pdf/1901.11409)\n4. [Ramalingam, et al., Less is more: Selecting informative and diverse subsets with balancing  constraints](https://arxiv.org/pdf/2104.12835)\n\nQUESTIONS:\n1. What is the rationale for choosing CoreSet in Figure, there are much more recent and useful active learning strategies that are specifically proposed for the problem of object detection? \n2. No comparisons were made with the works listed in `Data Selection with Pretrained Models` paragraph of related work though they seem very relevant, is there any specific reason for this? Including comparisons with these pertinent works could enhance the comprehensiveness and depth of the evaluation.\n\nLIMITATIONS:\nAll the limitations are clearly discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to introduce a unique method called FreeSel for data selection. This method is accompanied by a newly designed pipeline that incorporates existing general-purpose models, requiring minimal additional time. By leveraging a publicly available pre-trained model, FreeSel can select data from diverse datasets through a single-pass inference, eliminating the need for extra training or supervision. Specifically, the method involves extracting semantic patterns from intermediate features of the general-purpose model to capture nuanced local information within each image. Furthermore, it enables the selection of all data samples in a single pass using distance-based sampling at a fine-grained semantic pattern level.\n\nSTRENGTHS:\n1- For the first time, this paper presents a novel model that introduces a data selection pipeline. This pipeline incorporates crucial principles of a robust feature extractor, including generality, efficiency, and non-supervision, while incurring minimal time costs. This innovative idea holds potential for application in active learning scenarios.\n\n2- This approach leverages unsupervised extraction of intermediate features by harnessing the benefits of publicly available pre-trained vision transformer (LLM) models. These features are then utilized to define semantic patterns.\n\n3- In contrast to existing active learning approaches, this method selects data examples for a given budget in a single pass, focusing on the diversity found in the local patterns extracted by the pre-trained model. This approach is considerably more time-effective compared to existing methods, as it avoids the need for multiple passes or iterations.\n\n4- To demonstrate the generalizability of the proposed approach, experiments were conducted across various types of tasks, including image classification, object detection, and semantic segmentation. The results of these experiments reveal significant performance improvements, underscoring the effectiveness of the approach across different domains and applications.\n\nWEAKNESSES:\n1- We have noticed a decrease in the performance of the proposed model as the number of samples increases, which is in contrast to several active learning approaches. We require a justification for this observation. Is the model sensitive to the size of the datasets?\n\n2- In the paper, the variable \"c\" represents the number of pseudo categories, which defines the number of visual parts in an image. However, it is unclear how \"c\" is determined. Is it a fixed value for every image, or does it vary from image to image?\n\n3- The proposed method relies solely on local features to identify semantic patterns, disregarding global features which can be crucial in certain cases. Completely disregarding global features may not be the most desirable approach.\n\nQUESTIONS:\nPlease answer all the concerns raised in the weaknesses section.\n\nLIMITATIONS:\nThe authors have mentioned the limitations of the proposed approach in the conclusion section. The authors have also presented an ablation study to discriminate the contribution of pre-trained vision transformers (LLM) and other components. It is essential to understand that the performance of the proposed model is not only due to the capability of pre-trained LLM models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a diversity-based method for active learning. Given an image, features are extracted from a pretrained visual transformer model (DeiT-S), features with [CLS] token attention above a threshold are kept and clustered into K clusters. Each cluster is represented by the average of features belonging to the cluster. The cluster center, named semantic pattern in the paper, is used in a distance-based probabilistic sampling framework (similar to kmeans++) for sample selection. The proposed method is evaluated in various visual tasks including object detection, semantic segmentation and image classification and obtained promising results.\n\nSTRENGTHS:\n1. The proposed method utilized pretrained model for feature extraction and sample selection, eliminating the need to train a model at each AL cycle and thus is much more time efficient than traditional AL methods.\n2. The method is evaluated on various tasks and datasets and shown promising results.\n\nWEAKNESSES:\n1. The effectiveness of the proposed method is questionable. \n(1) From Fig.5 and Fig.7, it can be seen that the proposed method is always outperformed by competing methods at later stage, suggesting  that the pretrained features become less effective in selecting informative samples as the training progresses. Performance at later stage is more important as active learning is only useful when your method can achieve comparable performance (e.g., 95%) of the fully-supervised baseline. \n(2) It is not clear how the method performs when the target domain has large domain shift to the pretrained domain. Currently the evaluation datasets are all natural images that are similar to ImageNet. I would expect that the effectiveness of the proposed method would be undermined when the domain shift becomes large.\n(3) Benchmarking is not comprehensive, missing BADGE [1] for image classification, EnmsDivproto [2] for object detection.\n\n2. Technical contribution is limited. The proposed method combines existing techniques in a straightforward manner, lacking novelty and technical contribution. \n\n\n[1] Ash, Jordan T., et al. \"Deep batch active learning by diverse, uncertain gradient lower bounds.\" ICLR2020.\n[2] Wu, Jiaxi, Jiaxin Chen, and Di Huang. \"Entropy-based active learning for object detection with progressive diversity constraint.\" CVPR2022.\n\nQUESTIONS:\nThe proposed method is essentially doing class-wise aggregation to generate feature representation for an image, which has been explored in CDAL. Apart from the obvious difference that CDAL can obtain pseudo label directly from the target classifier while this paper need to employ clustering to generate pseudo labels, what are other differences and advantages of the proposed method compared to CDAL?\n\nLIMITATIONS:\nThe authors discuss the limitation of the proposed method (i.e., cannot beat all state-of-the-art methods) in Section 6 and leave it for future work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}