SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yichen Xie, Mingyu Ding*, Masayoshi Tomizuka, Wei Zhan
{yichen_xie, myding, tomizuka, wzhan}@berkeley.edu
A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. However, current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. In this paper, we challenge this status quo by designing a distinct data selection pipeline that utilizes existing general-purpose models to select data from various datasets with a single-pass inference without the need for additional training or supervision. A novel free data selection (FreeSel) method is proposed following this new pipeline. Specifically, we define semantic patterns extracted from intermediate features of the general-purpose model to capture subtle local information in each image. We then enable the selection of all data samples in a single pass through distance-based sampling at the fine-grained semantic pattern level. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being  $530 \times$  faster than existing active learning methods. Extensive experiments verify the effectiveness of FreeSel on various computer vision tasks. Our code is available at https://github.com/yichen928/FreeSel.
Deep Neural Network (DNN) models have achieved remarkable progress in various tasks, benefiting from abundant training samples and labels. Unfortunately, data labeling tends to be time-consuming and costly, especially for dense prediction tasks such as object detection and semantic segmentation, where experts may spend up to 90 minutes per image [33]. As such, effectively exploiting the limited annotation budget has become a long-standing problem in the advancement of computer vision.
Many methods have been proposed to identify the most suitable samples for annotation, where the mainstream follows the active learning [43, 45] or subset selection [42] pipelines. However, both kinds of methods rely on task-specific models. As the most popular data selection strategy, active learning algorithms employ a time-consuming and computationally expensive batch selection strategy [44], as shown in Fig. 1a. Specifically, a task-specific model is first trained using a small initial set of labeled samples. Then, the model is utilized to select images within a specified batch budget size. These selected images are annotated and added to the labeled pool, after which the model is retrained or fine-tuned using all the labeled samples. This iterative process is repeated multiple times for a large unlabeled data pool. Since the selection of data is tightly coupled with the task-specific model, the entire pipeline needs to be restarted from scratch and repeated when working on different tasks or datasets. In many cases, it even requires up to several days to select sufficient samples from a medium-sized data pool (e.g., Core-Set [44] in Tab. 1).
(a) Active Learning Pipeline: It follows the batchselection strategy with iterative training.
(b) Our Free Data Selection Pipeline: Samples are selected in a single-pass without extra model training.
Figure 1: Comparisons between active learning pipeline and our proposed free selection pipeline.
In this paper, we challenge this status quo by introducing an efficient data selection pipeline that enables the selection of data within a single pass (as illustrated in Fig. 1b), therefore achieving comparable efficiency to random selection. We identify the efficiency bottleneck of data selection methods as the training of the task-specific model. Building upon insights from recent research on unsupervised learning [2, 64], we recognize that pretrained models [5, 63] possess the ability to encode the semantic information of images in a fine-grained level. This observation inspires us to integrate pretrained models into the data selection process, thereby decoupling data selection from task-specific models and leveraging the inherent diversity captured by pretrained models. By leveraging publicly available pretrained models, our pipeline incurs no additional training costs. To provide a concrete foundation for our design, we consider the following three guiding principles.
- Generality: We strive for decoupling data selection from task-specific models. It is desired that a general model works on the data selection of multiple tasks or datasets.
- Efficiency: The batch selection setting of active learning (Fig. 1a) is known to be time-consuming due to its iterative nature. It is expected to be replaced with a single-pass model inference on unlabeled data pools.
- Non-supervision: Annotators may not always respond in time, and the entire data selection progress may be delayed by frequent requests for labels. It is preferred that annotations are not required until the end of data selection.
In view of the above principles, we propose the first free data selection (FreeSel) method, to the best of our knowledge, satisfying all the above principles simultaneously. FreeSel selects data samples based on the diversity of local features. The features are extracted by a publicly available pretrained vision transformer [12], which is generic enough to facilitate data selection for different networks, datasets, and tasks after pretraining on large-scale datasets [11] in an unsupervised manner, e.g., DINO [5]. We extract our newly defined semantic patterns by clustering the intermediate local features after an attention filter. The images are selected following a distance-based sampling strategy at the level of semantic patterns. In pursuit of efficiency, this whole process is finished within a single-pass model inference without any extra training. The data selection process is indeed unsupervised, which relieves the troubles of assigning responsive annotators.
As a result, our method pursues a free data selection using public pretrained models with a time efficiency close to random selection. We conduct extensive experiments on different tasks, datasets, and networks. When compared with existing active learning methods, our algorithm can achieve comparable performance with significantly advantageous efficiency.
Our contributions are three-fold. 1) We for the first time, introduce a new free data selection pipeline that adheres to three important principles of generality, efficiency, and non-supervision with negligible time costs. 2) We propose FreeSel, a novel method following our proposed pipeline. It can fill in the annotation budget in a single pass based on the inherent diversity of semantic patterns captured by pretrained models. 3) Extensive experiments on image classification, object detection, and semantic segmentation demonstrate the effectiveness of our pipeline.
Active Learning. Active learning aims to choose the most suitable samples for annotation so that model performance can be optimized with a limited annotation budget. Most existing work in this
Table 1: Principles of Data Selection Methods: Task Model refers to the coupling between data selection and a task-specific model. Batch Selection shows whether the method repeats the data selection in batch multiple times. Multi-time Labeling denotes whether it requests ground-truth labels in the data selection process. Time estimates the approximate time to select 8000 images from PASCAL VOC datasets (Sec. 5.5).
Methods	Task Model	Batch Selection	Multi-time Labeling	Time
Core-Set [44]	✓	✓	✓	~ 42 hours
Learn-Loss [57]	✓	✓	✓	+
CDAL [1]	✓	✓	✓	label query
FreeSel (ours)	✗	✗	✗	285 s (~530× faster)
field [47, 57, 44, 18, 59, 60] follows a pool-based protocol, selecting samples based on the ranking of the whole dataset. There exist two mainstream sampling strategies for pool-based methods i.e. uncertainty and diversity. Uncertainty inside the model prediction reflects the difficulty of data samples, estimated by different heuristics such as probabilistic models [16, 13], entropy [26, 36], ensembles[3, 32], and loss function [57, 24]. Some other algorithms try to find the diverse subset which well represents the entire data pool. They measure the diversity with the Euclidean distance between global features [44], adversarial loss [47], or KL-divergence between local representations [1]. However, all these methods couple the data selection with a task model and require repetitive model training in the batch selection pipeline, resulting in inefficiency. Differently, our proposed pipeline selects samples through a single-pass model inference on each unlabeled pool.
Subset Selection. As another category of data selection algorithms, subset selection methods often select all the required samples in a single pass with the model trained on a labeled seed set. The subset is usually selected based on some criterion of uncertainty [27], diversity [6, 4], or their combination [42]. In contrast, our proposed pipeline needs neither extra training on the target dataset nor knowledge about the label space.
Unsupervised Learning. Both contrastive methods [17, 20, 61, 53, 25, 48, 39] and generative models [52, 19, 49] have achieved great success in unsupervised representation learning. Contrastive methods discriminate different images without using any explicit categories. In contrast, generative methods directly predict masked visual information inside images. We exploit a general pretrained model [5] to represent input images for task-agnostic data selection. As a result, we do not train models specific to each task like the traditional active learning pipeline.
Data Selection with Pretrained Models. There are some attempts to combine unsupervised pretraining and data selection. [56] selects data samples by the loss of pretext tasks, but requires different pretext tasks for different downstream tasks. [37] formulates active learning as an integer programming problem in the feature space, handling low-budget cases. [51] and [54] select samples based on the diversity of global features, targeted for semi-supervised learning and model finetuning settings respectively. Active labeling proposed in [23] is the most similar to our paper, but their method considers selective partial labeling in each sample instead of sample selection and is limited to 3D tasks with the same networks for pretraining and downstream tasks.
3 Preliminary Study: Off-the-Shelf Features for Data Selection
Active learning work [44, 1] often selects representative samples based on the features extracted by task-specific models trained separately for each task. A straightforward alternative is to use off-the-shelf features instead, which are extracted by general-purpose models pretrained on a large-scale dataset. If it performs well, we can trivially improve the efficiency by eliminating the training step on each dataset.
We conduct this preliminary study on the object detection task over the PASCAL VOC dataset [14].
Figure 2: Core-Set over Off-the-Shelf Features
Figure 3: Overview of Our Proposed FreeSel: Our method uses a general pretrained vision transformer to extract features from images. Semantic patterns are derived from the intermediate features. Afterwards, we perform a distance-based sampling algorithm to select semantic patterns as well as the associated images. These selected images are labeled for downstream task model training.
Consistent with our following experiments, we
apply DeiT-S [50] for feature extraction in data selection. The model is pretrained in either supervised or unsupervised (with DINO framework [5]) manners on ImageNet [11]. For data selection, we implement the classical Core-Set algorithm [44] over the extracted global features, i.e. the [CLS] token feature in the last layer. We use Core-Set with these features to select various numbers of training samples, and train object detection models (SSD-300 [34]) over the selected subsets.
Fig. 2 shows results in comparison with random selection. Unfortunately, we find that this naive combination of off-the-shelf features and Core-Set algorithms degrades the object detection performance, especially with relatively low sampling ratios. We consider two potential reasons for this failure: 1) Complex scenes are hard to represent globally. Images may contain multiple objects including some very small ones. It is difficult for a global feature to represent all useful details in the image. 2) K-Center selects corner cases. In the feature space, in order to cover all the data samples with a small radius, the K-Center algorithm of Core-Set tends to select all the corner cases.
The above two concerns motivate our design in Sec. 4. We represent each image with dense semantic patterns to maintain useful local information. Images are sampled based on some probability related to the distance between local semantic patterns to relieve the harmful preference for corner cases.
We detail our new data selection method FreeSel, formulated in Sec. 4.1. We define a concept called semantic pattern in Sec. 4.2. Afterward, the sample selection strategy is explained in Sec. 4.3. An overview of our approach is illustrated in Fig. 3.
We aim to select a diverse subset from the unlabeled data pool for annotation, which covers as much discriminative regional information in the original pool as possible. The regional information inside an image  $I$  is reflected by the local features  $\mathbf{f}^I = \{f_r^I | r = 1,2,\dots ,HW\}$  of a pretrained DNN.  $H,W$  are the height and width of the feature map. The  $r$ -th region feature  $f_{r}^{I}\in \mathbb{R}^{K}$  in the feature map mainly describes the  $r$ -th region of the image [62, 41]. The discriminative power of all regional features  $\mathbf{f}^I$  can be represented by countable knowledge points [31].  $f_{r}^{I}$  is considered as a knowledge point w.r.t. a pseudo-category  $c$  if it is similar enough to the corresponding direction vector  $\mu_c$ .
$$
p (c | f _ {r} ^ {I}) = \frac {\pi_ {c} \cdot p _ {v M F} (f _ {r} ^ {I} | c)}{\sum_ {c ^ {\prime}} \pi_ {c ^ {\prime}} \cdot p _ {v M F} (f _ {r} ^ {I} | c ^ {\prime})} > \tau , \quad p _ {v M F} (f | c) = C _ {d} (\kappa_ {c}) \cdot \exp (\kappa_ {c} \cdot \cos (f _ {r} ^ {I}, \mu_ {c})) \tag {1}
$$
$c$  is a pseudo-category describing some specific visual patterns, e.g. an object part, which is represented by a vector  $\mu_c$  in the feature space.  $\pi_c$  is the prior probability of pseudo-category  $c$ ,  $\kappa_c$  is a concentration parameter, and  $C_d(\kappa_c)$  is a normalizing constant.
Inversely, given knowledge points inside an image  $I$ , they can be clustered to estimate the  $K$  pseudocategories inside the image as  $\hat{\mu}_j^I,j = 1,2,\ldots ,K$ . We define the estimation as semantic patterns in
2We follow the name of networks in [50] in our paper. DeiT-S is also called ViT-small in [5].
Figure 4: Visualization of Semantic Patterns: Every two images are considered as a group. Left: The filtered local features (dots) of each image are grouped into semantic patterns (arrows). Gray features are eliminated in Eq. 2. Dimensions are reduced by PCA for visualization. Right: Regions inside images can be associated with corresponding local features and then semantic patterns.
Algorithm 1: Semantic Pattern Extraction
Input: Pretrained transformer  $g$  unlabeled image pool  $\mathcal{I}$  , attention ratio  $\tau$  , centroid number  $K$  Output: Semantic patterns  $\hat{\mu}^I = \{\hat{\mu}_j^I\} ,I\in \mathcal{I}$    
1 for  $I\in \mathcal{I}$  do   
2  $\mathbf{ca}^I,\mathbf{pa}^I,\mathbf{f}^I = g(I)$  /\* last layer [CLS] and patch token attention and patch-wise features. \*/   
3 Sort  $ca_{r}^{I},pa_{r}^{I},f_{r}^{I},r = 1,2,\ldots ,HW$  in the decreasing order of attention  $ca_r^I$    
4  $F(\mathbf{f}^I) = \{f_r^I |r = 1,2,\dots,t,\sum_{j = 1}^t ca_j^I\leq \tau <  \sum_{j = 1}^{t + 1}ca_j^I\}$  /\* filter important regions based on the sorted attention (Eq. 2). \*/   
5 Derive local similarity  $\widehat{pa}_{ij}$  with Eq. 3 /\* ignore attention between faraway regions. \*/   
6  $\widehat{\mathbf{pa}}^I = [\widehat{pa}_{ij}^I ]_{i,j = 1,2,\dots,t}$  /\* only consider filtered t regions. \*/   
7  $\{C_j^I\}_{j = 1}^K =$  SpectralCluster(  $\widehat{\mathbf{pa}}^I,K$  /\* divide t regions into  $K$  clusters with spectral clustering algorithm. \*/   
8  $\hat{\mu}_j^I = \frac{1}{|C_j|}\sum_{r\in C_j}f_r^I,\quad j = 1,2,\ldots ,K$  /\* calculate the representation of each semantic pattern. \*/
Sec. 4.2. To ensure the diversity of our selection, our algorithm desires to find a subset of images  $S_{\mathcal{I}}$  in Sec. 4.3, whose semantic patterns  $\bigcup_{I \in S_{\mathcal{I}}} \{\hat{\mu}_j^I\}_{j=1}^K$  can be representative in the unlabeled pool.
4.2 Per-Image Semantic Patterns Extraction
To estimate the pseudo-categories, we define a novel notion called semantic patterns, which are extracted from each image separately. Given a pretrained vision transformer [12], we consider its last layer features for image  $I$  as  $\mathbf{f}^I = \{f_r^I\}_{r=1}^{HW}$ , where each patch corresponds to a region  $r$ .
According to Eq. 1, only a few regional features may be considered as meaningful knowledge points, while other regions are useless or even distracting. However, it is non-trivial to distill these knowledge points without any information about the pseudo-categories. To this end, we resort to the [CLS] token self-attention map of the transformer, which serves as a natural filter for regional importance even without the supervision of category information [5].
Attention Filter. For image  $I$ , the last layer [CLS] token attention map (average of multi-heads) is denoted as  $\mathbf{ca}^I = \{ca_r^I \in \mathbb{R}^+ | r = 1, 2, \ldots, HW\}$ ,  $\sum_{r=1}^{HW} ca_r^I = 1$ . We can filter the important regional features that jointly represent the most useful information in the entire image with Eq. 2.
$$
F \left(\mathbf {f} ^ {I}\right) = \left\{f _ {r} ^ {I} \mid r = 1, 2, \dots , t, \sum_ {j = 1} ^ {t} c a _ {j} ^ {I} \leq \tau <   \sum_ {j = 1} ^ {t + 1} c a _ {j} ^ {I} \right\} \tag {2}
$$
where regions  $r = 1,2,\ldots ,HW$  are sorted in the decreasing order of  $ca_r^I$ , and  $\tau \in (0,1)$  is a hyper-parameter, meaning the maintenance ratio of information represented by the filtered important features. The filtered features  $F(\mathbf{f}^{I})$  are considered as the knowledge points inside the images.
Feature Clustering. To estimate the feature vectors for pseudo-categories, we perform clustering over the filtered  $t$  knowledge points inside each image separately. Since K-Means is unreliable
able in the high-dimensional feature space (details in supplementary materials), we adopt spectral clustering instead. The self-attention map provides strong cues about the region-wise similarity inside each image. We denote the last layer attention map between patch tokens for image  $I$  as  $\mathbf{pa}^I = [pa_{ij}^I\in \mathbb{R}]_{i,j = 1,2,\dots ,HW},\sum_{j = 1}^{HW}pa_{ij}^I = 1,\forall i.$  It is more likely for nearby regions to interact with each other, so we only consider the attention between nearby patches [22].
$$
\widehat {p a} _ {i j} ^ {I} = \left\{ \begin{array}{l l} p a _ {i j} ^ {I} & d (i, j) \leq d _ {0} \\ 0 & d (i, j) > d _ {0} \end{array} \right. \tag {3}
$$
where  $d(i,j)$  is the Chebyshev distance between regions  $i,j$  in the feature map. We empirically set the threshold  $d_0 = 2$  in our experiments. Besides, we only consider the  $t$  regions after the filter in Eq. 2. In this case, we denote the new similarity matrix between patches as  $\widehat{\mathbf{pa}}^I = \left[\widehat{pa}_{ij}^I\right]_{i,j=1,2,\ldots,t}$ .
With this above  $t \times t$  similarity matrix, we utilize spectral clustering algorithms [38, 55] to divide the remaining  $t$  regions after filtering (Eq. 2) into  $K$  clusters  $C_j$ ,  $j = 1, 2, \ldots, K$ , each corresponding to a pseudo-category, where  $K$  is a hyper-parameter. The details of the spectral clustering algorithm are in our supplementary materials. We average the corresponding feature  $f_r$ ,  $r = 1, 2, \ldots, t$  of each region  $r$  belonging to each cluster  $C_j$  as follows.
$$
\hat {\mu} _ {j} ^ {I} = \frac {1}{| C _ {j} |} \sum_ {r \in C _ {j}} f _ {r} ^ {I}, \quad j = 1, 2, \dots , K \tag {4}
$$
where  $f_{r}^{I}\in F(\mathbf{f}^{I}),r\in C_{j}$  are local features of image  $I$  grouped into cluster  $j$  through spectral clustering.  $\hat{\mu}^I = \{\hat{\mu}_j^I\}$  represents semantic patterns inside the image  $I$ . Fig. 4 visualizes some examples of  $\hat{\mu}_j^I$ . The whole process of semantic pattern extraction is shown in Alg. 1
4.3 Sample Selection with Semantic Patterns
Our main target of data selection is to make the distributions of selected samples diverse and representative in the level of local semantic patterns instead of the global feature level. This fine-grained strategy guarantees that our selected subset can cover rich local visual patterns represented by different pseudo-categories, which are crucial for detection and segmentation tasks.
To this end, we adopt a distance-based sampling strategy at the semantic pattern level. The detailed algorithm is shown in Alg. 2. Given an unlabeled image pool  $\mathcal{I}$ , this process starts from randomly selecting an initial image  $I_0$  i.e. selecting all semantic patterns  $\hat{\mu}^{I_0}$  inside it. Then, we choose the next semantic pattern  $\hat{\mu}_j^I$  inside image  $I$  with probability in proportion to its squared distances from the nearest already selected semantic pattern (Eq. 5).
$$
p \left(\hat {\mu} _ {j} ^ {I}\right) \propto \min  _ {\hat {\mu} \in S _ {\mathcal {K}}} \left[ D \left(\hat {\mu} _ {j} ^ {I}, \hat {\mu}\right) \right] ^ {2}, I \in \mathcal {I}, j = 1, 2, \dots , K \tag {5}
$$
where  $S_{\mathcal{K}}$  is the pool of all the already selected semantic patterns. When we choose a semantic pattern  $\hat{\mu}_j^I$ , all the semantic patterns  $\hat{\mu}^I$  inside the image  $I$  that contains  $\hat{\mu}_j^I$  are put into the selected pool  $S_{\mathcal{K}}$ . We use cosine distance for  $D(\cdot, \cdot)$  as analyzed in the supplementary materials. This process continues until enough images have been selected. The selection only requires semantic patterns constructed from intermediate features offline beforehand. Consequently, only a single-pass model inference without any training or supervision is required in the entire data selection pipeline.
We evaluate FreeSel on object detection (Sec. 5.2), semantic segmentation (Sec. 5.3), and image classification (Sec. 5.4). The results of FreeSel are averaged over three independent selections with different random seeds. Features are extracted by the same general pretrained model for all the tasks (Sec. 5.1). We make some analysis of our proposed pipeline and method in Sec. 5.5. Finally, we examine the roles of different modules inside FreeSel in Sec. 5.6. We refer readers to supplementary materials for more implementation details, results, and ablation studies.
5.1 General Model for Feature Extraction
We adopt DeiT-S [50] (path size  $16 \times 16$ ) pretrained with the unsupervised framework DINO [5] on ImageNet [11] to extract features for data selection. The same pretrained model is used for all tasks.
Algorithm 2: Distance-based Selection
Input: all semantic patterns  $\hat{\mu}^I = \{\hat{\mu}_j^I\}$  for each image  $I$ , total annotation budget size  $b$   
Output: selected image pool  $S_{\mathcal{I}}$   
1 Initialize  $S_{\mathcal{I}} = \{I_0\}$   
/* initialize with a random image  $I_0$   
2 Initialize  $S_{\mathcal{K}} = \{\hat{\mu}_j^{I_0}, j = 1, \dots, K\}$   
/* initialize selected semantic pattern pool with all semantic patterns in  $I_0$   
3 repeat  
4 Sample  $\hat{\mu}_j^I$  with probability  $p(\hat{\mu}_j^I) \propto \min_{\hat{\mu} \in S_{\mathcal{K}}} [D(\hat{\mu}_j^I, \hat{\mu})]^2$   
5 /* sample next semantic pattern  $\hat{\mu}_j^I$  with the distance-based probability.  
6 /* add image  $I$  containing sampled  $\hat{\mu}_j^I$  to selected image pool  
7 /* add all semantic patterns in image  $I$  to selected semantic pattern pool  
8 until  $|S_{\mathcal{I}}| = b$ ;
Figure 5: Results on Object Detection: The mAP on  $100\%$  training data is 77.43.