SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Results on Semantic Segmentation: The mIoU on  $100\%$  training data is 62.95.
FreeSel can also fit other frameworks as well, as shown in supplementary materials. We emphasize that this pretrained DeiT-S model is only applied to the data selection process. For the downstream tasks, we still train the convolutional task models from scratch in accordance with prior work.
5.2 Object Detection
Dataset and Task Model. We carry out experiments on PASCAL VOC [14]. In line with prior work [1, 57], we combine the training and validation sets of PASCAL VOC 2007 and 2012 as the training data pool with 16,551 images. The performance of task model is evaluated on PASCAL VOC 2007 test set using  $mAP$  metric. We follow previous work [57, 1] to train an SSD-300 model [34] with VGG-16 backbone [46] on the selected samples. It reaches  $77.43$  mAP with  $100\%$  training data.
Results and Comparison. We compare our performance with existing active learning methods (Fig. 5) for multiple sampling ratios. For fairness, we only include task-agnostic methods instead of those designed specifically for object detection [59, 8] which should naturally perform better. Results show that FreeSel outperforms most traditional pipeline methods and remains competitive with the best ones. Besides, all these previous methods require repetitive model training and batch selection on each target dataset separately, while FreeSel can efficiently select all samples in a single pass. Sec. 5.6 also shows that FreeSel can outperform other alternative general-purpose model baselines.
5.3 Semantic Segmentation
Dataset and Task Model. We use Cityscapes [9] dataset for semantic segmentation. This dataset is composed of 3,475 frames with pixel-level annotation of 19 object classes. We report the result using mIoU metric. We follow previous active learning research to apply DRN [58] model for this task. It reaches  $62.95\mathrm{mIoU}$  with  $100\%$  training data.
Figure 7: Results on Image Classification: The accuracy on  $100\%$  training data is  $93.02\%$ .
Figure 8: Effect of Pretraining Methods: Experiments are conducted on PASCAL VOC.
Results. We compare the performance of FreeSel with traditional active learning methods in Fig. 6. Given the domain gap between the pretraining dataset (i.e. ImageNet) and Cityscapes, it is quite challenging to utilize the general pretrained model for data selection. However, with much higher efficiency, FreeSel still beats most traditional pipeline methods in performance.
5.4 Image Classification
Dataset and Task Model. We use CIFAR-10 [29] datasets and ResNet-18 [21] model in line with prior work [35, 57]. CIFAR-10 contains 60,000 images with size  $32 \times 32$  (50,000 for training and 10,000 for test) belonging to 10 categories. We report the results using Top-1 Accuracy metric. The model reaches  $93.02\%$  Top-1 Accuracy with  $100\%$  training data on CIFAR-10.
Results. We demonstrate the results of data selection methods in Fig. 7. Our performance is compared with traditional active learning methods as well. Since image classification focuses on global information, the advantage of semantic patterns cannot be fully demonstrated. However, with most sampling ratios, FreeSel still beats all its counterparts.
5.5 Analysis
Time Efficiency Analysis. Time efficiency of data selection is crucial for its practical use. Tab. 1 shows the comparison between FreeSel and other existing counterparts. The estimation is conducted on PASCAL VOC to choose 8,000 samples. We follow prior papers [1, 44, 57] to use SSD [34] as the task model (same as Sec. 5.2). The time is estimated on a single NVIDIA TITAN RTX GPU. Since FreeSel directly utilizes the publicly available pretrained model instead of training models separately for each dataset, only the feature extraction, semantic pattern construction, and data selection time should be considered, i.e. 285 seconds in total. In contrast, for other active learning methods, networks are trained repetitively on each dataset. We follow [57, 1] to set their initial set size and batch selection budget both as  $1k$ , so their model should be trained for 7 times over subsets of size  $1k \sim 7k$  to select 8,000 samples. These previous methods have similar time efficiency, requiring about 42 hours in total. They also need to wait for the oracle for ground-truth labels after selecting each batch of data. Based on the above information, our method can be  $530x$  faster than prior work.
Single-Pass Data Selection. Unlike prior active learning methods, FreeSel follows the new pipeline to select all the data samples in a single pass. This allows for great practical use. Firstly, it makes our method free of a random initial set. For one thing, FreeSel can bring performance gain in the lowest sampling ratio. This is beneficial in practice when the annotation budget is extremely low. For another thing, FreeSel would not suffer from the imbalanced initial set. As discussed in [47], low-quality initial sets would hurt the performance of prior active learning work significantly. Secondly, FreeSel simplifies the active learning pipeline from the iterative model training  $\rightarrow$  batch data selection  $\rightarrow$  batch annotation  $\rightarrow$  …… to a single-pass data selection  $\rightarrow$  data annotation, which saves notable efforts in the management, communication, and coordination of traditional sequential steps.
Introduction of Pretrained Model. Our proposed pipeline introduces a pretrained model (Fig. 1b) to satisfy the three principles of our new pipeline. Since the pretraining is not designed specifically for the data selection, directly using a publicly available model would not lead to extra time-cost or expense. According to Sec. 3, it is non-trivial to improve the efficiency of active learning with a
pretrained model. We further show that our great performance does not come from the pretrained model in Sec. 5.6.
Effect of Different Pretraining Algorithms. In this part, we pay attention to the effect of pretraining on the final performance of FreeSel. In addition to DeiT-S [50] pretrained with DINO framework [5] in Sec. 5.1, we also adopt two alternative pretraining frameworks MoCoV3 [7] and iBOT [63] as well as a larger DeiT-B model [50]. Those different pretrained models are applied to the data selection on PASCAL VOC dataset [14]. Same as Sec. 5.2, we train an SSD-300 model [34] on the selected samples for the object detection task. Fig. 8 demonstrates that FreeSel with different pretrained models for data selection only has marginal differences in the performance of the downstream object detection task. This result verifies that FreeSel can widely fit different pretraining algorithms. The great performance of data selection comes from our carefully designed modules in FreeSel instead of the strong representative ability of some specific pretrained models.
We delve into different parts of our method. Firstly, we analyze the contribution of each module inside FreeSel to the final performance. Then, the role of the pretrained DeiT-S model is also discussed.
Each Module Contribution. Starting from the failure case in Fig. 2, modules of FreeSel are added to it one by one. Tab. 2 demonstrates the step-by-step performance improvement. This experiment is conducted on PASCAL VOC in the same setting as Sec. 5.2. The following three modules are analyzed. More quantitative analysis of hyper-parameters is available in the supplementary materials.
- Feature Extraction Manner: In Sec. 3, the global feature of [CLS] token is directly used. We replace it with the proposed semantic patterns defined in Eq. 4.
- Attention Filter: We apply the attention filter in Eq. 2 to filter local features.
- **Selection Strategy:** Apart from the distance-based sampling in Eq. 5, we consider the alternative farthest-distance-sampling (FDS) w.r.t. semantic patterns, which is theoretically justified in [44] as an approximation of K-Centers. It chooses the next semantic pattern farthest from the nearest selected one as  $\hat{\mu}_j^I = \arg \max_{\hat{\mu}_j^I} \min_{\hat{\mu} \in s_{\mathcal{K}}} d(\hat{\mu}_j^I, \hat{\mu})$ .
The failure case of Core-Set on off-the-shelf features is shown in the first line of Tab. 2. Then, we extract features by constructing semantic patterns  $(K = 5)$  without applying the attention filter in the second line. It improves notably compared with the first line because the semantic patterns can represent useful local information important for object detection. However, it is only slightly better than random selection since the semantic patterns
Table 2: Module Contribution: We discuss the contribution of each module inside FreeSel. SP means semantic pattern. Experiments are conducted on PASCAL VOC.
Feature	Filter	Select	Image Number
3k	5k	7k
global	X	FDS	60.59	66.65	70.30
SP	X	FDS	64.15	68.22	70.42
SP	τ = 0.5	FDS	64.45	68.49	71.35
SP	τ = 0.5	Prob.	65.66	69.24	71.79
random sampling	64.21	67.53	69.32
are dominated by local noisy information in this stage. We apply attention ratio  $\tau = 0.5$  (Eq. 2) in the third line of the table, and the performance gets improved again. Finally, the FDS selection strategy is replaced by the distance-based probability sampling in Eq. 5. It provides extra performance gain because it would select more representative samples with fewer corner cases.
Role of Pretrained Model. There is a concern that the performance gain of FreeSel comes from the great representation ability of the pretrained vision transformer for data selection instead of our designed method. About this, we conduct an ablation study on CIFAR-10 in the same setting as Sec. 5.4. We equip Core-Set [44] and Learn-Loss [57] with the same pretrained network for data selection, i.e. DeiT-S [50] pretrained with DINO [5]. During the data selection period, pretrained DeiT-S is finetuned supervisedly to select data samples with Core-Set and Learn-Loss algorithms. After selection, we still train ResNet-18 [21] over the selected samples from scratch. In Fig. 9, this pretrained DeiT-S damages the performance of Core-Set and Learn-Loss. A potential explanation comes from the differences in the representation spaces of pretrained DeiT and from-scratch ResNet. The samples selected by DeiT-S with Core-Set and Learn-Loss algorithms may not be suitable for the
from-scratch training of the ResNet-18 model. This reflects that our performance gain does not come from the use of pretrained DeiT-S. Instead, the proposed FreeSel method plays an important role.
General-Purpose Model Baselines. To further disentangle the roles of the general-purpose model and our designed FreeSel framework, we compare FreeSel with the following baselines which can also select a subset from the data pool using the general-purpose models. 1) K-Means: We perform the K-Means algorithm on the global features extracted by the pretrained DeiT-S
Table 3: Baselines Using General-Purpose Model: We compare FreeSel with other baselines using the general-purpose model. Experiments are conducted on PASCAL VOC object detection task.
Methods	Pretrained Model	Image Number
3k	5k	7k
K-Means	DeiT-S (DINO)	64.85	68.05	71.50
Inconsistency	DeiT-S (DINO)	63.29	67.65	71.35
Entropy	DeiT-S (supervised)	56.33	66.03	69.72
FreeSel	DeiT-S (DINO)	65.66	69.24	71.79
model [50, 5], choosing the sample closest to each cluster center. 2) Inconsistency: We select the most difficult samples based on the inconsistency of multiple-time model predictions. To measure the inconsistency, we perform data augmentations (RandAugment [10]) to generate 10 different augmented copies for each image and calculate the average pairwise distances of global features between these copies extracted by the pretrained DeiT-S model [50, 5]. We select data samples by the order of inconsistency. 3) Entropy: We select the most ambiguous samples based on the classification uncertainty of the pretrained model. Since the classification score is required, we adopt the DeiT-S model [50] pretrained on ImageNet in a supervised manner and measure the uncertainty with the entropy of classification scores. We select data samples by the order of entropy. Experiments are conducted on object detection task in the same settings as Sec. 5.2. Tab. 3 shows that all the above baselines perform notably worse than FreeSel, especially with low sampling ratios. This reflects the importance of our proposed FreeSel algorithm. Trivial utilization of a general-purpose model would not lead to great performance of data selection.
The main goal of this paper is to enable a free data selection pipeline by proposing a novel pipeline with three key principles: generality, efficiency, and non-supervision. We verify its feasibility by designing the first method FreeSel following this new pipeline. Through a single-pass model inference, the semantic patterns are constructed based on the intermediate features of a general pretrained model, over which a distance-based selection strategy is performed to find the most diverse and informative data samples. Our method outperforms most existing counterparts with remarkably superior efficiency on different tasks including detection, segmentation, and classification.
We realize that FreeSel cannot beat all the other data selection methods in current stage due to the absence of training on the target datasets. Never
theless, this direction matters in boosting the training of downstream models without any extra time and cost on the shoulders of existing general pretrained models. It gains more significance given the current landscape dominated by large foundation models pretrained on multi-modality data [40, 15], which we believe can help to extend our method to a wide range of domains and modalities.
Figure 9: Effect of Pretraining Methods: Experiments are conducted on PASCAL VOC.
Acknowledgement. This work is partially supported by Berkeley DeepDrive.
[54] Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. arXiv preprint arXiv:2303.14382, 2023. 3
[55] D. Yan, L. Huang, and M. I. Jordan. Fast approximate spectral clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 907–916, 2009. 6, 15
[56] J. S. K. Yi, M. Seo, J. Park, and D.-G. Choi. Pt4al: Using self-supervised pretext tasks for active learning. In European Conference on Computer Vision, pages 596-612. Springer, 2022. 3
[57] D. Yoo and I. S. Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 93–102, 2019. 3, 7, 8, 9, 17
[58] F. Yu, V. Koltun, and T. Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472-480, 2017. 7, 16, 17
[59] T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5330-5339, 2021. 3, 7
[60] B. Zhang, L. Li, S. Yang, S. Wang, Z.-J. Zha, and Q. Huang. State-relabeling adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8756-8765, 2020. 3
[61] Z. Zhang, R. Girdhar, A. Joulin, and I. Misra. Self-supervised pretraining of 3d features on any point-cloud. arXiv preprint arXiv:2101.02691, 2021. 3
[62] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:1412.6856, 2014. 4
[63] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021. 2, 9
[64] J. Zhu, J. Qi, M. Ding, X. Chen, P. Luo, X. Wang, W. Liu, L. Wang, and J. Wang. Understanding self-supervised pretraining with part-aware representation learning. arXiv preprint arXiv:2301.11915, 2023. 2
In this supplementary material, we first explain the details of spectral clustering algorithm (Sec. 4.2) in Sec. A. We then analyze the sensitivity of FreeSel to the values of hyperparameters in Sec. B. Finally, implementation details of our experiments are explained in Sec. C.
A Spectral Clustering Algorithm
In this section, we explain the spectral clustering algorithm [38, 55] in the semantic pattern extraction process for each image  $I$  (Sec. 4.2 and Alg. 1). The detailed spectral clustering algorithm is shown in Alg. 3. This spectral clustering algorithm should be inserted into line 7 of Alg. 1.
To justify the use of spectral clustering algorithm for semantic pattern extraction, we also try another alternative which directly performs K-Means w.r.t. the local features  $f_{r}^{I}, r = 1,2,\dots,t$  to divide the  $t$  regions of image  $I$  into  $K$  clusters without using the patch token attention to achieve the same feature clustering goal in Sec. 4.2. Tab. 4 shows the comparison between spectral clustering and K-Means. Interestingly, these two feature clustering strategies lead to similar data selection performance on PASCAL VOC [14] object detection task. However, spectral clustering is stably superior when selecting data samples for Cityscapes [9] semantic segmentation task. We attribute this difference to the large domain gap between Cityscapes dataset and ImageNet dataset [11]. The DeiT-S model pretrained on ImageNet may extract local features with weaker discriminative ability from images inside Cityscapes dataset. Since spectral clustering algorithm depends less on the feature quality, it can bring better performance than direct K-Means over intermediate local features on Cityscapes.
Algorithm 3: Spectral Clustering
Input: Similarity matrix between patches  $\widehat{\mathbf{pa}}^I = [\widehat{pa}_{ij}]_{i,j=1,2,\dots,t}$ , semantic pattern number  $K$
Output: Clusters  $C_j^I, j = 1,2,\dots ,K$ , where each region  $r = 1,2,\ldots ,t$  of image  $I$  belongs to a unique  $C_j^I$ .
1 Derive the symmetric adjacent matrix  $\mathbf{A}$  from  $\widehat{\mathbf{pa}}^T$ :
$$
\mathbf {A} = \left(\widehat {\mathbf {p a}} ^ {I} + \widehat {\mathbf {p a}} ^ {I ^ {T}}\right) / 2, \quad \mathbf {A} \in \mathbb {R} ^ {t \times t}
$$
2 Derive the diagonal degree matrix  $\mathbf{D}$ :
$$
\mathbf {D} _ {i j} = \left\{ \begin{array}{l l} \sum_ {l = 1} ^ {t} \mathbf {A} _ {i l} & i = j \\ 0 & i \neq j \end{array} \right., \qquad \mathbf {D} \in \mathbb {R} ^ {t \times t}
$$
3 Calculate the normalized Laplacian matrix  $\mathbf{L}$ :
$$
\mathbf {L} = \mathbf {D} ^ {- \frac {1}{2}} (\mathbf {D} - \mathbf {A}) \mathbf {D} ^ {- \frac {1}{2}}, \quad \mathbf {L} \in \mathbb {R} ^ {t \times t}
$$
4 Obtain the  $K$  eigenvectors  $v_{l}, l = 1,2,\ldots,K$  corresponding to the  $K$  smallest eigenvalues  $\sigma_{l}, l = 1,2,\ldots,K$  of matrix  $\mathbf{L}$ .
5 Compose the matrix  $\mathbf{V}$  based on the  $K$  eigenvectors
$$
\mathbf {V} = \left[ v _ {1}, v _ {2}, \dots , v _ {K} \right], \mathbf {V} \in \mathbf {R} ^ {t \times K}
$$
6 Denote  $u_{i}^{T}$  as the  $i$ -th row of  $\mathbf{V}$ ,  $i = 1,2,\dots,t$ .
7Normalize each row of  $\mathbf{V}$  ..  $\hat{u}_i = u_i / \sqrt{\sum_{j = 1}^K u_{i,j}^2}$
8 Perform K-Means to divide  $\hat{u}_i, i = 1,2,\dots ,t$  into  $K$  clusters  $C_j^I,j = 1,2,\ldots ,K$ :
$$
\left\{C _ {j} ^ {I} \right\} _ {j = 1, 2, \dots , K} = K M e a n s (\left\{\hat {u} _ {i} \right\} _ {i = 1, 2, \dots , t})
$$
B Sensitivity to Hyperparameters
In this part, we analyze the sensitivity of our FreeSel to some hyperparameters including the maintenance ratio  $\tau$  in the attention filter (Eq. 2), the semantic pattern number  $K$  (Eq. 4), the neighborhood threshold  $d_0$  (Eq. 3), the distance function  $D(\cdot ,\cdot)$  (Eq. 5), and pretraining manner for the general
Table 4: Effect of Feature Clustering Strategies: We compare spectral clustering and K-Means for feature clustering. Experiments are conducted on PASCAL VOC object detection task and Cityscapes semantic segmentation task.
(a) Performance on PASCAL VOC Object Detection Task: The task model is SSD-300 [34].
Feature Clustering	Image Number
3k	5k	7k
K-Means	65.35	69.43	71.76
Spectral Clustering	65.66	69.24	71.79
(b) Performance on Cityscapes Semantic Segmentation Task: The task model is DRN [58].
Feature Clustering	Sampling Ratio
15%	25%	35%
K-Means	51.43	54.84	57.96
Spectral Clustering	51.77	55.72	58.58
Table 5: Sensitivity to Hyperparameters:  $\tau, K, d_0, D(\cdot, \cdot)$  separately denote the maintenance ratio, semantic pattern number, neighborhood threshold, and distance function. Experiments are conducted on the PASCAL VOC object detection task.
τ	K	d0	D(·,·)	Pretraining	Image Number
3k	5k	7k
0.3	65.22	69.00	70.69
0.5	5	2	cos.	unsupervised	65.66	69.24	71.79
0.7	64.77	69.33	71.64
0.5	1	2	cos.	unsupervised	64.90	69.01	71.02
10	65.21	69.13	71.50
0.5	5	1	cos.	unsupervised	65.48	68.73	71.39
3	65.37	69.41	71.74
0.5	5	2	euc.	unsupervised	64.77	69.42	71.31
0.5	5	2	cos.	supervised	64.40	68.82	71.43
model. Experiments are conducted on the object detection task, where samples are selected from PASCAL VOC dataset and SSD-300 is the downstream task model in the same settings as Sec. 5.2. Results are shown in Tab. 5.
Maintenance Ratio  $\tau$  (Eq. 2) Maintenance ratio  $\tau$  notably affects the final performance of FreeSel. Too low ratios lead to the ignorance of some crucial local visual patterns, while too high ratios introduce some harmful noisy information to the semantic patterns. Thus, a moderate attention ratio plays an important role in the high performance of FreeSel.
Semantic Pattern Number  $K$  (Eq. 4) When  $K = 1$ , the performance is hurt since semantic patterns degrade to global features in this case. When  $K = 10$ , a slight performance drop may be witnessed in comparison with  $K = 5$ .
Neighborhood Threshold  $d_0$  (Eq. 3) When  $d_0 = 1$ , the neighborhood is too small to represent the relationship between nearby regions. When  $d_0 = 3$ , the performance is a little worse than  $d_0 = 2$ . We think each region feature mainly interacts with nearby regions with distance  $d \leq 2$ .
Distance Function  $D(\cdot, \cdot)$  (Eq. 5) We find the cosine distance can lead to better performance than Euclidean distance. This result shows that the directions of local feature vectors are important to reflect the diversity of local visual patterns.
Pretraining Manner Instead of using the unsupervised pretraining framework DINO [5], we also try the DeiT-S model [50] pretrained in a supervised manner on ImageNet [30]. Results show a performance drop with supervised pretraining. We think this is because supervised pretraining introduces some biases of categories to the pretrained model.
C.1 Object Detection Implementation
C.1.1 Implementation of FreeSel
We set attention ratio  $\tau = 0.5$  and semantic pattern number  $K = 5$ . The input images are resized to  $224 \times 224$  when fed into the pretrained DeiT-S [50] model in the data selection process.
C.1.2 Implementation of Task Model
The implementation of task model is same as previous active learning research [57, 1]. The SSD-300 model [34] with VGG-16 [46] backbone is adopted for this experiment. The model is implemented based on mmdetection<sup>3</sup>. We follow [57, 1] to train the model for 300 epochs with batch size 32 using SGD optimizer (momentum 0.9). The initial learning rate is 0.001, which decays to 0.0001 after 240 epochs.
C.2 Semantic Segmentation Implementation
C.2.1 Implementation of FreeSel
The input images are resized to  $448 \times 224$  in line with their original aspect ratios when fed into the pretrained DeiT-S [50] model in the data selection process. Same as object detection, we set attention ratio  $\tau = 0.5$  and semantic pattern number is doubled to  $K = 10$  in line with the doubled input size compared to object detection task.
C.2.2 Implementation of Task Model
We follow prior active learning work [47, 24] to apply DRN [58] model for semantic segmentation task. The model is trained for 50 epochs with batch size 8 and learning rate 5e-4 using Adam optimizer [28].
C.3 Image Classification Implementation
C.3.1 Implementation of FreeSel