Boundary Matters: A Bi-Level Active Finetuning Method
Authors: Han Lu
Yichen Xie
Xiaokang Yang
Junchi Yan
Venue: NeurIPS 2024 poster (2024)
Keywords: Active Finetuning
Primary area: active_learning

Abstract: The pretraining-finetuning paradigm has gained widespread adoption in vision tasks and other fields. However, the finetuning phase still requires high-quality annotated samples. To overcome this challenge, the concept of active finetuning has emerged, aiming to select the most appropriate samples for model finetuning within a limited budget. Existing active learning methods  struggle in this scenario due to their inherent bias in batch selection. Meanwhile, the recent active finetuning approach focuses solely on global distribution alignment but neglects the contributions of samples to local boundaries. Therefore, we propose a Bi-Level Active Finetuning framework (BiLAF) to select the samples for annotation in one shot, encompassing two stages: core sample selection for global diversity and boundary sample selection for local decision uncertainty. Without the need of ground-truth labels, our method can successfully identify pseudo-class centers, apply a novel denoising technique, and iteratively select boundary samples  with designed evaluation metric. Extensive experiments provide  qualitative and quantitative evidence of our method's superior efficacy, consistently outperforming the existing baselines.

Boundary Matters: A Bi-Level
Active Finetuning Method
Han Lu', Yichen Xie?, Xiaokang Yang's Junchi Yan
Dept. of CSE & School of AI & Moe Key Lab of AI, Shanghai Jiao Tong University
University of California, Berkeley
xkyang, yanjunchi] @sjtw.edu.cn. yichen_xie weeee_iieeeeeeeeeeee
ht https: com/ http:///h.....a.tii.aaaaaaaa//// Btt.//iiniaaa--S///LaaEEE
Abstract
The cetraininng-iiettniin paradigm has gained widespread adoption in vision tasks and other fields. However, the finetuning phase still requires high-quality annotated samples. To overcome this challenge, the concept of active finetuning has emerged, aiming to select the most appropriate samples for model finetuning within limited budget. Existing active learning methods struggle in this scenario due their inherent bias in batch selection. Meanwhile, the recent active finetuning approach focuses solely on global distribution alignment but neglects the contri- butions of samples to local boundaries. Therefore, we propose Bi-Level Active Finetuning framework (BiLAF) to select the samples for annotation in one shot, encompassing two stages: core sample selection for global diversity and boundary sample selection for local decision uncertainty. Without the need of ground-truth labels, our method can successfully identify pseudo-class centers, apply a novel denoising technique, and iteratively select boundary samples with designed evalua- tion metric. Extensive experiments provide qualitative and quantitative evidence of our method's superior efficacy, consistently outperforming the existing baselines.
Introduction
The advancement of deep learning significantly relies on extensive training data. However, annotating large-scale datasets is challenging, requiring significant human labor and resources. To address this challenge, the pretraining- -inttuninnnnnningggggg paradigm has gained widespread adoption. In this paradigm, models are first pretrained in an unsupervised manner large datasets and then finetuned on smaller, labeled subset. While there substantial research about both unsupervised pretraining 7 [I 1 17
and supervised finetuning [15] (7), the optimization of sample set selection for annotation has received less attention, especially in scenarios with limited labeling resources.
Active learning methods 33 36 31
though effective in identifying valuable samples for training from scratch, face significant challenges when integrated into the pretraining-finee finettinnn--nnuinnggggg framework [3]
41. The primary limitation stems from the batch-sclection strategy commonly used by these methods. Allocating a limited annotation budget across multiple iterations can introduce harmful biases, which leads to overfitting. Consequently, this undermines the general representational quality of the pretrained model and leads the accumulation of errors in the iterative selection process.
To fill in the research gap. the Active Finetuning task has been formulated in [1] which focuses on the selection of samples for supervised finetuning using pretrained models. This method optimizes sample selection by minimizing the distributional gap between the selected subset and the entire data pool.
Corressponding author. This work was supported by NSFC (92370201, 62222607) and Shanghai Municipal Science and Technology Major Project under Grant 202ISHZDZX0102..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Feature distribution
Core samples
*
Boundary samples
O
O
True class boundary
Previous Method
BiLAF (Ours)
Figure 1: Design philosophy of our BiLAF framework. In contrast to the previous method, our method ensures the selection of central samples to maintain diversity while also reserving capacity to choose boundary samples enhance decision boundary learning.
Despite its notable performance, it fundamentally concentrates on the global diversity of selected samples, which neglects the local decision boundaries- also referred as sample uncertainty in data selection. As the volume of data increases, we found the model's capabilities drops dramatically, which indicates the selected samples become increasingly redundant and less informative.
To mitigate the inherent limitations of existing approaches, we introduce the innovative Bi-Level Active Finetuning Framework (BiLAF), which can effectively capture both diversity and uncer- tainty of selected samples, as depicted in Fig. [ The key challenge lies in measuring uncertainty within the pretraining-inetuiinng finetuming paradigm. Without labels and trained classifier head, traditional methods based on posterior probability [24] 39]. entropy [18] [29 and loss metrics [16] 45] are
infeasible. Notably, we observe that the global feature space of pretrained models inherently captures the interrelations among samples from different classes. Consequently, we can effectively utilize their feature outputs to facilitate the identification of support samples that are proximate to the model's 's decision boundary. This concept has been extensively supported by theoretical 23 and methodological research
5 across various domains.
To elucidate, our BiLAF framework operates through two distinct stages. The initial phase, Core Samples Selection, is dedicated identifying pivotal samples for each class. The selection mechanism employed can vary, including options such as K-Means or ActiveFT [41]. The second stage, Boundary Samples Selection, begins with our innovative unsupervised denoising technique that preciscly isolates noisy outliers. Following this, we systematically identify boundary samples adjacent to each core sample and efficiently eliminate redundant samples, employing our newly proposed boundary score.
In conclusion, our contributions are summarized as follows:
We propose Bi-Level Active Finetuming Framework (BiLAF) emphasizing boundary importance to balance sample diversity and uncertainty. This framework exhibits remarkable flexibility and can accommodate various methods seamlessly.
The proposed unsupervised denoising technique in BiLAF can effectively eliminate the outlier samples and an iterative strategy with newly designed metric can identify the marginal boundary samples in feature space. Compared to other methods, our approach has high accuracy and efficiency.
Extensive experiments and ablation studies demonstrate the effectiveness of our method. Com- pared to the current state-of-the-arr approach, our method achieves a remarkable improvement of nearly 3% absolute accuracy on CIFAR100 and approximately 1 on ImageNet. What's more, outperforms the other baselines in object detection, semantic segmentation, and the long-tail tasks.
2
Related Work
2.1 Active Learning / Finetuning
Active learning maximizes annotation efficiency by selecting the most informative samples. Typically, uncertaint--rriee methods select difficult samples based on heuristics like posterior probability 24

---

Unlabeled Pool
Feature Space
Core Sample Selection
Boundary Samples Selection
Denoising
Repeat
Inference
Boundary Score Calculation
Pretrained Model
Labeled Pool
Annotate
Iterative Selection and Removal
truck
Figure 2:
Our BiLAF framework in the Active Finetuning task. In the high-dimensional feature space, the Core Sample Selection focuses on pinpointing pseudo-class centers. Following this, we have devised a demoising method eliminate noise samples. Subsequently, we compute the Boundary Score metric for each sample, which aids in the iterative selection of samples and the removal of candidates from the pool. Ultimately, the selected samples are labeled for supervised finetuning.
39]. entropy [8. 29]. or loss metrics [6..5), while diversity- -aased approaches approximate the original data distribution using metries like sample distance or gradient directions [3]
20
However, these methods struggle within the pretraining fiatingg-fettinngggg paradigm [12 41
ActiveFT [1
addresses this by aligning the sample distribution with the unlabeled pool, often prioritizing high- density over boundary areas. Our algorithm, by integrating both diversity and uncertainty, refines the selection process to enhance decision boundary samples selection for following supervised finetuning.
2.2 Decision Boundaries in Neural Networks.
Decision boundaries are pivotal in neural network- classifiers, influencing both performance and interpretability 23 25 Their optimization enhances generalizability and accuracy in complex data In SVMs, decision boundaries separate hyperplanes and maximize geometric margins This concept extends to neural networks, where margin maximization also promotes generalization. For imbalanced datasets, adjusting the decision boundary is crucial for accurate minority class classification, with approaches like LDAM loss [5] and ELM loss [I9 developed to modify the boundary and balance errors across classes. Neural networks tend to utilize simple or highly discriminative features for decision boundaries (30 34 theoretical framework exploring decision boundary complexity and its inverse relation to generalizability is introduced in (23). Notably, this not trivial in unsupervised scenarios. Despite the challenge, we effectively leverage features from pretrained models, thereby introducing innovative denoising and selection methods without labels. This approach addresses a significant gap the active finetuning domain.
spaces 42 5
for robust classification
BiLAF: Bi-Level Active Finetuning
In this section, introduce novel Bi-Level Active Finetuning Framework (BiLAF), illustrated in Fig. BiLAF operates in two distinct stages: Firstly, Core Samples Selection (Sec. (Sec.3.2) involves identifying multiple pseudo-class centers, which ensures comprehensive coverage across all classes. Secondly, Boundary Samples Selection (Sec. 3.3 focuses on the accurate identification of boundary samples for each pseudo-class, using the novel denoising and iterative selection method. Algorithm 1 summarizes the complete workflow. The theoretical time complexity is analyzed in Appendix
3.1 Preliminary: Active Finetuning Task
The active finetuning task is defined in 41 Apart from large unlabeled data pool pa (x, [iEM) where [N] {1, ...,, N) identical to the traditional active learning task, we have the access to deep neural network model f(; wo) with the pretrained weight utp should be noted that the function f(; wo) can be pretrained either on the current data pool pa or any other external data

---

sources. Using the pretrained model, we can map the data sample x; to the high dimensional feature space = f wi) E RR where f; is the normalized feature x,. From this, we can derive the feature pool F (E(N) from Pu, which assists us in selecting the optimal samples.
Our task is select the subset P3S from P for annotation and the subsequent supervised finetuning. The subset P = {xa, Jet is determined by the sampling strategy S = {ss E [[]]) where B represents the annotation budget. The labels (y,, ]E(( C y, a subset of the label space y are accessed through an oracle, resulting in a labeled data pool P' = {Xe, : [ Subsequently, the pretrained model [(; we) undergoes supervised finetuning on P's. Our objective is to optimize the sampling strategy S to select the labeled set that minimizes the expected error of the finetuned model under the given annotation budget constraints.
3.2 Core Samples Selection
We start data selection procedure by selecting core samples. Those core samples can represent the distribution of the entire dataset. Popular methods include K-Means, Coreset [33], and ActiveFT [41], differing in their design targets and optimization procedures. We employ ActiveFT, the most advanced method to date, as our primary approach in the initial phase. For detailed implementation of ActiveFT, see Appendix Using ActiveFT, we obtain K core samples, which serve as the center for each aseudo-class. where K represents the predefined budget for core samples.
3.3 Boundary Samples Selection
In addition to those K core samples, continue to select samples close to the boundaries of semantic categories, which can enhance the training of the model's decision boundaries.
By leveraging pretrained models, data samples are mapped to robust feature representations that elucidate the relationships among samples, their intra-class counterparts, and inter-class samples from diverse classes. We introduce a novel method for boundary sample selection- the first in this field. Given the pseudo-class centers, our method involves three steps: 1) Identifying the samples associated with each pseudo-class center; 2) Implementing denoising processes to refine these samples; 3) Developing precise metrics to select boundary samples for each pseudo-class.
3.3.1 Sample Clustering
Given K pscudo-class centers denoted by C= {c1, .,...., .K} where e E [N]. a sample x, with its feature vector fş E Idd is assigned to the pseudo-class center that minimizes the distance D(G f.). This assignment can be mathematically represented as:
C arg min D(f,, cEC
where D)-..) denotes the distance function. While the choice of D can vary based on the design of the clustering model, we employ the Euclidean distance our implementation, which efficiently aligns each sample point with corresponding pseudo-class center during the optimization process.
3.3.2 Boundary Sample Denoising
Boundary samples within pseudo-class crucial for optimizing decision boundaries. However, they can also introduce noise, which potentially hinders the model's performance. For each class with N₁ samples, we define a removal ratio Pro. identify and eliminate Ni.rm N, Pro peripheral noisy samples from the candidate boundary set. This climination process is based on the density of samples in the feature space, where we define the concept of density distance follows: Definition (Density Distance). The density distance of sample is defined as the average distance its nearest neighbors. Formally, for a given sample z, characterized by its feature vector fy, its density distance p(x;) is defined as:
p(x,)
1 k
Σ Eng)
where nitt represents the index of the 1-th nearest neighhor of xz in the feature space, and denotes distance function.

---

Inspired by the classical clustering method DBSCAN 8 we propose an Iterative Density-based Clustering (IDC) algorithm. IDC clusters the candidate samples X; "u. where the index ax E [N] and j [N.]. The algorithm initiates with the core sample index C; as the seed set U, = {.i}. In each subsequent iteration, fraction P.n. of samples are included, which corresponds to Ni,in = N. P.n peripheral samples being integrated into the existing cluster U,.
The density distance of each candidate sample redefined from Eq. 2 as the average distance to the nearest k selected sample in the cluster U, Specifically, the density distance p(x,) for each remaining point xj is calculated as:
Algorithm 1 Pseudo-code for BiLAF
Input: Unlabeled data pool PM = he) pretrained model ((::%) annotation budget B.
Parameter: Core samples budget K. removal ratio Pro. den- sity neighbours IDC cluster ratio P.. opponent penalty coefficient 6, distance function D(.-).
Output: The selected samples index $ = {8)]je[8)
p(x,) = E
D(f,, Eng)s
b Stage I: Core Samples Selection
1: for i E [N] do
2:
f; f(x;; wo)
Select K centers feature space F using ActiveFT
where nt represents the index of the I-th nearest sample in U, to X,. In each iteration, the Ni,in samples with the lowest density distance are selected. This process repeats until all samples have been included. The order of inclusion into the cluster reflects each point's ' proximity to the center, with later-added samples more likely to be peripheral and noisy. Consequently, the last Nirm samples can be removed from U..
Can be K-Means and any other core selection method
b. Stage 2: Boundary Samples Selection
4: for each pseudo-class 64 do
Stepl: Assign the pseudo-class samples Initial the candidate index set U, = ( for j E [N] do
5:
6:
7:
if G, == arg min.ec D(f,, f.) then Add the index j to the set U.
8:
Step2: Iterative Density-based cluster then denoise Initialize the pscudo-class cluster U, = {e.).
9:
10:
while [0 > [U/] do
3.3.3 Iterative Sample Selection
II:
for each candidate index j c U. and j U; do
12:
p(x,) -i-.ane P(fy,fn,,) where Ry the nearest I-th sample index xy- Sort samples by increasing density distance p(x,) Add the previous Pi: (U samples to U;
After successfully eliminating noise samples, we advance to the selection of boundary samples from the centers of pscudo-classes. To stream- line this process, we introduce a set of defini- tions that facilitate the computation of a bound- ary score for each sample.
13:
14:
15:
Remove the last Pr.. - [U,] samples from U, denoise
Step3: Selection Process Initialize opponent penalty counts 1 0 for $
16:
17:
for yn = to B; do
Definition 2 (Intra-class Distance). Let xj be sample in the set U, of the pseudo-class c,. The intra-class distance for xj is the average distance from f, to all other samples within the same class set U.. It can be formally defined as:
18:
for each candidate index j E U: do dinte((X)) = াo Lier, &(f,. 122 ) BS(x;) = min cyec mas( Jodhere
19:
20:
Select j..... with the lowest Boundary Score if otherwiss set Jmin 6;.
22:
Add the index Jmmn to selected samples set S t ft + I where nearest opponent class is if m > 1.
1
dintra (X2)
Σ D(f,,fi) IEU.
(4)
23:
[UA]]
24: Remove the nearest [U,]/B, samples of Xj.... from
where D is the distance function and |U&| is the number of samples in pseudo-class C..
25:
return the selected samples index S
Definition 3 (Inter-class Distance). The inter-class distance for sample Xj in the pseudo-class o; is defined as the distance from f, to the nearest center of other pseudo-classes. It is defined as:
dinter(Xx) =
min ecEC,
(5)
where D the distance function, C= (cI,..., CK3 are the pseudo-class centers, and i + ensures that the pseudo-class e; of the sample xj is excluded from the calculation.
Definition& 4 (Boundary Score). The Boundary Score for sample x; in psuedo-class e; can be defined as a function of both intra-class and inter-class distances. It is defined as:
dinter (Xx) dinntra (X₂)
BS(x,) mrax(dsnter& (X;), dintrr
where disttr and dinter are the intra-class and inter-class distances, respectively. A smaller Boundary Score indicates closer proximity to the boundary.
5

---

For each pseudo- class & with the samples set U, we allocate the sample selection budget in proportion to the size of the pseudo-class. Specifically, the budget B, for pseudo-class o calculated as: B, B. -UAl // |0yl where [U;] represents the number of samples in pseudo-class C₁.
Initially, one might consider simply selecting the top B, samples with the lowest boundary scores. However, this approach risks over-concentration of selections within specific areas of the feature space. To address this, we employ an iterative selection and removal strategy, which progressively selects and eliminates candidate samples. Furthermore, to prevent the aggregation of multiple samples near the same pseudo-class boundary that faces the same opposing pseudo-class center, we introduce an opponent penalty. This mechanism increases the influence of previously selected boundary samples on subsequent selections, encouraging greater diversity across different boundaries.
Iterative selection and removal begins by selecting samples with the lowest Boundary Score in each iteration. Subsequently, the nearest ||U:|/ B:] samples surrounding the selected sample are removed to prevent clustering. This process is repeated B; times, starting from the center samples.
Opponent penalty monitors the relationship between the currently selected samples and the bound- aries to other pseudo-classes, imposing a penalty on opponent pseudo-classes whose boundaries have already been selected. Specifically, for the selected pool of pseudo-class c;. if boundary samples related to an opposing pseudo-class 7 * i have been selected f times, the inter-class distance for the subsequent samples to this pseudo-class I will be scaled by factor of 5*1, where the opponent penalty coefficient is hyperparameter greater than 1. This indicates that the more often a sample is selected in relation to pseudo-class I's boundary, the higher its Boundary Score with that pseudo-class becomes, making it less likely to be chosen in future rounds. Consequently, we recalibrate the Boundary Score for each sample in each iteration, leading to a modification from Eq. 6
8 D(f,, for) dintra (xx) acE, max (f(y, ). dintral (x²))
BS(x,)
min
where (ff fe,)) measures the distance from sample x, to the opponent pseudo-class I and the 8t reflects the imposed penalty.
In each pseudo-class i, we select the B; samples through an iterative selection and removal strategy that incorporates this opponent penalty in the increasing order of BSxx)). The selected samples from various pseudo-classes are ultimately aggregated to form the comprehensive annotation subset Py.
4 Experiments
Our approach has been rigorously evaluated using three primary image classification benchmarks, alongside various tasks with different sampling ratios, as detailed in Sec. 4.1 We compare its performance against multiple baselines and conventional active learning techniques, with results discussed in Sec. 4.2] The analysis of our method is presented in Sec. 4.3] and a comprehensive ablation study is provided in Sec. 4.4 All experiments were conducted using GeForce RTX 3090(24G) GPUs and Intel(R) Core(TM) 19- 10920X CPUs. The source code will be made publicly available.
4.1 Experiment Setup
Datasets and Evaluation Metrics. Firstly, evaluate our method using three widely recognized classification datasets: CIFARIO, CIFARI00 (22), and ImageNet- (32). The raw training data from these datasets constitute the unlabeled pool pu from which selections are made. For performance evaluation, we employ the Top-/ Accuracy metric. In addition to these classification tasks, extensive experiments have also been conducted in object detection, semantic segmentation, and long-tail tasks. Further details on these experiments can be found in Appendix E and F
Baselines. We compare our approach with three heuristic baselines Random, FDS and K-Means; five active learning methods CoreSet [33], VAAL (66,, LearnLoss [45), TA-VAAL (20), and ALFA-
Mix [31: and the well-designed active finetuning method ActiveFT [4 We utilize the overlapping results reported in
[11. The detailed information of the baselines is listed in Appendix
Implementation Details. In line with the SOTA method ActiveFT model, pretrained using the DINO 6 framework on ImageNet- in the unsupervised pretraining phase. For all the datasets, we resize images to 224 x 224 consistent with the pretraining for both
we use DeiT-Small
38

---

Table 1: Benchmark Results. Experiments conducted on three popular datasets with different annotation ratios. We report the mean and standard deviation over three trials. Traditional active learning methods require random initial data to start, thus use "-" to represent. BiLAF has shown a significant competitive advantage across the majority of scenarios, affirming its effectiveness.
CIFARIO
CIFARIO0
ImageNet
Methods
0.5%
1%
1%
10%
Random
77.3+2.6
82.2+1.9
88.940.4
14.9=1.9
24.3+20
50.843.4
69.3:0.77
45.14008
52.1=0.6
64.3+0.3
6.55
7.22:12
81.4:0.7
8.1410.6
12.8:0.3
16.9114
12.
26.7:10.6
43.1::0.4
05.5
K-Means
83.013.5
85.920.8
89.6:1066
17.6=1.1
31.910.1
42.41:10
70.7:0.3
49.8:0.55
55.420.3
64.010.2
CoreSet
81.610.3
88.41022
30.61044
48.3110.5
62.9.00.6
52.7.10.4
61.7100.2
VAAL
36
80.920.5
88.820.3
24.62111
46.420.8
70.12044
54.7=0.5
64.0+0.3
LearnLoss
81.6:0066
86.710.4
19.24222
38.242.8
65.741.1
54.3::0.6
63.210.4
TA-VAAL
20
82,6+0,4
88.7+0.2
34.7=0077
46,4+1.1
6688+0.5
55 00.000044
64.3+0.2
ALFA-Mix
11
83,4.0.33
89.6.10.2
35.31.0.8
50.4.10.9
69.9.10.6
55. 55.3.10.3
64.51.0.2
ActiveFT A1
85.0+0.4
88, 28.2+004
90.1++0.2
26.1-2.6
40.7+-0.9
$4.6+2233
71,0+00.5
50.1+003
55 35.80
65.3+0.1
BiLAF (ours)
81.++12
89.2#0.6
92.5=004
31881.6
43.5=0.8
6.,8+122
73.7=00.5
50.8204
56.9=0.3
66.220.2
Table 2: Performance on Object Detection, Semantic Segmentation and Long- -aileT Dataset.
PASCAL VOC (nAP)
ADE20K (mloU)
CIFARLOOL (IR=100)
#1000
#2000
#3000
#5000
5%
10%
15%
5%
10%
15%
Random
51.1+0.9
60.9:0.6
64.240.5
67.540.4
14.54#1022
20.27:1.68
13.55=1335
21.4922.77
28.55=1193
34.58
FDS
19.110.6
58.1004
62.110.3
61.510.2
12.1410.27
17.6610.33
23.5510.53
16.53106655
23.1510.57
28.76110.66
K-Menns
52.3+044
60.7:0.4
64.9:203
67.8103
13,6240.52
19.12:40.47
23.10=0.09
22.98+0.99
29.26+0082
34,47:0061
Coreset
33
61.540.3
66.0+0.3
69.0+0.2
20.25+0.44
2366+00.22
28.37+0.71
35,1240.55
YAAL 36
61.2:10.5
65.71004
68.910.3
20.56:10.75
23.87.0.88
29.97.1066
35.62.10.55
Learaloss
60.8+066
64.9+0.4
68.9+0.3
19.71+1.005
22.94-1.19
28. 18-16++1443
34.02+1.01
TA-YAAL
61.420.4
65.92003
69.1::02
2063::0.77
24.24:10:3
30.01:0096
35.5910064
AdtivoFT
31
54.9--03
61 61.8+0.3
66.0+0.2
69.1+0.2
15.3760111
21,60+0.40
25.03+0.87
24.60+102
31,58+0.76
37.01+0633
BilrAF(Our))
$8.7+0.5
62.3-003
66.3+033
69.2+0.2
16.54+0.37
22.03+0.53
25.71-0.79
26.56+1.18
32.35+0.81
37.33+0.7
data selection and supervised finetuning. In the core samples selection stage, we utilize ActiveFT and optimize the parameters Os using the Adam (21] optimizer (learning rate 1e-3) until convergence. We set the core number K as 50 (0),..)....)....)))) 250(0.5%) (405(0.5%) for CIFARIO, CIFAR and ImageNet separately. In the boundary samples selection stage, consistently set nearest neighbors number k as 10, both removal ratio Prm and clustering fraction Pm. as 10%, opponent penalty coefficient d as 1.1. The experiment details of supervised finetuning are listed in the Appendix E
4.2 Overall Peformance Comparison
The average performance and standard deviation from three independent runs are presented in Tab. Under a low sampling ratio of 0.5% for CIFARIO, our method is marginally outperformed by ActiveFT. This can be attributed to the more stable model training aided by core point selection at extremely low budgets, whereas boundary samples tend to introduce greater instability and perturbation. However, as the volume of data increases, the advantages of constructing precise boundaries become more pronounced. Our approach BiLAF exhibits significant superiority across most scenarios, markedly outperforming competing methods. Notably. on C00 BiLAF consistently outperforms the previously best-performing model, ActiveFT. by approximately 3%. On ImageNet, BiLAF achieves a consistent improvement about 1%. These significan enhancements underscore the effectiveness of the BiLAFF method. Further details on the robust performance of our method across various pretraining paradigms and model architcctures, which substantiate both its effectiveness and generality, provided in Appendix F
Tab. presents the performance of our method across diverse tasks and scenarios. Although our design, based on boundary sample selection, appears to be tailored for classification tasks, the performance of BiLAF still effectively surpasses that of other models in these domains. We attribute this success to our effective denoising of outlier samples and the method's 's focus on selected sample uncertainty. The less pronounced advantage in certain tasks may be due to global features not fully representing task requirements such as in object detection involving multiple objects, and the default denoising removal ratio parameters might remove minority samples in long-tail distributions. Despite these challenges, BiLAF demonstrates robust performance across various scenarios, confirming its
7

---

Table 4: Ablation CIFAR100. In the denoising process, "DG", "DB" and "IDC" indicate the basic gestacce-guide method, density- based method and iterative density-based clustering method. In the selection criterion, "BD", "BS" indicate the basic distance metric and the boundary score metric. In the selection process, "OS", "ISR", "OP" indicate selecting the top samples in one shot, iterative selection and removal and whether to use opponent penalty. BiLAF represents the complete implementation and we explore the influence of three designs separately.
Denoising Process
Selection Criterion BD
Selection Process OS
Annotation badget
DG DB IDC
BS
ISR OP
10%
BILAF
31.82
43.48
62.75
73.67
31.10 (-0.22)
41.06 (- 2.42)
61.60 (-1.15)
( (-0.28)
29.27 ( 2.55)
36.42 ( T.06)
61.16 (-1.80)
72.87 (-0.80)
/
28.88 (2244)
36.78 (-8.70)
(0.83(11900)
(-0.91)
4
v
/
( (-3.88)
33.76
9.72))
54.03 (-8.72)
(-
32.13 (40.33))
0.39)
62.07 0.68)
72,92 (-0.75)
30.52 (-130)
42.13
1.35)
60.24 (- 2.51)
69.66 (-4.01)
strength as a general sample selection method. As a flexible framework, there is still considerable scope for improvement in our method by further designing it based on task-specific features.
4.3
Analysis
Data Selection Efficiency. Traditional active learning methods typically follow a paradigm involv- ing multiple iterations of selection and training. However, this approach not only demands substantial computational resources due to repeated training but also results in prolonged data selection times and increased management costs associated with multiple annotation processes in practical scenarios. In contrast, our method, alongside ActiveFT [41), adopts a one-shot selection strategy, offering distinct advantages. Tab. 3 presents comparison of the time spent in selecting varying proportions of data across different methods on CIFAR 100. For a deeper understanding of the theoretical time complexity and further analysis, please refer Appendix D
Selected Samples Visualization. Fig. 3
Table 3: Time compelxity. Our method exhibits signif- icant advantages beyond conventional active learning approaches and comparable speed to ActiveFT.
illustrates the sample selection process our method. Firstly, we identify the central samples represented by pentagrams, and then expand to the boundary points denoted by circles from each center. Our method focuses on the boundaries between two cat- egories rather than purely fitting the entire distribution. This approach allows for the selection of more valuable samples. For more detailed comparison with other methods and the visualization of denoising process, please refer to the Appendix
Ratio
K-Means
CoreSel
VAAL
LeurnLoss
ActiveFT
BiLAPlours)
25%
16.64
1h57m
7h:2m
20m
12.66
18.6s
37.0s
7b44as
12h13m
1137m
21.96
19.2s
10%6
70.24
200:88m
36h24m
9hi25m
37.36
20.34
4.4 Ablation Study
Effectiveness of Designs. Tab. 4 demonstrates the contributions of all proposed components out- lined in Sec. [3.3 to the model performance. Our framework is structured around three core compo- nents: Denoising Process, Selection Criterion, and Selection Process. IDs to 3 evaluate the impact of the Denoising Process, comparing iterative density- based clustering with other strategies detailed in Appendix C and scenarios without denoising. We observe that performance degradation is substantial under smaller budgets but diminishes as the bud- get increases. This suggests that with fewer data selected, the adverse effects of noisy samples are more pronounced; however, with an increased data volume, these deficiencies are mitigated, underscor- ing the efficacy of our denoising strategy. ID 4 explores the impact of our Selection Criterion de- sign. In the absence of the Boundary Score metric,
Figure 3: tSNE Embeddings on CIFAR10 with 1 % annotation budget of BiLAF. Penta- grams represent the chosen core samples, while circles denote the chosen boundary samples.
8

---

models struggle to accurately identify potential boundary samples and are negatively impacted by irrelevant marginal samples, significantly reducing performance. IDs 5 and 6 assess the effects of our Selection Process design. While removing the opponent penalty does not critically undermine performance and might even enhance it at a 1% data volume, iterative selection and removal are essential, particularly as data volume increases. These steps effectively increase the diversity of boundary samples selected. Conversely, a one-time selection approach at increased data volumes leads to the accumulation of redundant samples, resulting in wastage and performance degradation.
Core Samples Selection Number. Without ground truth labels, ensuring that each pseudo-class center selected during the core sample selection stage of our BiLAF method represents a distinct category is challenging. Tab. [5) illlstratee the impact of varying the number of core samples on accuracy across different annotation budgets within the CIFAR 100 dataset. We found that perfor- mance significantly suffers when fewer centers are selected. For instance, selecting 125 centers for 100 categories resulted in suboptimal performance, primarily due to the limited number of centers being unable to represent all categories adequately. This limitation poses significant challenges for subsequent boundary sample selection. However, performance stabilizes once a sufficient number of centers encompass all categories established. Optimizing the ratio of center samples boundary samples can yield the performance gains. For example, the best performance at 5% data volume was achieved with 375 centers, and at 10% data volume, 500 centers were optimal. Although there is potential to further enhance our method, we maintained the number of centers across all experiments with different annotation ratio for consistency.
Table 5: Ablation for Core Samples Numbers.
Table 6: Ablation for Core Selection Method.
Core Ratio/ / Core Number 0.50% / 250
Core Selection Method
Rudget
0.25% 125
0.75% 375
1.00% 500
Budget
Random
FDS
K-Mcans
ActiveFT
1%
21.51
31,82
28.37
27.24
20.69
31.82
2%
36.64
43.48
42.68
42.18
2%
36.22
33.22
41.17
43.48
6%
59. 20
62.75
63.32
62.46
60.68
60.13
62.39
62.75
10%
71.86
73.67
73.58
74.32
10%
72.84
71.05
74.27
73.67
Core Samples Selection Method. In the core sample selection, we primarily utilized ActiveFT. However, there numerous existing methods, such as Random, FDS, and K-Means. Tab. apbesessssntts the model performance based on boundary selection using different pseudo-class centers. We found that the accuracy of the BiLAF framework is closely tied to the quality of the method used for selecting pseudo-class centers. Mis-selection of centers can introduce significant bias, adversely affecting subsequent sample selection. Notably, ActiveFT tended to yield the highest performance, while the traditional K-Means method also demonstrated strong results and outperform ActiveFT even in 10% budget, validating the robustness of our framework with well-defined centers.
Hyperparameter Influence. Our pri- mary handcrafted parameters include the removal rate of noise samples dur- ing the denoising process and the oppo- nent penalty applied during the selection process to penalize boundaries within the same class. Fig. 4examines the im- pact of varying these parameters model's 's performance on the CIFARIO 10 dataset, with annotation budgets of 2% and 5%. We observe that the removal rate significantly influences performance. A low setting allows excessive noise accumulation, while high setting depletes boundary sample points the budget increases, adversely affecting performance. In contrast, the opponent penalty exerts more subtle effect and can modestly enhance model performance.
CHAILOO with 2% Budge:
CIFA
Budget
$3.75
G0:00
$1.22
Opporeent Rejaat
$3.33
62.44
G
Opperent
4::00
12
53
0.11
$2.11
60,60
62:12
50
0.3
22
0.3
Semoool Rati
A1
02
Figure 4: The Hyperparameter Influence.
Threshold of Core Numbers. Tab. 5 iillustrates the impact of different Core Numbers on perfor- mance. In practical applications, determining the optimal Core Number directly is question worth exploring- Similarly, in traditional active learning, the effectiveness of different methods often varies with the scale of the data. The classic Coreset (33 seeks to cover all samples using selected points. However, ProbCover 44 highlights that Coreset struggles when the dataset very small, prompting the introduction of a coverage radius to ensure each selected point influences a fixed area.

---

Table 7: Threshold of Core Numbers. Distance and Rate Return on CIFARIO and C0
Core Num
50
100
150
250
375
|
500
1000
1500
2500
5000
CIFARIO
Distance
0.7821
0.7588
0.7472
0.7307
0.7203
0.7111
0.6896
0.6746
0.6506
0.6010
Rate of Return x10-1
4.6575
2.3264
1.6422
0.8362
0.6887
0.4420
0.3002
0.2398
0.1984
CIFARI00
Distance
0.8378
0.8082
0.7913
0.7724
0.7564
0.7478
0.7229
0.7059
0.6800
0.6272
Rafe of Return x10
5.9221
3.3791
1.8906
1.2797
0.6845
0.4985
0.3404
0.2589
0,2112
In our case, using central points to approximate dense distributions seems more appropriate, especially when the budget is highly constrained, without selecting additional boundary samples. However, as the budget increases, the selection of boundary samples becomes more meaningful. To further investigate, explore what appropriate threshold might look like. We employed the Core Sample Selection method to derive different numbers of central points and analyzed their benefits. Specifically, we define Distance the average Euclidean distance from each sample to its nearest selected sample in the feature space. Additionally, we examine the Rate of Return (incremental benefit per core sample) within different ranges, where Rate of Return Distance Difference / Core Number Difference between two adjacent columns.
In Tab. 7 We found that the Rate of Return diminishes gradually, indicating that core samples are crucial in the early stages, while the benefits decrease significantly later on. A clear demarcation point serve guide for when to begin Boundary Sample Selection, such as the range of 250-375 for CIFAR- 10 and 375-500 for CIFAR-100. This provides a simple yet effective guideline. Additionally, in practical applications, we discovered that introducing boundary points earlier may yield better results, such as CIFAR 100 with 1 % (500 samples) annotation samples.
Table 8: Performance Different Pretraining Frameworks and Models CIFAR10.
DeiT-S Pretained with iBOT
ResNet50 Pretrained with DINO
Methods
1%
2%
1%
2%
Random
83.0
89.8
76.2
83.7
CoreSet
82.8
89.2
70.4
83.2
LearnLoss
83.6
89.2
71.7
81.3
VAAL
85.1
89.3
75.0
83.3
ActiveFT
88.3
90.9
78.6
84.9
BiLAFFOurs) 89.1
92.2
79.3
85.8
Generality Pretraining Frameworks and Model Architectures. Our method BiLAF demon- strates versatility across various pretraining frameworks and models. BiLAF has effectively integrated with the DINO 6 framework and the DeiT-Small (38 model. Here, we apply the method
DeiT-Small
38 trained with generative unsupervised pretraining framework iBOT [47] and CNN
model ResNet50
trained with DINO (6. All models are pretrained on ImageNet- k and finetuned
on CIFARIO 10 with other same implementation details described in Appendix Tab. 8 hiighlights our approach's substantial improvement over other sample selection baseline across different sampling ratios, illustrating our method's broad applicability to diverse pretraining strategies and model types.
5 Conclusion
In this paper, we underscore the significance of active finetuning tasks and critically examine existing methods, which often overlook uncertainty aspects, particularly under the pretraining finetuniig paradigm. We propose an innovative solution: the Bi-Level Active Finetuning Framework (BiLAF). This framework not only ensures diversity in the selection of central points but also prioritizes boundary samples with higher uncertainty. BiLAF effectively amalgamates existing core sample selection models and introduces a novel strategy for boundary sample selection in unsupervised scenarios. Our extensive empirical studies validate BiLAF''s effectiveness, demonstrating its capability to enhance predictive performance. Through comparative experiments, we explore new avenues, such as finding the optimal balance between central and boundary points. We believe our work offers valuable insights into Active Finetuning and will serve catalyst for further research in this field.
10

---

References
[1] S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In European Conference on Computer Vision, pages 137-153. Springer, 2020.
[2] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, arXiv preprint arXiv:/906.0388 03001,, 2019.
[3] J. Z. Bengar, J. van de Weijer, B. Twardowski, and B. Raducanu. Reducing label effort: Self- supervised meets active learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1631-1639, 2021.
[4] C. J. Burges. A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery, 2(2):121-167, 1998.
[5] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with renelldistriution-aweee margin loss. Advances in neural information processing systems, 32, 2019.
[6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.044: 04800, 2018.
[8] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pages 226-231, 1996.
[9] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-333 338, 2010.
[10] A. Freytag, E. Rodner, and J. Denzler. Selecting influential examples: Active learning with expected model output changes. In European conference on computer vision, pages 562-577. Springer, 2014.
[11] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-21284, 2020.
[12] G. Hacohen, A. Dekel, and D. Weinshall. Active learning budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794, 2022.
[13] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable 2021.
vision learners. arXiv preprint arXiv:2111.0665
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[15] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2021.
[16] Huang, T. Wang, H. Xiong, J. Huan, and D. Dou. Semi-supervised active learning with temporal output discrepancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 3447-3456, 2021.
[17] S. Huang, Y. Xie, S.-C. Zhu, and Y. Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6535-6545, 2021.

---

[18] A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classifica- tion. In 2009 ieee conference on computer vision and pattern recognition, pages 2372-2379. IEFE, 2009.
[19] Kato and K. Hotta. Enlarged large margin loss for imbalanced classification. arXiv preprint arXiv:2306.09133 09132, 2023.
[20] K. Kim, D. Park, K. Kim, and S. Y. Chun. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8166-8175, 2021.
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6988 2014.
[22] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] S. Lei, F. He, Y. Yuan, and D. Tao. Understanding deep learning via decision boundary. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[24) D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pages 148-156. Elsevier, 1994.
[25] H. Li. Support vector machine. In Machine Learning Methods, pages 127-177. Springer, 2023.
[26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In Computer ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 21-37. Springer, 2016.
[27] X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10.00221,0.07600, 2021.
[28] Z. Liu, H. Ding, H. Zhong, W. J. Dai, and C. He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9274-9283, 2021.
[29] W. Luo, A. Schwing, and R. Urtasun. Latent structured active learning. Advances in Neural Information Processing Systems, 26, 2013.
[30] G. Ortiz-Jimenez, Modas, S.-M. Moosavi, and Frossard. Hold me tight! influence of discriminative features on deep network boundaries. Advances in Neural Information Processing Systems, 33:2935-2946 2020.
[31] A. Parvaneh, E. Abbasnejad, D. Teney, G. R. Haffari, A. van den Hengel, and J. Q. Shi. Active learning by feature mixing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12237-12246, 2022.
[32] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2111-222 2015.
[33] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint (el:::0080008 2017.
[34] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
[35] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[36] S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
12

---

[37] R. Strudel, R. Garcia, I. Laptev. and C. Schmid. Segmenter: Transformer for semantic segmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7262-7272, 2021.
[38] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[39] K. Wang. D. Zhang. Y. Li, R. Zhang, and L. Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology. 27(12):2591- 2600, 2016.
[40] Y. Xie, M. Ding, M. Tomizuka, and W. Zhan, Towards free data selection with general-purpose models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[41] Y. Xie, H. Lu, J. Yan, X. Yang. M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining-fineuiingg paradigm. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23715-23724, 2023.
[42] B. Xu, S. Shen, F. Shen, and J. Zhao. Locally linear svms based on boundary anchor points encoding. Neural Networks, 117:274-284, 2019.
[43] W. Xu, Z. Hu, Y. Lu, J. Meng, Q. Liu, and Y. Wang. Activede: Distribution calibration for active finetuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16996-17005, 2024.
[44] O. Yehuda, A. Dekel, G. Hacohen, and D. Weinshall. Active learning through covering lens. Advances Neural Information Processing Systems, 35:22354 22367, 2022.
[45] D. Yoo and I. S. Kweon. Learning loss for active learning. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 93-102, 2019.
[46] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[47] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ihot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:211.007832, 2021.
13

---

A Related Work
Due to space limitations in the main part, a more detailed account of related work is provided here.
A.1 Active Learning/ Finetuning
Active learning aims to maximize the efficacy of a limited annotation budget by strategically selecting the most informative data samples. In pool-based scenarios, existing algorithms generally adopt criteria such as uncertainty or diversity for sample selection. Uncertainty- -ased methods make selections based on the difficulty of each sample, assessed through various heuristics including posterior probability 24 39], entropy [8] 29), loss function [66 45]. or their impact on model performance [T0 28 Converscly, diversity- -based algorithms aim to select representative samples that closely approximate the distribution of the entire data pool. This concept of diversity can be
measured by the distance between global 33 and local [ 40) representations, through other
metrics such as gradient directions
[2] or adversarial loss [20 36
Nevertheless, the algorithms previously discussed are primarily tailored for training from scratch and encounter significant obstacles within the -inettininniinnuinggggg paradigm [22 41]. In response to this, ActiveFT 41 has been developed specifically for this framework. It operates by aligning the distribution of selected samples with that of the original unlabeled pool within the feature space. However, ActiveFT tends to prioritize high-density areas, frequently neglecting boundary samples. In contrast, our proposed algorithm integrates both diversity and uncertainty into the selection process, which is important for determining the decision boundary in the supervised finetuning.
recent work ActiveDC [43 employs pseudo-labeling techniques from semi-supervised learning. However, this approach focuses fine-tuning after sample selection rather than optimizing the sample selection process itself. While performs exceptionally well under extremely small budgets, our tests show that it falls short when compared to the K-Nearest Neighbors or Linear Probing methods we discuss in Appendix G
A.2 Decision Boundaries in Neural Networks.
Decision boundaries play a pivotal role in neural sowwork-based classification models, significantly affecting both performance and interpretability 23 25 Optimizing these boundaries can substan- tially improve model generalizability and accuracy, especially in complex, high sigh-dimensional data the case of support vector machines (SVMs), decision boundaries are integral to defining separating hyperplanes, and maximizing the geometric margin around these boundaries crucial for robust classification [ This principle is equally pertinent to neural networks, where enhancing the margin can similarly boost generalization capabilities. When dealing with imbalanced datasets, adjusting the decision boundary becomes essential for accurately classifying minority classes. To address this, techniques such LabellDistributionn-
environments [42 5)
Margin (LDAM) loss (5) and Enlarged Large Margin (ELM) loss [19] have been developed to refine the decision boundary, thereby improving the balance of generalization errors across different classes. Studies have shown that neural networks tend to utilize the most discriminative or simplest features in constructing decision boundaries 30 34]. Additionally, theoretical framework has been proposed for evaluating the complexity of decision boundaries using the novel metric of decision boundary variability, which is inversely related to generalizability 2
However, in the active finetuning task, the selection of decision boundaries has not traditionally been emphasized. Although challenging in unsupervised scenarios, we effectively leverage features from pretrained models, thereby introducing innovative denoising and selection methods without dependency on labels. This approach addresses significant gap in the active finetuning domain.
B Details of ActiveFT Method
selects the most useful data samples in the feature space of the pretrained model under the guidance of two basic intuitions: 1) bringing close the distributions between the selected subset P and the original pool P", maintaining the diversity of Pg. Formally, the goal is to find the
ActiveFT 41
14

---

optimal selection strategy S as follows.
Sopt = arg min DPy, Pıs) AR(FS)
(8)
where ((,,)) :) is some distance metric between distributions Ps. of pd and Pʃs of Ps. R(-) is to measure the diversity of a set, and À is scale balance these two terms.
Due to the difficulty in directly optimizing the discrete selection strategy S, Pfs is alternatively modeled with Рag. where fs = {60) jE(K) are the continuous parameters and K is the budget size of core samples. Each B' after optimization corresponds to the feature of a selected sample fay. We would find fs, closest to each 0's after optimization to determine the selection strategy S. The goal of this continuous optimization is written in Eq. 9
Spp arg min (pp, Pes) AR(0s) s.t.110'sl2
Ug
Following the mathematical derivation in [41] Eq. 9 can be solved by optimizing the following loss function through gradient descent.
L = (P.r..PP A- AR((s))
E fEFx
sim (f, 05://]++ + jE(B] (
log
exp sim (sim((g.()//7)
(10)
where sim(::)) is cosine similarity between normalized features, the temperature 0.07 and the balance weight À is empirically set 1.
After the optimization process, it finds features (fs, ]E([]] with the highest similarity to 0'g.
faz = arg max sfm 0'/) fe feFu
(11)
The corresponding data samples (xs,)eE/8 selected the subset P with selection strategy S (5ybeeık)
C Denoising Algorithm
We propose three progressive methods to address the challenge of denoising: the distance-guide method, the density-based method, and iterative density-based clustering (IDC), with the latter elaborated in detail in the main body of the paper. In this appendix section, we introduce two additional methods.
Distance-guide method: This method removes the top Nr.mm samples that are furthest from the current class center, based their distance. Although straightforward and simple, has limitations, particularly in scenarios where the class distribution varies significantly in different directions, resembling elliptical shape. In such cases, this rudimentary method may inadvertently remove distant cluster of samples instead of the actual noise.
Density-based method: This approach removes the top Nirm samples with the highest density distance from their class. The underlying rationale is that noise samples are generally farther away from their neighboring samples. Thus, using the distance nearby samples an auxiliary measure to identify noise is reasonable. The definition of density distance provided in Eq. 2
D Time Complexity of BiLAF
Theoretical Time Complexity. Given that our approach is bi-level method, the overall time is influenced by two components: Core Sample Selection and Boundary Sample Selection. Here, we analyse the theoretical complexity of the Boundary Sample Selection component in Algorithm
Define N the total number of samples, as the annotation budget, M aN as the number of core samples, T as the number of cluster iterations, where
0.5% in CIAFR 100 and T
15

---

1/P.n 1/10% = 10 in all the datasets. Based on Algorithm| | Samples Selection into three steps:
we divide the Stage 2 Boundary
1. Assigning the pseudo-class samples is O(N = O because we need to assign each data sample to its corresponding pseudo-center.
2. Iteratively density-basee clustering then denoising is O(TN/ /a). Considering that for each pseudo-center, the expected number of samples is N/M, we need to compute pairwise distances and find the k nearest samples, resulting in a complexity of O( O(N²/M²) Subsequently, we must sort these distances based density distance, which has complexity of ON/MM log N/M). We need to repeat the process for T times in the iterative clustering. Therefore the time is O(TN² /M² TN/M log N /M) for each pseudo-center. The total complexity is then multiplied by M, yielding an overall complexity of DTTN3/M + TN log N/M) = N/TN/a + TN log 1/a) = N(TN/o).
3. Selection process is O(N/a BN). Considering each pseudo-center, the expected number of samples is N/M. We need to calculate the pairwise distances to compute the intra-class distance, dintro. resulting in complexity of O(N² /M²). When considering the inter-class distances, due to the presence of the coefficient 8 adjustments are required each time, thus the complexity becomes [B,] X (N/M x M) (N),, where [B,] represents the number of selections within this pseudo-category, and Ei= [&] B. Therefore, the overall complexity in this process is O(N2/M BN) O(N/a BN).
Thus, the overall complexity O(aN2 + TN/a + BN). In practice, matrix multiplication and constants impact the speed across different steps.
Experimental Results. Tab. 3 provides the specific execution times of our method (including the time spent on selecting core samples using ActiveFT). Actually, we should not omit T, which is relatively small constant (e.g., 10), because T/a can be comparable in scale to B when B is small. Therefore, the complexity remains almost consistent according to different B in Tab. 3
E Experiment Setup
Due to space limitations in the main part, we provide a comprehensive overview of the experiment setup for classification tasks in this section.
Datasets and Evaluation Metrics. Our approach is evaluated using three widely recognized datasets:
CIFARIO, CIFARTOO (2]. and ImageNet-Ik (32 Both CIFARIO and CIFAR100 contain 60,000 images with resolutions of 32x32, but they differ in classification complexity, offering 10 and 100 categories, respectively. Each comprises 50,000 images for training and 10,000 for testing. The large-scale dataset ImageNet-IK includes 1,000 categories and a total of 1,281,167 training images along with 50,000 validation images. The raw training data serve as the unlabeled pool pd from which selections are made. We employ the Top- Accuracy metric for performance evaluation.
Baselines. We compare approach with three heuristic baselines, five active learning methods and the well-designed active finetuning method ActiveFT 41
Three heuristic baselines are listed as follows:
1. Random:
The selection of samples for annotation is entirely stochastic.
2. FDS: The method is also known as the K- -renter-Greedy algorithm, which selects the next sample feature that is farthest from the current selections. As proven in (33 it minimizes the expected loss over the entire pool and the selected subset.
3. K-Means: We implement the K-Means method on the feature pool F and select samples nearest to the centroids, where the number K equals to the budget size B.
Five active learning methods are listed as follows:
1. CoreSet 33
CoreSet selects samples that are central to the data distribution, effectively minimizing the maximum distance between any data point and the nearest selected sample, thus representing the core characteristics of the dataset.
16

---

2. VAAL [6]
The Variational Adversarial Active Learning (VAAL) framework combines variational autoencoders with adversarial learning facilitate active learning in an unsuper- vised setting. It utilizes a discriminator to distinguish between labeled and unlabeled data, guiding the selection of samples likely to enhance model performance upon labeling.
3. LearnLoss 45 LearnLoss employs a deep learning strategy to estimate the value of laheling each data sample. It involves training a secondary network alongside the primary model to predict the potential reduction in loss from labeling specific unlabeled samples.
4. TA-VAAL [20]: Twsk-Aware Variational Adversarial Active Learning (TA-VA VAAL) enhances the VAAL framework by integrating task-specific objectives into the adversarial model, making the active learning process more attuned to the requirements of the specific task, thereby optimizing data selection.
5. ALFA-Mix 31: ALFA-Mix merges active learning with data augmentation techniques to boost model performance. It selects informative samples and employs mixup strategy to create synthetic samples by blending selected data points, thereby enriching the training dataset and promoting robust feature learning.
These methods has been modified and applied specifically for active finetuning tasks following the instructions 4T. These approaches utilize batch-selection strategy for data sampling. Initially, the model is trained on a randomly chosen initial dataset. Subsequently, it employs this trained model to pick a batch of images from the training set. The model is then retrained using the cumulatively selected samples. The chosen active learning techniques encompass strategies based on both diversity and uncertainty in the active learning domain, serving as the baselines in the active finetuning field.
In this scenario, ActiveFT [1 emerges as the strongest baseline due to its tailored design for the task. This approach generates representative subset from the unlabeled pool by aligning its distribution with the entire dataset and enhances diversity through the optimization of parametric models within a continuous space.
Implementation Details, In the unsupervised pretraining phase, we use DeiT-Small 38 model, pretrained using the DINO [6] framework on ImageNet- 1k. due to its recognized efficiency and popularity. For all three datasets, we resize images to 224 x 224 consistent with the pretraining for both data selection and supervised finetuning. In core samples selection stage, we utilize ActiveFT and optimize the parameters Os using the Adam 21 optimizer (learning rate 1e-3) until convergence. number K as 50(0.1%) 1%), 250(0.5%) (405().5%) for CIFAR1O, CIFAR100 and
We set the ImageNet separately. In the boundary samples selection stage. we set nearest neighbors number k as 10, both removal ratio P.... and clustering fraction P... as 10%, opponent penalty coefficient& d as 1.1. In the supervised finetuning phase, we finetune DeiT-Small model following the setting in [4]. We finetune the models using the SGD optimizer with learning rate as 3e-3, weight decay as 1e-4 and momentum as 0.9. We employ cosine learning rate decay with a batch size of 256 distributed across two GPUs The models are finetuned for 1000 epochs on all datasets with different sampling ratios, except for ImageNet with sampling ratio 5%, where we finetune for 300 epochs. Our experiments are implemented using the mmclassification framework
F
More Extensive Experiments
To further investigate the generalizability of our method, we conducted extensive experiments. Firstly, We applied our method to diverse downstream tasks, including Object Detection (F.I) and Semantic Segmentation datasets (F2). Then, we explored the performance of our method in different scenarios, such as the long-tail distribution scenarios
[.3). Finally, we conduct experiments on fine-grained
classification scenarios
F11 Object Detection
Datasets. We conduct our experiments on the PASCAL VOC dataset 99.
Following established protocols from prior studies (5] we merge the training and validation sets of PASCAL VOC 2007 and 2012, forming a training data pool of 16,551 images. We assess the performance of the task
17

---

model on the PASCAL VOC 2007 test set, utilizing the mAP (mean Average Precision) metric for evaluation.
Implementation Details. To facilitate comparison and model alignment in testing, we only use the Dei'--mmall for feature extraction and data selection in this dataset. In the data selection stage, We set the core number K as 500. The other settings are the same as described in Sec. [.1 in main part. Then in the finetuning process, for alignment with methods previously documented in [45], we train an SSD-300 model [26 with a VGG-16 backbone [35] on the selected samples. Following protocols from prior studies [45], we train the model for 300 epochs with batch size of 32. We utilize an SGD optimizer with momentum of 0.9. The initial learning rate is set at 1e-3, decaving to 1e-4 after 240 epochs. Our experiments are implemented using the mmdetection framework
F.2 Semantic Segmentation
Datasets. For the segmentation task, we utilize the ADE20K dataset [46], which comprises 20,210 images for training. 2,000 images for validation, and 3,352 images for testing. Each image is annotated with fine-grained labels across 150 semantic classes. We designate the training set as the unlabeled pool 7d for selection purposes. The performance is assessed using the mloU (mean Intersection over Union) metric.
Implementation Details. We utilize DeiT-Small the same as in image classification tasks. In the data selection stage, We set the core number K as 2.5% of data samples. The other settings are the same as described Sec. [41 in main part. We resize the images to 224 224 well and adopt Segmenter (37] for finetuning in the segmentation task, which is pure transformer model, using the same DeiT-Small model as its backbone for finetuning. The model is trained for 127 epochs following [37], corresponding to 16k/32k/4SK iterations 5%/10% 15% of the training data, respectively. Training employs the SGD optimizer (learning rate 1e-3, momentum 0.9) with batch size of 8 and polynomial decay of the learning rate. Our experiments are implemented using the mmsegmentation framework
model pretrained with DINO framework (6]
F.3 Long- Taig-T Classification
Datasets. In the long-tailed datasets, define the imbalance ratio (IR) as IR Tmmaz/ Remins where Nyanr and Temin represent the number of training samples in the largest and smallest class, respectively. CIFARIO0LY derived from the original balanced CIFAR 100 dataset (22] which includes 50,000 training images and 10,000 test images, each sized 32x32 across 100 classes. Following the methodology of [5), we constructed the long-tailed version, CIFARI0OLT, by applying exponential decay to the sampling of training images per class, while keeping the balanced test set unchanged. We utilize CIFAR 100LT with imbalance factors (IR) as 100 our experiments. CIFAR 100TT (IR=100) has 10847 training samples.
Implementation Details. We set the core number K 217(2%) for CIFAR10OLT. Other settings are consistent with the main experiment. See Appendix for reference.
Table 9: CUB-200-2011 Dataset. Comparison of Random, ActiveFT, and BiLAF on Different
Selection Budget.
Budget
Select Number
Random
ActiveFT
BiLAF (Ours)
20%
1198
46.32
47.83
48.53
30%
1798
58.85
59.67
60.52
40%
2397
66.75
67.36
68.31
50%
2997
72.98
73.25
74.06
Phupp://githubbb phtpp://gituu...oo//ppoop http:///ww...cmooooooommmm////me
18

---

F.4 Fine-Grained Classification
We utilized the CUB-200-2011 dataset, which includes 200 bird species with a total of 11, 788 images. According to the default configuration of the dataset, 5, 994 samples are used for training, with the remainder used for testing. Given the large number of classes, Samples to select Boundary Samples, while all other parameters were set according to the default values reported the paper. Tab. Tab.Dcless Clearly demonstrates that our approach continues hold leading position in fine-grained classification datasets.
used 10% of the data Core
G Extra Finetuning Methods: Linear Probing and K-Nearest Neighbors
Aside from the Full Fine-Tuming, Linear Probing or even K-Nearest Neighbors (KNN) classifiers are also effective approaches. In this section, explore whether BiLAF remains selecting high-quality data points with these fine-tuning methods. We conducted experiments using Linear Probing and KNN on the CIFARIO dataset, comparing the results across Random, ActiveFT, and BiLAF (ours).
Table 10: Results CIFAR10 using KNN, Linear Probing, and Full Fine- Tine-Tuning across different budgets.
Finetuning Methods
Selection Method
B = 0.5%
=
1%
2%
B 5%
K-Nearest Neighbors
Random
82.1
86.7
88.2
90.8
K-Ncarest Neighbors
ActiveFT
86.8
87.2
88.5
91.7
K-Nearest Neighbors
BiLAF (ours)
85.7
87.4
88.7
91.8
Linear Probing
Random
85.1
87.6
90.1
92.5
Linear Probing
ActiveFT
87.8
88.7
90.6
92.8
Linear Probing
BiLAF (ours)
86.5
89.1
91.0
93.0
Full Fine-Tuning
Random
77.3
82.2
88.9
94.8
Full Fine-Tuning
ActiveFT
85.0
88.2
90.1
95.2
Full Fine-Tuning
BiLAF (ours)
81.0
89.2
92.5
95.7
From the table, the following conclusions can be drawn:
1. At extremely low data volumes, Linear Probing and K-Nearest Neighbors outperform Full Fine-Tuning.
2. As the data volume increases, the performance improvements of Linear Probing and K-Nearest Neighbors start to slow down, which gradually necessitates the use of Full Fine-Tuning Tuning.
3. Interestingly, the quality of data selected by different methods shows a consistent trend across K- Nearest Neighbors classifiers, Linear Probing, and Full Fine-Tuning Our method, compared to competitors, is able select more suitable data, which effective across different fine-tuning paradigms.
H Analysis of Selection Samples to the Decision Boundary.
We conduct linear probing on all the samples with true labels using features from both the pre- pre-trained model and the oracle model which is fully finetuned all samples. We analyze whether samples selected using different boundaries. We uncertainty and a propensity towards boundary samples. 2) ProbDiff, calculated as the difference between the highest and second-highest probabilities. A smaller value indicates that the sample is closer to the boundary between these two classes
ActiveFT, BiLAFF (ours)-tend to be near the decision metrics for this analysis: 1) Entropy, where higher value indicates greater
From the Tab boundary samples, mantaining consistency across models with various capabilities.
S and 14 the results demonstrate that method effectively select
19

---

Table 11: Results of Entropy and ProbDiff on CIFAR1O using Pretrained Model.
CIFAR10 (Pretrained)
Selected Nums
Entropy
ProbDiff 1 Entropy (Top50) 1 ProbDiff (Top50)
Random
250
0.0935
0.9486
0.4260
0.7536
ActiveFT
250
0.0424
0.9747
0.2073
0.8743
BiLAF (ours)
250
0.1023
0,9366
0.4769
0.6924
Random
500
0.0960
0.9416
0.7022
0.5056
ActiveFT
500
0.0433
0,9763
0.3690
0,7799
BiLAF (ours)
500
0.1149
0.9273
0.7089
0,4500
Random
1000
0.0955
0.9430
0.9181
0.308
ActiveFT
1000
0.0849
0.9495
0.8810
0.3560
BiLAF (ours)
1000
0.1461
0.9064
1.0262
0,2005
Table 12: Results of Entropy and ProbDiff on CIFAR 100 using Pretrained Model.
CIFARIO0 (Pretrained)
Selected Nums
Entropy
ProbDiff
Entropy (Top50) f ProbDiff (Top50)
Random
500
0.6240
0.7295
2.2516
0.0876
ActiveFT
500
0.2962
0.8664
1.5663
0,2369
BiLAF((ours)
500
0,3933
0.8167
1.7832
0.1423
Random
1000
0.5317
0.7766
2.2375
0.0594
ActiveFT
1000
0.3650
0.8430
2,0812
0,0833
BiLAF
(iLAF(oous) (ours)
1000
0.4751
0.7815
2.2606
0,0542
Random
2500
0.5253
0.7749
2.6790
0.0232
ActiveFT
2500
0.4851
0.7936
2.6653
0,0206
BiLAF (ours)
(ours)
2500
0,5795
0.7476
2.7196
0.0192
Random
5000
0.5442
0.7652
2.8791
0.0151
ActiveFT
5000
0.5197
0.7768
2.8487
0.01
BourF(
5000
0.6219
0.7336
2.9194
0.0090
Table 13: Results of Entropy and ProbDiff on CIFARIO using Oracle Model.
CIFAR10 (Oracle)
Selected Nums
Entropy
ProbDiff
Entropy (Top50) f ProbDiff (Top50)
Random
250
0.001512
0.999831
0.002437
0.999729
ActiveFT
0.001521
0,999830
0.002403
0.999717
BiLAF (ours)
250
0.001541
0.999827
0.002473
0,999714
Random
500
0,001483
0.999835
0,002782
0.999676
ActiveFT
500
0,001542
0,999828
0.002958
0,999644
BiLAF (ours)
500
0.001605
0,999820
0.003020
0.999641
Random
1000
0.001551
0,999823
0.003543
0.999575
ActiveFT
1000
0.001527
0,999829
0.003472
0,999586
BiLAF (ours)
0.001598
0.999820
0.003588
0.999567
Table 14: Results of Entropy and ProbDiff on CIFAR100 using Oracle Model.
CIFAR100 (Oracle)
Selected Nums
Entropy
ProbDiff
Entropy (Top50) 1 ProbDiff (Top50)
Random
500
0.049238
0.992216
0.176811
0.956287
ActiveFT
500
0.040140
0.993335
0.124594
0.962662
BiLAF (ours)
500
0.042792
0.994406
0.137114
0.965457
Random
0.047730
0.993905
0.202417
0.963037
ActiveFT
1000
0.042763
0.993913
0.198896
0,961337
BiLAF (ours)
1000
0.045447
0.993854
0.203449
0.960462
Random
2500
0.046654
0.993407
0,297133
0.919803
ActiveFT
2500
0.047036
0.993245
0,303767
0.919122
BiLAF (ours)
2500
0.047357
0,993103
0.309907
0.917868
Random
5000
0.047028
0.993741
0.407092
0.897825
ActiveFT
5000
0.045173
0.994264
0.340326
0.925870
BiLAF (ours)
5000
0.049476
0.992885
0.466601
0.842029
20

---

I Limitation and Social Impact.
Limitation. In this paper, our approach is based on general features and purposes. For specific tasks, such as object detection task with multi-insttacess, our method does not show a significant advantage. What's more, in dealing with sample features long-tail problems, denoising methods might remove outliers that are key samples. However, as a flexible framework, we can make corresponding improvements for different downstream tasks, which can be considered as future work.
Social Impact. We firmly believe that this work has positive impact society. By replacing iterative sample selection with one-shot sample selection, our approach not only saves substantial resources that would otherwise be wasted in training, but also enhances labeling efficiency without the need for multiple arrangements. Additionally, our method achieves state-of- of-thef-- performance among similar approaches based on small number of samples, effectively reducing resource waste caused by ineffective labeling.
J Qualitative Visualization
(a) Denoising Ratio Prm 0%
(b) Denoising Ratio Prm 10%
(c) Denoising Ratio P... 20%
(d) Denoising Ratio P... 30%
Figure 5: Denoising Strength Visualization using tSNE Embeddings of CIFAR10.
Denoising Visualization. Fig. 5 illustrates the impact of variations in the removal rate Pro during the denoising process on the retained samples. The red bounding boxes highlight regions with significant changes. These areas include obvious outliers and zones of confusion where multiple class intermingle. As the removal rate increases, the number of samples in these areas tends to decrease gradually, thereby reducing their influence on subsequent boundary point selection, while the relatively dense boundaries are often preserved. However, this represents a trade-off as some important samples might also be removed. Therefore, we have conducted a quantitative analysis in the main text to address this concern.
Selected Samples Visualization. Fig. 6 displays the sample selection by different methods when annotating 1% samples from the CIFARI00 dataset. It is evident that the FDS method has significant
21

---

(a) FDS
(b) K-Mcans
(c) ActiveFT
(d) BiLAF (ours)
Figure 6: ISNE Embeddings on CIFAR10 with 1% annotation budget of our BiLAF method and other methods.
drawbacks, leading to insufficient selection of sample samples from central classes in feature space, which greatly impedes the model's learning capability. Both the K-Means and ActiveFT methods focus selecting central samples, differing in their optimization goals and processes. As seen in Fig. 66 both methods achieve their intended purpose, resulting in a relatively uniform selection of samples. Our method further refines this approach. Firstly, identify the central samples, represented by pentagrams, and then expand to boundary samples, denoted by circles, from each center. Our strategy emphasizes the boundaries between categories, rather than conforming to the entire distribution. For example, considering the 'brown' sample class, our method selects fewer samples its internal area, far from other categories, and focuses more on locating boundary samples. Extensive experimental results quantitatively demonstrate the substantial improvement our method brings model performance, validating its effectiveness.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The main claims are clearly presented in the abstract and introduction, and further elaborated and substantiated throughout the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Appendix [
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
Refer to Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In paper, we provide detailed descriptions of each step of the algorithm, the pseudocode, and all settings required to replicate the experiments. Additionally, we commit to making the code publicly available by the time of the camera-ready submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions each step of the algorithm, the pseudocode, and all settings required to replicate the experiments. Meanwhile, we make the code publicly available with the GitHub link in the title.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (ht.:ps:/nniss. cc/
public/ gesds///////////u////os/ssss/
des CoodeSubme ss annolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 httppsscc/ppbbbccuu.ccooo CCoee
ss ceiiinnolliyy Stt for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to the main part and Appendix D and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean and variance of the experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///waaarar.coo/////cc/cccaa
Answer: [Yes]
Justification: Our work adheres the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer to Appendix
D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of all assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "yCf6JBHOIY", "reviewer": "Reviewer_V2Vc", "summary": "This paper introduces BiLAF, a novel approach for selecting boundary examples alongside core samples to enhance the fine-tuning of pre-trained models for downstream tasks. Specifically, the boundary selection strategy leverages the distinction between intra- and inter-class distances within the pre-trained feature space and incorporates an opponent penalty to promote diversity across different boundaries. Experiments and ablation studies demonstrate the effectiveness of the proposed BiLAF in achieving state-of-the-art results.", "questions": "Besides the weakness shown in the above section, please also see the following questions: \n\n- The efficacy of pre-trained features for the fine-tuning task is essential for BiLAF. Would there be a consistent improvement if we first apply unsupervised pre-training on the unlabeled fine-tuning set?  \n\n- Algorithm 1's pseudo-code indicates that ActiveFT is the first stage of BiLAF. However, Table 3 suggests that BiLAF is faster than ActiveFT. Why BiLAF is faster than ActiveFT if it needs to run ActiveFT first? \n\n- It is an interesting idea to consdier an opponent penalty to encourage diversity across boundaries. I wonder if this penalty has varying effects on different pseudo classes. Specifically, does it help more on a pseudo class with more neighboring classes (which could potentially be the more confusing class), like the pink one in Figure 3, while helping less for those on the border, such as the green one Figure 3?", "limitations": "In the appendix, the paper acknowledges a limitation in the proposed method, as it is based on general features. However, as I indicated in the weakness section, the paper lacks a discussion/study to investigate the severity of this limitation and its potential impact on the performance.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. \n- In addition to classification tasks, the experiments include different scenarios such as detection and segmentation. In these settings, BiLAF demonstrates the state-of-the-art results. \n- Ablation studies and execution time are included in the analysis to help provide a deeper understanding of the proposed BiLAF.", "weaknesses": "- BiLAF seems to heavily rely on pre-trained features, as all of its data selection processes are based on them. BiLAF's reliance on pre-trained features for data selection raises concerns due to potential discrepancies between pre-training and fine-tuning tasks. Note that the features from pre-trained classifiers are commonly used in traditional active learning because the labeled and unlabeled datasets are usually from the same task. However, this might not hold true in general pre-training and fine-tuning paradigms. Moreover, Line 120 mentions that “By leveraging pre-trained models, data samples are mapped to robust feature representations that elucidate the relationships among samples, their intra-class counterparts, and inter-class samples from diverse classes.” It remains unanswered and uncertain to what extent these pre-trained features are robust enough to effectively apply BiLAF. \n\n- In the typical pre-training and fine-tuning paradigm, full fine-tuning may not always be the optimal approach. For instance, when we only have limited/few-shot data, techniques like linear probing or even nearest-neighbor classifiers can yield better results. Furthermore, the choice of fine-tuning method can also be influenced by the similarity between pre-training and downstream tasks [1]. However, the paper lacks a discussion of this aspect and solely focuses on full fine-tuning, neglecting the potential benefits of alternative methods. Specifically, does BiLAF remain necessary for selecting high-quality data points, if we can apply more suitable fine-tuning methods? Or would simple random sampling suffice? \n\n[1] Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning, ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Concern about pre-trained features. Would there be a consistent improvement if we first apply unsupervised pre-training on the unlabeled fine-tuning set?***\n\n**R1:** \n1. Our main experiment utilized a checkpoint pre-trained on ImageNet using DINO. We conducted studies both **on consistent tasks (ImageNet) and inconsistent ones (CIFAR & Others)**, demonstrating that our method is effective regardless of whether the pre-trained features come from the same task.\n2. Furthermore, we explored features trained **using different paradigms and architectures**, such as iBOT and DINO, as well as ResNet50 and DeiT-S, detailed in Appendix F’s Ablation Study. This highlights the generalizability of our approach across different feature sets.\n3. It is widely acknowledged and applied that many pre-trained models, such as DINOv2 and CLIP, claim robust transferability across diverse downstream tasks after extensive pre-training on large datasets.\n4. Due to constraints of time and resources, we were unable to perform additional unsupervised training. However, we conduct experiments to show that the boundary points selected based on pseudo-classes are **consistent** with the actual boundaries, and the boundaries selected based on pre-trained features are consistent with using fully-trained Oracle models. The experiments results will be detailed in the following offical comments due to the characters limitations. \n\n\n>***Q2: Full fine-tuning may not always be the optimal approach? Linear probing or even nearest-neighbor classifiers can yield better results. Specifically, does BiLAF remain necessary for selecting high-quality data points with more suitable fine-tuning methods?***\n\n**R2:** Thank you for your insightful comments and suggestions. Following your ideas, we experimented with Linear Probing and KNN on the CIFAR10 dataset and compared the results across Random, ActiveFT, and **BiLAF(ours)**.\n\n| CIFAR10       | 0.5% | 1%   | 2%   | 5%   |\n|-------------|------|------|------|------|\n| **KNN**     |\n| Random      | 82.1 | 86.7 | 88.2 | 90.8 |\n| ActiveFT    | 86.8 | 87.2 | 88.5 | 91.7 |\n| BiLAF       | 85.7 | 87.4 | 88.7 | 91.8 |\n| **Linear**  |\n| Random      | 85.1 | 87.6 | 90.1 | 92.5 |\n| ActiveFT    | **87.8** | 88.7 | 90.6 | 92.8 |\n| BiLAF       | 86.5 | 89.1 | 91.0 | 93.0 |\n| **Finetune**|\n| Random      | 77.3 | 82.2 | 88.9 | 94.8 |\n| ActiveFT    | 85.0 | 88.2 | 90.1 | 95.2 |\n| BiLAF       | 81.0 | **89.2** | **92.5** | **95.7** |\n\nBased on these results, we have supplemented our experiments with 5% budget. From the table, the following conclusions can be drawn:\n\n- At extremely low data volumes, Linear Probing and KNN outperform full fine-tuning.\n- As the data volume increases, the performance improvements of Linear Probing and KNN start to slow down, which gradually **necessitates the use of full fine-tuning**.\n- Interestingly, the quality of data selected by different methods shows a consistent trend across KNN, Linear, and full fine-tuning. **Our method, compared to competitors, is able to select more suitable data, which is effective across different fine-tuning paradigms.**\n- Following prior work, our original focus was on full fine-tuning. We appreciate your comments, which have helped enhance the comprehensiveness of our work.\n\n>***Q3: Why BiLAF is faster than ActiveFT if it needs to run ActiveFT first?***\n\n**R3:** The primary reason is that ActiveFT in our approach is utilized solely for selecting a core sample set, not the entire dudget $B$ as 2%, 5%, 10%. In our CIFAR100 experiments, we fixed the core number as 0.5%. Therefore, the time consumption for BiLAF mainly comprises the time ActiveFT takes to select 0.5% of the samples and the time for further boundary sample selection, which can be faster than directly selecting all samples using ActiveFT. For example, when B=2%, the denoising process takes $5.98$ seconds and iterative boundary sample selection takes $4.12$ seconds, with additional time allocated to ActiveFT tasks selecting core points and other operations. A significant portion of the time, including the denoising, is independent of $B$. When $B$ is small, the impact of boundary sample selection based on $B$ is minimal, resulting in a negligible increase in runtime for our method across different $B$ values. For a comprehensive time complexity analysis, please see Appendix D.\n\n>***Q4: Specifically, does it help more on a pseudo class with more neighboring classes while helping less for those on the border?***\n\n**R4:** In an ideal scenario, if a central representative is identified for each class, the class on the border would select samples closer to other classes rather than samples farther away, as demonstrated in Figure 1. For instance, as shown in Figure 3, the green class would select adjacent samples from the blue, pink, orange, or even light blue categories, thereby promoting diversity at these boundaries. Such selections can significantly enhance the distinguishability of boundary classes from others, demonstrating that the 'opponent penalty to encourage diversity across boundaries' is a **universally applicable strategy**. However, in light of your concerns and upon further reflection, we recognize that as the number of boundary selections increases, the advantages for border classes may not be as pronounced as for central classes, which may require more samples to depict their more complex and varied boundaries accurately. Optimizing the distribution of boundary points based on the positions of the centers presents a compelling and valuable new direction for future research.\n\nThank you once again for your valuable time and insightful comments, which have greatly enhanced our work. We hope our responses have addressed your concerns and demonstrated the versatility of our method. We look forward to further discussions and sincerely appreciate it if you could reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HebatvccDO", "reviewer": "Reviewer_2Ep4", "summary": "This paper proposes a Bi-Level Active Finetuning Framework (BiLAF) for optimizing sample selection in the pretraining-finetuning paradigm. BiLAF combines global diversity and local decision uncertainty through two stages: core sample selection and boundary sample selection. Without requiring labels, the method successfully identifies pseudo-class centers, employs a tailored denoising technique, and iteratively selects boundary samples. Experimental results demonstrate that BiLAF consistently outperforms existing baseline methods across various vision tasks, demonstrating its superior efficacy in improving model performance.", "questions": "1. What does c_i in equation (1) refer to? Previously, c_i was used to denote the cluster centers, but here it seems the authors use c_i to represent the assignment of f_j. Additionally, what does a_i of X_i in section 3.3.2 represent? The inconsistent use of symbols is confusing.\n2. On what basis do the authors claim that this algorithm is effective on imbalanced datasets? Particularly when the authors state that classes with fewer samples might disappear during the denoising and iterative selection processes.\n3. In the explanation of Fig. 3, the authors state that \"Pentagrams represent the selected core samples, while circles denote the chosen boundary samples.\" However, the figure shows that the learned center points are still biased towards the boundaries. Does this not risk the model focusing too much on the boundaries and neglecting the intra-class distribution characteristics?", "limitations": "The authors acknowledge the issue that in long-tail scenarios, the denoising methods might remove outliers that are key samples. However, what the authors do not mention is that the numerous hyperparameters in their algorithm can lead to difficulties in achieving optimal performance and even cause instability in different scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured, with clear organization into subsections that introduce the different stages of the method. The use of an algorithm to summarize the entire process is effective.\n2. The approach not only focuses on the centers of each class but also pays attention to the boundary samples between classes, effectively balancing global diversity and local decision uncertainty when selecting samples for annotation.\n3. The iterative selection and removal strategy, along with the opponent penalty, effectively prevents the aggregation of multiple samples near the same pseudo-class boundary that faces the same opposing pseudo-class center.", "weaknesses": "1. There is ambiguity in the use of symbols in the method description, leading to unclear explanations.\n2. The authors claim that the method is effective for imbalanced datasets but do not provide a thorough theoretical explanation. Additionally, experimental results show that the improvement over ActiveFT on the CIFAR100LT dataset with a 15% annotation ratio is only 0.3%, whereas the improvement is much more significant on balanced datasets. This contradicts the authors' claim. The authors explain that \"the default denoising removal ratio parameters might remove minority samples in long-tail distributions,\" which highlights a potential issue with their algorithm design under long-tail conditions, conflicting with their earlier statement.\n3. The ablation study shows that the inclusion of the opponent penalty contributes little to the performance improvement and even causes performance degradation under lower budgets. The necessity and design of the opponent penalty need to be reconsidered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Important Notes***. Reviewer’s recognition of \"numerous hyperparameters can cause instability in different scenarios\" in Limitations is not justified. Except *Core Number*, we **consistently employs the same hyperparameters across all six different tasks**, such as *nearest neighbors number*, *denoising rate*, *clustering fraction* and *opponent penalty*. The adjustment in Core Number is tailored to the varying class numbers across tasks, and we **ensure uniformity in Core Number for different budgets within the same task**. This uniformity **underscores the stability and scalability of our framework**. Changes in these parameters could potentially elevate the performance ceiling of our method. In our Ablation Study, alterations in the Core Number and Core Selection Method indicated further enhancements.\n\n>***Q1: There is ambiguity in the use of symbols, such as $c_i$ in equation (1), $a_i$ of $X_i$ in Sec.3.3.2.***\n\n**R1:** The sample set is $X=\\{x_1,\\ldots,x_N\\}$, where the subscript $i$ represents the sample $x_i$ and its corresponding feature $f_i$. In Section 3.2, Core Sample Selection identifies $|C|$ center points, represented by the index set $C=\\{c_1,\\ldots,c_K\\}, c_i\\in[N]$. In Section 3.3.1, Equation (1), for each sample $x_j, j\\in[N]$, we find its nearest center point $f_{c_i}$ in the feature space, which is closest to its feature $f_j$. We then assign this sample $x_j$ to the cluster controlled by the center $c_i$, denoted as $U_i$. \n\nIn Sec.3.3.2, we introduce $a_{i,j}$, where $j \\in [|U_i|]$, to represent the index of the $j$-th element in the set $U_i$ controlled by the $i$-th center $c_i$. Since subsequent operations are performed on each set $U_i$ individually, $a_{i,j}$ serves as the index of the sample $x_{a_{i,j}}$ and its feature $f_{a_{i,j}}$. Actually, $U_i$ records all the indices of samples $X_i$ belong to the center $c_i$.\n\nThe variable $j$ serves as an enumeration variable in different contexts, which might have led to some ambiguity. We will address these ambiguities and provide clearer explanations in the revised version.\n\n\n>***Q2: Why is the method effective on imbalanced datasets? Improvement on the CIFAR100LT dataset with 15% ratio is only 0.3% with the claim \"the default denoising removal ratio parameters might remove minority samples\".***\n\n**R2:** Firstly, these methods are not specifically tailored for long-tailed distributions. Our method, which focuses on boundary-based approaches, exhibits strong representational capabilities in general scenarios, providing us with a natural advantage. This supports our claim of the method's universality based on its performance in long tailed dataset.\n1. Denoising involves a trade-off; excessive denoising can eliminate minority classes at the boundary. However, totally without any denoising, selecting boundary samples based on minority class centers could mistakenly include majority class samples because minority and majority classes are often close in boundaries. \n2. Objectively, as the annotation rate increases, ActiveFT tends to select long-tailed samples as well, which may reduce the gap as the data volume grows.\n3. The default core number and denoising ratio were selected in the paper, which could potentially exclude long-tailed samples due to their isolated distribution or outlier status. Upon adjusting the Core Number and Denoising Ratio in experiments, we discovered that the model has considerable improvement.\n\n|Denoising Ratio|0%|5%|10%|20%|\n|-|-|-|-|-|\n|Core Number 2.5%|37.10±0.60|37.61±0.66|37.33±0.71|37.04±0.89|\n|Core Number 5%|37.24±0.56|**37.85**±0.59|37.67±0.64|37.17±0.78|\n\n>***Q3: The necessity and design of the opponent penalty need to be reconsidered with little contribution to the performance improvement.***\n\n**R3:** In Ablation Study, we observed an intriguing trend: the negative impact of the components examined in IDs 1-4 generally diminishes as the number of selected samples increases, while the benefits of the opponent penalty grow with a larger sample size. The design intention behind this metric is to prevent the selection of too many similar boundary points, which can degrade model performance. At sample sizes of 5% and 10%, the opponent penalty yields an accuracy improvement of over 0.5%, which we find significant.\n\nAs the important notes explained, we used the same hyperparameters in all experiments. In practice, parameters such as Core Number, Denoising Rate, and Opponent Penalty can be further tailored to specific scenarios to enhance the model's training capacity. This metric can also be applied flexibly depending on the amount of data selected.\n\n>***Q4: Learned center points are still biased towards the boundaries? Does this not risk the model focusing too much on the boundaries and neglecting the intra-class distribution characteristics?***\n\n**R4:** \nThe possible reasons are 1)Each class have multiple centers 2)TSNE visualizes data in two dimensions, which can be the centers in high-dimensional space. Actually, in our method, Opponent Penalty **encourages diverse explorations across boundaries**, allowing a more varied distribution. Iterative Selection removes nearby points of selected samples, **maintaining diversity by choosing boundary-nearer points instead of regional centers**. What's more, we might sample multiple points within a class as pseudo-centers. Boundary points between pseudo-classes of the same label can act as internal points of the class to better fit the distribution. Therefore, our method not only focuses on the classification boundaries but can also enhance the intra-class distribution.\n\nThank you for valuable time and insightful comments. We hope our responses have addressed your concerns. We welcome further discussions and sincerely appreciate it if you could reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r12Qc0iC5i", "reviewer": "Reviewer_8trC", "summary": "In this paper, the authors introduce a novel Bi-Level Active Finetuning Framework (BiLAF) designed to address the limitations of existing active learning methods in the context of the pretraining-finetuning paradigm. The framework aims to optimize sample selection for finetuning models within a limited annotation budget. The authors propose an innovative unsupervised denoising technique to eliminate noisy samples and use a newly designed boundary score metric for iterative boundary sample selection. Extensive experiments demonstrate that BiLAF outperforms existing methods across various datasets and tasks.", "questions": "Same as weakness.", "limitations": "As described in the Weakness section, there might be some critical aspects that are not verified enough. The authors are encouraged to show additional results to support their claims.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The bi-level framework that combines core sample selection with boundary sample selection effectively addresses limitations in existing active learning methods.\n- The novel unsupervised denoising technique effectively eliminates noisy samples, improving the reliability of sample selection.\n-  Extensive experiments on multiple datasets and tasks consistently show BiLAF outperforms state-of-the-art methods.\n- The paper is well-organized with a clear explanation of the motivation, proposed method, and experimental results.", "weaknesses": "Activate Finetuning is advantageous for model fine-tuning in scenarios with limited data, and the motivation behind the proposed BiLAF is clear. However, I have concerns regarding the generalizability of this method to different data sizes. As shown in Table I, there is a significant performance drop in the CIFAR10 dataset under the 0.5% setting. This decline might be attributed to the small size of CIFAR10 images and the low 0.5% ratio, making it challenging to select and denoise boundary samples. It raises the question of whether BiLAF requires a certain threshold of fine-tuning data to be effective, which warrants further investigation by the authors. Additionally, the CVPR 2024 paper (see reference below) reports on the performance of CIFAR10 under 0.1% and 0.2% settings, which is highly relevant. I believe that Activate Finetuning would be more meaningful in scenarios with extremely limited data (e.g., few-shot fine-tuning), where changing the random seed can lead to significant accuracy fluctuations. Therefore, I recommend the authors include experiments and analyses on such scenarios. Furthermore, comparative experiments with other settings in the paper should be included to substantiate the method's effectiveness.\nMoreover, when the amount of fine-tuning data increases, BiLAF's performance becomes comparable to the baseline ActiveFT. Hence, it would be insightful for the authors to discuss whether BiLAF remains effective with larger amounts of fine-tuning data, such as 20% of the ImageNet 1k dataset. This additional discussion would enhance the paper's comprehensiveness.\n\nXu, Wenshuai, et al. \"ActiveDC: Distribution Calibration for Active Finetuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Threshold: Whether BiLAF requires a certain threshold of fine-tuning data to be effective?.***\n\n**R1:** This is an excellent question. The performance gap with just 0.5% of CIFAR-10 data indeed triggers considerations about scale. In traditional active learning, the applicability of different methods varies with scale. The classic Coreset[1] method aims to cover all samples with the selected points, whereas ProbCover[2] notes that Coreset performs poorly when the data scale is very small, leading to the proposal of a coverage radius to ensure each selected point controls a fixed area. In our scenario, it is evident that using central points to fit different dense distributions appears to be a more rational choice when the budget is extremely limited, without using additional samples for boundary selection. We believe that this situation is both normal and objectively present.\n\nFollowing this, we **explored what such a threshold** might look like. We employed the Core Sample Selection method to obtain different numbers of central points and then analyzed the benefits these central points brought. Specifically, we define ***Distance*** as the mean Euclidean distance from each sample to the nearest selected point in the feature space and investigate the ***Rate of Return*** (Incremental Benefit per Core Sample) in different range, where *Rate of Return = Distance Difference / Core Number Difference between two adjacent columns.*\n- CIFAR10\n\n|Core Num|50|100|150|250|375|500|1000|1500|2500|5000|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Distance|0.7821|0.7588|0.7472|0.7307|0.7203|0.7117|0.6896|0.6746|0.6506|0.6010|\n|Rate of Return $\\times 10^{-4}$|-|4.6575|2.3264|1.6422|0.8362|0.6887|0.4420|0.3002|0.2398|0.1984|\n- CIFAR100\n\n|Core Num|50|100|150|250|375|500|1000|1500|2500|5000|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Distance|0.8378|0.8082|0.7913|0.7724|0.7564|0.7478|0.7229|0.7059|0.6800|0.6272|\n|Rate of Return $\\times 10^{-4}$|-|5.9221|3.3791|1.8906|1.2797|0.6845|0.4985|0.3404|0.2589|0.2112|\n\nWe found that the rate of return diminishes gradually, indicating that core samples are crucial in the early stages, while the benefits decrease significantly later on. **A clear demarcation point can serve as a guide for when to begin Boundary Sample Selection**, such as the range of 250-375 for CIFAR-10 and 375-500 for CIFAR-100. **This provides a simple yet effective guideline.** Additionally, in practical applications, we discovered that **introducing boundary points earlier may yield better results**, such as CIFAR100 with 1% (500 samples) annotation samples.\n\nSimilarly, how to allocate proportions remains a question. Our ablation study shows that for different budgets, there are varying optimal core numbers, which remains an open question. To some extent, we can determine this based on the rate of return and the number of sample categories. In the paper, we adhered to a uniform hyperparameter setting, which actually demonstrates that our framework offers more flexible parameter choices with significant potential for performance improvement.\n\n[1] Sener, Ozan, and Silvio Savarese. \"Active Learning for Convolutional Neural Networks: A Core-Set Approach.\" ICLR18\n\n[2] Yehuda, Ofer, et al. \"Active learning through a covering lens.\" NIPS22\n\n>***Q2: Limited Data Condition: ActiveDC reports on the performance of CIFAR10 under 0.1% and 0.2% settings. Active Finetuning would be more meaningful in scenarios with extremely limited data?***\n\n**R2:** ActiveDC employs pseudo-labeling from semi-supervised learning, which pertains to how to fine-tune after sample selection, rather than the selection of the samples themselves. Therefore, the appropriate comparisons should be with other semi-supervised training or fine-tuning methods. Following Reviewer V2Vc's suggestion in Q2, we experimented with KNN and Linear Probing in scenarios with very small data volumes and achieved results that significantly surpassed those of ActiveDC.\n\n|CIFAR10|0.1%|0.2%|\n|-|-|-|\n|Random+KNN|64.2|76.6|\n|Random+Linear|67.1|78.7|\n|Finetune(ActiveDC)|61.3|73.1|\n\nWe also found that these basic methods become ineffective as the data volume increases. Consequently, we believe that **active selection for full-scale fine-tuning should focus on scenarios where the data volume is not extremely small**. Our method selects higher-quality samples and achieves superior results across different fine-tuning paradigms. We will reference the ActiveDC paper in the revised version and discuss the differences between our work and ActiveDC.\n\n>***Q3: Larger Data Condition: How about larger amounts of finetuning data, such as 20% of ImageNet.***\n\n**R3:** **We have supplemented our results with 10% and 20% budget scenarios on ImageNet, and our method continues to maintain a strong advantage.** Theoretically, as data volume increases, the selection method ActiveFT tends to lean towards Random, thus tending towards a uniform distribution. However, our method effectively identifies boundaries, providing better separability. Empirically, our method has shown impressive results. **The Improvement Ratio has increased**, where the Improvement Ratio is defined as *Improvement Ratio=(BiLAF - Random) / (ActiveFT - Random)*. In active learning, it is normal for accuracy differences to decrease as data volumes increase, making the improvement ratio a reasonable metric to assess performance.\n\n|Selection Ratio|Random|ActiveFT|BiLAF(ours)|Improvement Ratio|\n|-|-|-|-|-|\n|1%|45.1|50.1|50.8|1.14|\n|2%|52.1|55.8|56.9|1.30|\n|5%|64.3|65.3|66.2|1.9|\n|10%|71.2|71.8|72.5|2.17|\n|20%|74.5|74.8|75.2|2.33|\n\nOverall, we **have effectively analyzed our method across different scales**, proposed effective boundary metrics for smaller data volumes, and empirically demonstrated the superior results and feasibility of our method with larger data volumes. We hope our responses have addressed your concerns. We welcome further discussions and sincerely appreciate it if you could reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RmMgkZlfbt", "reviewer": "Reviewer_kN8s", "summary": "The paper proposes an active fine-tuning method that considers both diversity and uncertainty. This method selects uncertain samples through an unsupervised denoising approach and boundary score evaluation. The efficiency and effectiveness of this method, which involves selecting central and boundary samples, have been validated through multiple experiments.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The writing is generally clear, and the content is relatively comprehensive.\n2. The method considers both diversity and uncertainty, which is a good idea.\n3. The experiments are quite thorough.", "weaknesses": "1. The main diagram, Figure 2, lacks detail and clarity, and its meaning overlaps with the schematic in Figure 1. Figure 2 should focus on explaining how the methods \"Boundary Score Calculation\" and \"Iterative Selection and Removal\" are implemented (in detail) rather than simply outlining the overall process.\n2. Is there consistency between the decision boundaries of the K pseudo-classes and the decision boundaries of the true task categories? Are the decision boundaries of the unsupervised method consistent with the decision boundaries when the pre-trained model is fine-tuned on downstream tasks? (Different types and capabilities of models have different decision boundaries.) It seems these issues were neither considered nor analyzed.\n3. The theoretical foundation is weak. Is it necessary to introduce data on class decision boundaries? Is there a theoretical basis for applying \"diversity and uncertainty\" sampling to active fine-tuning methods? Is there a theoretical basis for the design and implementation methods?\n4. Exploration of the number of uncertain sample points: How is K determined? Having too many such sample points presents two problems: incorrectly labeled samples and disruption of learning the general features of the categories.\n5. \"but neglects the contributions of samples to local boundaries.\" \nIs the statement in the abstract deviating from the original meaning? Should it be \"but neglects the contributions of local boundary samples\" instead of \"but neglects the contributions of samples to local boundaries\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Figure 2 should be modified.***\n\n**R1:** Thank you for your suggestion. In the revised version, we will enhance the figure to improve its detail and clarity.\n\n>***Q2: Is there consistency between the decision boundaries of the K pseudo-classes and the decision boundaries of the true task categories? Are the decision boundaries of the unsupervised method consistent with the decision boundaries when the pretrained model is fine-tuned on downstream tasks?***\n\n**R2:** Yes, these conditions are consistent! **We conducted linear probing on all the samples with true labels using features from both the pre-trained model and the oracle model (fine-tuned on all samples)**. We analyzed whether samples selected using different methods—Random, ActiveFT, BiLAF (ours)—tend to be near the decision boundaries. We used two metrics for this analysis: 1) **Entropy**, where a higher value indicates greater uncertainty and a propensity towards boundary samples. 2) **Prob_Diff** (probability difference between the top two classes) calculated as the difference between the highest and second-highest probabilities. A smaller value indicates that the sample is closer to the boundary between these two classes. \n\nWe computed the mean values for all samples and the top 50 samples. We found that samples selected based on pre-trained features exhibit higher Entropy and smaller Probability Differences in both scenarios, sufficiently demonstrating that **our method aligns with the true decision boundaries and the samples selected in unsupervised pretrained features aligns with the pretrained model is finetuned on downstream tasks**.\n\n>***Q3: The theoretical foundation is weak. Is it necessary to introduce data on class decision boundaries? Is there a theoretical basis for applying \"diversity and uncertainty\" sampling to active fine-tuning methods? Is there a theoretical basis for the design and implementation methods?***\n\n**R3:** \n1. **Theoretical Assurance**: Assuming a binary classification problem that is linearly separable. Our solid theoretical foundation lies in Support Vector Machines (SVM). The objective of SVM is to find a hyperplane that maximizes the margin, enhancing the classifier's generalization ability. Only the points on the boundary (support vectors) contribute to the margin, while internal points do not affect its size. This classical design emphasizes the **importance of boundary samples**.\n\n2. **Related Work**: Previous works in Active Learning invariably involve diversity and uncertainty. Diversity aims to fit the original data distribution better with fewer samples. As the labeled data increases, allocating the remaining budget to uncertainty becomes common. \n\n3. **Design Philosophy**: Faced with multiple challenges in a single iteration scenario for data selection, we must determine class centers and choose boundary samples. We also need to consider the potential noise, where different classes mix at the boundaries and each class may have multiple feature centers. Thus, we select more pseudo-classes than actual classes to ensure coverage of different classes, and we employ denoising to reduce sample mixing at boundaries and avoid noise as much as possible. **Denoising process mainly targets noise in non-ideal scenarios.** **Opponent Penalty and Iterative Selection encourage the boundary samples with the regional diversity.** Overall, based on solid boundary theory, we have designed each part for real-world noisy scenarios, using a set default of parameters throughout except for the number of pseudo-classes, which varies with different sample sizes, further demonstrating our method's effectiveness.\n\n4. **Oracle Verification**: We apply our method an feature Oracle model(finetuned on all samples). At 5% budget of CIFAR100, we achieved 65.7% and at 10% budget, we reached 75.3%—both over 1.5% higher than selecting central points alone, **proving the practical upper bound**. We also confirmed our method's **consistency** with actual conditions in Q2. Thus, this also validates the effectiveness of our approach from this perspective.\n\n>***Q4: Exploration of the number of uncertain sample points: How is K determined? Having too many such sample points presents two problems: incorrectly labeled samples and disruption of learning the general features of the categories.***\n\n**R4:** \n1. **Core Number K**: We conduct experiments to estimate an approximate threshold for the Core Number needed.\n2. **Boundary Sample Selection**: Our method's components tackle the problem:\n    - **Denoising**: Removes highly uncertain boundary points to prevent class mixing, conservatively choosing more accurate points.\n    - **Opponent Penalty**: Encourages diverse explorations across boundaries, allowing a more varied distribution.\n    - **Iterative Selection**: Removes nearby points of selected samples, maintaining diversity by choosing boundary-nearer points instead of regional centers.\n3. **Practical Experiments**: Since exact sample centers are unknown, we use numerous pseudo-classes, K, with boundaries between same-label pseudo-classes acting as internal points.\n4. **Experiment Consistency**: This flexible framework can achieve SOTA results using the same default hyperparameters. The ablation study showed that a suitable K enhances sample quality which represent much room for improvement.\n\n>***Q5: Should it be \"but neglects the contributions of local boundary samples\" in abstract?***\n\n**R5:** We will adopt your suggestion in the revised version, as it appears more reasonable. Thank you for helping to further enhance the readability of our paper.\n\nThank you for valuable time and insightful comments. We hope our responses have addressed your concerns. We welcome further discussions and sincerely appreciate it if you could reconsider your rating.\n\n[Finally, experiments for Q2 and Q4 will be detailed in official comments due to the characters limitation.]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rx8GGlUcuA", "reviewer": "Reviewer_vV4Z", "summary": "In this paper, the authors aim to address the problem of active fine-tuning learning. The difference with active learning is that active fine-tuning models are pre-trained rather than trained from scratch. Therefore, the characteristics of the pre-trained model are important for this problem. While the uncertain methods of active learning disturb the quality of the pre-trained model, ActiveFT in active fine-tuning learning ensures stable training by selecting the most representative sample solution. The authors' idea lies in refining the boundaries by selecting samples through uncertainty based on representative samples. Their active fine-tuning framework, BiLAF, selects central and boundary samples to learn diversity and uncertainty. Besides, it has novel improvements, including unsupervised denoising and an iterative strategy for boundary sample selection. Finally, this paper conducts extensive experiments and ablation studies.", "questions": "I have the following questions/comments for the authors:\n1. Does using a pre-trained model to select fine-tuned samples amplify errors in the pre-trained model? I noticed significant performance differences between the different pre-trained models for Table 7.\n2. Is it reasonable to consider samples that deviate from the center of the pseudo-class as noise?\n3. On different datasets, when the labeled sample size is less than, which approximate range is boundary sample selection not required?\n4. Is the proposed method stable for tasks like fine-grained classification?\n5. The notation for intra-class and inter-class distances should be changed from Xi to Ci, using the cluster notation to describe intra-class and inter-class distances.", "limitations": "For downstream task fine-tuning, which in many cases is application-specific, one of the problems I often encounter is how to determine the data collection conditions as best as possible in order to minimize costs. Therefore, the biggest problem I encountered was not having enough samples for classification.\nTherefore, I suggest the authors to further improve the results for tasks with higher annotation costs, such as detection and segmentation tasks or domains requiring expert knowledge annotation such as remote sensing and medicine.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors provide a new active fine-tuning framework that combines the advantages of traditional uncertainty methods with the state-of-the-art ActiveFT method. They also provide extensive data experiments and analyses to illustrate the effectiveness of the proposed method.", "weaknesses": "Boundary sample selection is not discussed for fine-grained categorization, a case with less category differentiation. Besides, what is the gap between active fine-tuning learning and the upper performance of fine-tuning using all samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***Q1: Fine-grained classification?***\n\n**R1:** Following your suggestion, we utilized the CUB-200-2011 dataset, which includes $200$ bird species with a total of $11,788$ images. According to the default configuration of the dataset, $5,994$ samples are used for training, with the remainder used for testing. Given the large number of classes, we used 10% of the data as Core Samples to select Boundary Samples, while all other parameters were set according to the default values reported in the paper. **The following table clearly demonstrates that our approach continues to hold a leading position in fine-grained classification datasets.**\n\n|Percent|Select Number|Random|ActiveFT|BiLAF(Ours)|\n|-|-|-|-|-|\n|20%|1198|46.32|47.83|**48.53**|\n|30%|1798|58.85|59.67|**60.52**|\n|40%|2397|66.75|67.36|**68.31**|\n|50%|2997|72.98|73.25|**74.06**|\n\n>***Q2: What is the gap between active fine-tuning learning and the upper performance of fine-tuning using all samples?***\n\n**R2:**  The table presents the upper bound performance achieved by training with all available data. Despite a significant gap between training with a subset of samples and full selection, our method consistently outperforms random sampling at equivalent annotation rates.\n|DataSet|0.5% Rand|0.5% BiLAF|1% Rand|1% BiLAF|2% Rand|2% BiLAF|100% Full|\n|-|-|-|-|-|-|-|-|\n|Cifar10|77.3|81.0|82.2|89.2|88.9|92.5|**99.0**|\n\n|DataSet|1% Rand|1% BiLAF|2% Rand|2% BiLAF| 5% Rand|5% BiLAF|5% Rand|5% BiLAF| 100% Full |\n|-|-|-|-|-|-|-|-|-|-|\n|Cifar100|14.9|31.8|24.3|43.5|50.8|62.8|69.3|73.7|**90.5**|\n\n\n|DataSet|1% Rand|1% BiLAF|2% Rand|2% BiLAF|5% Rand|5% BiLAF|100% Full|\n|-|-|-|-|-|-|-|-|\n|ImageNet|45.1|50.8|52.1|56.9|64.3|66.2|**81.5**|\n\n>***Q3:Does using a pre-trained model to select fine-tuned samples amplify errors in the pre-trained model? Significant performance differences in Table 7.***\n\n**R3:** In Table 7 and the principal experiments of our paper, **ViT-S models pretrained with iBOT and DINO frameworks exhibit comparable performance, whereas ResNet50 differs due to its unique architecture and model size**. Objectively speaking, variations in the samples selected can arise based on the quality of features and the upper limits of the pretrained models. Nonetheless, Table 7 underscores the universality of our approach, showcasing its capability to **effectively select superior samples across various pretrained models**.\n\n>***Q4: Is it reasonable to consider samples that deviate from the center of the pseudo-class as noise?***\n\n**R4:** 1. Similar to **typical denoising scenarios**, points significantly distant from their cluster are often deemed as noise. In our approach, we implement **localized denoising**, where each category has a center, and noise specific to that locality is removed. 2. In practice, due to inadequate feature learning, it's common for samples potentially belonging to two categories to intermingle at the boundaries. Therefore, our denoising aims to **clear these mixed areas to avoid erroneous sample selection**. This is a trade-off: denoising allows us to conservatively select boundary samples to address poorly learned features and classifications, while opting not to denoise would constitute a bolder choice.\n\n>***Q5: On different datasets, when the labeled sample size is less than, which approximate range is boundary sample selection not required?***\n\n**R5:** We employed the Core Sample Selection method to obtain different numbers of central points and then analyzed the benefits these central points brought. Specifically, we define ***Distance*** as the mean Euclidean distance from each sample to the nearest selected point in the feature space and investigate the ***Rate of Return*** (Incremental Benefit per Core Sample) in different range, , where *Rate of Return = Distance Difference / Core Number Difference between two adjacent columns.*\n- CIFAR10\n\n|Core Num|50|100|150|250|375|500|1000|1500|2500|5000|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Distance|0.7821|0.7588|0.7472|0.7307|0.7203|0.7117|0.6896|0.6746|0.6506|0.6010|\n|Rate of Return $\\times 10^{-4}$|-|4.6575|2.3264|1.6422|0.8362|0.6887|0.4420|0.3002|0.2398|0.1984|\n- CIFAR100\n\n|Core Num|50|100|150|250|375|500|1000|1500|2500|5000|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Distance|0.8378|0.8082|0.7913|0.7724|0.7564|0.7478|0.7229|0.7059|0.6800|0.6272|\n|Rate of Return $\\times 10^{-4}$|-|5.9221|3.3791|1.8906|1.2797|0.6845|0.4985|0.3404|0.2589|0.2112|\n\nWe found that the rate of return diminishes gradually, indicating that core samples are crucial in the early stages, while the benefits decrease significantly later on. **A clear demarcation point can serve as a guide for when to begin Boundary Sample Selection**, such as the range of 250-375 for CIFAR-10 and 375-500 for CIFAR-100. **This provides a simple yet effective guideline.** Additionally, in practical applications, we discovered that **introducing boundary points earlier may yield better results**, such as CIFAR100 with 1% (500 samples) annotation samples.\n\n>***Q6: The notation for intra-class and inter-class distances should be changed from Xi to Ci.***\n\n**R6:** Thank you for your suggestion. Given that our distance metric applies to each individual sample, I understand your recommendation to express $d_{intra}(x_j)$ as $d_{intra}(x_j, C_i)$ for greater clarity. We will adopt your suggestion and implement this change in the revised version.\n\n\n>***Q7: Recommend Tasks with higher annotation costs, such as detection segmentation, remote sensing and medicine.***\n\n**R7:** There are two aspects to consider: which data should be annotated and how to increase the speed of annotators. Due to time constraints and our lack of exposure to remote sensing and medical data, we plan to conduct further research in the future. As for detection and segmentation tasks, our experiments have also covered these, specifically in PASCAL VOC and ADE20k datasets.\n\nWe hope this response could help address your concerns, and wish to receive your further feedback soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their time and insightful feedback.  Overall, we are heartened by their recognition of our paper's clear writing and structure (V2Vc, 2Ep4, 8trC, kN8s), insightful balance of diversity and uncertainty (2Ep4, kN8s, vV4Z), technical novelty and effectiveness (2Ep4, 8trC), extensive experiments and analysis (V2Vc, 8trC, kN8s, vV4Z) with the state-of-the-art performance (V2Vc, 8trC).


We have carefully considered and responded to each of the critical insights and suggestions provided by the reviewers. Our aim is to address all concerns and enhance our work through this collaborative process. We will incorporate these constructive comments into the revised version of our paper. We are confident that incorporating the reviewers' suggestions will significantly improve the quality of our paper and contribute to the field.

Meta-review: **Summary:** The paper proposes a novel, bi-level fine-tuning method that actively selects uncertain and diverse samples through an unsupervised denoising approach and boundary score evaluation. Extensive experiments demonstrate that BiLAF outperforms existing methods across various datasets and tasks.

**Strength:** Clear, well-organized, clearly motivated writing; thorough experiments; interesting and meaningful ideas; a new active fine-tuning framework that is effective and combines the advantages of traditional uncertainty methods with the state-of-the-art method and effectively balances global diversity and local decision uncertainty; the novel unsupervised denoising technique effectively eliminates noisy samples, improving the reliability of sample selection. 

**Weakness:** The main diagram can be improved; analysis or discussions on the consistency of decision boundaries, the quality and effect of pre-trained models, fine-grained categorization (a case with less category differentiation) were missing; the theoretical foundation is weak; some technical details (e.g., hyperparameters) and ablation were missing; the generalizability of this method to different data sizes was not clear; comparative experiments with other settings should be included; further justification on imbalanced datasets was missing; the necessity and effectiveness of the opponent penalty need to be reconsidered; only full fine-tuning is considered.

**After rebuttal:** The authors provided a rebuttal, including new experimental results. Three out of five reviewers read the rebuttal and provided feedback. These reviewers have acknowledged that most of their concerns are well-addressed and have raised the ratings. **After rebuttal, the paper received increased ratings 5-5-6-6-7, with an average rating of 5.8.**

**Recommendation:** The AC checked the review, rebuttal, and the reviewers’ further feedback. The AC agreed with the reviewers about the multiple strengths of the papers and appreciated the authors’ efforts to effectively address weaknesses. The AC thus recommended **acceptance** and suggested the authors incorporate reviewers’ comments in their final version.
Decision: Accept (poster)