Enhancing Deep Batch Active Learning for Regression with Imperfect Data Guided Selection
Authors: Yinjie Min
Furong Xu
Xinyao Li
Changliang Zou
Yongdao Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Batch mode deep active learning; Auxiliary data; Regression
Primary area: general_machine_learning

Abstract: Active learning (AL) reduces annotation costs by  selecting the most informative samples based on both model sensitivity and predictive uncertainty. While sensitivity can be measured through parameter gradients in an unsupervised manner, predictive uncertainty can hardly be estimated without true labels especially for  regression tasks, reducing the informativeness of actively selected samples. This paper proposes the concept of \textit{auxiliary data} to aid the uncertainty estimation for regression tasks. With detailed theoretical analysis, we reveal that auxiliary data, despite potential distribution shifts, can provide a promising uncertainty surrogate when properly weighted. Such finding inspires our design of AGBAL, a novel AL framework that recalibrates auxiliary data losses through density ratio weighting to obtain reliable uncertainty estimates for sample selection. Extensive experiments show that AGBAL consistently outperforms existing approaches without auxiliary data across diverse synthetic and real-world datasets.

Enhancing Deep Batch Active Learning for Regression with Imperfect Data Guided Selection
Yinjie Min" School of Statistics and Data Science
Furong Xu' School of Statistics and Data Science
Nankai University nk nk. me.yjjin@gmaii.co
Nankai University frxu@mail nankai edu. cn
Xinyao Li
School of Computer Science and Engineering University of Electronic Science and Technology of China xinyao3260std. uestc. edu. cn
Changliang Zout
School of Statistics and Data Science, LPMC and KLMDASR and LEBPS
Nankai University nk. www.chollgmaallooomm
Yongdao Zhou'
School of Statistics and Data Science
Nankai University
ydzhou@nankai. edu. en
Abstract
Active learning (AL) reduces annotation costs by selecting the most informative samples based on both model sensitivity and predictive uncertainty. While sen- sitivity can be measured through parameter gradients in unsupervised manner, predictive uncertainty can hardly be estimated without true labels especially for regression tasks, reducing the informativeness of actively selected samples. This paper proposes the concept of auxiliary data to aid the uncertainty estimation for regression tasks. With detailed theoretical analysis, despite potential distribution shifts, can provide promising uncertainty surrogate when properly weighted. Such finding inspires our design of AGBAL, novel AL framework that recalibrates auxiliary data losses through density ratio weighting obtain reliable uncertainty estimates for sample selection. Extensive experi- ments show that AGBAL consistently outperforms existing approaches without auxiliary data across diverse synthetic and real-world datasets.
reveal that auxiliary data,
Introduction
Supervised machine learning often requires large amounts of labeled data to achieve good perfor-
mance, but obtaining high-quality labels can be prohibitively expensive or time-consuming many
real-world applications (Snn atall [020 Panayıdes et al, 2020 Thompson el all 2024]] As more
Equual contribution
"Correpponiigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

efficient alternative. active learning (AL) strategically queries the most informative samples from the target unlabeled data pool to maximize model performance with minimal annotation [Gal and Ghahramani, 2016, Sener and Savarese, 2018, Beluch 2018, Holzmüller et all, 2023).
Active learning aims select the most informative samples for annotation. With the objective of minimizing the model's generalization loss through gradient-based parameter optimization, we establish that informativeness consists of two fundamental components: model sensitivity and pre- dictive uncertainty. Highly sensitive samples produce stronger gradients during training, leading to more significant parameter updates|Koh and Liang 2017, Chen etall 2022], while those with high predictive uncertainty indicate regions where the predictions of the trained model are less ac- curate[[Gal and Ghahramani, 2016, Lakshminarayanal et all, 201]] By jointly considering both factors, we achieve more efficient model optimization through targeted sample selection.
The model sensitivity can be directly quantified using model gradients [Caietaal 2013, Pinsler et all 2019 Holzmiller et all, 2023]. For classification tasks, predictive uncertainty can be intuitively rep- resented by certain metries of class probabílities, such as entropy [[Krsch et all 2012 Assh et all 2020, Bang et al, DO22IJ. However, for regression tasks, the absence of labeled target data makes uncertainty estimation impossible. particularly since the model and training samples are not indepen- dent, and the loss on training data may approach zero during optimization. Various approaches have been proposed to find an alternative of predictive uncertainty in regression. including committee- based methods(losee However, these solutions remain computationally intensive while offering only heuristic proxies rather than theoretically grounded measures. To address this, we propose using auxiliary data for robust uncertainty estimation and selecting samples with high informativeness.
[022 and deep network dropout techniques [Gal and Ghahramami, 2016].
We formalize auxiliary data as distributionally shifted yet relevant supplements to the target data in active learning, where the auxiliary distribution is more continuous than the target distribution. While target datasets are often limited by annotation costs, e.g. expert-labeled medical data, safety- critical driving scenarios or precise industrial design data records, auxiliary data naturally abound: hospitals accumulate medical images from patients with varying symptom manifestations
stall, 2017 Zhou et all 2021, Tsai all 20224,, autonomous vehicles continuously log driving data in varied Devironments(Sum et all, 20205, Yu et all, 2020. Bai et all. [2224], and industrial systems retain sensor logs with recording inaccuracies or hardware degradation[[ et all 2018. Rezazadeh et all 2024]. Crucially, these auxiliary data share underlying physical or statistical re- lationships with the target distribution [Ganin et all, 2016. Kang et all 2019, Zhang et all 2020]. However, these imperfect datasets exhibit inherent limitations that constrain their utility for model training. Specifically, distributional shifts undermine the theoretical foundations of conventional generalization frameworks (e.g.. Empirical Risk Minimization). leading to significant performance degradation amwini and Gebrn, 2018, Alcom el all 2019, Koh el all 202M]] Moreover, data contamination poses security threats, potentially enabling backdoor attacks and compromising data privacy in machine learning [Dagıeiski et all, 2018, Li all 20211, Carlim of all 2021]. Thus, these data are frequently neglected by current AL methods.
Our key insight is that while auxiliary data cannot be directly combined with target data to improve model training performance, it can nevertheless provide reliable predictive uncertainty estimation through density vatio-weightt
loss approximation, as shown in Figure MI The proposed method can be briefly summarized in three steps: (1) estimating the density ratio between auxiliary and target distributions, (2) computing the model's loss on auxiliary data, and (3) applying density ratio weighting to obtain approximation of the true loss. While the density ratio estimation is still dependent of the trained model, it successfully addresses the zero-loss problem and our analysis reveal that it yields reliable surrogate for the true loss.
Inspired by such findings, we propose Auxiliary data Guided Batch Active Learning (AGBAL) to complement current AL. selection frameworks. The contributions of this paper are as follows:
We formally decompose informativeness in active learning into model sensitivity and predictive uncertainty for pradient-based parameter optimization. While the latter cannot be directly esti- mated due to model-training data dependence, we find that typically discarded imperfect auxiliary data can provide an uncertainty estimate surrogate when properly weighted.
We propose Auxiliary data Guided Batch Active Learning (AGBAL), novel framework that uses imperfect auxiliary data to estimate predictive uncertainty and selects more informative data. We validate the predictive uncertainty estimation through Neural Tangent Kernel (NTK) theory.

---

Through extensive experiments on synthetic and real-world datasets, we demonstrate consistent performance advantages of AGBAL. Our comparative analysis reveals that auxiliary data- data-guided gradient kernels consistently outperform gradient kernels across various selection strategies.
2 Proposed Method
2.1 Problem Formulation
We consider multivariate regression and the goal is to learn a function f Rd + R from training dataset D, which consists of samples drawn i.i.d. from P Px Pyix. Typically, we consider E 0),0 Ih and the goal is to find 0" = arg mingee R00; P), where R(0; P) = Ex I(Y,J(X;;)) and I(:,-) is a loss function. In practice, optimize 0 by minimizing R(0; D) |DI E( (1,, ((;
parameterized function family F(O) {f(:
The goal of active learning (AL) is to select which data should be annotated in order to learn the model as quickly as possible. Given training task objective R(0; P). a batch mode active learning (BMAL) algorithm starts with data pool Dx, which consists of samples drawn i.i.d. from Px. The initial set be labeled is B x Dx. initial labeled set is Lo = Bo = {(ri. 21) I; E Bo,x, y1 ~ Py|x=,, } and the unlabeled set is Uo Dx Bo,x-
Assume θ_1 initialized randomly. At each BMAL step 2 we update the predictor parameter 0, using A,0,-1, Lt), where A : θ x (X' x y)" + θ the learning algorithm mapping previous parameter 0;-1 and labeled data L. to an updated parameter Or, Also assumes next batch selection
algorithm S x (X x 7 x X x x K. + X that selects batch of unlabeled data by
BB+1X S(O,, C,, Ue, NN, K), where K ((:-) E K. is kernel that measures similarities between inputs. Label 1,+ and we get labeled dataset B++1 = {(xis 3)) x4 Be+1,X, Py|x=
We update data pool by L L, B U441 = 24, \ B,41,x.
2.2 Motivation and Methods
Holzmülller et all [2025] has demonstrated that sample selection strategies can be decomposed into two components: (1) feature mapping, which projects input data into an informative representation space; and (2) selection method operating this space identify samples of maximal value. Typically, the selection methods are implemented based on distance metric in the representation space. In practice, this distance can be directly characterized by a kernel function defined over pairs of inputs. For example, the gradient kernel is defined as K (grad( (x, x²; 0) = {σ (0;,")) T Φb (0; where @1 (0; z) = (z; 000 E RT is the model parameter gradient. Also kernel function consists of base kernel and kernel transformation, where base kernel (like gradient kernel) serves for feature extraction and optional kernel transformations in section A.2 enable additional adaptation. For notation simplicity, kernel transformations are omitted in the main-text notation. Notably, the gradient kernel is essential for active learning frameworks by encoding the informativeness. through parameter model gradients. While existing work Holzmüller all [2023]] has demonstrated its superiority over alternatives like last-layer activations or NNGP kernels, we identify key limitations in its current formulation under regression tasks. The gradient of R(0; P) at 0 can be formulated
ØR(0: P). 00
f(XY,f(X:6)
Ol (Xx; 0)) (
Ex
EX.Y~
00
0
00
OIYY ((x;)) Ex-Px dn (6; (0; X),
Ex-Px 0n (0: X) Ey Prix 0f
(1)
where (0;0x) = Ey a1YY, [(x; ()) 0) is the expected loss gradient. Given 0 and input x. Φ1 (0; x) can be directly computed, but evaluating (²( x:) requires access to the conditional distribution Pyyx [y|x= In an active learning setting, we face two fundamental challenges: (1) collect- ing additional samples from P to estimate Φz (0: 2)) is impractical, and (2) reusing existing training data introduces the double double-dipping problem. Current methods Holzmüller et all 2023 directly set 22 1.
The gradient kernel relies solely on Φi (0) to characterize the informativeness at point æ. However, 2 (0; z) captures the predictive uncertainty. When f(;; 0) arg min, Ey Pr|xer ((Y,)), acquir-
3

---

Audilaary
Target
High confidence point
point
Auxillary
Gradient
More well-trained selected
Measure by
Hotamutirrt et al [2023]
Festure map 9h by taking gracient of
Selection
A
Methods
Reqdir additionall labeling x for
masder
Low Loss
bee
mandde
X
Estimate density ratio A7
expected
gradient map di
Amwwnge
gradient of using density
Jane
40
ratio estimator and data from
Origin a
Mare
Our Method
Auxilary Guided & Estimation
Rescaled Gradient Space
Figure 1: The AGBAL framework processes target data from P and auxiliary data from Q. The rescaled gradient feature map projects high-confidence points near the origin O (as their magnitudes of expected loss gradient small), while maintaining high-uncertainty points greater distances. Crucially, conventional methods without auxiliary data cannot estimate the expected loss gradient, causing high and low uncertainty points distribute chaotically in the gradient space, hindering the selection of the most uncertain points.
ing additional samples is meaningless. As shown in Figure M in the red box, many well-trained points can be selected. In practical scenarios, active learning task rarely operates in isolation: there often exists related tasks that possess auxiliary dataset that has underlying similarities but exhibit different data distribution. This motivates our Auxiliary data Guided Batch Active Learning (AG- BAL) framework in Figure MI suppose we have access to a labeled dataset D' = {(X;, YO drawn from different distribution Q Qx Qrix * P. but Q > is absolutely continuous.
Let g(r, y) and (x, y) denote the joint density functions of P and Q, and g(x), Iaux be their marginal density covariate functions. The conditional density defined as g(y x) = g(x, y) g)/g(1)))))))))
and Jaux (y x) = Jaux (x, (z,3)/ (aux(()) (I). Define r(x. y) g(x, y) Jaux (I. v). r(x) = ( Jaux
and r(y z) r(z, (or,y)//le (z)) We can reformulate Φ2(0; x)) as follows:
<2(0;)) = Ey~Prix- {
aI(Y. f(x; f(;Y)((x;0))
glv xx) Olyy f(x; 0))
dy
0f(;; 0)
0f(x;0)
= f Jaux(y (y [x)(( xx)
al(y, ffr; dy = By r(Y | x))
a((Y, f(x;))
0ʃ(;; 0)
0s(;;0)
arg min Ey Xmm {s f B)) } r(Y x)) 7(x)
0f 0)
sER
arg min Ey x== {
ol [(;))((x r(2,
0f(z;
sEl
Assume f(x., y) is the density ratio estimator of r(x, y) obtained through algorithm Adr. which is detailed in section A. we can estimate Φ2 in function space & by:
n'' Σ F(X) Y;) (x)xx) aUYY f(X:0)) Os({;0)
62 = arg min
(2)
As shown in green dashed box in Figure 0L define auxiliary data guided gradient feature map as danx (w; 0; Φi, 22) 52(0; (0; 2)). and auxiliary data guided gradient kernel
Kgrad-aus (x, x': 0, )2) {(aax (x': 0 Φ1 22)3 Φaux (2; 0; Φ1, +2)
(3)
We present the complete algorithm for AGBAL Algorithm II
2.3 Theoretical Analysis
In our proposed approach, the methodology originated from an error re-assessment framework de- signed to achieve more precise feature characterization at each data point. We now proceed to

---

Algorithm Auxiliary Data Guided Batch Active Learning (AGBAL)
Initial labeled and unlabeled set Lo and Up, auxiliary data set D', training algorithm A, density ratio estimating algorithm Aar, next batch selecting algorithm S, loss function L, gradient mapping 11, initial parameter 0-1, label budget N each step and training epochs T.
Input:
I: Set f e
2: While 1
< do
3:
Update predictor parameter with 0, A(0,-,,, Lx) Estimate density ratio F; = Aar(L,,
4:
5: Train estimator of expected loss gradient as 22 in (24 using i. and D' Construct auxiliary data guided gradient kernel K Kr.grad-uu
(L) with 0z, Φ1 and क2.
6:
7:
Select unlabeled batch as B = S(O,, L, NN Kr grad-aaduu
8: Obtain B, by labeling B and update <- 1 ++ 1, Lt t- L--1 Br, Ue + U ) Br,x
9:
End while
10: return (Lo....,fe) Lx) and (fo....,0)) 0.)
theoretically analyze the performance of auxiliary data based loss estimation under squared error 32) = (y1 32)² /2 in neural network architectures.
loss (2(11.32)
Setting: Consider data (#1, 311 vi)...... ..... , (I yu) generated from model y = fay; + where & @ C Rd represents the true parameters, and are independent noise terms with zero mean and finite variance o². Define the empirical risk with Le penalty as (0) n Ei= X||0 /2/, where À > 0.
Di=1 (2(1), 0))
Pointwise convergence of f(:0): Let (0) E R be the semidefinite matrix with (entry főr (01(0;z;) a:)) 01 10;,;). The NTK kernel is defined as (mt(x1,22) Ea- [(01(0) (0; " Φi (0; x2)] For x (II,..., (Tn) and y = (vi,..., Mn), define K Kutkar, x),,,,)xx, = (Katk (z, x1) 1....... KatkT (I, ). As width goes to infinity and 0 ~ Pe. Hgrad converges in prob- ability to a deterministic matrix Kntk(x,
Raxn with entries Kmtk (Zi, [Uacot et all 2018].
Denote (r) = (a, x){ (Kenk< (x TT under certain conditions we have [zo; 0) asymptotically equivalent to fridge(Zn) when the network width goes to infinity, and fridge is asymptotically normal with + Do. Details presented in A.4 in supplementary materials.
Re-formulation of Φ2: For l2 loss (12 U2) /2y 1/2 V/. Therefore with Y = f(X:
can calculate (b (0; = (X:: ; f(x; - Xx x] f(x; f(;; , 0*) Let (0;) = 2 (1) (0: 20) + 02
(2) (0)))). where σ6 ) (0: zo) = (f(=u: frudge (zv) (B( fx;
and 0 (0: xO) = [fringoo fridg (x))) Assume f is independent of X Y/ ~ Q, let dis- Ef
tribution QF satisfies dQr(I, /)Q = f(z, y). When n' + co, the (L) can be re-formulated
02 arg minB de#
(/)(0()))(()))))))) f(x';0) + s(x';00)² arg minE, y If (1; - 0) -)(==))))
dE
Thus [(xo; 0) s( 0) 22 00 vo) is estimate of [xx; 0*) and denote the corresponding
asymptotic equivalent ridge kernel estimator as Fridge (2o). Similarly define ø2 (0) f(zo: ((x;0'), 02 (0: [f(zo:)) 0) (ridg((x))) + [[iddegge s(: 0*)] and 712) (0: (0: =
Tridge (Iro) E [((dddige
Theorem 2.1. Assume equivalence f(zo; 0) Fridco + op(((m)), Flzo: 0) Fridge (zo) (C((m))), where m represents the neural network width and ((m) - 0 when m - oo, well as asymptotic normality conditions
(5 (0) 4, ((,)))(z))), ( (0:20) 4 N( (zo)) and
Corr( (2) (0; (0) 22 (0)))) p. We have
7(2
(0; o² (z0) + +z((z))= (00 +o((((m)).
5

---

Remark 2.2. As (0: is estimated on the holdout auxiliary dataset independent of f ( 0) and the only dependence is the density ratio estimator, expect p to be small. Also f(ro; estimated similar to the process of the estimation of [ 0), thus we expect (20) & (wn). Therefore, (0) has variance approximate (ro) which is proportion to the variance of fudge(Zo) (2o). Therefore x2) serves as meaningful surrogate for the expected loss gradient.
Remark 2.3. In 202 (0; zo), the first term negligible according to Lemma A and the second term represents the bias of ridge estimation, which is not inffuenced by sampling randommess of Y given X 2o and cannot be eliminated with limited samples under given NTK kernel. The 02 (2) (0; xo) is the variance component that characterizes the deviation from the expected estimation due to the training of finite samples. At locations where this term is large, additional sampling can improve the accuracy of the estimate. Although ? (0; zo) cannot be directly calculated from Fridge 52(0; xo) can be calculated.
3 Experiments
To evaluate our auxiliary data guided gradient kernel against the origi- nal gradient kernel framework, we conduct comprehensive comparisons across seven representa- tive selection methods: (1) maxdiag (maximum diagonal selection) as the uncertainty-basee base- line, (2) maxdet (determinant maximization) [Seo et all (000] (3) bait-FB
Comparison Methods:
Greedy total uncertainty minimization) [[sh et all, 2020J, (4) Frank-Wolfe (fw) optimization for
kernel embedding approximation [Pinsler et al, 2019), (5) maxdist (maximum distance) repre- senting geometric diversity approaches [Yu and Kim [0TI] (6) kmeanspp (next point proba- bility proportional to squared distance) |Arhhrr and Vassi vitski. 2006, Ostrovsky et all, 2013] and (7) lemd (largest cluster maximum distance method) [Holzmüllerr et all 2023], and (8) ran- dom selection. This systematic evaluation covers the spectrum from simple uncertainty sampling (maxdiag) hybrid diversity/uncertainty methods (lcmd), allowing us to assess how our ker- nel optimization affects different selection paradigms. The codes are available in the repository
https://github.com http://itttuu...mmaasaaee hwtpm//wwiiniinnnnnnn http:///tthb.ccoooo/oss////////a
To ensure fair comparison, we standardize: (1) the neural network architecture across all methods, (2) initialization using parameters from identical Lo pretraining, (3) uniform hyperparameters (learn- ing rates, epoch counts) during model updates, and (4) both kernel configurations (K aux and K Kgrad) adopt the identical optimal transformation settings as established in Holzmiller et all (20233]
All methods start with [Co = 200 initially labeled samples and select N 200 additional samples per step, running for T = 15 steps with an unlabeled pool size of [24]] 9, 000. The auxiliary dataset has size Naux = 1000.
Evaluation Metrics: In each experiment, the objective of optimization is the loss 12(31, 92) (y1 - y2)², and the test set = {[ē,, 77) for evaluating the learner is distributed independently and identically (i..d.) with respect to the training data, with size [T| = 2000. The mean squared error (MSE) of learner ((::0) is defined as: MSE Σ." 12 (Th, (T;; For each data setting, every method is repeated 20 times.
Let Eu, denote the sequence of average MSE values for a given method; the area under the + f2)
curve (AUC) of this sequence is calculated as: AUC =
We report the average MSE for each method at specific training steps (i.e. the step 10), illustrating the training efficiency of different approaches under a fixed budget of training samples.
Since AGBAL utilizes auxiliary data, we must verify that these data cannot be directly used for target task training. We compare the MSE of two models: one trained solely on T-step annotated samples, and another trained on both annotated samples and auxiliary data.
Summary of Results:
We evaluate AGBAL (our method based on auxiliary data guided gradient kernel) against BMDAL (gradient kemel method without auxiliary data) and the random baseline across seven datasets: synthetic data $1, $2 and real-world datasets BIO, BIKE, DIAMOND, CT,
STOCK. Details are in Section I and 630

---

Table 1:
Comparison of 8 selection methods across synthetic and real-world datasets in terms of
AUC, where Avg Impro represents improvement over BMDAL averaged across experiments.
S1
S2
BIO
BIKE
DIAMOND
CT
STOCK
Avg Impro
random
0.928
1.421
0.451
0.459
21,009
0.380
0.392
lemd
1.111
1.517
0.417
0.394
19.714
0.255
0.370
lemd (ours)
0.846
1.279
0.420
0.435
20.687
0.291
0.363
0.6%
maxdist
0.863
1.310
0.428
0.439
20.6433
0.270
0.389
maxdist (ours)
0.834
1.266
0.417
0.401
21.179
0.298
0.361
1.8%
kmeanspp
0.894
1.378
0.414
0.404
19.691
0.271
0.372
kmeanspp (ours)
0.842
1.294
0.406
0.364
19.613
0.264
0.355
4.5%
fw
0.953
1.448
0.434
0.455
21.115
0.347
0.388
fw (ours)
0.899
1.341
0.418
0.404
21.840
0.310
0.377
5.5%
hait
0.853
1.340
0.441
0.481
22.057
0.435
0.392
bait (ours)
0.835
1.293
0.431
0.398
22.134
0.374
0.371
6.3%
maxdet
0.876
1.318
0.418
0.486
19.702
0.320
0.378
maxdet (ours)
0.833*
1.254'
0.409
0.362
19.846
0.254
0.359
8.9%
maxdiag
0.903
1.401
0.451
0,597
24.594
0.526
0.415
maxdiag (ours)
0.836
1.270
0.420
0.410
20.745
0.304
0.361
18.0%
Table 0I reports the AUC of the learning curves, where lower values indicate faster convergence. AG- BAL consistently outperforms BMDAL in most settings, particularly with maxdet (achieving the best results on S1, $2, BIKE, and CT) and kmeanspp (leading BIO, DIAMOND and STOCK). Table @ compares the RMSE of AGBAL and BMDAL after 10 training steps across all selection strategies and datasets. AGBAL achieves significantly lower errors in almost all settings, demon- strating faster carly-stage convergence.
Table B presents the test MSE comparison across all scenarios and methods after 7-step active learn- ing. To highlight the distributional discrepancy of auxiliary data, we deliberately select a large auxiliary dataset size (Naux 10000) to amplify the performance degradation caused by naively incorporating auxiliary data. The universally increased MSE values observed when incorporating auxiliary data into the training set demonstrates: (1) degraded model generalization performance, and (2) significant distributional shifts between auxiliary and target data. These results show that auxiliary data cannot be directly utilized for model training, thereby justifying our design of employ- ing them exclusively for sample selection guidance.
Together, the empirical evidence establishes that while auxiliary data provide effective selection signals, their inherent distribution shifts prevent their direct inclusion in training sets.
We also investigate two other factors affecting AGBAL's performance: (1) auxiliary data quality (distributional discrepancy relative to the target distribution) in Section B.T. and (2) auxiliary data quantity (sample size) under fixed quality conditions in Section A.7.
The random strategy, serving naive baseline, performs poorly, underscoring the importance of active selection. Notably, AGBALs gains are most pronounced for uncertainty-drive strategies (e.g.. maxdet, kmeanspp), while methods perform less significant for hybrid and diversity- based selections like lemd and maxdist. This demonstrates our methods compatibility with different selection criteria while maintaining superior convergence. For BMDAL without auxiliary data, kmeanspp and lemd give the best performance.
3.1 Synthetic Data
In our synthetic experiments, the regression model for data from P is Y, =; + <(x,), where (=x) = E(Y, X; ===u .x) and the residual c(X;) may depend on X,. The model for data from Q is Y = +x()). The covariates X; and X/ are sampled from the d-dimensional standard nor- mal distribution (N, La). We consider two data generating settings: for x = (II,...,Id)) let exx)
and c'(x) ~ NO |x(z))/4), /4,, f((x) is defined by shifting plx) with 8(x; Ç) = Ç cos( O |z l)/4:
S1: m(x)= Σ [oog(1z)) + 1)1/4 and (a(xx) = p(x)) + 5(x; Q);
S2: μ(z)= ΣL= 1/2
1//2/4/ /4, and p'(x) p(x) + ((z;();
7

---

Figure & and H in appendix present the MSE learning curves versus the train- ing steps for different sample selection methods with Ç = 8. The results demonstrate that auxiliary data guided gradient kernel in AGBAL consistenty outperforms the gradient kernel approach across nearly all experimental settings. Notably, while the gradient kernel may underperform random se- lection in certain cases (e.g., maxdiag on SI; maxdiag and fw on S2), AGBAL maintains superior performance over random selection in most these scenarios. Refer to Table BHE for detailed MSE results.
Performance Result:
Experiments on auxiliary data quality: We configure Ç (,, 4, 8, 16, 32, 64) induce exponen- tially increasing distributional shifts in auxiliary data. As shown in Figure E. AUC grows linearly with the exponential progression of Ç. But our method still outperforms direct gradient kernel ap- proaches within tested limits, as shown in Table I2 This demonstrates that: (1) higher-quality auxiliary data better guides active sample selection, and (2) even when auxiliary data quality deteri- orates substantially, our loss estimation serving as error magnitude indicator rather than precise measurement remains useful. More details are provided in Section A.7.I
maxdlag
maxdet
balt
tw
maxdist
kmeanspp
land
1.5
1.00
0.35
0.90
1.3
0.85
1.2
1.6
16 32
64
Figure 2: AUC plots of AGBAL under varying Ç.
Visual Analysis: Further, validate AGBAL's ability identify sample points with higher pre- dictive uncertainty through a controlled toy example. Using identical initially trained models and unlabeled data pools, compare the predictive uncertainty of points selected via auxiliary- guided gradient kernel versus conventional gradient kernels across different selection methods.
Let (ax) Ei= Ti, (xx)) " Iú, and noise exx), ((x) ~ ~N d). The initial labeled set Lo and the unlabeled pool Lo each contain 200 data points. In the subsequent batch selection step, we select 10 points from Un- For visualization, we project the unlabeled pool Up onto a 2D space
via PCA, with color mapping representing the expected loss gradient f(r; 0) E(Y X z). Darker shades indicate higher predictive uncertainty, which should be prioritized for selection. As illustrated in Figure B. while both AGBAL and BMDAL aim to select representative points in the representation space, AGBAL consistently identifies more high-uncertainty samples. The result plots for other selection methods provided in Figure B-d.
3.2 Real Data
We evaluate method five public regression datasets also considered by (Holzmüller all 2023]: physicochemical properties of protein tertiary structure (BIO) [Ranal 2013), bike sharing
(BIKE) [FnaeeeT and Gamal 40,,), prices of diamonds (DIAMOND) relative location of CT
slices (CT) Graa and Cavallard 2011] and BNG stock price data (STOCK) Detailed dataset descriptions are provided in Section
Result: Figure presents the MSE learning curves comparing different methods on dataset BIKE (with additional results four other real-world datasets shown in Figures M| and Table G-g in appendix due to space constraints). The results demonstrate that our auxiliary data guided gradi- ent kernel based methods achieves the fastest MSE reduction compared with gradient kernel based methods, which indicates Kgrad-an aux better measures the informativeness of data points. Moreover, while lemd and kmeanspp typically perform best without auxiliary data, the maxdet and kmeanspp
On http://wwwwaag..ccommnmssssseees
On htwp://wwwweemeloooeeeheeeeettoe

---

AGBAL-kmm
BMDAL-Iomd
PC2
1
20
20
PC1
PC1
PC1
PC1
Figure 3: Visualization of loss of selected points across four AL configurations. Left, right panels display lemd, fw results of AGBAL and BMDAL, respectively.
selection methods show superior performance when guided by auxiliary data. This consistent pat- tern across all real-data scenarios confirms the effectiveness of incorporating auxiliary information into the sample selection process.
maxdiagl0: maxdlagiAI
kmeanspp(()
mandettA)
bak(())
(w(A)
maxdist(A)
kmeanspp(A)
lamdIA]
MS5
0.4
0.4
0.3
0.3
0.3
Epoch
Epoch
Spoch
Figure 4: MSEs decreasing plots during AL AL steps for different selection method and different kernels for STOCK dataset (B for gradient kernel, for auxiliary data guided gradient kernel)
3.3 Computational Burden Analysis
Our method introduces additional components (density ratio estimation and auxiliary loss estima- tion) compared gradient-only approaches BMDAL. However the selection kernel itself remains unchanged; we only modify the features used for selection. The primary computational overhead stems from processing auxiliary data. For efficiency, lightweight machine learning methods (e.g.. random forests) can be adopted for density ratio and loss estimation without compromising perfor- mance. To illustrate scalability, we conducted experiments across real-world datasets with varying auxiliary dataset sizes (100, 500, 1000, 10,000 samples) server with dual Intel Xeon Gold 6330 CPUs (1 12 threads total) and 125GB RAM.
The results are shown in the Table M. Even with only 100 auxiliary data, the maxdet selection strat- egy, which performs best among the compared approaches, still achieves meaningful performance improvements. The gain grows with larger auxiliary datasets. Crucially, for auxiliary datasets below 1,000 samples, the additional time and memory overhead remains negligible. In practical applica- tions, we can choose the auxiliary data size based on their specific needs for either better perfor- mance (larger auxiliary sets) or faster computation (smaller auxiliary sets).
Related Work
In Section and 2.22 we propose that regression tasks under gradient- optimization, infor- mativeness can be decomposed into model sensitivity and predictive uncertaintytwo dimensions that

---

have been extensively studied separately. In this section, we systematically review existing work along these two research lines.
Expected model change maximization plays vital role in model sensitivity-based methods. Early work by Catetall [20]3]] introduced bootstrap ensemble modeling to identify samples with maximal gradient variations across ensemble members, explicity capturing model sensitivity. Subsequent approaches integrated representation learning: Ash el [2020] developed clustering strategy using last-layer gradient embeddings to select samples that induce divergent model updates, while Holzmilller et all [2025]] proposed gradient- gradient-based kernel transformation method that strategically selects the most distant points within the largest cluster, optimizing both representativeness and
diversity.
As predictive uncertainty cannot be directly estimated, various alternative methods have been pro- posed. Early work includes RayChaudhuri and Hamey [T995] and [Burbidge al] [2007]], who introduced the Query-by-Committe (QBC) framework to select samples with maximal model dis- agreement as a measure uncertainty. With the advent of deep learning, more sophisticated uncer- lainty estimation techniques emerged. Gal and Ghahramani [2016] developed Monte Carlo dropout approximate predictive uncertainty, while BBluch etall [2018] employed ensembles with varied parameter initializations to characterize sample uncertainty.
Altemative approaches focus diversity and representativeness. Sener and Savarese [20T]] intro- duced the core-sct approach with theoretical guarantees via minimum covering radius, and Wu et all [201%] developed greedy maxmin distance algorithm for data selection. Subsequent developments include Liu et all [2021]]s 's iterative optimization method balancing average and minimum distances,
and Kim and Shin [ hybrid approach combining density-based clustering and core-set selec- trade-offs.
tion to enhance
A distinct line of research explores loss-driven approaches. Konyushkova et all 200 proposed a data-driven method that learns query strategies by predicting expected error reduction. Yoo and Kweon [2019]] learned loss predictor module to estimate target losses of unlabeled data. Sinha at
[ introduced VAAL, which uses adversarial training between a variational autoencoder
and discriminator to learn latent representations while implicitly quantifying sample loss.
Several works also leverage auxiliary data for active learning. Our approach differs from transfer active learning methods such as Waan et al. [2014]] in how auxiliary data is defined: transfer learning and domain adaptation typically assume covariate shift with different P(X) but similar P(Y X) across domains, whereas our setting allows both P(X) and P(Y | X) to differ in imperfect auxiliary data. This distinction is critical, as standard transfer methods may experience more negative transfer under such joint distribution shifts.
Conclusion
In this work, we address fundamental challenge in active learning for regression tasks - the inability to reliably estimate predictive uncertainty. We propose AGBAL, a novel active learning framework that overcomes this limitation by leveraging auxiliary data through density ratio- weighted loss ap- proximation. Theoretically grounded in NTK analysis, our method transforms typically discarded auxiliary data into valuable uncertainty estimates, while properly accounting for distributional shifts between auxiliary and target domains. Through extensive evaluations, we demonstrate that AGBAL consistently outperforms conventional active learning approaches across diverse application scenar- ios.
Acknowledgments and Disclosure of Funding
Zou was supported by the National Key R&D Program of China (Grant No. 1022YFA1003800) and the National Natural Science Foundation of China (Grant Nos. 12231011 and 1253101
Zhou was supported by the National Natural Science Foundation of China (Grant Nos. 12131001), and the Fundamental Research Funds for the Central Universities in Nankai University.
10

---

References
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4845 4854, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020.
Xiangyu Bai, Yedi Luo, Le Jiang, Aniket Gupta, Pushyami Kaveti, Hanumant Singh, and Sarah Ostadabbas. Bridging the domain gap between synthetic and real-world data for autonomous driving. Journal on Autonomous Transportation Systems, 1(2):1-15, 2024.
Jihwan Bang, Sumyeong Ahn, and Jae-Gil Lee. Active prompt learning in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27004-27014, 2024.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9368 116889977,20 2018.
Christopher M Bishop and Nasser M Nasrabadi. Pottern recognition and machine learning, vol- ume 4. Springer, 2006.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Conference on Fairness, Accountability and Transparency, pages 77-91. PMLR, 2018.
Robert Burbidge, Jem Rowland, and Ross D King. Active learning for regression based query by committee. In Intelligent Data Engineering and Automated Leurning-IDEAI. 2007: 8th Inter- national Conference, Birmingham, UK, December 16-19, 2007. Proceedings 8, pages 209-218. Springer, 2007.
Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing expected model change for active learning in regression. In 2013 IEEE 13th International Conference on Data Mining, pages 51-60. IEEE, 2013.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 27), pages 0633-2650, 2021.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems, 35:9727-995 2022.
Hadi Fanaee-T and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2: 2:113-127, 2014.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty deep learning. In International Conference on Machine Learning, pages 1050- 1059. PMLR, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35,, 2016.

---

Kriegel H.-P. Schubert M. Poelsterl S. Graf, F. and A. Cavallaro. Relative location of CT slices on axial axis. UCI Machine Learning Repository, 2011. DOI:
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):83-85, 2005.
David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. A framework and bench- mark for deep batch active learning for regression. Journal of Machine Learning Research, 24 (164):1-81, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gener- alization in neural networks. Advances in Neural Information Processing Systems, 31:8571-8855 2018.
Matthew Jagielski. Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pages 19-35. IEEE, 2018.
William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary Mathematics, 26(189-206):1, 1984.
Ashna Jose, João Paulo Almeida de Mendonça, Emilie Devijver, Noël Jakse, Valérie Monbet, and Roberta Poloni. Regression tree-based active learning. Duta Mining and Knowledge Discovery, 38(2):2004460 2024.
Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct im- portance estimation. Journal of Machine Learning Research, 10:1391-1445, 2009.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
Yeachan Kim and Bonggun Shin. In defense of core-set: A density-aware core-set selection for active learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 804-812, 2022.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. Advances in Neural Information Processing Systems, 32, 2019.
Pang Wei Koh and Percy Liang Understanding black-box predictions influence functions. In International Conference Machine Learning, pages 1885-1894 PMLR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pages 5637-5664. PMLR, 2021.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Ad- vances in Neural Information Processing Systems, 30, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. Advances in Neural Information Processing Systems, 34:14900-144 19900-14912. 2021.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. survey on deep learning in medical image analysis. Medical Image Analysis, 42: 60-88, 2017.
12

---

Ziang Liu, Xue Jiang, Hanbin Luo, Weili Fang, Jiajing Liu, and Dongrui Wu. Pool-based unsu- pervised active learning for regression using iterative representativenes--dieerr maximization (irdm). Patlern Recognition Letters, 142:11-19, 2021.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 36(11):5847-566 2010.
Rafail Ostrovsky, Yuval Rabani, Leonard J Schulman, and Chaitanya Swamy. The effectiveness of lloyd-type methods for the k-means problem. Journal of the ACM (JACM), 59():1121)))))) 2013.
Andreas S Panayides, Amir Amini, Nenad D Filipovic, Ashish Sharma, Sotirios A Tsaftaris, Alistair Young. David Foran, Nhan Do, Spyretta Golemati, Tahsin Kurc, et al. Ai in medical imaging informatics: Current challenges and future directions. IEEE Journal of Biomedical and Health Informatics, 24(7):1837-1877, 2020.
Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. Advances Neural Information Processing Systems, 32, 2019.
John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Huihui Qiao, Taiyong Wang, Peng Wang, Shibin Qiao, and Lan Zhang. time-distributed spa- tiotemporal feature learning method for machine health monitoring with multi-sensor time series. Sensors, 1899),9332, 2018.
Prashant S Rana. Physicochemical properties of protein tertiary structure data set. UCI Machine Learning Repository, 2013.
Tirthankar RayChaudhuri and Leonard GC Hamey. Minimisation of data collection by active learn- ing. In Proceedings of ICNN'95 (CNN'95-Intern) 95-IIntrnaational Conference on Neural Networks, volume 3, pages 1338-1341 1341. IEEE, 1995.
Nima Rezazadeh, Donato Perfetto, Mario de Oliveira, Alessandro De Luca, and Giuseppe Lamanna. A fine-tuning deep learning framework to palliate data distribution shift effects in rotary machine fault detection. Structural Health Monitoring, page 14759217241295955 2024.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Sambu Seo, Marko Wallat, Thore Graepel, and Klaus Obermayer. Gaussian process regression: Active data selection and test point rejection. In Mustererkemnung 2000: 22. AAGM-Symposium
Kiel, 13.-15. September 2000, pages 27 27-34. Springer, 2000.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
Chen Sun, Jean M Uwabeza Vianney, Ying Li, Long Chen, Li Li, Fei-Yue Wang, Amir Khajepour, and Dongpu Cao. Proximity based automatic data annotation for autonomous driving. JEEE/CAA Journal of Automatica Sinicu, 7(2):395-404 2020a.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pages 2446-2454, 2020b.
Neil Thompson, Martin Fleming, Benny J Tang, Anna M Pastwa, Nicholas Borge, Brian C Goehring, and Subhro Das. A model for estimating the economic costs of computer vision systems that use deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23012-23018 2024.
13

---

Cheng-Chand Tsai, Yuan-Chie Chen, and Chun-Shien Lu. Test-time stain adaptation with diffusion models for histopathology image classification. European Conference on Computer Vision, pages 257-275. Springer, 2024.
Rui Tuo and Lu Zou, Asymptotic theory for linear functionals of kernel ridge regression. arXiv preprint Stti::24033022455 2024.
Xuezhi Wang, Tzu-Kuo Huang, and Jeff Schneider. Active transfer learning under model shift. In International Conference Machine Learning, pages 1305-1313. PMLR, 2014.
Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Information Sciences, 474:90-105, 2019.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv: 1902 04760, 2019.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 93-102, 2019.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2636-2645, 2020.
Hwanjo Yu and Sungchul Kim. Passive sampling for regression. In 2010 IEEE International Con- ference on Data Mining, pages 1151-1156. IEEE, 2010.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 694-699, 2002.
Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, Qingsong Liu, and Clark Glymour. Do- main adaptation as a problem of inference on graphical models. Advances in Neural Information Processing Systems, 33:4965-4976, 2020.
S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L Prince, Daniel Rueckert, and Ronald M Summers. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE, 18205)88 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The formulation of AGBAL can be found in section A
and M
Theoretical
analysis can be found in section 2X and experiments can be found in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of proposed method discussed in section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
For the theoretical result in section AX details of assumptions and proofs can be found in section A44 and A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions and formulations required to reproduce the main experimental results, including information the model, datasets, baselines, and evaluation metrics, Additionally, we present the algorithms and implementation details in section and to offer clearer implementation guidance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The datasets utilized in this study publicly accessible via the cited refer- ences, and the implementation code is available upon request.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
7 //ips..cc/pblli//uuudda Coaeee//ooeSSummmieeee oagguadeeeeoeeeebbaiaaaaaaoooooo seiobmmmiiioooooooyyc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings are detailed in Section B. and additional experi- ments presented in Section A.G to provide further insights into the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For both synthetic and real-world datasets, we report MSE results with 2x етror bars to quantify variability. Due to computational constraints, we omit AUC error bars which would require extensive repeated experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
The experiments implicitly reflect computational resource requirements
through experimental scale parameters (e.g., dataset size) in section B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and have ensured that all aspects of our research fully comply with its guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of our proposed method are discussed in section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that our work poses no foresecable risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our experiments properly cited, and source links provided for reference.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: Our code is well documented with the documentation provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We do not use LLM to impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy /uTeee//Jeeriiooccc///ofeerrrers for what should or should not be described.
21

---

A Technical Appendices and Supplementary Material
A.1 Density Ratio Estimation
Consider two datasets Dp = {m:)==1 ~ P with density p(z), and Dq = [- ~ Q with density
q(z). Assume P < Q, the target is to estimate the density ratio r(x) = dP/dQ(x) = p(x)/q(2))
which quantifying the discrepancy between the distribution P and Q.
There are two kind of density ratio estimation algorithms Aar(-): direct estimation and classifier- based indirect estimation.
In direct density ratio estimation, we typically use a parametric function go(x)) to estimate r(x)) The estimation problem is then formulated as minimizing minimizing the Bregman divergence between p(z) and go (z) ()): given a certain function u(x),
min
(x))(x))) - Ez- [e=-(((((0((=)))
There are many choices for the function v(xx). such as: (1) u(x) (x 1)2/2 used by LSIF [Kanamorietta al 2009), (2) u(æ) = x log(z) (r) x used by UKL [Nuuyen et all 20100 (3) u(z) = =log(z) - (1 + x) log(1 + z) used by BKL (LR) [Hastrc.cLal al PINSS.
For undirected density ratio estimation, we adopt a probabilistic classification framework. We con- struct labeled dataset by assigning label samples from Dy and label 0 to samples from Dq- Specifically, we first train a classifier to distinguish samples from the target versus mixture distribu- tions, then apply calibration methods [Zadrozny and Elkan, 2002], i.e., Platt-calibration atl 1999 to convert classifier outputs into well-calibrated probability estimates.
The combined dataset ((W, ) contains + samples, where the first observations are from Dp, (W,, Y.) = (Ia, 1), S n and (W,, Yi) = (z,, 0), i n. The conditional class probability can be expressed as:
p(w)
r(w) r(w) +1
P(Y W w) p(w) (w))
where r(w) pqw)/q(w) is the density ratio function. This relationship allows us recover the density ratio through:
P(Y = W w) r(w) T- = W = w)
Let Ō(x) be probabilistic classifier trained to estimate P(Y classifier as g(x)= = (ag(x> + b), where a is the sigmoid function and a, are trained on another dataset. practice we can split (W,, K into two parts: one to train ī(x) and another to fit a and 6. The corresponding density ratio estimator is then given by:
W w). Assume calibrated
7(x)
f(x) = 1- 1-g(x)
A.2 Kernel Transformations
The framework by [Holzmiiterr et all 2023 introduces modular set of kemel transformations to adapt base kernels for active learning objectives. These transformations modify base kernels to improve uncertainty estimation (e.g., Gaussian process posterior transformation), diversity (e.g.. acs random fcature transformation and acs gradient transformation), and computational efficiency (c.g., scaling, sketching). All transformations combine their effects. We denote this transformation chain as K-T,,TT,- where Ti is first applied to the base kernel K, followed by Tz. In the following, we formalize each transformation and its implementation.
composable and can be sequentially chained to
To simplify our notation, we consider a dataset Dx, which is partitioned into two disjoint subsets: a labeled subset Lx = {r,) C Dx that has been annotated by domain experts, resulting in the labeled dataset L (1 /=11 An unlabeled subset 1 Dx Lx containing the remaining unannotated data points. We assume the existence of a positive semidefinite kernel function K
X x X + R defined in the input space.
22

---

(1) Scaling transformation - scale(Lx) normalizes kernel outputs to have unit variance:
-1/2
K (x
<x),Lx)= -
1
Σ
K(xxx)
|Cx
mELx
(2) Sketching transformation - sktch(d2) summarizes feature mapping with projection to lower dimension space:
K (d [ No(=')
where (xx) the map corresponding to the base kernel K that K(x, (d(x)) where ((x) Rdi and N RRdx d2 < dj is random matrix with i.i.d. standard normal entries, approximates high-dimensional features via random projection to d2 dimensions. Using variants of the celebrated Johnson-Lindensraanl
Johnson all 1984], we can prove that for Gaussian random projection to dimension d2 2 log log(df/0)) /6)/ with probability 2 - we have:
Vx, x' E x : c)dx (z, x²) S dx (aterch (I, x') - (1 + (x,x),
where dx (x, / (K( + - 2K (x, x²) denotes the kernel distance between x and x'.
(3) Gaussian process posterior transformation: For the feature map (xx) corresponding to the kernel K, consider Gaussian process with kernel K. Assume a Gaussian process linear regression model using ((x) as features: Y/ (xx;) + 6 with a weight prior w (V,0, I) and i.i.d. observation noise E (0, o²), Bishop and Nasrabadi [2006] proves the posterior distribution for a Gaussian process after observing the training data C is also Gaussian process with kernel
K (_.0 = Cov(w" 6(x), w (ox(() C)
= =((xxx') - K( (X(x +x) " x(xx,z'),
where K( Lx) (K(x, K( 22), k(,, x( (K(x, Lx)) and K ((Lx,Lxx x;);,, Calculate the posterior covariance of the Gaussian process
after observing data L with noise variance o². For convenience, we use K+c to denote
K K=sscae(C)
(4) ACS random feature transformation: Pinsler et all [2019] applied the ACS-FW method to Gaussian process linear regression. We the Gaussian process model parameterized by
with noise variance o², employing the kernel K (=scale(C)) as described above. Let face (I, w)
1 1/2 log(1 + (=<(z,))/o") - - & (x))
(2 12 + K K-,c(æ, (x)x,),))/// 2002 where @=scale((xx) de- notes the feature mapping associated with the kernel function K (=scale(Cx)) Then,
K<acs(r,z') (w- (2, (z),
where P(w| L) denotes the posterior distribution of parameter observing data L.
(5) ACS gradient transformation: Pinsler
201 propose the weighted Fisher inner product
given by:
K- Bracs-grad(a (I, x') B VV w)
Vu.fac ,w)].
Under the Gaussian process model, they show explicit formula for K K-acss (2, x¹) is given by:
1 K K (rads--rad(E,d" (2, ga (KK
A.3 Selection Methods
After applying the transformed base kernels to the unlabeled data, multiple selection strategies can be employed to identify the most informative unlabeled candidates for annotation.
Let L. = { Hi represent the labeled dataset where L. = far contains only the input features, U4 fe denotes the unlabeled data pool, Or represents the current model
23

---

parameters, and K denotes the kernel of the base kernel after kernel transformations. The active learning objective is select N points from U4 for annotation in the next batch.
We employ an iterative approach to select samples (except bait-FB). Let B,.x, denote the selected sample batch after the i-th iteration, where x,x i. By repeating the selection process N times for different methods and combining the selected samples, we obtain the next batch of points for labeling. The next batch selection algorithm is denoted as S θ (X y) X
K. + X
The next batch of unlabeled data is B4+1.X S(O,. L, K), where is the selected batch size. We denote the selection at the i-th iteration with x,,x,; as S S(O,, L2, Ue B,,x,i, K)
Maxdiag selection maximizes the diagonal entries of the kernel matrix prioritize high-uncertainty samples:
S(O,, Lt, Ue B,, K) argmax K K(x, x),
zEll:
Maxdet [Seo ctal all 20T0 selection optimizes the determinant the kernel matrix, balancing infor- mativeness and diversity;
S(O,, St, Ue, Br K) = argmax K __
æElh
where x² a tuning parameter.
Bait-FB | et all 2020 selection leverages Fisher information for batch selection to enhance
representativeness. Bait-FB greedily selects 2NV samples and then removes N samples, and the greedy selection of 2NN samples is:
S(O,, Mr, B,, K) = argmax DE &
Σ
K x (Utt),,))
Frank-Woo Wolfe selection employs convex optimization approach to iteratively select batches with near-optimal submodular guarantees. Details can be referred to et [2019].
Maxdist selects samples that maximize pairwise distances, and explicitly promotes diversity and and Kim 2010:
s(,, Ue Br,x,,, K)= = argmax min dx (z,
zE4f 2'EB,,x,i
For Bi,x,o, an arbitrary maximizer from U4 is chosen.
Kmeanspp ensures representative coverage via k-means++ initialization and Vassilvitskii, 2006, Ostrovsky all 2013]:
minseB.,x. dx (x,
Yx E u; P(S(O,, B,x,i, K) = x) Ex' minseB,,x. dx (x'
Lemd Holzmüller et all [0233] selects the point with the maximum distance to point in B, that has largest cluster size. For each point E Ur define its associated center as c(z) = arg mingeB..x, dx (z, For any & B define its cluster size as s(ā)
E Exetioot)== dx (x. Z)². The selection is defined as:
s S(ō, u,, 1,K)
argmax
maxses 8(8) dx(x, c(x)).
xE(h:s(e())) (ax.
For Bi,x,,, select argmax K(x,x))
A.4 Theoretical Details
With the NTK kernel defined as (atk( Eg ((:((0211) T1 (0; the key equiva- lence holds:
.,. (Inz Br), as layer widths go to infinity and the parameters of neural network are initialized by ~ Po. the optimized predictor f(x; 0) under R2 converges in probability to:
Lemma A.1. [Yang 2079 Arora 2019] For data (æ1, v1), ........
Fridge()) KKtk(I, x) Kntx KKtx + Aun) -
24

---

Therefore, under the infinite-wide regime, the equivalence between neural network training and kernel ridge regression allows us to leverage pointwise behavior of fridge(z) (x) to characterize the neural network predictor at fixed point E IRd Denote ridge estimate of NTK kernel as:
Kntk(z (z, mo) (z x) Kntx(xx x) + Anln] -1 { (no x)}
Let Lo norm be || (,, ro) Kntk( ( to) Iac sup, |Ritk (2,, 2o) Kitk (2. mo)], and empirical norm be |e 2o) - Kutk(t ml [n- E (Rutk To) - Kuek (2;, } 1/2
Assumption 1. The noise terms e; are independent and identically distributed with E(es) - 0 and Var(e,) = o < 00
in Lemma A.2. [[uo and Zoud 2024] Under assumption 1-2, Fridge is asymptotically normal:
Assumption n/1/21|Bmx((s.0
2o0
(=
probability.
(Var(feigge 1/2 (ridgo( (TT) (
E(Jridge((oo) 4 N(,,
where Var(Fridge (zo)) = of Kut x) [x, + ) (Knekc x
Remark A.3. Assumption 2 can be verified using the upper and lower bounds of (
Knak' ( and ([,nk((; Kntk (: (0) in Tuo and Zol [20224]
A.5 Proof of Theorem
We decompose the (0; as
e2(0; ro) = f(ru: 0) f( 8)
{f(zo 8 f( 0*) (f(zo:0) f(xo:0"}}
02(0;=0) - 22 (0:
21550) (0: xo) 02 (0; mm)} + { (0 20) - 7(2) (0: xn)}
(11
[[idge (mn) (xn)) B FF (2o)) (r([m)))] (6) - (0:
Therefore the first part only contains bias and an asymptotically negligible item and the variance comes from the second part:
Var (2(0);0))
-Var [B (2o) E(Jrriegoo (ro)) + or(((m)) (0 (0:=0)))))))))))) - -350) (0:z0))
=Var(op(((m)) + (0! (0; xo) 0/ (0:2(00)))) (0:20)))
-Var (2) (0: - 02 (0; x0) + +p(()m))
=Var (0) (0; zo)) Var (²,)) (2) (0: _0)) 2Cov (o ( 20) T (0: + op(((m))
-x²(x0) + 82 (xo) (z0) (2o) + op(((m)).
A.6
Addtional Information for Experiments
A.6.1 Real Data Descriptions
Dataset Information: The datasets BIO, BIKE, DIAMOND, and STOCK have feature dimension d=9, and CT has original dimension d=379 (reduced to 50 features through correlation-banl selec- tion).
The auxiliary datasets are generated through three distinct approaches: (1) Data partitioning by feature: for the DIAMOND dataset, we select samples with the highest clarity as the training and test set (representing rare, high-cost-to-labell instances) while using lower- clarity samples as auxiliary data, creating significant distributional heterogeneity; (2) Shared-origin
Auxiliary Data Descríption:
25

---

Avg Impro data with label corruption: for CT and STOCK datasets, the target and auxliary initially share the same distribution, but we systematically corrupt parts of the auxiliary data (10% relabeling from (UCT, aCT) for CT, 40% from [JSTOOKK, "STOCK) for STOCK) to simulate annotation/trans. mission errors or data missing, where PCt, MSSTCK and c:. &STock represent the expectation and variance of response variable in CT, STOCK dataset; (3) Feature-depende
label perturbation:
for BIO and BIKE datasets, we modify auxiliary data (z, y) to be (x, Ynew), Ynew = y (x)). where
5(x) models instrumentation or recording errors. For BIO dataset, 8(x) = cos cos(E 24) and for
BIKE dataset ((x)) 2 cos 2x uz). Table B conclusively demonstrates significant distributional shifts between auxiliary and target data.
A.6.2 Additional Results
In this section provide additional and detailed results for experiments.
Table 2: Comparison of 8 selection methods across synthetic and real-world datasets in terms of
RMSE at step 10, where Avg Impro represents improvement over BMDAL averaged across 7 exper- iments.
S1
S2
BIO
BIKE
DIAMOND
CT
STOCK
random
0.890
1.352
0.412
0.295
17.854
0.308
0.346
lemd
0.971
1.443
0.390
0.256
16.644
0.168
0.328
lemd (ours)
0.803
1.206
0.392
0.291
17.045
0.197
0.315
0.6%
maxdist
0.833
1.255
0.404
0.285
16.691
0.177
0.343
maxdist (ours)
0.797
1.202
0.386
0.272
16.657
0.186
0.321
2.7%
kmeanspp
0.856
1.307
0.389
0.259
16.139
0.181
0.327
kmeanspp (ours)
0.808
1.230
0,381
0.246
15,974
0.176
0.318
3.5%
fw
0.915
1.382
0.407
0.295
17.582
0.272
0.341
[w (ours)
0.856
1.268
0.391
0.273
18.389
0.231
0.341
5.2%
bait
0.812
1.267
0.404
0.307
18.218
0.375
0.350
bait (ours)
0.797
1.227
0.398
0.271
19.114
0.299
0.335
5.4%
maxdet
0.843
1.260
0.393
0.333
16.257
0.240
0.336
maxdet (ours)
0.802
1.200'
0.383
0.258
16.587
0.161
0.328
9.8%
maxdiag
0.880
1.363
0.426
0.431
20.508
0.451
0.372
maxdiag (ours)
0.800
1.207
0.388
0.280
16.223
0.196
0.323
22.2%
Table RMSE comparison at step 15: target-only trained versus target and auxiliary combined data (Aux) trained models.
S1
BIO
BIKE
DIAMOND
CT
STOCK
maxdiag
0.819
1.205
0.384
0.233
14.029
0.150
0.306
maxdiag (Aux)
1.514
1.958
0.639
1.008
19.256
0.199
0.325
maxdet
0.819
1.223
0.374
0.223
13.817
0.131
0.309
maxdet (Aux)
1.512
1.956
0.631
1.048
19.215
0.193
0.337
bait
0.818
1.254
0.396
0.230
17.131
0.200
0.309
bait (Aux)
1.527
1.977
0.628
1.056
19.610
0.248
0.329
fw
0.889
1.332
0.381
0.219
16.957
0.178
0.317
fw (Aux)
1.547
1.985
0.620
1.096
19.165
0.213
0.329
maxdist
0.829
1.223
0.384
0.230
13.675
0.141
0.304
maxdist (Aux)
1.528
1.967
0.636
1.003
19.161
0.200
0.337
kmeanspp
0.831
1.244
0.378
0.214
13.708
0.140
0.301
kmeanspp (Aux)
1.521
1.973
0.625
1.012
19.155
0.199
0.330
lemd
0.837
1.235
0.383
0.240
14.421
0.148
0.301
lemd (Aux)
1.529
1.964
0.635
1.004
19.191
0.198
0.338
random
1.011
1.490
0.397
0.235
16.307
0.233
0.319
random (Aux)
1.541
1.994
0.622
1.077
19.455
0.257
0.333
A.6.3 Additional Computational Burden Analysis Results
For different methods, in Table D we present the computational resource consumption, where Av- gUsg denotes the memory usage and AvgTTime represents the average runtime per step of the method. The first row reports the consumption of BMDAL, followed by the consumption of our method under
26

---

AGBAL-maxdaag
BMDAL-maxdiag
maxdet
AMDAL-maxder
0-
0-
0-
Figure 5: Visualization of the loss of selected points for maxdiag and maxdet methods.
AGBAL -aait
BMDAL-bait
AGRAI-maxdist
BMDAL-maxdist
40
40
-2
PC-1
Figure 6: Visualization of the loss of selected points for bait and maxdist methods.
AGBAL-kmeanspp
BMDAL-kmeanspe
PC2
0
20
PC11
PC1
Figure 7: Visualization of the loss of selected points for kmeanspp and random methods.
Table 4: MSE results of S1.
5
6
7
11
12
13
14
15
random
1.077
1.058
0.993
0.971
0.955
0.942
0.930
0.921
0.911
0.900
0.890
0.882
0.875
0.868
0.862
0.857
maxdiag
1.077
0.972 0.928
0.921
0.917
0.
0.912 0.9
0.908 0. 0,900
0.894
0.886
0.880
0.873
0. 867
0.
0.902
0.857
0.853
maxdiag (ours)
1.777
0.977 0.879
0.859
0.843
0.834
0.826
0.818
0.810
0.805
0,800
0.795
0.791
0.788
0,786
0.783
muxdet
1.077
0.982 0.924
0.906
0.893
0.881
0.874 0.866 0.858
0.850
0.843
0.837
0.831
0.826 0.821 0.818
maxdet (ours)
L077 0.949 0.867
0.846
0.838
0,828
0.822
0.817
0.811
0.506
0.802
0.798
0.795
0.792
0.790
0.788
bait
1.077 1.014
0.917
0.885
0.864
0. 808
0.839
0. 0.831
0.824
0.818
0.812
0.807
0. 8.802
0. 799
0.796
0.794
(ours)
0.830
fw
1.077 1.062 1.016
0.995
0.987
0.978
0.967
0.903
0.939
0.926
0.915
0.906
0.898
0.891
0.884
0.879
fw (ours)
1.077 1.046 0.963
0.942
0,930
0.915
0.899
0,887
0.876
0.865
0.856
0,847
0.840
0.834
0.829
0.825
maxdist
1.077
0.963 0.900
0.884
0.875
0.867
0.859
0.851
0.845
0.839
0.833
0.826
0.822
0.817
0.814
0.810
maxdist (ours)
1,077
0.966 0.881
0.861
0.846
0,833
0.824
0.816
0.808
0.802
0.797
0.793
0.789
0.786
0.783
0.781
kmeanspp
1.077
1.020
0.954
0.926
0.915
0.906
0.896
0.888
0.876
0.565
0.856
0.848
0.841
0.835
0.831
0.827
kmeanspp (ours)
1.777
0.967
0,892
0.864
0.849
0.838
0.831
0.825
0.819
0.813
0.308
0.803
0.799
0.795
0.792
0.790
lamd
1.075
1.066
1.046
1.027
0.956
0.942
0.931
0.921
0.913
Jemd (ours)
1,077
0,988
0.948
0.876
0.853
0.840
0.831
0.822
0,815
0.308
0.803
0.798
0.794
0.791
0.788
0.786
27

---

Table 5: MSE results of S2.
12
13
14
random
1.703
1.652
1.541
1.492
1.460
1.441
1.424
1.403
1.384
1.367
1.352
1.339
1.328
1.318
1.310
1.303
maxdiag
1.703
1.516
1.451
1.427
1.423
1.415
1.409
1.395
1.383
1.373
1.363
1.352
1.342
1.334
1.326
1.319
maxdiag (ours)
1.703
1.504
1.346
1.309
1.279
1.261
1.246
1.235
1.225
1.216
1.207
1.200
1.195
1.191
1.187
1.184
musdet
1.703
1.402
1.364
1.341
1.324
1.308
1.295
1.283
1.270
1.260
1.251
1.244
1.238
1.233
1.228
maxdet (ours)
1.703 1.453
1,310
1.277
1.259
1.245
1.231
1.221
1.212
1.206
1.200
1.195
1.190
1.186
1.182
1.180
bait
1.703 1.632 1.459
1.396
1.360
1:336
1.319
1.303
1.290
1.277
1.267
1.258
1.250
1.244
1.238
1.234
bait (ours)
1.703 1.568
1.382
1.326
1.303
1.287
1,270
1.256
1.245
1.235
1.227
1.220
1.214
1.209
1.205
1.201
Iw
1.703 1.663
1.571
1.528
1.496
1.473
1.451
1.434
1.415
1.398
1.382
1.367
1.355
1.344
1.335
1.328
Fwr (surs)
1.703 1,586
1,459
1.408
1.376
1.350
1.329
1,311
1.296
1.282
1,268
1,257
1.248
1.241
1.234
1.229
maxdist
1.703
1.491
1.390
1.348
1.322
1.306
1.292
1.281
1.271
1.263
1.255
1.248
1.243
1.238
1.233
1.230
maxdist (ours)
1.703
1,358
1,273
1.255
1.241
1.226
1,216
1.208
1.196
1.192
1,188
1,185
1,182
kmeanspp
1.703 1.601 1.502
1.454
1.417
1.393
1.374
1.354
1.336
1.321
1.307
1.295
1.285
1.276
1.270
1.264
kmeanspp (ours)
1.703
1.532
1,383
1.336
1.308
1.291
1,276
1.263
1.249
1.239
1.230
1.222
1.216
1,210
1,205
1.202
lomd
1.703
1.748
1.686
1.607
1.582
1.563
1.538
1.511
1.486
1.466
1.443
1.425
1.405
1.390
1.376
1.365
lemd (ours)
1.703
1.534
1.422
1.326
1.291
1,270
1.254
1.239
1.224
1.215
1.206
1.198
1.193
1.188
1.183
1.180
Table 6: MSE results of BIO.
0
5
6
7
10
11
12
13
14
15
random
0.677 0.536
0.512
0.
0.468
0. 164
0.449
0.428
0.416
0.409
0.
405
0.
401
0,399
0,397
maxdiag
0.677
0.525
0.490
0.472
0.457
0.449
0.447
0.442
0.435
0.430
0.426
0.420
0.415
0.412
0.409
0.406
maxdiag (ours)
0.677
0.470
0.451
0.463
0.487
0.503
0.503
0.515
0.519
0.523
0.524
0.526
0.533
0.537
maxdet
0.677 0.472 0.448
0.434
0.418
0.412
0.407
0.402
0.399
0.397
0.393
0.391
0.389
0.388
0.387
0.39667
maxdet (ours)
0.677 0.483 0.443
0.418
0.408
0.400
0.395
0.391
0.387
0.385
0.383
0.382
0.381
0.380
0.379
0.380
bait
0.677 0.558 0.503
0.467
0.448
0. 439
0.422
0.424
0.412
4.4
0.404
0.401
0.398
0.397
0.394
0
buit (ours)
0.677 0.513
0.478
0.460
0.439
0.429
0.422
0.413
0.407
0.402
0.398
0.396
0.395
0.393
0.390
0.388
Iw
0.470
0.457
0.433
0. 430
0.422
0.411
0.403
0.
0.400
0,398
0,396
0,394
Iw (ours)
0.677 0.484 0.451
0.434
0.419
0.411
0.406
0.400
0.398
0.394
0.391
0.391
0.389
0.388
0.387
0.386
maxdist
0.677 01.486 0.460
0.448
0.433
0.423
0.419
0.414
0.409
0.406
0.404
0.401
0.398
0.396
0,305
0.394
maxdist (ours)
0.677 0,501 0,466
0.440
0,421
0.408
0,400
0.394
0.391
0.388
0.386
0.384
0.382
0.381
0.379
0.378
kmeanspp
0.677 0.174 0.443
0.9
0.415
0. 406
0. 0.4
0.396
0.393
0.390
0.389
0.387
0
0.385
0,380
0.382
0,382
kmeanspp (ours)
0.677 0.467 0.431
0.417
0.407
0.399
0.394
0.390
0.385
0.384
0.381
0.379
0.378
0.377
0.376
0.375
lamd
0.677 0..772 0.452
0.432
0.422
0.415
0.405
4.400
0.397
0.393
0.390
0.388
0.386
0.385
0.384
0.383
lemd (ours)
0.677
0.472
0.450
0.432
0.421
0.412
0.402
0.387
0.385
0.384
Table 7: MSE results of BIKE.
5
6
7
10
11
12
13
14
15
random
1.413
0.923
0.764
0.613
0.537
0. 0.462
0.408
0.368
0.340
0.315
0.295
0.278
0.263
0.252
0.243
0.235
maxdiag
1.413 1.051
0.929
0.798
0. 705
0.
0.622
0.574
0.524
0.483
0.454
0.431
0.409
0.389
0.367
0.349
0.332
maxdiag (ours)
1.413
0.830
0.620
0.505
0.426
0.378
0.349
0.324
0.307
0.292
0.280
0.269
0.259
0.249
0.242
0.236
maxdet
1.413 0.890 0.751
0.631
0.561
0.504
0.452
0.
0.381
0.356
0.333
0.315
0.300
0.286
0.276
0.268
maxdet (ours)
1.413 0.698 0.476
0.397
0.351
0.327
0.312
0.291
0,280
0.268
0.258
0.249
0.242
0.237
0.231
0.226
bait
1.413 0.971 0.789
0.661
0.574
0.
191
0.431
0.390
0.355
0.328
0.307
0.292
0.278
0.267
0.255
0.246
(ours)
1,133
0.480
0.416
0.374
0.346
0.261
0.235
fw
1.413 0.947 0.745 0.614
0.514
0.443
0.399
0.365
0.335
0.312
0.295
0.278
0.263
0.253
0.245 0.237
fw (ours)
1.413 0.809 0.577
0.491
0.429
0.390
0.357
0.330
0.308
0.289
0.273
0.261
0.251
0.243
0.236 0.229
maxdist
1.113 0.915 0.716 0.579
0.486
0. 123
0.377
0.316
0.321
0.300
082
0.271
0.259
0.249
0.230 0.232
maxdist (ours)
L413 0.829 0.597
0.484
0.412
0.368
0.340
0.316
0.300
0.285
0.272
0.260
0.251
0.242
0.234
0.227
kmeanspp
1.413 0.852 0.654 0.511
0.431
0. 380
0.342
0.313
0.273
0.
0.219
0.239
0.231
0.224 0.219
kmeanspp (ours)
1.433 0.762 0,522
0.416
0.362
0.327
0.300
0.283
0.268
0.257
0.246
0.238
0.230
0.224
0.218 0.213
lomd
1.413
0.819 0.620
0.502
0.420
0.369
0.331
0.302
0.281
0.266
0.256
0.245
0.237
0.229
0.223
0.218
lemd (ours)
1,133 0.819 0.620
0.559
0.489
0.450
0.411
0.369
0.341
0.317
0.291
0.276
0.261
0.252
0.242
0.234
28

---

Table 8: MSE results of DIAMOND.
15
random
33.608
29. 686
26.771
220, 4865 225. 0:235
78U
20.
474
19.
429
18 923
19,309
17.804
39.3
17.090
10,
809
16. 5212
16 307
mexdiag
33.668
34.604
83.749
9:29
26.469
864
22.
21
20.508
19.791
19.487
18.451
18.232
masding (ouns)
33,666
85.427
27.964
25.435
23,301
22,054
20.700
19.560
18.331
16.854
16.223
15.840
15.467
15,038
14,841
14,628
masdet
66.
30.538
145
21.
20.044
326
17
15.887
15.462
0:0
14.652
maxdet (ours)
33,666
31.333
26.014
24.047
21.857
20.019
18.761
18.211
17.210
17.039
16.987
16.273
15.725
15.288
15.081
14:826
buit
33. 666
31.433
25. 557
26. 973
25, 002
23. 365
22, 6:10
20, 566
19. 652
18. 897
218
18.127
17.578
17. 17.345
17.183
16. 16.977
buit (ours)
33,066
31.764
28.619
26.016
24.163
22.536
21.492
20.835
20.311
19,377
19.114
18.677
18,249
17.902
17.557
17,330
tw
33.66B
31 .308
229
4512
223 0:34 21, 2622
20, 6:25
19. 738
18.
17.968
17.082
16:979
16.690
16,
216
15,
15.608
33,666
31662
28.651
23,475
19,677
19,077
17,313
16.73
15.914
mandiist
33 63.666
33.311
37.960
517 22 198
21.050
19. 973
19. 212
18 238
17:297
16. 691
16.460
16.095
15.988
15.775
15.408
maxdist (ouns)
33,666
36.023
29.659
26.134
23,642
22.181
20.896
19.661
18.201
17.259
16.657
16.290
15,980
15.466
15:346
14,930
kmeussgp
333 G16 30.
26.346
23. 9A15
21. 830
20.
200
19.
375
17 934
17.U55
16.410
16. 139
15,809
15.617
15.309
15.029
14.580
knicunspp (ours)
33,666 31,448
26.154
22,890
21.410
20.365
19.123
18.30
17,335
16.468
15.974
15.656
15.188
15,075
14.715
14,523
lamd
33. 666 30.379
26.531
23.009
21.025
19.093
884
18.003
17 37.322
17,037
16.644
15,
15.914
10.302
15.322
15. 008
kmdd (ours)
33,666
30.179
26.501
26.659
22,573
21,850
21.215
19.384
18,087
17,673
17.045
16.671
16.032
16,048
15.667
15.366
Table 9: MSE results of CT.
0
5
6
7
11
12
13
14
15
random
0.595
0.503
0.445
0.419
0.309
0.372
0.358
0.351
0.337
0.308
0.292
0.276
0.258
0.245
0.233
maxdiag
0.848
0.641
0.608
0.597
0.593
0.594
0.598
0.564
0.538
0.503
0.451
0.431
0.407
0.387
0.371
0.351
maxdiag (ours)
0.848
0.595
0.457
0.411
0.379
0.331
0.295
0.266
0.236
0.215
0.196
0.183
0.173
0.166
0.158
0.153
muxdet
0.848
0.527 0.440
0.391
0.364
0.342
0.318
0.299
0.278
0.259
0.240
0.224
0:211
0.202
0.193
0.184
maxdet (ours)
0.848 0.517
0.394
0.333
0.286
0,260
0.230
0.203
0.186
0.171
0.161
0.153
0.146
0.140
0.135
0.131
bail
0.554
0.514
0.470
0.462
0.438
031
0.393
0.356
0.327
0.313
0.298
buit (ours)
0.848 0.620
0,488
0.454
0.425
0.395
0.378
0.359
0.354
0.322
0.299
0.277
0.255
0.234
0.225
0.211
0.848 0.554
0.463
0.417
0.388
0.367
0.355
0.330
0.299
0.286
0.272
0.259
0.241
0.227
0,216 0.207
fw (ours)
0.848 0.498
0.430
0.391
0.346
0.323
0,303
0.286
0,266
0.249
0,231
0.216
0.206
0.196
0.187
0.179
maxdist
0.848 0.538 0.415
0.356
0.309
0.277
0.250
0.225
0.205 0.190
0.177 0.168
0.1
0.150
0.143 0.138
maxdist (ours)
0,848 0.610 0.467
0.407
0.363
0.328
0.291
0.264
0,232
0.204
0,186
0.171
0.160
0,151
0.146
0,142
kmeanspp
0.818 0.507 0.115
0.356
0.311
0.280
0.252
0. 0.232
0.214
0.191
0. 181 0.168
0. 059
0.152
0.146 0.139
kmeanspp (ours)
0.848
0.335
0.292
0.269
0.247
0.222
0.189
0.166
0.157
0.149
0.143 0,138
lemd
0.848
0.485
0.371
0.331
0.293
0.266
0.244
0.217
0.198
0.179
0.168
0.158
0. 041
0.140
0.1 0.132
lemd (ours)
0.848
0.485
0.433
0.392
0.354
0.321
0.283
0.257
0.242
0.216
0.197
0.185
0.174
0.165
0.157
0.151
Table 10: MSE results of STOCK.
5
6
7
10
11
12
13
14
15
random
0.588
0..98
0.468
0.451
0.430
0.
409
0.389
0.379
0.367 0.355
0.346
0.340
0.334
0.328
0.324
0.319
maxdiag
0.588
0.541
0.496
0.465
0.448
0.
429
0.416
0. 0.400
0.392
0.383
0.372
0.365
0.358
0.352
0.346
0.342
muxdiag (ours)
0.588
0.476 0.434
0.395
0.374
0.360
0.349
0.339
0.333
0.328
0.323
0.319
0.315
0.312
0.309
0.307
muxdet
0.588 0.488 0.456
0.429
0.405
0.387
0.372 0.361 0.350
0.343
0.336
0.330
0.324
0.321
0.316
0.313
maxdet (ours)
0.588 0.449 0.407
0.388
0.367
0.357
0.350
0.341
0.337
0.328
0.325
0.322
0.319
0.316
0.313
bait
0.588 0.595 0.168
0.447
0.424
0.
004
0.388
0.377
0.369
0.360
0.350
0.343
0
0.336
0.331
0.326
0.321
(ours)
0.588
0.406
0.387
0.378
0.365
0.355
0.342
0.329
0.324
0.321
0.317
0.312
fw
0.588 0.513
0.477
0.442
0.421
0.404
0.385
0.368
0.360
0.351
0.341
0.336
0.330
0.324
0.319
0.316
Iw (ours)
0.588 0.480 0.445
0.416
0.392
0.382
0.369
0.360
0.352
0.345
0.341
0.335
0.330
0.327
0.323
0.319
maxdist
0.588
0.511 0.481 0.445
0.416
0.402
0.386
0.372
0.362
0.350
0.343
0.337
0.330
0.325
0.321
0.317
maxdist (ours)
0,588 0.477 0.436
0.400
0.375
0.359
0.345
0.341
0.333
0.327
0.321
0.318
0.315
0.312
0.309
0.306
kmeanspp
0.588
0.193 0.157
0.398
0.
382
0.365
0.353
0.3/12
0.331
0.327
0.320
0.315
0.310
0.306
0.303
kmeanspp (ours)
0,588 0.452 0.412
0.389
0.371
0.357
0.346
0.338
0.329
0.324
0.318
0.314
0,311
0.308
0.305
0.302
lomd
0.498
0.452
0.421
0.396
0.374
0.358
0.347
0.339
0.332
0.328
0.321
0.316
0.312
0.308
0.305
lemd (ours)
0.588
0.498
0.464
0.419
0.392
0.365
0.345
0.332
0,325
0.319
0.315
0.311
0
0.306
0.303
0.301
29

---

maxdlagiA)
maxdettA)
baR(A)
Tw(A)
maxdist(A)
kmeanspp(A)
lomdIA]
1.1
1.1
1.1
1.0
1.0
1.0-
0.3
0.8
1.1
1.1
1.1
1.0
1.0
0.9
Epoch
Epoch
Figure 8: S1 MSEs plot.
maxdiagi8) maxdiagiA)
..... maxdellB)
bai(())
fwiR)
..... maxdist(B)
kmeanspp(8)
lomdi8
kmeanspp(A)
1.6-
1.4
1.4
1.2
5
10
15
10
15
1.8
1.6
1.4
1.2
10
15
15
Epoch
Epoch
Epech
Figure 9: S2 MSEs plot.
maxdiagi8) maxdiag/A)
maxdettB) maxdetlA)
baR(B)
maxdiist(B) maxdist(A)
kmeanspp(B) kmeenspp(A)
londie]
bak(()
fw(A)
landAA]
0.7
0.7
0.7
0.7
MSLE
0.5
0.5
0.5
0.5
0.4
0.4
10
15
0.7
0.7
0.7
0.5
0.5
0.5
10
15
Epach
Epoch
Spoch
Figure 10: BIO MSEs plot.
30

---

maxdiapi8i maxdlagiA)
landiB lomdIA]
maxdettA)
baz(A)
Tw(A)
maxdist(A)
kmeanspp(A)
1.5
0.5
0.5
Epoch
Epech
Spoch
Figure 11: BIKE MSEs plot.
maxdiagi8)
...... maxdellB)
bai(())
fwi8)
maxdist(B)
kmeanspp(8) kmeanspp(A)
lomdi8
maxdiagiA)
35
30
30
MSG
25
20
20
15
0
1.0
15
0
5
10
15
30
5
10
40
30
30
30
20
0
5
10
15
5
10
15
10
Epoch
Epoch
Epoch
Figure 12: DIAMOND MSEs plot.
maxdiagi8) maxdiagiA
maxdettB maxdet(A)
baR(B)
maaxdist(B) maxdist(A)
kmeanspp(B) kmeenspp(A)
londiB] landAA]
bak(()
fw(A)
0.8
0.8
M8b
0.6
0.6-
0.4
0.2
0.2
10
15
10
0.8
0.8
0.8
0.6
0.6
0.6
0.7
30
15
Epach
Epoch
Spoch
Figure 13: CT MSEs plot.
31

---

varying sizes of auxiliary data samples, along with the corresponding AUC improvement achieved each sample size.
Table 11: Computational Burden Analysis Results.
lemd
maxdiag
maxdet
fw
maxdist
bait
kmeanspp
AvgUsg (M)
BMDAL
571.4
502.2
553.8
544.0
576.0
596.6
562.2
ours (100)
575.2
501.4
561.4
570.6
570.0
599.0
511.0
(500)
593.4
522.0
578.6
567.0
587.0
609.0
520.2
ours (1000)
595.0
529.0
584.8
578.0
585.8
610.8
539.2
ours (10000)
766.0
713.4
750.0
752.4
770.6
769.2
765.4
AvgTime (s)
BMDAL
1.39
0.06
0.32
0.23
1.19
1.10
1.28
ours (100)
2.35
0.91
1.23
1.12
2.10
2.02
2.41
(500)
3.08
1.62
1.97
1.83
2.80
2.72
2.81
ours (1000)
3.57
2.27
2.62
2.46
3.39
3.32
3.17
ours (10000)
15.04
14.49
15.23
14.74
15.95
15.44
15.89
Avglmpro (%)
ours (100)
-1..77
14.91
5.83
2.28
-9.17
5.86
-066
ours (500)
-6.17
20.31
10.69
4.66
-0.40
8.95
2.60
ours (1000)
5.65
17.99
10.47
5.03
1.13
7.71
3.86
ours (10000)
-6.08
22.56
12.02
5.20
1.91
8.76
4.47
A.7 Additional Experiments
A.7.1
Experiments Auxiliary Data Quality
Since the core of method lies in improving loss estimation through auxiliary data guidance, an intuitive expectation that better alignment between auxiliary and target distributions should yield superior estimation performance. This naturally raises the question: how severely will our method degrade as auxiliary data quality deteriorates? To investigate this, we adopt the same experimental settings defined in SI and S2.
At ( = 64, where the distribution shift is most severe, AGBAL achieves its highest AUC for the MSE curve while still outperforming BMDAL (which no auxiliary data). As shown in Ta- ble I2 AGBAL maintains superior performance across most selection methods, except for a slight degradation under maxdist. This demonstrates our method's robustness: when distribution discrep- ancy becomes extreme, the density ratio estimation effectively nullifies the auxiliary data's influence (weights approach zero). preventing negative learning while maintaining comparable performance to not using auxiliary data at all.
Table 12: Worst case AUC comparison between AGBAL and BMDAL.
maxdiag
maxdet
bait
fw
maxdist
kmeanspp
lemd
S1
BMDAL
0.956
0.952
0.914
1.038
0.933
0.975
1.131
AGBAL (ours)
0.942
0.918
0.904
1.038
0.947
0.940
0.975
Improvement
1.5%
3.6%
1.1%
0.0%
-1.5%
3.6%
13.8%
S2
BMDAL
1.501
1.430
1.417
1.583
1.406
1.479
1.647
AGBAL (ours)
1.437
1.398
1.390
1.543
1.436
1.426
1.468
Improvement
4.3%
2.2%
1.9%
2.5%
-2.1%
3.6%
10.9%
A.7.2
Experiments on Auxiliary Data Quantity
On the other hand, the volume of auxiliary data warrants investigation. While real-world scenarios typically provide abundant auxiliary data (e.g., corrupted data or related-task data), limited-quantity cases do exist such as small historical archives with long-term records. To examine the perfor- mance of our method with scarce auxiliary data, conduct experiments with varying volumes (50, 100, 300, 500, 1000 samples) under the S1 and S2 configurations, fixing Ç = 8. For each selec- tion method, evaluate the performance by computing the AUC of MSE curves.
Figure 144S presents the AUC values of MSE curves for AGBAL (solid lines) versus BMDAL (dashed lines, without auxiliary data) across varying auxiliary data sizes (Naux). The results demon-
32

---

strate: (1) AGBAL's performance improves monotonically with increasing Naux: (2) At minimal data volume aNx 50), AGBAL achieves comparable performance to BMDAL while maintain- ing superiority in most cases; (3) The only exception occurs under maxdist selection, where AGBAL marginally underperforms BMDAL. These findings confirm that even with modest amounts of aux- iliary data, AGBAL can be still useful.
maxdiag
mixdee
boit (A)
for maxdiss
kmeanspp kmmanssp
land (A]
maxdlag
balt (8)
tv
0.90
0.94
0.88
0.86
0.845
0.85
0.840
0.92
0.84
0.835
0.830
0.90
500
0.87
0.86
0.86
0.50
0.84
0.84
0.85
300
500
2000
100
300 Nax
1000
100
500
1000
Naw
Figure 14: AUC plots of varying Naux for SI.
maxdiag
kmeanspp kmeanspp
maxdiag
maxdet
- boitt (1)
for
maxdist (8)
land (9)
1.32
1.450
1.425
1.30
AUC
1.35
1.20
1.375
1.30
1.26
1.375
1.32
1.350
1.4
1.28
1.300
1.3
300
500
300
1000
Figure 15: AUC plots of varying Naux for S2.
We evaluate our method on five real-world datasets with varying auxiliary data sizes Naux. Fig- 2 illustrates the AUC trends against auxiliary data volume. Due to the inherent distribu- tional complexity of real-world data, the AUC-n curves exhibit non-smooth variations. However, aggregating results across all five datasets and selection methods reveals: (1) consistent decreasing trend in AUC as Naux increases; (2) at minimal auxiliary data (Naux = 100), while AGBAL's ad- vantage over BMDAL (measured by optimal selection method AUC) becomes marginal, remains competitive demonstrating the framework's robustness.
A.8 Limitations
The performance of AGBAL is fundamentally constrained by the quality and relevance of the auxil- iary data. While the method demonstrates robustness to moderate distribution shifts between auxil- iary and target distributions, its effectiveness diminishes when the auxiliary data becomes too noisy or exhibits systematic biases.
The theoretical analysis relies on NTK assumptions that may not hold in practical settings. While the infinite-widie network approximation provides valuable insights, its applicability to modern deep architectures with finite width and complex layer interactions remains uncertain. This paper does
33

---

maxdiag
macdet
mandist
kmeanspo
0.435
0.4400
0.48
0.415
0.4375
0.430
0.46
0.4350
0.420
0.4325
0.42
1000
10000
100
1000
10000
500
1000
10000
100
10000
0.04
0.412
0.425
0.43
0.410
0.408
0.420
0.42
0.406
1000
10000
100
1000
10000
500
1000
Figure 16: AUC plots of varying Naux for BIO.
maxdiag IA]
maxdet (A)
bailt (A)
fu IA]
maxdist (A)
kmeanspp (A)
lamd [A]
maxdiag
maxdist
kmaanspp
0.00
0.48
0.55
0.46
0.44
0.50
0.44
0.45
0.42
0.42
0.40
0.35
0.40
100
500
1000
10000
300
500
1000
10000
100
500
1000
10000
100
500
1000
10000
0.46
0.44
0.40
0.44
0.42
0.38
0.42
0.37
0.40
100
500
1000
10000
100
500
1000
10000
100
500
1000
10000
Nas
N...
Figure 17: AUC plots of varying Naux for BIKE.
maxdiag
mazdet
(A)
fy
kmeanspp kmeenspp (B)
maxding
maxdet 09)
- bait (8)
for EBI
maxdist (8)
lamd [9]
20.5
22.00
22.0
21.75
21.50
19.0
21.25
22.0
21.5
20.5
20.50
20.0
20.25
21.0
19.5
20.00
20.5
19.0
19.75
1000
10000
100
Nex
Figure 18: AUC plots of varying Nanx for DIAMOND.
34

---

maxdiag
macdet
balt
mandist
kmeanspp
0.50
0.34
0.45
0.30
0.33
0.40
0.28
0.32
0.35
0.30
0.31
100
500
1000
10000
100
500
1000
10000
1000
500
1000
10000
100
500
10000
10000
0.375
0.29
0.28
0.325
0.30
300
0.27
0.28
0.275
100
500
1000
10000
100
500
1000
10000
Masc
Figure 19: AUC plots of varying Naux for CT.
maxdist
maxdiag
macdet
balt
tv
kmeanssp
0.375
0.385
0.370
0.365
0.380
0.37
0.36
0.375
500
1000
10000
10000
100
500
1000
10000
500
1000
10000
0.39
0.380
0.370
0.38
0.365
0.375
0.37
0.360
0.370
0.36
0.355
0.365
500
1000
10000
10000
100
500
1000
10000
Nexc
Figure 20: AUC plots varying Naux for STOCK.
investigate how deviations from these ideal conditions might affect the method's performance in real-world applications.
The experimental validation, while comprehensive, focuses primarily regression tasks with rel- atively low-dimensional input. The performance of AGBAL on high-dimensional structured data, such images or time series, remains unexplored. Additionally, all experiments assume the avail- ability of auxiliary data with similar feature spaces, leaving open the question of how the method would perform when auxiliary data come from substantially different modalities.
A.9 Broader Impacts
The proposed AGBAL framework has several potential positive social impacts. By reducing anno- tation costs through more efficient active learning, our method could make machine learning more accessible in resource-constraine domains such as healthcare in developing regions or small-scale industrial applications. The ability leverage imperfect auxiliary data aligns well with real-world scenarios where perfect datasets are rare, potentially enabling more applications in safety-critical domains such as medical diagnosis autonomous driving.
While effective, our method raises privacy concerns when auxiliary data contains sensitive informa- tion, and may require fairness considerations when adapted to classification tasks. Though efficiency gains could lower barriers for misuse, this risk is mitigated by the method's domain aagnosii nature. We encourage future work preseevvin and fairnes--awe awwrre extensions.
35


Reviews:
[{"review_id": "DwJyK5F4R2", "reviewer": "Reviewer_Xdew", "summary": "This paper proposes AGBAL (Auxiliary data Guided Batch Active Learning), a new framework for regression-based active learning that incorporates imperfect auxiliary data to better estimate predictive uncertainty. Unlike prior methods which rely heavily on model gradients alone, AGBAL estimates the expected loss gradient by weighting auxiliary data losses using a density ratio between auxiliary and target distributions. The estimated gradient is then used to define an auxiliary-data-guided gradient kernel, improving sample selection strategies. The method is theoretically grounded with derivations based on NTK (Neural Tangent Kernel) theory and evaluated on synthetic and real-world regression datasets, showing consistent performance gains over conventional approaches.", "questions": "1. How robust is AGBAL to mislabeled or low-quality auxiliary data? Can performance degrade catastrophically if auxiliary data is noisy, even if the density ratio is correctly estimated?\n2. How is AGBAL fundamentally different from importance sampling or domain adaptation weighting schemes? Could this approach be viewed as an instance of transfer-active learning with a known source domain?\n3. Given that Lₜ is actively selected and thus biased, how reliable is it as a reference for estimating the density ratio or surrogate gradients? Would using a small, randomly labeled validation set instead improve stability?\n4. Could the auxiliary data be adversarially exploited? Since the method uses them for uncertainty estimation, poorly aligned auxiliary data could potentially misguide the acquisition function.\n5. Can this framework be extended to classification tasks, or is it strictly limited to regression? The kernel-based formulation and surrogate gradient approximation might generalize.", "limitations": "- Biased Labeled Set (Lₜ): The framework assumes Lₜ to be a reliable proxy for the target distribution, but active learning selection is inherently biased.\n- Data Quality Sensitivity: The performance is highly sensitive to the quality of the auxiliary dataset and the correctness of the density ratio estimation.\n- Distribution Matching Assumption: The method assumes that the auxiliary distribution is absolutely continuous with respect to the target distribution, which may not hold in real-world cases (e.g., outlier auxiliary sources).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Novelty: The idea of leveraging auxiliary (distribution-shifted) data via density ratio-weighted gradient approximation is conceptually fresh and well-motivated for regression tasks, where predictive uncertainty is hard to estimate without labels.\n- Theoretical Rigor: The authors provide solid mathematical grounding, including an NTK-based analysis showing how the auxiliary-guided gradient approximates the true expected loss gradient.\n- Empirical Effectiveness: Extensive experiments on multiple datasets and across various selection strategies demonstrate that AGBAL consistently improves MSE and convergence over baseline methods.\n\nWeaknesses\n- Dependency on Auxiliary Data Quality: The method assumes the availability of auxiliary data that is not only relevant to the target task but also sufficiently high quality. As shown in the authors’ own ablation study (Fig. 2), performance deteriorates when the auxiliary data distribution is heavily shifted. In practice, ensuring auxiliary data quality may be non-trivial.\n- Resemblance to Importance Sampling: The core mechanism of re-weighting auxiliary losses by a density ratio is reminiscent of importance sampling in off-policy learning or domain adaptation. While the authors mention this connection, a clearer differentiation—conceptually or in results—is needed.\n- Bias in Target Labeled Set (Lₜ): The estimation of the density ratio relies on the labeled target dataset Lₜ, which is obtained through active learning. However, such data is typically biased, as AL tends to over-sample uncertain or diverse examples. This raises concerns about the validity of the density ratio estimation and the generalizability of the reweighted losses.\n- Auxiliary Uncertainty Estimation is Still Model-Dependent: Although the method avoids relying solely on model gradients, the uncertainty estimation through auxiliary loss still depends on a trained model, and errors or biases in that model can propagate into the selection process.\n- Borderline Scalability: The full method requires training and maintaining auxiliary loss estimators, density ratio estimators, and selection kernels, which could add computational burden compared to simpler gradient-based AL strategies.\n- Readability: Certain sections of the paper-particulary Section 2.3- are difficult to follow. The exposition is mathematically dense and lacks intuition-building commentary. The derivation involving NTK-based decomposition of loss gradient estimation is correct but could benefit from clearer structure, step-wise annotations, or illustrative figures.\n- Scalability and Complexity: The full pipeline involves model training, auxiliary data loss computation, density ratio estimation, and guided kernel construction. The practical cost of running AGBAL is not clearly discussed, making it less attractive for low-resource or fast-iteration settings.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive assessment recognizing the novelty of our density ratio-weighted gradient method for regression uncertainty, the theoretical rigor of our NTK analysis, and the empirical effectiveness across diverse datasets. Below we address all comments and provide additional results.\n\n**W1\\&Q1\\&Q4:** Thank you for your thoughtful comment regarding the dependency on auxiliary data quality. We appreciate your attention to this important aspect of our method. \n- As shown in Figure 2, the performance degrades only linearly even when the parameter $\\zeta$ controlling the distribution shift increases exponentially. This suggests that our approach is not overly sensitive to moderate mismatches in auxiliary data quality. \n- We acknowledge that when auxiliary data is poorly aligned with the target distribution (e.g., having completely disjoint support), the estimated loss may become uncorrelated with the true loss. In such extreme cases, if the estimated loss is small where the true loss is large, this could indeed misguide the acquisition function. \n- Currently, we determine whether the auxiliary data is suitable based on prior domain knowledge. We fully agree that developing quantitative measures to evaluate auxiliary data quality would be valuable to prevent such adversarial scenarios. This is an important direction we plan to explore in future work including both auxiliary data quality measurement and high-quality auxiliary data selection.\n\n**W2\\&Q2:** Thank you for this thoughtful question regarding the relationship between our method and importance sampling or domain adaptation weighting schemes. \n- Our method differs from standard domain adaptation in two key aspects: \n    1. most domain adaptation methods assume a different distribution of $X$ but consistent distribution of $Y\\mid X$ across domains (covariate shift setting)\n    2. in the setting of our paper, we assume arbitrary distribution shifts between auxiliary and target data (including both covariate and label shifts), due to the corrupted nature of the auxiliary data. This makes traditional covariate weighting schemes unsuitable for loss estimation in our setting. \n- On the other hand, while the estimation of $\\phi_2$ indeed incorporates importance sampling principles, our methodology extends beyond conventional implementations. Specifically, we utilize the auxiliary data's joint distribution over both covariates and labels as our proposal distribution for target loss estimation, rather than relying solely on marginal covariate distributions.\n\n\n**W3\\&Q3:** Thank you for raising this important concern about potential bias in the labeled target set $L_t$. This is an insightful observation about our density ratio estimation procedure. \n- First, your point is particularly relevant, and we would like to clarify that our method's design naturally accommodates this form of bias. While active learning does indeed induce a shift in the covariate distribution of $L_t$ (as certain examples are preferentially selected), the conditional distribution of $Y\\mid X$ remains unaffected since we always observe the true labels for selected points. \n- Crucially, as shown in Eq. (1) of our origin paper and the derivation between lines 127-128, our estimation of $\\phi_2$ depends only on the ratio $r(Y\\mid x)$ of conditional distributions rather than the marginal covariate distributions. This means that while the composition of $L_t$ may be biased in terms of $x$, this bias does not compromise the validity of our density ratio estimates for the purpose of loss reweighting, since the key quantity $r(y\\mid x)$ can still be accurately estimated from the available data.\n\n\n**W4:** Thank you for raising this important point about model dependency in our auxiliary uncertainty estimation. \n- We acknowledge that our method is model-dependent and that errors in the trained auxiliary model will propagate into the selection process. However, we emphasize that these estimation errors are correlated with the true loss, which provides meaningful signal for sample selection. \n- Our empirical results demonstrate that simple models like random forests can provide meaningful uncertainty estimates for effective sample selection. Crucially, when using appropriate auxiliary data, the selection performance with these imperfect estimates still outperforms the BMDAL baseline that ignores this term (by setting it to 1).\n\n**W5:** Thank you for raising these important concerns regarding computational complexity and scalability. \n- Our method introduces additional components (density ratio estimation and auxiliary loss estimation) compared to gradient-only approaches BMDAL. However the selection kernel itself remains unchanged; we only modify the features used for selection. The primary computational overhead stems from processing auxiliary data. For efficiency, lightweight machine learning methods (e.g., random forests) can be adopted for density ratio and loss estimation without compromising performance.\n- To illustrate scalability, we conducted experiments across 5 real-world datasets with varying auxiliary dataset sizes (100, 500, 1000, 10,000 samples) on a server with dual Intel Xeon Gold 6330 CPUs (112 threads total) and 125GB RAM. \n- The results are shown in the table below. Even with only 100 auxiliary data, the maxdet selection strategy, which performs best among the compared approaches, still achieves meaningful performance improvements. The gain grows with larger auxiliary datasets. Crucially, for auxiliary datasets below 1,000 samples, the additional time and memory overhead remains negligible. \n- In practical applications, we can choose the auxiliary data size based on their specific needs for either better performance (larger auxiliary sets) or faster computation (smaller auxiliary sets).\n\nWe'll add these discussions in the revised version.\n\n|||lcmd|maxdiag|maxdet|fw|maxdist|bait|kmeanspp|\n|-|-|-|-|-|-|-|-|-|\n|**AvgUsg(M)**|BMDAL|571.40|502.20|553.80|544.00|576.00|596.60|562.20|\n||ours(100)|575.20|501.40|561.40|570.60|570.00|599.00|511.00|\n||ours(500)|593.40|522.00|578.60|567.00|587.00|609.00|520.20|\n||ours(1000)|595.00|529.00|584.80|578.00|585.80|610.80|539.20|\n||ours(10000)|766.00|713.40|750.00|752.40|770.60|769.20|765.40|\n|**AvgTime(s)**|BMDAL|1.39|0.06|0.32|0.23|1.19|1.10|1.28|\n||ours(100)|2.35|0.91|1.23|1.12|2.10|2.02|2.41|\n||ours(500)|3.08|1.62|1.97|1.83|2.80|2.72|2.81|\n||ours(1000)|3.57|2.27|2.62|2.46|3.39|3.32|3.17|\n||ours(10000)|15.04|14.49|15.23|14.74|15.95|15.44|15.89|\n|**AvgImpro(%)**|ours(100)|-11.47|14.91|**5.83**|2.28|-9.17|5.86|-0.66|\n||ours(500)|-6.17|20.31|**10.69**|4.66|-0.40|8.95|2.60|\n||ours(1000)|-5.65|17.99|**10.47**|5.03|1.13|7.71|3.86|\n||ours(10000)|-6.08|22.56|**12.02**|5.20|1.91|8.76|4.47|\n\n**W6:** Thank you for your valuable feedback regarding the readability of Section 2.3. We apologize for the current mathematical density in this section and fully agree that additional intuitive explanations would enhance understanding. In our revision we will add more intuitive struture about the NTK-based decomposition.\n\n**Q5:** Our approach can be extended to classification tasks by using appropriate loss functions such as cross-entropy. However, we note an important technical distinction: since class labels are discrete rather than continuous, the joint density ratio estimation would decompose into estimating separate covariate density ratios within each class-specific subset of the auxiliary data.\n\n*We sincerely appreciate your thoughtful and constructive feedback. Many of the points you raised have helped us identify important directions for future research. We are more than happy to answer any further questions during the rebuttal period.*"}, {"review_id": "2MaiwGLEbj", "reviewer": "Reviewer_XtfN", "summary": "This paper focuses on deep batch active learning for regression tasks where obtaining labels is typically costly, and the objective is to identify a small set of informative unlabeled instances to label, thereby improving the predictive performance of deep learning models. The active selection process typically relies on measures such as model sensitivity and predictive uncertainty. While model sensitivity can be estimated using only unlabeled data, obtaining reliable uncertainty estimates can be markedly challenged by the scarcity of available labeled data. To address this, the paper proposes leveraging auxiliary data, typically drawn from a different distribution than the target task, to guide the active selection process. Although such auxiliary data cannot be directly used to train deep learning models for the main task of interest, the paper shows that they can be used to efficiently estimate the predictive uncertainty to guide the AL process. The proposed method approximates the model's loss by first estimating the density ratio between the auxiliary and main/target data distributions, then computing the model’s loss on the auxiliary data, and finally applying a density ratio weighting scheme. Numerical tests on both synthetic and real-world datasets demonstrate that incorporating auxiliary data in this manner leads to improved performance over the gradient kernel-based BMDAL method, which does not use auxiliary data. The proposed approach is further supported by theoretical analysis via the notion of neural tangent kernels.", "questions": "Please refer to weaknesses W1,W2 above. In addition, some questions are listed below:\n\nQ1. What are the properties of the auxiliary dataset relative to the target dataset of the main task, so that the proposed method can be effective?\n\nQ2. What is the role of the size of the auxiliary dataset in the performance of the proposed approach?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "STRENGTHS \n\n(S1) The idea of judiciously leveraging auxiliary data to obtain reliable estimates of the model predictive uncertainty that can readily guide the active selection process seems interesting, intuitive, and effective. \n\n(S2) The background, motivation and key concepts of the proposed method are well presented in general. \n\n(S3) The proposed approach is supported by a well-motivated theoretical analysis. \n\n(S4) The literature review is comprehensive. \n\n\nWEAKNESSES\n\n(W1) The paper does not provide an analysis or discussion of the computational complexity of the proposed method, and it does not report the running time of the proposed method compared to the baselines that do not process auxiliary data in the numerical evaluation. Since leveraging and processing auxiliary data using the density ratio weighting approach introduces additional computational overhead, in my opinion it is important to discuss this additional complexity and show the running time demonstrating the trade-off between prediction performance and running time. \n\n(W2) The proposed method is not compared against effective model predictive uncertainty estimation approaches that can be used for AL; see e.g. [1], [2]. Such an additional test/comparison both in terms of prediction performance and running time, in my opinion is important to further demonstrate the benefits of the proposed approach.  \n\n[1] Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.\n\n[2] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning (pp. 1050-1059). PMLR.", "was_revised": true, "final_justification": "The authors have adequately addressed my comments and concerns. Accordingly, I will raise my rating and recommend acceptance of the paper.", "rebuttal": "Thank you for your positive assessment of our intuitive approach to leveraging auxiliary data for uncertainty estimation, the clear presentation of motivations and key concepts, the well-founded theoretical analysis, and the comprehensive literature review. Below we address all comments and provide additional results.\n\n**W1:** We sincerely appreciate your valuable feedback regarding the computational complexity analysis of our method. \n- While our approach requires processing auxiliary data through density ratio weighting, we balance computational efficiency through two design choices: \n    - using lightweight machine learning methods (e.g., random forests); \n    - controlling the size of auxiliary datasets. \n- The core selection mechanism preserves the same complexity as gradient-only methods. The additional overhead comes from the density ratio estimation and auxiliary loss computation.\n- Our experiments shown in table below demonstrate that both the required resource and the performance improvement scale with the auxiliary data size. We conducted experiments across 5 real-world datasets with varying auxiliary dataset sizes (100, 500, 1000, 10,000 samples) on a server with dual Intel Xeon Gold 6330 CPUs (112 threads total) and 125GB RAM. \n- Even with only 100 auxiliary data, the maxdet selection strategy, which performs best among the compared approaches, still achieves meaningful performance improvements. For auxiliary datasets below 1,000 samples, the additional time and memory overhead remain negligible. \n- This allows us to flexibly balance performance and computational cost by adjusting the auxiliary data size according to their specific requirements. The consistent improvements over gradient-only baselines justify the additional computation, particularly since active learning typically operates in label-efficient regimes where query costs dominate computational costs.\n\nWe'll add these discussions in the revised version.\n\n|||lcmd|maxdiag|maxdet|fw|maxdist|bait|kmeanspp|\n|-|-|-|-|-|-|-|-|-|\n|**AvgUsg(M)**|BMDAL|571.40|502.20|553.80|544.00|576.00|596.60|562.20|\n||ours(100)|575.20|501.40|561.40|570.60|570.00|599.00|511.00|\n||ours(500)|593.40|522.00|578.60|567.00|587.00|609.00|520.20|\n||ours(1000)|595.00|529.00|584.80|578.00|585.80|610.80|539.20|\n||ours(10000)|766.00|713.40|750.00|752.40|770.60|769.20|765.40|\n|**AvgTime(s)**|BMDAL|1.39|0.06|0.32|0.23|1.19|1.10|1.28|\n||ours(100)|2.35|0.91|1.23|1.12|2.10|2.02|2.41|\n||ours(500)|3.08|1.62|1.97|1.83|2.80|2.72|2.81|\n||ours(1000)|3.57|2.27|2.62|2.46|3.39|3.32|3.17|\n||ours(10000)|15.04|14.49|15.23|14.74|15.95|15.44|15.89|\n|**AvgImpro(%)**|ours(100)|-11.47|14.91|**5.83**|2.28|-9.17|5.86|-0.66|\n||ours(500)|-6.17|20.31|**10.69**|4.66|-0.40|8.95|2.60|\n||ours(1000)|-5.65|17.99|**10.47**|5.03|1.13|7.71|3.86|\n||ours(10000)|-6.08|22.56|**12.02**|5.20|1.91|8.76|4.47|\n\n**W2:** We sincerely appreciate the reviewer's suggestion to compare with uncertainty-based active learning methods. \n- As shown in the table below, both deep ensemble and dropout methods underperform random sampling in our experiments, while AGBAL demonstrates consistent improvements (in Table 1 of our origin manuscript). \n- This observation aligns with findings in [1], which show that pure uncertainty batch sampling without diversity considerations can perform worse than random selection. The limitations of batch uncertainty-based approaches stem from several factors: \n    1. when selecting batches based solely on uncertainty, the chosen samples may cluster in localized high-uncertainty regions while neglecting other informative areas of the input space [1];\n    2. the fundamental assumption that high uncertainty necessarily correlates with prediction errors does not always hold in practice, particularly when model uncertainty estimates are miscalibrated [2];\n    3. the quality of uncertainty estimates is highly sensitive to architectural choices and hyperparameters (e.g., dropout rates, ensemble size) [2,3], requiring careful tuning that may not generalize across different datasets. \n- In contrast, our gradient-based approach directly optimizes the risk minimization objective, making it more robust to dataset characteristics.\n\n|Method|BIO|BIKE|DIAMOND|CT|STOCK|AvgTime(s)|AvgUsg(M)|\n|-|-|-|-|-|-|-|-|\n|random|0.451|0.459|21.009|0.380|0.391|0.00|451.20|\n|dropout|0.471|0.637|23.252|0.492|0.436|0.02|470.40|\n|ensemble|0.507|0.794|23.244|0.582|0.483|7.71|472.20|\n\n**Q1:** We sincerely appreciate this insightful question regarding the key properties of effective auxiliary datasets for our method. \n- Currently, the selection of suitable auxiliary data relies on prior knowledge, and our experimental setup in Appendix A.6.1 provides concrete examples of how we partition target and auxiliary datasets.\n- Through empirical observations, we find that higher-quality auxiliary datasets exhibit smaller variance in the estimated density ratios $\\hat{r}(x',y')$. This can be understood through the lens of distributional shift: when target and auxiliary distributions are identical (0\\% shift), $\\hat{r}(x',y')=1$ with zero variance. As the distributional discrepancy increases, the density ratios deviate more substantially from 1, leading to greater variance. However, this is only an empirical observation, and developing more rigorous quantitative measures for assessing auxiliary data utility is an important direction for future work.\n\n**Q2:** We sincerely thank the reviewer for highlighting this important question. \n- The size of the auxiliary dataset plays a crucial role in balancing performance and computing efficiency in our approach. The results of the experiment in the W1 table demonstrate that a larger auxiliary data set leads to both better performance and more resource consumption. \n- Notably, even minimal auxiliary dataset (100 samples) improves performance under our maxdet selection strategy, which performs best among the compared approaches. The additional time/memory incurred by auxiliary datasets below 1,000 samples is negligible.\n\n*We sincerely appreciate your thoughtful and constructive feedback. The issues you raised have brought to light important questions. We are more than happy to answer any further questions during the rebuttal period.*\n\n**Reference**\n\n[1] Settles, Burr. \"Active learning literature survey.\" (2009).\n\n[2] Tsymbalov, Evgenii, Maxim Panov, and Alexander Shapeev. \"Dropout-based active learning for regression.\" International conference on analysis of images, social networks and texts. Cham: Springer International Publishing, 2018.\n\n[3] Beluch, William H., et al. \"The power of ensembles for active learning in image classification.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018."}, {"review_id": "6nLBoGwURc", "reviewer": "Reviewer_6X3n", "summary": "The paper addresses a long standing problem that in order to know to the gradient for gradients which are required for active learning one usually needs to know the labels if not relaying on strong model assumptions. The paper attempts to solve this egg and chicken problem using similarly, but different, auxiliary data, for which the labels are known. Trough that, the problem is transferred from knowing $p(x,y)$ to knowing $r(x,y)\\coloneqq \\frac{p(x,y)}{q(x,y)}$.", "questions": "1. Why we need to assume $\\hat{r}$ is independent of $X_i',Y_i' \\sim Q$ (line 153) and why is it reasonable  to assume so.\n2. Line 90: (a) Is $\\mathcal A$ a function that takes in its second argument an **infinite** training set from $(\\mathcal{X}\\times\\mathcal{Y})^\\mathbb{N}$; (b) why does $\\mathcal A$ needs the initial parameter in its first argument, how is it been used in the rest of the paper?\n3. Why do we need to expect that the estimation.\n4. Line 162: why do we expect $\\rho$ to be small?\n5. What is the  limit of $\\mathbb{E}[\\hat{\\phi}_2(\\hat{\\theta},x_0)]$ ?\n6. Table 1: why is lower AUC better (usually larger is better)?", "limitations": "Yes, in the checklist.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. The problem is important and the mathematics looks sound (I did not check proofs).\n2. The core idea is original interesting.\n3. Experimental results are encouraging.\n\n**Weaknesses**\n1. It is not clear for me what is the meaning of the theoretical result, what it guarantees and in which practical conditions.  \n2. The theoretical result is only applicable for in the infinite width regime. \n3. The paper only deals with the square loss.\n \n(Points 2 and 3 are pointed out as  limitations by the authors in the checklist)", "was_revised": true, "final_justification": "Comment:\nI decided to raise my score.\n\nMy main concerns were:\n\n(1) Although the paper presents itself as proposing an active learning method for deep networks, it is essentially focused on active learning for kernel methods.\n\n(2) Some assumptions were unclear.\n\n(3) General clarity.\n\nAlthough the response only partially addressed my concerns, I eventually raised the score due to the originality and the interesting analysis.", "rebuttal": "We are grateful for your valuable assessment of our work's significance and mathematical soundness, along with your appreciation of the core idea's originality and the encouraging experimental results. Below we provide detailed responses to all comments to further strengthen these aspects.\n\n**W1:** Thank you for your thoughtful question regarding the theoretical guarantees of our method. \n- Our theoretical result builds upon NTK theory to establish that in infinitely wide networks, the estimation error $\\phi_2^{(2)}$ at each point $x$ stems from the variance $\\sigma^2(x)$ of NTK kernel regression on the target sample set. The key insight is that when we have two low-correlation estimates $\\phi_2^{(2)}$ and $\\widetilde{\\phi}_2^{(2)}$ derived from this variance, the variance of their difference becomes approximately proportional to $\\sigma^2(x)$. This allows us to use the difference between two estimates as a proxy for estimation error.\n- The theoretical guarantee holds under practical conditions where (i) the auxiliary data provides reasonably accurate estimates (ensured by prior knowledge about information relevance), and (ii) the auxiliary data is either independent or exhibits low correlation with the target data. These conditions are satisfied by our definition of auxiliary data.\n\n**W2:** Thank you for your insightful comment regarding the applicability of our theoretical results. While our current analysis is presented in the infinite-width regime, we note that theoretical treatments of finite-width neural tangent kernels already exist in prior work [1], which contains extensive technical derivations. Our analysis could be extended to the finite-width case following [1], but this would introduce substantial additional complexity. Since our primary contribution focuses on the active learning framework rather than NTK theory itself, we have chosen to present the cleaner infinite-width case. We will consider expanding this theoretical aspect in future work.\n\n**W3:** We sincerely appreciate the reviewer's observation regarding the loss function used in our analysis. \n- While our theoretical development employs the $L_2$ loss, we emphasize that our core methodology is fundamentally applicable to any differentiable loss function.\n- The $L_2$ loss provides superior mathematical convenience due to its smooth, everywhere-differentiable nature and its analytical tractability. \n- In contrast, non-smooth losses like $L_1$ (whose gradient is discontinuous) introduce significant analytical challenges that would obscure our primary contributions. However, the proposed active learning framework itself remains valid for general loss functions in practice.\n\n**Q1:** We are grateful for the reviewer's careful attention to this technical detail. \n- The equality in lines 154-155 requires $\\hat{r}$ to be independent of $X_{i}^{\\prime},Y_{i}^{\\prime}\\sim Q$ to ensure the objective of argmin can be expressed as a sum of independent terms. \n- The assumption is practically reasonable because we assume the availability of abundant auxiliary data. In practice, we can split the auxiliary data into two parts: one subset for estimating the density ratio $\\hat{r}$ and another independent subset for loss estimation. This data-splitting strategy satisfies the independence requirement.\n\n**Q2:** Thanks for raising this detail.\n- The training set $L_t$ contains a finite number of samples $\\mathbb{N}$, but algorithm $\\mathcal{A}$ can incorporate arbitrarily many samples. The current expression was chosen for simplicity. Also, to maintain notational consistency, we require an initialization parameter for the model. \n- The initial parameters can be either 0 or randomly generated from a given distribution. This initialization has two purposes: (1) providing a starting point for the NTK parameters, and (2) following standard machine learning practice where parameter initialization is required for training procedures.\n\n**Q3:** We are not certain which specific expectation the reviewer is referring to. However, if the question concerns the expectation in Equation (1), we note that this operation is formally defined in Line 82 of our paper. The expectation appears because we aim to optimize the model parameters with respect to the true risk $R$ over distribution $P$, whose gradient direction requires this expectation. In practice, since we cannot compute this expectation directly, we resort to empirical estimation. This approach aligns with standard practice in statistical learning theory, where population objectives are approximated through finite samples.\n\n**Q4:** The assumption that $\\rho$ should be small stems from our core methodological design where we use the difference between two estimates as a proxy for estimation error (as explained in W1). For this approach to be valid, the two estimates must exhibit sufficiently low correlation, hence we require $\\rho$ to be small. When $\\rho=1$, the estimates become identical, making their difference vanish . This degenerate case provides no meaningful information.\n\n**Q5:** $\\hat{\\phi}_2(\\hat{\\theta},x_0)$ is the estimated loss gradient using auxiliary data. As long as the estimator $\\hat{\\theta}$ converges to the optimal parameter and the density ratio estimator converges to the true density ratio, the limit of $\\mathbb{E}[\\hat{\\phi}_2(\\hat{\\theta},x_0)]$  converges to $0$.\n\n**Q6:** In Table 1, Area Under the Curve (AUC) metric is employed to evaluate the overall convergence efficiency of different active learning methods. As specified in line 201 of our paper, the AUC metric is calculated by $\\text{AUC}=\\sum_{i=1}^T(\\xi_{i-1}+\\xi_i)/2$, where $\\xi_0,\\ldots,\\xi_T$ represents the sequence of average MSE values obtained by each method. In this context, a lower AUC value indicates superior performance, corresponding to smaller prediction errors and more effective sample selection. Conversely, when evaluating based on accuracy curves (where higher values are desirable), the interpretation of AUC values would be inverted.\n\n*We sincerely appreciate your thoughtful and constructive feedback. Your questions have opened up a broader space for discussion. We are more than happy to answer any further questions during the rebuttal period.*\n\n**Reference**\n\n[1] Novak, Roman, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. \"Fast finite width neural tangent kernel.\" International Conference on Machine Learning. PMLR, 2022."}, {"review_id": "cj53sPeSAd", "reviewer": "Reviewer_AvdT", "summary": "This paper proposes an active learning approach that leverages auxiliary data (data that is distributionally shifted but relevant) to help with predicting sample uncertainty. The approach uses this data to estimate density ratio between auxiliary and target data distributions, compute model's loss on auxiliary data, and then apply density ratio weighting to obtain approximation of true loss. The paper then verifies their approach through theoretical analysis based on Neural Tangent Kernel theory. Results show strong performance of their approch.", "questions": "1. How does your method compare against prior works that also use auxiliary data in active learning?\n2. How does your method compare against baselines in terms of compute resources and time?\n3. Are there any sort of analysis or arguments that can be extended to understand how the proposed approach may translate to real-world applications?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Intuitive yet innovative active learning approach to using auxiliary data \n* Well-supported analysis of their proposed approach \n* Writing and motivation is clear throughout the paper\n* Empirical results show consistent improvement with their proposed approach and appendix has very thorough experimentation ablations\n\nWeaknesses:\n* Lack of mentions/comparison against prior work that also make use of auxiliary data for active learning (eg \"Active Transfer Learning under Model Shift\" [2014])\n* As mentioned in paper's limitations, unsure how NTK analysis translates to real-world applications as it assumes infinite-width network approximation\n* Lack of clarity on experiments compute resource. While the paper runs many experiments, it is unclear how costly their proposed approach is in terms of compute resources and time.", "was_revised": true, "final_justification": "I keep my same score of Accept, as I believe this paper brings novel contributions to active learning field.", "rebuttal": "We sincerely appreciate your positive feedback highlighting our innovative yet intuitive approach to leveraging auxiliary data in active learning, along with your recognition of our rigorous theoretical analysis, clear presentation, and thorough experimental validation including detailed ablation studies. Below we address all comments to further strengthen these contributions.\n\n**W1\\&Q1:** We sincerely appreciate the reviewer's insightful comment regarding the comparison with transfer active learning methods. Our initial submission lacked explicit discussion of this important aspect, and we thank the reviewer for bringing this to our attention.\n- Our work differs from transfer learning approaches like [1] in the definition of auxiliary data:\n    - transfer learning and domain adaptation assume covariate shift with different $X$ distribution but consistent distribution of $P(Y\\mid X)$ between source and target domains [1];\n    - our setting considers imperfect auxiliary data where both $P(X)$ and $P(Y\\mid X)$ may differ. \n- This distinction is crucial because standard transfer methods could suffer from negative transfer under such joint distribution shift. To validate this empirically, we compared directly combining target and auxiliary data versus our approach, with results of our approach demonstrating superior performance (reported in Table 3 of our original submission).\n- We agree that exploring connections to transfer active learning methods would be valuable, and we will include this discussion in the revised version, along with potential extensions combining our method with transfer techniques when the $P(Y\\mid X)$ consistency holds. This clarification will better position our contribution relative to prior work.\n\n**W2/Q3:** We sincerely appreciate the reviewer's thoughtful question regarding the practical implications of our NTK-based analysis. \n- While our current theoretical analysis employs the infinite-width neural network approximation for tractability, we note that prior work [2] has established rigorous connections between finite-width networks and their neural tangent kernel behavior, suggesting our theoretical insights could be extended to practical settings. However, such extension would require substantial additional technical details.\n- The infinite-width analysis serves to provide theoretical justification for our approach, while empirical results demonstrate the method's performance with finite-width networks across multiple real-world datasets. We agree that further investigation of finite-width cases would be valuable and will consider this direction in future work.\n\n**W3/Q2:** We greatly appreciate the reviewer's important question regarding the computational efficiency of our method. \n- Our approach introduces computational overhead through density ratio estimation and auxiliary loss computation, but we mitigate this through careful design choices: \n    - employing efficient machine learning methods for density ratio and loss estimation; \n    - allowing flexible control of auxiliary dataset sizes. \n- The selection period maintains the same computational complexity as BMDAL.\n- Empirical results demonstrate that our method achieves meaningful performance improvements even with modest auxiliary data sizes (100, 500, 1000 samples), where the additional computational overhead remains negligible. With only 100 auxiliary data, the maxdet selection strategy, which performs best among the compared approaches, still outperforms BMDAL. \n- All experiments were conducted on a standard server (dual Intel Xeon Gold 6330 CPUs, 125GB RAM), with detailed time/memory comparisons provided in the table below. The performance gains consistently justify the modest computational cost, particularly in active learning settings where annotation costs typically dominate computation costs. \n\nWe will include the computational analysis in the revised version.\n\n|||lcmd|maxdiag|maxdet|fw|maxdist|bait|kmeanspp|\n|-|-|-|-|-|-|-|-|-|\n|**AvgUsg(M)**|BMDAL|571.40|502.20|553.80|544.00|576.00|596.60|562.20|\n||ours(100)|575.20|501.40|561.40|570.60|570.00|599.00|511.00|\n||ours(500)|593.40|522.00|578.60|567.00|587.00|609.00|520.20|\n||ours(1000)|595.00|529.00|584.80|578.00|585.80|610.80|539.20|\n||ours(10000)|766.00|713.40|750.00|752.40|770.60|769.20|765.40|\n|**AvgTime(s)**|BMDAL|1.39|0.06|0.32|0.23|1.19|1.10|1.28|\n||ours(100)|2.35|0.91|1.23|1.12|2.10|2.02|2.41|\n||ours(500)|3.08|1.62|1.97|1.83|2.80|2.72|2.81|\n||ours(1000)|3.57|2.27|2.62|2.46|3.39|3.32|3.17|\n||ours(10000)|15.04|14.49|15.23|14.74|15.95|15.44|15.89|\n|**AvgImpro(%)**|ours(100)|-11.47|14.91|**5.83**|2.28|-9.17|5.86|-0.66|\n||ours(500)|-6.17|20.31|**10.69**|4.66|-0.40|8.95|2.60|\n||ours(1000)|-5.65|17.99|**10.47**|5.03|1.13|7.71|3.86|\n||ours(10000)|-6.08|22.56|**12.02**|5.20|1.91|8.76|4.47|\n\n*We sincerely appreciate your thoughtful and constructive feedback. Your observations have helped pinpoint several significant areas for future study. We are more than happy to answer any further questions during the rebuttal period.*\n\n**Reference**\n\n[1] Wang, Xuezhi, Tzu-Kuo Huang, and Jeff Schneider. \"Active transfer learning under model shift.\" International Conference on Machine Learning. PMLR, 2014.\n\n[2] Novak, Roman, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. \"Fast finite width neural tangent kernel.\" International Conference on Machine Learning. PMLR, 2022."}]

Author rebuttal:

Meta-review: This paper introduces AGBAL, an active learning framework for regression that leverages auxiliary data to improve uncertainty estimation. By applying density ratio weighting to recalibrate auxiliary losses, AGBAL enables more reliable sample selection even under distribution shifts. Theoretical analysis supports the approach, and experiments on synthetic and real-world datasets demonstrate consistent improvements over methods that do not use auxiliary data.

Reviewers recognized the novelty of the approach and its effectiveness, validated through both theoretical analysis and experimental results. The authors’ rebuttal clarified several concerns raised in the original reviews. As a result, all reviewers supported acceptance. The authors are encouraged to incorporate the additional discussions and results into the final version.
Decision: Accept (poster)