Direct Acquisition Optimization for Low-Budget Active Learning

## Abstract

Abstract Active Learning (AL) has gained prominence in integrating data-intensive machine learning (ML)
models into domains with limited labeled data.
However, its effectiveness diminishes significantly when the labeling budget is low.
In this paper, we first empirically observe the performance degradation of existing AL
algorithms in the low-budget settings, and then introduce Direct Acquisition
Optimization ( DAO ), a novel AL algorithm that optimizes sample selections based on
expected true loss reduction.
Specifically, DAO utilizes influence functions to update model parameters and incorporates
an additional acquisition strategy to mitigate bias in loss estimation.
This approach facilitates a more accurate estimation of the overall error reduction, without
extensive computations or reliance on labeled data.
Experiments demonstrate DAO ’s effectiveness in low budget settings, outperforming
state-of-the-arts approaches across seven benchmarks.

## 1 Introduction

Active learning (AL) explores how adaptive data collection can reduce the amount of
data needed by machine learning (ML) models (Settles, ; Ren et al., ).
It is particularly useful when labeled data is scarce or expensive to obtain, which
significantly limits the adaptability of modern deep learning (DL) models due to their
data-hungry nature (van der Ploeg et al., ).
In these cases, AL algorithms selectively choose the most beneficial data points
for labeling, thereby maximizing the effectiveness of the training process even if the data
is limited in number.
In fact, AL has been broadly applied in many fields (Adadi, ), such as
medical image analysis (Budd et al., ), astronomy (Škoda et al., ), and
physics (Ding et al., ), where unlabeled samples are plentiful but the process of labeling
through human expert annotations or experiments is highly cost-intensive.
In these contexts, judiciously selecting samples for labeling can significantly lower
the expenses involved in compiling the datasets (Ren et al., ).

Many active learning algorithms have emerged over the past decades, with early
seminal contributions from (Lewis, ; Tong & Koller, ; Roy & McCallum, ), and a shift that focuses more on deep active learning - a branch of AL that targets more
towards DL models in more recent years (Huang, ).
Depending on the optimization objective, AL algorithms can be classified into two categories. The first category includes heuristic objectives that are not exactly the same as the evaluation
metric, i.e. error reduction.
Examples in this category are diversity (Sener & Savarese, ),
uncertainty (Gal et al., ), and hybrids of both (Ash et al., ).
Second category includes criteria that is exactly the same as the evaluation metric,
where notable approaches include expected error reduction (EER) (Roy & McCallum, ) and
its modern follow-up works (Killamsetty et al., ; Mussmann et al., ).

Despite the popularity of the first type of AL algorithms, we show in
§2 that these methods often suffer heavily in low-budget settings,
where the total (accumulative) sampling quota is less than 1% of the number of unlabeled data points,
making them less suitable for the extreme data scarcity scenarios.
In terms of the methods from the second category, their higher running time and
reliance on the availability of a validation or hold-out set remain
significant limitations, constraining their applicability in many data-scarcity
scenarios as well.
For example, EER (Roy & McCallum, ) re-trains the classifier for each candidate with all
its possible labels, where in each time also evaluates the updated model on all the
unlabeled data, making its runtime intractable especially for deep neural networks.
And GLISTER (Killamsetty et al., ), despite being much more computationally
efficient, requires a labeled, hold-out set for its sample selection process,
formulated as a mixed discrete-continuous bi-level optimization problem, to be optimized
properly.

While these constraints might not be a huge limitation a few years ago, it poses a more
important challenge currently as we are adopting deep learning models to more areas,
where labeled data may be extremely expensive to acquire.
More importantly, it is also worth noticing that under these scenarios, the highly limited
labeled data should have been better utilized for training than being reserved for AL
algorithms.

Above limitations highlight a critical gap between the capabilities of current AL
methodologies and the urgent demands from real-world applications, underscoring the need for
developing novel AL strategies that can operate both relatively efficient while presenting
little to none reliance on the labeled set.
To this end, we introduce Direct Acquisition Optimization (DAO), a novel
AL algorithm that selects new samples for labeling by efficiently estimating the expected
loss reduction.
Compared to EER and GLISTER, DAO solves the pain points of prohibitive running time and
the reliance on a separate labeled set through utilizing influence
function (Ling, ) in model parameters updates, and a more accurate,
efficient unbiased estimator of loss reduction through importance-weighted sampling.

To summarize, the contributions of this paper are: (1) an empirical analysis of
existing AL algorithms under low budget settings;
(2) a novel AL algorithm, Direct Acquisition Optimization (DAO), which optimizes sample
selections based on expected error reduction while operating efficiently through influence
function-based model parameters approximation and true overall reduced error estimation;
and (3) thorough experiments demonstrating DAO’s superior performance in the
low-budget settings, out-performing current popular AL methods across seven benchmarks.

## 2 Low-Budget Active Learning: A Motivating Case Study

In this section, we provide an empirical analysis to demonstrate that commonly used
heuristic-based AL algorithms do not work well under very low-budget settings.
Specifically, we analyze (1) uncertainty sampling methods including least
confidence (Lewis, ), minimum margin (Scheffer et al., ),
maximum entropy (Settles, ),
and Bayesian Active Learning by Disagreement (BALD) (Gal et al., ); (2) diversity sampling
methods such as Core-Set (Sener & Savarese, ) and Variational Adversarial Active Learning
(VAAL) (Sinha et al., ); and (3) hybrid method such as Batch Active learning by
Diverse Gradient Embeddings (BADGE) (Ash et al., ).

We test the above methods on the CIFAR10 (Krizhevsky et al., ) dataset starting with an
initial labeled set with size $|\mathcal{L}_{\text{init}}|=10$, and conducted 50 acquisition
rounds where after each round $B=10$ new samples are selected and labeled.
We use ResNet-18 (He et al., ) as our training model across all methods.
And we repeated the acquisitions five times with different random seeds.
The results are visualized in Fig. 1, where we plot the relative
performance between each method and random sampling acquisition through a diverging color map.

Figure: Figure 1: Existing methods fail to outperform random sampling with small budgets. This figure shows the relative performance between multiple methods and random acquisition. Within each subplot, $x$ axis represents the accumulative acquisition size, while $y$ axis indicates runs initiated with different random seeds. White color indicates on-par performance with random, blue indicates worse, and red indicates better.
Refer to caption: x1.png

Aligning with the general perceptions that
low-budget (Mittal et al., ; Hacohen et al., ) and
cold-start (Zhu et al., ; Chandra et al., ) AL tasks are especially challenging, we
empirically observe that almost all popular AL algorithms fail to outperform the naive random
sampling when acquisition quota is less than 1% (500 out of 50,000 in the case of CIFAR10) of
the unlabeled size.
More specifically, when the quota is less than 0.2% (less than 100 data points for CIFAR-10), all
methods fail to reliably outperform random sampling (as the beginning of each heatmap in
Fig. 1 are almost all blue), which greatly motivates the development of
DAO.
We also include the more conventional line plot of the empirical analysis which may provide more
detailed information of each run in Appendix A.1.

## 3 Methodology

Different from the heuristics-based AL algorithms that optimize criteria such as
diversity or uncertainty, DAO is built upon the EER formulation with the selection
objective being the largest reduced error evaluated on the entire unlabeled set.
More specifically, DAO majorly improves upon two aspects:
(1) instead of re-training the classifier, we employ influence
function (Cook & Weisberg, ), a concept with rich history in statistical learning, to
formulate the new candidate sample as a small perturbation to the existing labeled
set, so that the model parameters can be estimated without re-training;
and (2) instead of reserving a separate, relatively large labeled set for
validation (Killamsetty et al., ), we sample a very small subset directly from the
unlabeled set and estimate the loss reduction through bias correction.

Essentially, when considering each candidate from the unlabeled set, we optimize the EER
framework on two of its core components, which are model parameter update and true loss
estimation.
Additionally, we upgrade EER, which only supports single sequential acquisition, to offer
DAO in both single and batch acquisition variants by incorporating stochastic samplings to
the sorted estimated loss reductions. We illustrate our algorithmic framework in
Fig. 2.
In the following parts of this section, we first introduce a more formal problem statement
in §3.1, and then dive into each specific component of DAO from §3.2 to §3.5.

### 3.1 Problem Statement

The optimal sequential active learning acquisition function
can be formulated as selecting a single $\mathbf{x}^{\text{train}}_{t}$ from the current unlabeled
set $\mathcal{U}_{t}$ at each round $t$ such that

$$ $\mathbf{x}^{\text{train}}_{t}=\operatorname*{arg\,min}_{\mathbf{x}_{i}\in \mathcal{U}_{t-1}}\mathbb{E}_{(y_{i}|f^{*},\mathbf{x}_{i})}\left[L_{\text{true }}(f_{t|\mathbf{x}_{i},y_{i}})\right]$ (1) $$

where $f^{*}$ represents an optimal oracle that maps from any unlabeled data
$\mathbf{x}_{i}\in\mathcal{U}_{t-1}$ to its ground-truth label $y_{i}$, and
$f_{t|\mathbf{x}_{i},y_{i}}$
is the model that has been trained on the union of the current labeled set
$\mathcal{L}_{t-1}$ and the current unlabeled candidate $\mathbf{x}_{i}\in\mathcal{U}_{t-1}$.
In addition,
$L_{\text{true}}(f_{t|\mathbf{x}_{i},y_{i}})=\frac{1}{|\mathcal{U}_{t-1,i}|}
\sum_{\mathbf{x}\in\mathcal{U}_{t-1,i}}\ell(\mathbf{x};f_{t|\mathbf{x}_{i},y_{
i}})$
represents the loss estimator that can predict the unbiased error of
$f_{t|\mathbf{x}_{i},y_{i}}$,
where $\ell$ denotes the loss of $f_{t|\mathbf{x}_{i},y_{i}}$ evaluated on $\mathbf{x}_{i}$ (against its true label $f^{*}(\mathbf{x}_{i})$).
It is numerically the same as if
$f_{t|\mathbf{x}_{i},y_{i}}$
has been tested on the entire unlabeled set
$\mathcal{U}_{t-1,i}$, where $\mathcal{U}_{t-1,i}=\mathcal{U}_{t-1}\setminus\{\mathbf{x}_{i}\}$.
Such formulation represents the optimal AL algorithm and aligns with any existing
sequential active learning algorithm —
of which the goal is to select the new data point that
can most significantly improve the current model performance (Roy & McCallum, ).

Unfortunately, Eq. (1) cannot be directly implemented in practice.
Because, first, we do not have access to the optimal oracle $f^{*}$ to reveal the label $y_{i}$
for each $\mathbf{x}_{i}\in\mathcal{U}_{t-1}$;
second, even if we had $f^{*}$ and therefore $y_{i}$, we cannot afford the cost of retraining model
$f_{t-1}$ on each $\mathcal{L}_{t-1}\cup\mathbf{x}_{i}$ to obtain the updated $f_{t|\mathbf{x}_{i},y_{i}}$;
and third, we do not have the unbiased true loss estimator $L_{\text{true}}$, which demands
evaluating $f_{t|\mathbf{x}_{i},y_{i}}$ on the entire $\mathcal{U}_{t-1,i}$.

Therefore, the goal of DAO is to solve the above challenges and efficiently and
accurately approximate Eq. (1) for the
sample selection strategy.
It is also worth noting that, when $\mathbf{x}^{\text{train}}_{t}$ represents a set of
newly acquired data points, the above formulation becomes eligible for batch active learning,
which is more suitable for deep neural networks (Huang, ).

### 3.2 Label Approximation via Surrogate

In this section, we address the first challenge when approximating Eq. (1).
As we do not know the true label or true label distribution $p(y|\mathbf{x},f^{*})$
of each unlabeled sample $\mathbf{x}$, the best we can do is provide an approximation for
$p(y|\mathbf{x})$.
To this end, we introduce the concept of a surrogate (Kossen et al., ),
which is a model parameterized by some potentially infinite set of parameters $\theta$.
Specifically, $p(y|\mathbf{x})$ can be approximated using the marginal distribution
$\pi(y|\mathbf{x})=\mathbb{E}_{\pi(\theta)}[\pi(y|\mathbf{x},\theta)]$
with some proposal distribution $\pi(\theta)$ over model parameters $\theta$.
In other words, we have:

$$ $p(y|\mathbf{x})\approx\int_{\theta}\pi(\theta)\pi(y|\mathbf{x},\theta)\mathop{ }\!\mathrm{d}\theta$ (2) $$

As the sample selection process continues, new labeled points should also be used to train and update
the surrogate model $\pi(\theta)$ for better approximation of the true outcomes.

Although ideally, a more capable surrogate is preferred for better ground truth
approximations, we acknowledge that the choice of surrogate model can be very sensitive to
the computational constraints.
Therefore, if running time is at center of the concerns during sample acquisitions, using
$f_{t}$ at step $t$
also as the surrogate could be an efficient alternative, as we don’t need to update a
second model, nor do we need to run forward pass on the both models.
However, this will come with the cost that $\pi_{t}$ never disagrees with $f_{t}$, which causes
performance degradation for the unbiased true loss estimation, which will be illustrated
with more details in §3.4.
Therefore, in short, we do not recommend replicating $f_{t}$ as surrogate in practice, unless the
computational constraint is substantial.

Figure: Figure 2: Schematic of the algorithmic framework of DAO.
Refer to caption: x2.png

### 3.3 Model Parameters Update without Re-training

At acquisition round $t$, suppose we have labeled set $\mathcal{L}_{t-1}$ and unlabeled
set $\mathcal{U}_{t-1}$ as the results from the previous round $t-1$, and new sample
$\mathbf{x}_{i}\in\mathcal{U}_{t-1}$ that is currently under consideration for
acquisition, the goal of this section is to estimate the parameters of model $f_{t|\mathbf{x}_{i},y_{i}}$
that could has been obtained after training $f_{t-1}$ on the combined dataset
$\{\mathcal{L}_{t-1}\cup\mathbf{x}_{i}\}$.
In other words, if we suppose the conventional full training converges to parameters
$\hat{\theta}_{\mathbf{x}_{i}}$, we have:

$$ $\hat{\theta}_{\mathbf{x}_{i}}=\arg\,\min_{\theta\in\Theta}\frac{1}{|\mathcal{L }_{t-1}|+1}\sum_{\mathbf{x}\in\{\mathcal{L}_{t-1}\cup\mathbf{x}_{i}\}}\ell( \mathbf{x};\theta)$ (3) $$

where recall that $\ell(\mathbf{x};\theta)$ denotes the loss of $\theta$ on $\mathbf{x}$.
The core of our approach is that, instead of re-training as showed in
Eq. (3), we can approximate the effect of adding a new sample as
upweighting the influence function by
$\frac{1}{|\mathcal{L}_{t-1}|+1}$ (Koh & Liang, ) and then directly estimate
the updated model parameters.

Following (Cook & Weisberg, ),
we have the influence function defined as:

$$ $\mathcal{I}_{\text{up},\text{params}}(\mathbf{x}_{i})\coloneqq\frac{d\hat{ \theta}_{\epsilon,\mathbf{x}_{i}}}{d\epsilon}\bigg{|}_{\epsilon=0}=-H_{\hat{ \theta}}^{-1}\nabla_{\theta}\ell(\mathbf{x}_{i};\hat{\theta})$ (4) $$

where $H_{\hat{\theta}}$ is the positive definite Hessian matrix (Koh & Liang, ).
Next, we can estimate the model parameters after adding this new sample $\mathbf{x}_{i}$, as:

$$ $\begin{split}\hat{\theta}_{\mathbf{x}_{i}}-\hat{\theta}&\approx\frac{1}{| \mathcal{L}_{t-1}|+1}\mathcal{I}_{\text{up},\text{params}}(\mathbf{x}_{i})\\ &=-\frac{1}{|\mathcal{L}_{t-1}|+1}H_{\hat{\theta}}^{-1}\nabla_{\theta}\ell( \mathbf{x}_{i};\hat{\theta})\end{split}$ (5) $$

where $\nabla_{\theta}\ell(\mathbf{x}_{i};\hat{\theta})$ could be
approximated as the expected gradient of sample $\mathbf{x}_{i}$: By a slight abuse of notation of the training loss function $\ell$, we denote

$$ $\nabla_{\theta}\ell(\mathbf{x}_{i};\hat{\theta})\approx\sum_{k=1}^{K}\nabla_{ \theta}\ell(\mathbf{x}_{i},\hat{y}_{k};\hat{\theta})\cdot\hat{p}_{k}$ (6) $$

In Eq. (6), $\hat{y}_{k}$ and $\hat{p}_{k}$ represent model’s
label prediction and likelihood (e.g. confidence) respectively while $K$ represents the
total number of classes in the ground truths.

In practice, the inverse of $H_{\hat{\theta}}$ cannot be computed due to its prohibitive
$O(np^{2}+p^{3})$ runtime (Liu et al., ), with $p$ being the number of model
parameters.
The computation unavoidably becomes especially intensive when $f$ is a deep neural network
model (Fu et al., ).
Luckily, we have two optimization methods, conjugate gradients (CG) (Martens et al., )
and stochastic estimation (Agarwal et al., ) at our disposal.

#### Conjugate gradients.

As mentioned earlier, by assumption
we have $H_{\hat{\theta}}\succ 0$ and $\nabla_{\theta}\ell(\mathbf{x}^{\prime};\hat{\theta})$ as a vector.
Therefore, we can calculate the inverse Hessian vector product (IHVP) through first
transforming the matrix inverse into an optimization problem, i.e.

$$ $H_{\hat{\theta}}^{-1}\nabla_{\theta}\ell(\mathbf{x}_{i};\hat{\theta})\equiv \arg\,\min_{t}\;t^{T}H_{\hat{\theta}}t-v^{T}t$ (7) $$

and then solving it with CG (Martens et al., ), which speeds up the runtime effectively
to $O(np)$.

#### Stochastic estimation.

Besides CG, we can also efficiently compute the IHVP using the stochastic estimation
algorithm developed by Agarwal et al. (Agarwal et al., ).
From Neumann series, we have $A^{-1}\approx\sum_{i=0}^{\infty}(I-A)^{i}$ for any matrix $A$.
Similarly, suppose we define the first $j$ terms in the Taylor expansion of
$H_{\hat{\theta}}^{-1}$ as

$$ $H_{\hat{\theta},j}^{-1}=\sum_{i=0}^{j}(I-H_{\hat{\theta}})^{i}=I+(I-H_{\hat{ \theta}})H_{\hat{\theta},j-1}^{-1}$ (8) $$

we have
$H_{\hat{\theta},j}^{-1}\rightarrow H_{\hat{\theta}}^{-1}$ as $j\rightarrow\infty$.
The core idea of the stochastic estimation is that the Hessian matrix $H_{\hat{\theta}}$
can be substituted with any unbiased estimation when computing $H_{\hat{\theta}}^{-1}$.
In practice, we sample $n_{\text{ihvp}}$ data points from the existing labeled set
$\mathcal{L}_{t-1}$ and use $\nabla_{\theta}^{2}\ell(\mathbf{x}_{i};\hat{\theta})$ as the
estimator of $H_{\hat{\theta}}$ (Liu et al., ).
Notice that since $n_{\text{ihvp}}$ is usually very small (in our experiments we used
$n_{\text{ihvp}}=8$), it does not create a constraint on the size of the current labeled set,
which does not interfere with the low-budget settings.

Finally, we can approximate the model parameters after the addition of $\mathbf{x}_{i}$ as

$$ $\hat{\theta}_{\mathbf{x}_{i}}=\hat{\theta}-\frac{1}{n+1}H_{\hat{\theta}}^{-1} \nabla_{\theta}\ell(\mathbf{x}_{i};\hat{\theta})$ (9) $$

which does not require any re-training.
And we will demonstrate in §5.1 that this parameter update strategy
provides much better approximations than the naive single backpropagation as seen
in the existing AL literature (Killamsetty et al., ).

### 3.4 Efficient Unbiased Loss Estimation

Referring back to Eq. (1), the last challenge that we need to address
is to gain access to the unbiased true loss estimator $L_{\text{true}}$.
In other words, we want to predict the true performance of $f_{t|\mathbf{x}_{i},y_{i}}$ on the
unlabeled set $\mathcal{U}_{t,i}$ without exhaustive testing.
Strictly, such evaluation cannot be drawn until $f_{t|\mathbf{x}_{i},y_{i}}$ is evaluated on the entire unlabeled
set $\mathcal{U}_{t,i}$.
However, this is infeasible in practice.

Such approximation is typically carried out in other
approaches (Killamsetty et al., ; Mussmann et al., ) by randomly sampling a
labeled validation set $\mathcal{V}$ at the beginning of the entire acquisition process,
which will later be used for evaluations in all the subsequent acquisition episodes.
Despite the simplicity as well as being i.i.d., which makes the estimated loss unbiased
by nature, this approximation method suffers from large variance as the size of
$\mathcal{V}$ is usually much smaller than $\mathcal{U}$, which unavoidably hurts the
acquisition performance.
It is also contradictory to the goal of AL in general, especially under the low-budget
settings, as discussed in §1.

Different from the existing works, we propose to sample a subset $\mathcal{C}$ from
current $\mathcal{L}_{t-1}$ in each acquisition round based on an alternative acquisition
function, and then correct the bias in the loss induced from this acquisition function.
In the meantime, we also want to keep the variance low, so that the final corrected loss
enjoys both low bias and low variance, which is more preferable than the zero bias but
high variance that the random i.i.d. sampling has.

Specifically, continuing with the notations from §3.1, let
$\mathcal{C}=\{\mathbf{x}_{t,1},\dots,\mathbf{x}_{t,m},\dots,\mathbf{x}_{t,n_{
\mathcal{C}}}\}$, where
$\mathcal{C}\subset\mathcal{U}_{t-1}$, be the subset containing
$n_{\mathcal{C}}$ samples selected
for this true loss estimation at each round $t$. Farquhar et al. () shows that if
$\mathbf{x}_{t,m}$ is sampled in proportion to the true loss of each
data point, the bias originated from this selection can be corrected through the Monte
Carlo estimator
$\hat{R}_{\text{LURE}}$(^1^11LURE stands for Levelled Unbiased Risk Estimator).
Following our notations, it takes the form:

$$ $\hat{R}_{\text{LURE}}=\frac{1}{n_{\mathcal{C}}}\sum_{m=1}^{n_{\mathcal{C}}}{v_ {m}}\ell\left(\mathbf{x}_{t,m};f\right)$ (10) $$

where recall that $\ell$ denotes the loss of $f$, and the importance weight $v_{m}$ is

$$ $v_{m}=1+\frac{|\mathcal{U}_{t-1}|-n_{\mathcal{C}}}{|\mathcal{U}_{t-1}|-m}\left (\frac{1}{(|\mathcal{U}_{t-1}|-m+1)q^{*}_{t}(m)}-1\right)$ (11) $$

with $q^{*}_{t}(m)$ being the acquisition distribution of index $m$ at round t.
Importantly, the variance can be significantly reduced if the acquisition
distribution $q^{*}_{t}(m)$ is proportion to the true loss of each data point.
Again, this is not feasible as we do not have access to the labels for $\mathcal{U}_{t-1}$.
However, following Kossen et al. (), we can approximate $q^{*}_{t}(m)$ with

$$ $q_{t}(m)=-\sum_{y}\pi(y|\mathbf{x}_{t,m})\log f(\mathbf{x}_{t,m})$ (12) $$

for classification tasks when the loss function is the cross-entropy loss, and where $\pi$ is
conveniently just our surrogate discussed in §3.2.
Referring back to the discussion we had on choosing a good surrogate $\pi$,
with $f(\mathbf{x})$ being designed to approximate $p(y|\mathbf{x})$
as well, the surrogate $\pi$ should ideally be different from $f$ so that more diversity is
introduced in the acquisitions.

To put all the components together, our loss correction process involves selecting samples
in $\mathcal{C}$ based on

$$ $\mathbf{x}_{t,m}\propto-\sum_{y}\pi_{t-1}(y|\mathbf{x})\log f_{t-1}(y|\mathbf{ x})$ (13) $$

where $\pi_{t-1}$ is the surrogate model $\pi$ at round $t-1$.
Finally, the corrected loss $s_{i}$ can be approximated using $\hat{R}_{\text{LURE}}$ as

$$ $s_{i}=\frac{1}{n_{\mathcal{C}}}\sum_{m=1}^{n_{\mathcal{C}}}\hat{v}_{m}\ell \left(\mathbf{x}_{t,m};f_{t}\right)$ (14) $$

where $\hat{v}_{m}$, which depends on the choice of $\mathbf{x}_{t,m}$, is
the approximated version of the original $v_{m}$ defined in Eq. (11).
Specifically, $\hat{v}_{m}$ takes the form

$$ $\hat{v}_{m}=1+\frac{|\mathcal{U}_{t-1}|-n_{\mathcal{C}}}{|\mathcal{U}_{t-1}|-m }\left(\frac{1}{(|\mathcal{U}_{t-1}|-m+1)q_{t}(m)}-1\right)$ (15) $$

where $q_{t}(m)$ is the acquisition function defined in Eq. (13).

### 3.5 Batch Acquisition via Stochastic Sampling

In §3.1, we briefly discussed that when
$\mathbf{x}^{\text{train}}_{t}$ represents a set of data points (instead of a single one), the
formulation in Eq. (1) essentially represents the batch active
learning scenario.
Suppose the acquisition budget per round is $k$, although selecting the top $k$ samples with the
lowest estimated losses (or highest expected error reduction) is straightforward, this approach
is sub-optimal.
This is because top-$k$ acquisition, while effective to some degree due to its greedy nature,
overlooks the crucial interactions among data points in batch acquisitions.
Specifically, while aiming to select the most informative unlabeled points, top-$k$ acquisition
may lead to redundant choices, diminishing the overall benefit of the acquisition.

Inspired by Kirsch et al. (), we propose to similarly perturb the original
ranking of the estimated true losses so that the batch sampling provides better
acquisitions when the most informative data points may be duplicated.
Suppose at acquisition episode $t$, we rank the set of estimated true loss of each unlabeled
data point in ascending orders as
$\{\hat{l}_{\text{true},i}\}_{\mathbf{x}_{i}\in\mathcal{U}_{t-1}}$,
such that $\hat{l}_{\text{true},i}\leq\hat{l}_{\text{true},j},\forall i\leq j\text{ and }
\mathbf{x}_{i},\mathbf{x}_{j}\in\mathcal{U}_{t-1}$, we can perturb the ranking with three
strategies: soft-rank, soft-max, and power acquisition, to improve batch performance from the
naive top-$k$ sampling.

#### Soft-rank acquisition.

Soft-rank acquisition relies on the relative ordering of the scores while ignoring the
absolute score values.
It samples the data point ranked at index $i$ with probability
$p_{\text{softrank}}(i)=i^{-\beta}$, where $\beta$ is the “coldness” parameter and is
kept as 1 throughout this paper.
It is not hard to notice that $p_{\text{softrank}}(i)$ is invariant to
$\hat{l}_{\text{true},i}$, as long as the relative ranking remains the same.
More conveniently, with sampled Gumbel noise $\epsilon_{i}\sim\text{Gumbel}(0;\beta^{-1})$,
taking the top-$k$ data points from the perturbed ranked list

$$ $\hat{l}^{\text{softrank}}_{\text{true},i}=-\log i+\epsilon_{i}$ (16) $$

is equivalent to sampling $p_{\text{softrank}}(i)$
without replacement (Huijben et al., ).

#### Soft-max acquisition.

In contrast to soft-rank, soft-max acquisition uses the actual scores, i.e., the estimated
true losses, instead of their relative orderings.
However, this acquisition does not rely on the semantics of the actual values, resulting
in the transformed true loss simply being:

$$ $\hat{l}^{\text{softmax}}_{\text{true},i}=\hat{l}_{\text{true},i}+\epsilon_{i}$ (17) $$

where $\epsilon_{i}$ remains the same Gumbel noise as in the soft-rank acquisition.
Statistically, choosing the top-$k$ data points from this perturbed ranked list is equivalent
to sample from $p_{\text{softmax}}(i)=e^{\beta i}$ without replacement.

#### Power acquisition.

While neither soft-rank or soft-max acquisitions take the semantic meaning of the actual
score values into account when designing the acquisition distribution, power acquisition
uses the value directly when determining the perturbed values.
Specifically, the power acquisition perturbs the scores as

$$ $\hat{l}^{\text{power}}_{\text{true},i}=\log\hat{l}_{\text{true},i}+\epsilon_{i}$ (18) $$

where again $\epsilon_{i}$ is the Gumbel noise, and choosing the top-$k$ indices from this
new list is equivalent to sampling from $p_{\text{power}}(i)=i^{\beta}$ without replacement.
Results comparing DAO with different batch acquisition strategies discussed above are
showed in Appendix A.2.

Combining all the components,
the pseudocode of DAO is summarized in Algorithm 1.

Figure: Algorithm 1 Direct Acquisition Optimization (DAO)

## 4 Experiments

We evaluate DAO on seven classification benchmarks including
digit recognition datasets MNIST (LeCun et al., ),
Street-View House Numbers recognition (SVHN) (Sermanet et al., ),
object classification datasets STL-10 (Coates et al., ), CIFAR-10,
CIFAR-100 (Krizhevsky et al., ),
as well as domain-specific datasets Fashion-MNIST (Xiao et al., )
and Stanford Cars (Cars196) (Krause et al., ).

### 4.1 Experimental Setup

#### Baselines.

To ensure fair comparisons, besides baseline methods that we empirically surveyed
in §2, we also include other state-of-the-arts AL methods, including
Deep Bayesian Active Learning (DBAL) (Gal et al., )
and GLISTER (Killamsetty et al., ), where GLISTER is a direct competitor that
also optimizes the EER framework.

For all the baselines, we used the default/recommended parameters and their official
implementations if publically available.
In terms of earlier works such as least confidence (Lewis, ),
minimum margin (Scheffer et al., ), and maximum entropy (Settles, ),
we used the peer-reviewed deep active learning framework DeepAL+ (Zhan et al., ).
All experiments are repeated ten times with different random seeds.

#### Implementation details.

Throughout the experiment section, we set ResNet-18 (He et al., ) as the model
$f$ to be trained from scratch.
We employed VGG16 (Simonyan & Zisserman, ), initialized with random weights, as our
surrogate $\pi$.
We used stochastic estimation (Agarwal et al., )
when estimating the updated model parameters, as discussed in §3.3.
We choose $n_{\text{ihvp}}=8$ when approximating
the unbiased estimator of $H_{\hat{\theta}}$,
and set $n_{\mathcal{C}}=16$ for biased
loss correction as in
§3.4.

### 4.2 Digit Recognition

First, we demonstrate DAO’s effectiveness through two digit recognition benchmarks: MNIST (LeCun et al., ) and SVHM (Sermanet et al., ).
MNIST is a collection of handwritten digits consisting of 60k training and 10k test
images, while SVHN is a more challenging dataset containing over 600k real-world house
numbers images taken from street views.
Both datasets contain 10 classes corresponding to digits from 0 to 9.

Based on the insights from §2, we define a general rule of
low-budget setting as one image per class, which translates to initial label size
$|\mathcal{L}^{\text{MNIST}}_{\text{init}}|=10$ and per-episode budget
$B_{\text{MNIST}}=10$ for MNIST.
Given that SVHN is more challenging, and there are ten times more unlabeled images than
in MNIST (600k vs. 60k), we experiment both
$|\mathcal{L}^{\text{SVHN}}_{\text{init}}|=10,B_{\text{SVHN}}=10$
and
$|\mathcal{L}^{\text{SVHN}}_{\text{init}}|=100,B_{\text{SVHN}}=100$ for SVHN.
The results are showed in Fig. 2(b) and 2(c).

Figure: (a) MNIST
Refer to caption: x3.png

### 4.3 Object Classification

Next, we assess DAO on more general and complex object classification tasks.
STL-10 (Coates et al., ) is a benchmark dataset derived from labeled examples
in the ImageNet (Deng et al., ).
Specifically, STL-10 contains 5k labeled 96$\times$96 color images spread across 10 classes,
as well as 8k images in the test split.
CIFAR-10 (Krizhevsky et al., ) contains a collection of 60k 32x32 color
images in 10 different classes, with 6k images per class.
CIFAR-100 is similar to CIFAR-10, but covers a much wider range, containing 100 classes
where each class holds 600 images.

Continuing with the low-budget setting (1 image per class), we have
$|\mathcal{L}^{\text{STL-10}}_{\text{init}}|=10$, $B_{\text{STL-10}}=10$
for STL-10,
$|\mathcal{L}^{\text{CIFAR-10}}_{\text{init}}|=10$, $B_{\text{CIFAR-10}}=10$
for CIFAR-10
and $|\mathcal{L}^{\text{CIFAR-100}}_{\text{init}}|=100$, $B_{\text{CIFAR-100}}=100$
for CIFAR-100.
The results are showed in Fig. 2(d), 2(e)
and 2(f) respectively.

### 4.4 Domain Specific Tasks

The last part of our experiments involves case studies on applying DAO to
domain-specific tasks, which simulates many real-world applications.
Specifically, we use FashionMNIST (Xiao et al., ) and
StanfordCars (Krause et al., ), also known as Cars196, in this experiment.
FashionMNIST is structure-wise similar to MNIST, comprising 28$\times$28 images of
70k fashion products from 10 categories, with 7k images per category.
The training set contains 60k images, while the test set includes the rest.
StanfordCars is a large collection of car images, containing 16,185 images with a
near-balanced ratio on the train/test split, resulting in 8,144 and 8,041 images
for training and testing.
There are 196 classes in total, where each class consists of the year, make, model of
a car (e.g., 2012 Tesla Model S).
The results of both datasets are showed in Fig. 2(g)
and 2(h).

### 4.5 Discussion

From Fig. 3, we notice that the proposed DAO outperforms
popular AL state-of-the-arts by a clear margin across all seven benchmarks.
Especially, with SVHN, when the budget is extremely low ($B=10$, which is 0.0017% of the unlabeled size), DAO leads the performance by a very large
gap, indicating its superior capability in the low-budget setting.
Such performance does not degrade much as the budget constraint is relaxed.
As shown in Fig. 2(c), DAO still performs relatively well.
The only experiment that DAO does not improve as much is the StanfordCars.
However, the accuracy improvement from DAO is more smooth and has less variance,
indicating better robustness when applied to the more challenging (StanfordCars has 196 classes)
applications.

## 5 Component Analysis and Ablation Studies

We now analyze specific components of DAO and conduct ablation studies
on the strategies proposed for model parameters approximation (§5.1)
and true loss prediction (§5.2).

### 5.1 Accuracy on Model Approximation

First, we assess if estimating the model parameters updates through modelling the effect of
adding a new sample as upweighting the influence function provides a more accurate model
performance approximation than using single backpropagation as seen in the existing
work (Killamsetty et al., ).
Specifically, we conduct the experiments on CIFAR-10 (Krizhevsky et al., ),
with initial labeled size $|\mathcal{L}^{\text{CIFAR-10}}_{\text{init}}|=100$
(randomly sampled from the train split), per-episode budget $B_{\text{CIFAR-10}}=1$,
and number of acquisition episode $E=25$.
We compare the updated models performance (accuracy) on the test split of CIFAR-10.
Different from the experiments in §4,
we do not apply any AL algorithm when acquiring the sample in each round.
Instead, we randomly choose $B$ sample in each acquisition round from the
unlabeled set and then update the models through both methods with the same selected sample.

To access the difference between models updated with our influence function-based method
and single backpropagation, we compute the mean squared error (MSE) between the performance
of each model and the model updated by conventional full training, which is defined in
Eq. (3).

Based on the result showed in Fig. 3(a), we see that the
proposed method provides more accurate (smaller mean and median) and more robust (smaller std.)
model approximations than single backpropagation, contributing to the performance
gain we observe in §4.

### 5.2 Bias Correction vs. Random Sampling

Next, we conduct ablation studies on replacing the proposed loss estimation
(§3.4) with the average loss of randomly sampled data points.
More specifically, we replace the estimated loss $s_{i}$ from averaging the corrected
loss (Eq. (10)) of the acquired samples via an alternative acquisition criteria
(Eq. (13)) with averaging losses of the samples acquired uniformly,
i.e., at round $t$, we have
$s^{\text{random}}_{i}=\frac{1}{M_{\text{random}}}\sum_{m=1}^{M_{\text{random}}
}\ell(\mathbf{x}_{t,m};f_{t})$
where $\textbf{x}_{t,m}\sim U(1,|\mathcal{U}_{t,i}|)$.
We choose two $M_{\text{random}}=16$ and $256$, where former provides a direct comparison
with our proposed loss estimation approach, and latter represents a brute-force solution that
works relatively well but is often infeasible in practice due to intensive running time.

The results are showed in Fig. 3(b).
We see that the proposed method performs even better than the conventional random-sampling loss
estimation with large sampling size, while computationally being only 1/8 of the run time.
Additionally, the variance of our method is much smaller, indicating more robust loss
estimation, which translates to more robust acquisition performance.

Figure: (a)
Refer to caption: x12.png

## 6 Related Work

#### Active learning.

AL has gained a lot of attraction in recent years, with its goal to
achieve better model performance with fewer training data (Freund et al., ; Graepel & Herbrich, ; Schohn & Cohn, ; Fine et al., ; Hoi et al., ; Guillory & Bilmes, ; Yang & Carbonell, ; Chen et al., ).
There have been different selection criteria including uncertainty (Lewis, ),
query-by-committee (Seung et al., ), version space (Mitchell, ) and
information-theoretic heuristics (Hoi et al., ).
The uncertainty-based selection criteria is arguably the most popular, which includes methods
like least confidence (Lewis, ), minimum margin (Scheffer et al., ), and
maximum entropy (Settles, ).
At their core, these methods select points where the classifier is least certain.
However, uncertainty-based methods can be biased towards the current learner.
Query-by-committee and version space (Seung et al., ; Abe, ; Mitchell, ), on the other hand, keep a pool of models, and then select samples
that maximize the disagreements between them.
Information-theoretic methods (Hoi et al., ; Barz et al., ) typically utilizes
mutual information as criteria.

#### EER-based acquisition criterion.

Alternatively, EER was proposed to select new training
examples that result in the lowest expected error on future test examples, which directly
optimizes the metric by which the model will be evaluated (Roy & McCallum, ).
In essence, EER employs sample selection based on the estimated impact of adding a new
data point to the training set, rather than evaluating performance against a separate
validation set, meaning that it does not inherently require a validation hold-out set.
However, its necessity to retrain the model for every possible candidate sample and
every possible label renders its cost intractable in the context of deep neural
networks (Budd et al., ; Škoda et al., ; Ding et al., ).
More recent EER-based AL algorithms (Killamsetty et al., ; Mussmann et al., )
focus on addressing this efficiency concern.
However, these methods rely on a small set of
validation data to be used for the evaluation of the expected loss reduction, which is
not ideal for the low-budget AL settings.
In this paper, we present DAO, a novel AL algorithm that improves upon EER through
optimizations on both model updates as well as loss estimation, efficiently and
effectively broadening the applicability of EER-based algorithm.

## 7 Conclusions and Future Work

In this paper, we introduced Direct Acquisition Optimization (DAO), a novel algorithm
designed to optimize sample selections in low-budget settings.
DAO hinges on the utilization of influence functions for model parameter updates and a separate
acquisition strategy to mitigate bias in loss estimation, represents a significant optimization
of the EER method and its modern follow-ups.
Through empirical studies, DAO has demonstrated superior performance in low-budget settings,
outperforming existing state-of-the-art methods by a significant margin across seven datasets.

Looking ahead, several promising directions for future research can be explored.
First, further exploration into the scalability of DAO in larger and more complex datasets
will be crucial.
Second, an in-depth investigation into the influence function’s behavior in different model
architectures could yield insights that further refine and enhance the DAO framework.
And finally, integrating DAO with other machine learning paradigms, such as unsupervised
and semi-supervised learning, could lead to the development of more robust and versatile active
learning frameworks.

## Acknowledgement

This work was supported in part by NSF
RI:2313130, NSF 2037026, the Data Science Institute’s AI+Science Research Initiative, and the Research Computing Center at the University of Chicago. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of
any funding agencies.

## Appendix A Appendix

### A.1 More Empirical Analysis Results

Figure: Figure 5: Relative performance between existing popular AL methods and random acquisition. horizontal axis represents the accumulative size of the labeled set, while vertical axis indicates relative performance in percentage.
Refer to caption: x14.png

### A.2 Experiments on Different Stochastic Batch Acquisitions

In this section, we further study the performance of DAO when no batch sampling strategy
or other sampling strategy is used and compare the results with existing popular AL algorithms.
The results are showed in Fig. 6.
For all experiments, we used the same low-budget setting as discussed in
§4.3.

Figure: (a)
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.	How does the running time of the proposed approach compare to other methods? 
2.	Since this is an EER-based approach, the analysis of performance and computational efficiency between the proposed method and traditional EER algorithm would further substantiate the advantages claimed in the paper.
3.	In Section 3.3. the notion \bm{x} denotes only the data or represents the data-label pair?
soundness: 3
strengths: The study addresses notable limitations in existing EER methods, potentially enhancing their applicability within deep learning frameworks. Of particular interest is the use of influence functions to approximate model parameter updates, thereby alleviating the need for repeated model retraining. The experiments are well-designed, covering a broad range of popular benchmark datasets. The results are robust and persuasive, demonstrating performance gains over existing active learning methods.
summary: This paper introduces an EER-based active learning approach for a low-budget setting. The proposed method comprises three main components: (i) label estimation through a surrogate model, (ii) model parameter update approximation using influence functions, and (iii) unbiased loss estimation. Additionally, the authors explore various techniques for batch-mode query selection. Extensive experiments on popular benchmark datasets suggest that this approach delivers superior performance over existing state-of-the-art AL methods.
weaknesses: The main concern lies in the limited technical contributions of the methodology, as the primary techniques appear to have been adapted from existing studies. Besides, although influence functions offer an efficient approximation for parameter updates, they can be both memory and computationally intensive, especially with deep models. This may limit the practicality of applying the proposed method to deep models. A minor point is that the authors claim to use a surrogate model to estimate labels for unlabeled data. However, the implementation details suggest that the current model is merely assigning pseudo-labels rather than employing a distinct surrogate.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. On line 42, the citation to Ash. et al. seems broken.
2. On line 214, what is $v$?
3. On line 192, how is $\hat{\theta}_{\epsilon, x_i}$ defined as a function of $\epsilon$?
4. On line 250, shouldn't $C$ be a subset of the unlabeled examples instead of labeled examples?
5. The authors mention their estimation of the loss is unbiased and low bias at the same time. I think it's biased? Not sure why the authors call it unbiased.
soundness: 3
strengths: The paper conducts experiment across a large number of datasets, showing the consistent improvement in accuracy over baseline methods.

The paper also draws from multiple latest advances in deep active learning, and effectively integrated them together to ensure better performance and theoretical justification.
summary: The paper proposes DAO, an expected error reduction strategy for deep active learning. The method leverages influence functions in approximating the model update when training on a newly labeled data point. To further make this approximation computationally efficient, the authors propose to use conjugate gradients and stochastic estimation. In addition, the algorithm also utilizes a low variance approximation of the error rate on the unlabeled pool, utilizing a surrogate model.
weaknesses: 1. The paper does not compare against TypiClust [1], which is known to outperform all the baseline strategies here in low budget settings.
2. In the high budget experiments, the total budget is not sufficiently large, since each class only has 50 annotation budget on average. It would be interesting to see the algorithm's performance with even larger annotation budget.
3. The notation $\ell(x, f)$ is very confusing. The loss function should be a function of the label $y$ as well. This seems to have been omitted in 3.3 and 3.4. In section 3.3 equation (4), it seems this is approximated using the model's prediction. However, there is no mention of this in section 3.4. How is the loss computed for equations (6) and (9)?
4. If the author is already using the predicted label as ground truth y, instead of the method in section 3.4, the authors can use this to approximate $L_{\text{true}}$, and an ablation study should be conducted. In addition, the authors mention Mussmann et. al. as an expected error reduction algorithm, but never compares against it in the experiments.
5. There has also been quite a few works utilizing NTK to approximate the model's behavior when training on a new data point. The authors should distinguish from these methods, potentially by comparing against them.
6. The authors could consider rearranging the order between sections 3.2 and 3.3. This is because the current 3.2 naturally flows into 3.4, but is interrupted by 3.3 at the moment.

[1] Hacohen, Guy, Avihu Dekel, and Daphna Weinshall. "Active learning on a budget: Opposite strategies suit high and low budgets." arXiv preprint arXiv:2202.02794 (2022).

[2] Mohamadi, M. A., Bae, W., & Sutherland, D. J. (2022). Making look-ahead active learning strategies feasible with neural tangent kernels. Advances in Neural Information Processing Systems, 35, 12542-12553.

[3] Wang, H., Huang, W., Wu, Z., Tong, H., Margenot, A. J., & He, J. (2022). Deep active learning by leveraging training dynamics. Advances in Neural Information Processing Systems, 35, 25171-25184.

[4] Wen, Z., Pizarro, O., & Williams, S. (2023). NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage. arXiv preprint arXiv:2306.04099.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - page 6, Algorithm 1 description, line 5 "randomlly sample n_ivp from unlabeled data U_i,t" this is wrong from the description in page 4, 183 line. We should sample labeled data instead of unlabeled data to update the model parameter (see line 227) But U_i,t from the description is the unlabeled data

- how the label likelihood from surrogate is inferred? did you use the softmax score?

- what is the memory complexity for computing the influence function? 

- what is the lift of performance from the random re-ranking in batch acquisition in 3.5 ?

To address issue 1) related questions:
1. The criteria used for selecting and updating the surrogate model?
2. How the authors ensure the surrogate model remains a good estimate of the oracle in low-budget settings?
3. How the quality of the surrogate model affects the cross-entropy correction term?

To address issue 2) related questions:
1. Could author provide a detailed computational complexity analysis comparing DAO to model retraining, especially for larger models?
2. Could author provide empirical evidence comparing the runtime of their method to retraining with early stopping?
3. How DAO handles cases when the Hessian in influence function computation is indefinite? How the method performance near saddle points?
4. Could authors provide an analysis of how errors in parameter estimation propagate to the sample selection stage?

To address issue 3) related questions:
1. It will be nicer if we can see experiments to empirically demonstrate the method's robustness (or lack thereof) to errors in the surrogate function and parameter updates.
2. May need an ablation study showing the impact of each approximation step on the overall performance.
soundness: 2
strengths: Originality: 
This paper focus on extreme low budge setting while using expected loss reduction as criterion for sample selection. Most of active learning are based on heuristics, which emphasize the computational efficiency in sample selection process. But the training objective is different from the heuristic for sample selection, making them less effective. However, evaluating expected loss on test data requires some labeled validation set. This is not suitable for active learning with low budget. For this reason, this paper raised an important challenge in active learning. This paper combines several existing techniques such the the use of surrogate proposal (which is common for sampling-based methods), and the use of variance-reduction importance sampling for estimation of expected loss. It is interesting to see that the influence function is chosen.   

Quality:
Both derivations and results from this paper seem correct, and are supportive to the author's claim that the heuristic-based methods are sub-optimal and using surrogate function would help to mitigate the lack of labeled samples in low budget setting. No code is shared thus we cannot judge on repeatablilty of experiment.

Clarity:
This paper has well-organized structure. The objective is clear for the active learning.
summary: In this paper, the authors present a new Active Learning framework under the low budget setting. The proposed Direct Acquisition Optimization (DAO) algorithm uses the influence function to update the model parameter and estimate the true loss with importance sampling. The main contributions of this paper is a novel AL algorithm with the proposal of selecting samples based on expected loss reduction. Compared to existing approaches, this method do not rely on additional holdout labeled data for loss evaluation and requires less time for model update using influence function approximation. The experiment results indicate the proposed method is more efficient than the baseline methods including the random selection in image classification use cases.
weaknesses: Contribution: 
This paper focus on active learning with low budge setting and the algorithm presented solves three issues: 1) how to obtain pseudo-labels  in unlabeled data; 2) how to approximate new model parameter without retraining; 3) how provide low bias estimation on true expected loss on unlabled data. 

The issue for 1) is that the obtaining a good proposed surrogate function is difficult.This model plays a central role for the success of this algorithm both theoretically and practically showing in experiments. It is always possible to use the old model from last iteration for surrogate, but it is an poor estimate of the oracle given low budget setup. Thus although this framework claims that it is superior to the heuristic it still inject the heuristic within the surrogate model. More issue with surrogate model is the update of the model. Would you retrain the surrogate model and the main model using the same acquired labels? how to make sure that surrogate model guides us towards the true target better than current production? if not, the cross entropy correction would be a misleading term.
- BTW, the deep learning score is also used as confidence when computing the parameter update. It should be aware that the possible overconfidence issue for deep learning would be a factor to watch esp. when the data is noisy. 

The issue for 2) is that to approximate model parameter using influence function, we have to approximate the influence function itself with some iterative method. Both CG and Quasi-Newton like solution has its own challenge, e.g. in CG, in order to compute the influence of one unlabeled sample. we have to do two back-propagation, which is more time-consuming than retraining the model with early stopping. Another issue for this approximation method is additional memory requirement for large model, like LLM. Finally, when the Hessian is indefinite ( e.g. saddle points), this is not clear that if this second-order estimate would result in severe underwhelming result in parameter estimation. Since the new model parameter is used in cross-entropy correction term in importance sampling, it is likely that the error rate will affect the sample stage.

The issue for 3) is that since cross-entropy acquisition function is the key for bias-variance reduction, it is critical to have some guarantee that the impact of error in both surrogate function selection, and parameter update is bounded. It is expected that when the cross-entropy is low but both proposal and target function are limited, this active learning would be worse than random selection. 

All in all, a major concern for this model is that it is a very complicated system with multiple approximation steps, which has its own errors in each step. The good performance is thus not guaranteed. The uninformative choice of surrogate function and the approximation of parameter update would damage the performance of this model in larger and more complicated dataset. As for the computation cost as compared to model retraiin, it is not clear parameter update with second-order gradient like estimation would reduce the cost of computation as compared to partial retrain (like using Contrastive Divergence).