Calibrated Uncertainty Sampling for Active Learning

## Abstract

Abstract We study the problem of actively learning a classifier with a low calibration error. One of the most popular Acquisition Functions (AFs) in pool-based Active Learning (AL) is querying by the model’s uncertainty. However, we recognize that an uncalibrated uncertainty model on the unlabeled pool may significantly affect the AF effectiveness, leading to sub-optimal generalization and high calibration error on unseen data. Deep Neural Networks (DNNs) make it even worse as the model uncertainty from DNN is usually uncalibrated. Therefore, we propose a new AF by estimating calibration errors and query samples with the highest calibration error before leveraging DNN uncertainty. Specifically, we utilize a kernel calibration error estimator under the covariate shift and formally show that AL with this AF eventually leads to a bounded calibration error on the unlabeled pool and unseen test data. Empirically, our proposed method surpasses other AF baselines by having a lower calibration and generalization error across pool-based AL settings.

## 1 Introduction

Active Learning (AL) has recently become a crucial topic in machine learning due to the development of Deep Neural Network (DNN) and Big data. In the pool-based sequential AL, we consider situations where the unlabeled data pool is abundant but manual labeling is expensive (e.g., medical diagnosis, etc.). Given a fixed budget labeling cost per round, our goal is to design an Acquisition Function (AF) to actively query informative samples from the expert for labels, such that the model can quickly learn and generalize well on unseen data (Roy & McCallum, 2001; Settles, 2009).

To achieve this goal, uncertainty-based approaches (e.g., least-confident, entropy (Shannon, 1948; Wang & Shang, 2014), margin sampling (Roth & Small, 2006), BALD (Gal et al., 2017)) and diversity-based approaches (e.g., Coreset (Sener & Savarese, 2018), BADGE (Ash et al., 2020)) have shown promising results by achieving a better generalization than the naive random-sampling baseline (Lüth et al., 2023; Citovsky et al., 2021; Kim et al., 2020). However, such approaches often focus solely on generalization (e.g., accuracy) and overlook the verification of model uncertainty estimation quality (e.g., calibration) (Tran et al., 2022; Nalisnick et al., 2019).

Meanwhile, the ability of models to produce high-quality uncertainty estimation is crucial for a reliable DNN in high-stakes applications (e.g., forecasting, healthcare, finance, etc.). Beyond the generalization, in principle, a reliable model also includes uncertainty estimation ability to permit graceful failure, signaling when it is likely to be wrong (Liu et al., 2020; Bui & Liu, 2024). That said, the uncertainty estimation quality of the model, especially derived from DNN, is often poorly calibrated (Kuleshov et al., 2018; Nado et al., 2021). Therefore, this results in two critical issues in the literature on AL (e.g., Fig. [1](https://arxiv.org/html/2510.03162v1#S1.F1)). Firstly, the uncertainty quantification quality of the aforementioned AL baselines on unseen test data is not verified, leading to high risks in high-stakes applications. Secondly, suppose the model is uncalibrated on the unlabeled pool during the query times, then the AF with uncertainty-based sampling is also unreliable, resulting in non-informative query samples and leading to sub-optimal generalization and high calibration error on unseen data.

Figure: Figure 1: Accuracy and calibration comparison on MNIST with $T=50$ and $k=10$. Uncalibrated-least-conf is the least-conf method, but is additionally made to be uncalibrated by randomly scaling the logit vectors for every sample. We can see that when the model is uncalibrated, the least-confident sampling not only has a worse ECE than our method but also has a lower accuracy because of querying non-informative samples. A short demo is available at [this Google Colab link](https://colab.research.google.com/drive/1QRLmzHyET-heDZ4IUuhU1v1Zd2gIrIXF?usp=sharing).
Refer to caption: x1.png

Therefore, with the goal of developing a reliable model in this pool-based AL setting, we propose a new AF strategy in Fig. [3](https://arxiv.org/html/2510.03162v1#S3.F3) by ranking based on the lexicographic order of the calibration error and the model uncertainty. Specifically, we estimate the calibration error of each sample on the unlabeled pool by using the kernel trick with the cumulative labeled data. Then, using the lexicographic ordering of Cartesian products (Duffus, 2006; Murphy, 2023), we select samples with the highest calibration errors. If samples share the same calibration error, we continue to select samples with the highest model uncertainty. The key ideas of this approach are: (1) selecting the samples with the highest calibration error can help the model with self-calibrated predictive certainty, maintaining a high uncertainty estimation quality on the unlabeled pool and unseen dataset; (2) using this well-calibrated model can help uncertainty sampling discover informative samples during query times, improving calibration and generalization.

Our contributions can be summarized as follows:

- 1.
We propose Calibrated Uncertainty Sampling for AL (CUSAL), an AF by estimating calibration error on the unlabeled pool and using the lexicographic order of calibration and uncertainty, prioritizing querying samples with the highest calibration error before leveraging model uncertainty.
- 2.
We theoretically provide an upper bound for our calibration estimator error under the covariate shift of AL. Furthermore, we prove that using our AF, we can bound the expected calibration error of the learned classifier on the unlabeled pool and the unseen test data.
- 3.
We empirically confirm our proposed method surpasses other acquisition strategies in the pool-based AL by having a lower calibration and generalization error across several settings on the MNIST, F-MNIST, SVHN, CIFAR-10, CIFAR-10-LT, and ImageNet datasets.

## 2 Preliminary

### 2.1 Problem setting

We denote $\mathcal{X}$ and $\mathcal{Y}$ as the sample and label space. A dataset is defined by a joint distribution $\mathbb{P}(X,Y)\in\mathcal{P}_{\mathcal{X}\times\mathcal{Y}}$, where $\mathcal{P}_{\mathcal{X}\times\mathcal{Y}}$ is the set of joint probability distributions on $\mathcal{X}\times\mathcal{Y}$. We consider the pool-based AL setup (Roy & McCallum, 2001), i.e., we are given a warm-up dataset $S_{0}=\{(x_{i},y_{i})\}_{i=1}^{n_{0}}$, where $n_{0}$ is the number of data points in $S_{0}$ and $(x_{i},y_{i})\sim\mathbb{P}(X,Y)$, $\forall i\in[n_{0}]$. We are also given an unlabeled dataset $U_{0}=\{x_{j}\}_{j=1}^{m_{0}}$, where $m_{0}$ is the number of data points in $U_{0}$, s.t., $x_{j}\sim\mathbb{P}(X)$ and can request the true corresponding label sample $y_{j}$ according to $\mathbb{P}(Y|X)$ for every $j\in[m_{0}]$. We aim to test a learning model on the test set $D_{te}=\{(x_{i},y_{i})\}_{i=1}^{n_{te}}$, where $n_{te}$ is the number of data points in $D_{te}$, s.t., $(x_{i},y_{i})\sim\mathbb{P}(X,Y)$, $\forall i\in[n_{te}]$.

We can consider the standard pool-based AL as a sequential decision-making process, i.e., at round $t\in[T]$, where $T$ is the number of labeling rounds, the model agent parameterized by a learnable parameter $\theta_{t}$, is provided a sequence of unlabeled pool $U_{t}\subseteq\mathcal{X}$ and seeks to find a global minimum

$$ $\displaystyle x_{t}^{*}:=\operatorname*{arg\,max}_{x\in U_{t}}AF(x;\theta_{t-1}),$ (1) $$

where $AF$ is an Acquisition Function. Once the set of points $\{x_{t}^{*}\}$ has been chosen, the oracle $\mathbb{P}(Y|X)$ provides its corresponding label $\{y_{t}^{*}\}$. Hence, the new-labeled set and the pool set become

$$ $\displaystyle S_{t+1}=S_{t}\cup\{x_{t}^{*},y_{t}^{*}\},\quad U_{t+1}=U_{t}\setminus\{x_{t}^{*}\}.$ (2) $$

And, the model parameter $\theta_{t}$ is optimized (i.e., trained) with the labeled set $S_{t+1}$, after which it receives a loss outcome $\ell_{te}(\theta_{t})$ on the test set $D_{te}$, and repeat this process at the next round $t+1$.

### 2.2 Generalization

Under the classification setting, the model predicts $y\in\mathcal{Y}$, where $\mathcal{Y}$ is discrete with $K$ possible categories by using a forecast $h(\cdot):=h(\cdot;\theta)=\sigma\circ f(\cdot;\theta)$, which composites a backbone $f(\cdot;\theta):\mathcal{X}\rightarrow\mathbb{R}^{K}$, parameterized by $\theta$, and a softmax layer $\sigma:\mathbb{R}^{K}\rightarrow\Delta_{y}$ which outputs a probability distribution $W(y):\mathcal{Y}\rightarrow[0,1]$ within the set $\Delta_{y}$ of distributions over $\mathcal{Y}$; the value of probability density function of $W$ is $w$. Similar to the literature on AL, we consider $\ell_{te}(\theta_{t})$ to include the generalization error by the accuracy of $h$, i.e.,

$$ $\mathbb{E}_{(x,y)\sim\mathbb{P}(X,Y)}\left[\ell_{01}(h(x),y)\right]:=\mathbb{E}_{(x,y)\sim\mathbb{P}(X,Y)}\left[\mathbb{I}[\operatorname*{arg\,max}_{y_{k}\in\mathcal{Y}}\left[h(x)]_{y_{k}}\neq y\right]\right].$ (3) $$

### 2.3 Calibration

Furthermore, we consider the calibration error in Eq. [4](https://arxiv.org/html/2510.03162v1#S2.E4) as the second term of $\ell_{te}(\theta_{t})$. Specifically, let us first recall the definition of distribution calibration (Dawid, 1982) for the forecast:

###### Definition 2.1 .

(Song et al., 2019; Kuleshov et al., 2018)
A forecast $h$ is said to be distributionally calibrated if

$$ $\mathbb{P}(Y=y|h(x)=W)=w(y)\text{, }\forall y\in\mathcal{Y},W\in\Delta_{y}.$ $$

Intuitively, the forecast $h$ is well-calibrated if its outputs truthfully quantify the predictive uncertainty. For instance, if we take all data points $x$ for which the model predicts $[h(x)]_{y}=0.8$, we expect $80\%$ of them to take on the label $y$ indeed. This yields the definition of the $L_{p}$ calibration error of $h$ as follows

$$ $CE_{p}(h):=\left(\mathbb{E}\left[\left\|\mathbb{E}[Y|h(X)]-h(X)\right\|_{p}^{p}\right]\right)^{\frac{1}{p}},$ (4) $$

where

$$ $\displaystyle\mathbb{E}[Y|h(X)=h(x)]=\frac{\sum_{y_{k}\in\mathcal{Y}}y_{k}p(h(X)=h(x),Y=y_{k})}{p(h(X)=h(x))}.$ (5) $$

Since the distributionally calibrated Def. [2.1](https://arxiv.org/html/2510.03162v1#S2.Thmtheorem1), we can say $h$ is perfectly calibrated if $CE_{p}(h)^{p}=0$.

###### Definition 2.1 .

## 3 Calibrated Uncertainty Sampling Method

To improve calibration and generalization performance, we introduce our novel AF by ranking based on the lexicographic order of the calibration error and the model uncertainty. We formally present our framework as follows:

### 3.1 Estimating calibration error on unlabeled pool

To select the highest calibration error, we need to know the calibration error of each sample on the unlabeled pool. Unfortunately, as provided in Eq. [4](https://arxiv.org/html/2510.03162v1#S2.E4), this quantity requires the knowledge of the true label $y$, and this is an unknown quantity for our model in the AL setting.

Therefore, to estimate the calibration error of each sample on the unlabeled pool, firstly, let us recall the consistent and differentiable Lp canonical calibration error estimator for $n$ Independent-identically-distributed (IID) labeled samples $\{(x_{i},y_{i})\}_{i=1}^{n}$ from the joint distribution $\mathbb{P}(X,Y)$ (Popordanoska et al., 2022). In particular, let us consider the estimator of canonical calibration error, where the expected value outside in Eq. [4](https://arxiv.org/html/2510.03162v1#S2.E4) is measured by the empirical data, i.e.,

$$ $\displaystyle\widehat{CE_{p}(h)^{p}}:=\frac{1}{n}\sum_{j=1}^{n}\widehat{CE_{p}(h(x_{j}))^{p}}=\frac{1}{n}\sum_{j=1}^{n}\left\|\mathbb{E}_{iid}\left[\widehat{Y|h(x_{j})}\right]-h(x_{j})\right\|_{p}^{p},$ (6) $$

where

$$ $\displaystyle\mathbb{E}_{iid}\left[\widehat{Y|h(x_{j})}\right]:=\frac{\sum_{i=1}^{n}k\left(h(x_{j});h(x_{i})\right)y_{i}}{\sum_{i=1}^{n}k\left(h(x_{j});h(x_{i})\right)},\forall j\in[n].$ (7) $$

As verified by Popordanoska et al. (2022), when $p(h(x))$ is Lipschitz continuous over the interior of the simplex, $\exists$ a kernel s.t. $\mathbb{E}_{iid}\left[\widehat{Y|h(x_{j})}\right]$ is a pointwise consistent estimator of $\mathbb{E}\left[{Y|h(x_{j})}\right]$, that is:

$$ $\displaystyle\operatorname*{plim}_{n\rightarrow\infty}\frac{\sum_{i=1}^{n}k\left(h(x_{j});h(x_{i})\right)y_{i}}{\sum_{i=1}^{n}k\left(h(x_{j});h(x_{i})\right)}=\mathbb{E}\left[Y|h(x_{j})\right].$ (8) $$

Figure: Figure 2: Overview of our calibrated uncertainty sampling framework for Active Learning.
Refer to caption: x4.png

From Eq. [8](https://arxiv.org/html/2510.03162v1#S3.E8), firstly, we can see that to estimate calibration error $\mathbb{E}\left[Y|h(x_{j})\right]$ for every sample $j\in[n]$, it requires the knowledge of $n$ labels. This requirement is not met on the unlabeled pool of AL setting. Secondly, it is worth noticing that to verify the consistency, i.e., Eq. [8](https://arxiv.org/html/2510.03162v1#S3.E8), Popordanoska et al. (2022) requires $n$ samples are sampled IID. That said, this assumption does not hold in AL setting by the covariate shift, i.e., the difference in distribution between unlabeled pool samples $U$ and the queried samples $S$ from the model $h$ (Liu et al., 2015; Prinster et al., 2024). So, the challenge here is how we can modify the estimator in Eq. [6](https://arxiv.org/html/2510.03162v1#S3.E6) such that it is still consistent on the unlabeled pool.

To address this challenge in AL setting, we modify the estimator in Eq. [6](https://arxiv.org/html/2510.03162v1#S3.E6) to estimate the calibration error quantity on the unlabeled pool. Specifically, for every round $t\in[T]$, given $n_{t}$ samples in the new-labeled set $S_{t}$ and $m_{t}$ samples in the unlabeled pool set $U_{t}$, our new estimator for every sample on the unlabeled pool $x_{j}$, is as follows:

$$ $\displaystyle\widehat{CE_{p}(h(x_{j}))^{p}}:=\left\|\frac{\sum_{i=1}^{n_{t}}k\left(h(x_{j});h(x_{i})\right)y_{i}}{\sum_{i=1}^{n_{t}}k\left(h(x_{j});h(x_{i})\right)}-h(x_{j})\right\|_{p}^{p},\forall j\in[m_{t}],$ (9) $$

where $k$ is a Dirichlet kernel, i.e.,

$$ $\displaystyle k\left(h(x_{j});h(x_{i})\right):=\frac{\Gamma(\sum_{k=1}^{K}\alpha_{ik})}{\prod_{k=1}^{K}\Gamma(\alpha_{ik})}\prod_{k=1}^{K}h(x_{j})_{k}^{\alpha_{ik}-1},$ (10) $$

with $\alpha_{i}=\frac{h(x_{i})}{b}+1$, where $b$ is a bandwidth parameter in the kernel density estimate. It is worth noting that while the estimator in Eq. [6](https://arxiv.org/html/2510.03162v1#S3.E6) & Eq. [7](https://arxiv.org/html/2510.03162v1#S3.E7) requires both $i,j$ in the same set $[n]$, our estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) estimate $j$ in $m_{t}$ unlabeled sample $U_{t}$ from $i$ in $n_{t}$ queried datapoints $S_{t}$. These $S_{t}$ and $U_{t}$ are two separate sets. Hence, we formally prove our calibration estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) is also point-wise consistent and provides its estimator error bound under the covariate shift of AL setting in Thm [4.1](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem1). Intuitively, if the model knows the calibration error of every sample on the unlabeled pool, it can query the highest calibration error samples to self-correct their confidence prediction in the future. Following this idea, we propose our new AF in the next section.

### 3.2 Our lexicographic order acquisition function

Following our calibration error estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9), we leverage it for our novel AF as follows. Our key idea is that if the model is unreliable, we should fix its calibration error, i.e., choose the least calibrated samples first, and then use uncertainty sampling later to query informative samples in the unlabeled pool. Formally, we consider the two partially ordered sets on the unlabeled pool, including the calibrated error set $A$ and the uncertainty set $B$ from the softmax layer, defined as follows

$$ $\displaystyle A:=\left\{\left\|\frac{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)y_{i}}{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)}-h(x)\right\|_{p}^{p}\mid x\in U_{t}\right\},\quad B:=\left\{\max_{y\in[K]}[h(x)]_{y}\mid x\in U_{t}\right\}.$ (11) $$

Then, our AF is

$$ $x_{t}^{*}=\operatorname*{arg\,max}_{x\in U_{t}}AF(x;\theta_{t-1}),$ (12) $$

where $x$ is chosen by the lexicographical order $(a,b)$ on the Cartesian product $A\times B$, which is defined as

$$ $\displaystyle(a,b)\leq(a^{\prime},b^{\prime})\Leftrightarrow a<a^{\prime}\text{ or }(a=a^{\prime}\text{ and }b\geq b^{\prime}),$ (13) $$

for all $a,a^{\prime}\in A$ and $b,b^{\prime}\in B$. Since we denote $A$ as the set of estimated calibration errors and $B$ as the set of model uncertainties for samples on the unlabeled pool $U_{t}$, the AF in Eq. [12](https://arxiv.org/html/2510.03162v1#S3.E12) means we select samples with the highest calibration errors. If samples share the same calibration error, we continue to select samples with the highest model uncertainty. The pseudo-code for our algorithm is presented in Alg. [1](https://arxiv.org/html/2510.03162v1#alg1). Intuitively, the acquisition strategy in Eq. [12](https://arxiv.org/html/2510.03162v1#S3.E12) suggests that when $h$ is uncalibrated, we will select the least calibrated samples in the unlabeled pool to improve uncertainty quantification quality. And, when $h$ becomes reliable by being perfectly calibrated (i.e., the calibration error across samples is the same and small), the AF will select the least confident samples to improve generalization performance. We formally explain this behavior in the following section.

Figure: Algorithm 1 Our AF: Calibrated Uncertainty Sampling for AL (CUSAL) (code is in Apd. [B.2](https://arxiv.org/html/2510.03162v1#A2.SS2))

## 4 Theoretical Analysis of our Acquisition Function

Firstly, recall that in Eq. [8](https://arxiv.org/html/2510.03162v1#S3.E8), Popordanoska et al. (2022) shows that their estimator $\mathbb{E}_{iid}\left[\widehat{Y|h(x_{j})}\right]$, $\forall j\in n$ in Eq. [7](https://arxiv.org/html/2510.03162v1#S3.E7) is a pointwise consistent estimator of $\mathbb{E}\left[Y|h(x_{j})\right]$. Yet, this result only holds when the $n$ data points are sampled IID, i.e., $\{x_{i},y_{i}\}_{i\in[n]}\sim\mathbb{P}(X,Y)$. Unfortunately, this does not hold in AL. Specifically, this is due to the biased sampling during query times from the AF of the model leading to a covariate shift existing between the cumulative labeled data $S$ and samples from the unlabeled pool $U$ (Liu et al., 2015; Dasgupta, 2011; Fannjiang et al., 2022). Formally:

$$ $\displaystyle S\sim\mathbb{P}(X)\mathbb{P}(Y|X),\quad U\sim\tilde{\mathbb{P}}(X_{S})\mathbb{P}(Y|X),$ (14) $$

where $\tilde{\mathbb{P}}(X_{S})$ denotes the marginal distribution of $X$ excluding the samples random variable $X_{S}$ in the cumulative labeled dataset $S$. It is worth noticing that at the start of AL, although all data points, i.e., $S_{0}\cup U_{0}$ could be sampled IID from $\mathbb{P}(X,Y)$, the covariate shift happens when we update our model on $S_{0}$, then we use this trained model to query in the next round. For example, if we were trying to predict the next president based on voting during a presidential election in the U.S., although the votes of people are sampled
from the same distribution of U.S. citizens, after querying for the most informative samples, the model could end up biased to a sub-population distribution (e.g., citizens in N.Y. state), resulting in a covariate shift between the cumulative and unlabeled pool.

Therefore, we next guarantee that our modified estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) is a point-wise consistent estimator under the covariate shift of AL by the following theorem:

###### Theorem 4.1 .

Given a sample $x$ on the unlabeled pool with $m_{t}$ data points and $n_{t}$ samples in the cumulative labeled data, our estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) is a point-wise consistent estimator under active learning with covariate shift, i.e.,

$$ $\displaystyle\operatorname*{plim}_{n_{t}\rightarrow\infty}\left\{\mathbb{E}_{\pi_{U}}\left[\widehat{Y|h(x)}\right]:=\frac{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)y_{i}}{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)}\right\}=\mathbb{E}_{\pi_{U}}[Y|h(x)],$ $$

where $\pi_{U}$ denotes the sample distribution on the unlabeled pool. And, the mean square error of our estimator in Eq. [7](https://arxiv.org/html/2510.03162v1#S3.E7) is bounded by

$$ $\displaystyle\mathbb{E}\left[\left|\mathbb{E}_{\pi_{U}}\left[\widehat{Y|h(x)}\right]-\mathbb{E}_{\pi_{U}}\left[Y|h(x)\right]\right|^{2}\right]\leq\mathcal{O}(n_{t}^{-1}b^{\frac{-K+1}{2}}+b^{2}).$ $$

The proof is in Apd. [A.1](https://arxiv.org/html/2510.03162v1#A1.SS1). From Thm. [4.1](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem1), we can see that when $n_{t}$, i.e., the number of samples in the warm-up data or cumulative pool, is large enough, we can achieve a perfect calibration error estimator. In addition, when $n_{t}$ increases, the quality of the estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) also improves correspondingly. We verify this behavior in Fig. [3](https://arxiv.org/html/2510.03162v1#S3.F3) with a decrease in the gap between our estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) and the estimator with the knowledge of true labels on the unlabeled pool.

Since our goal is improving the model uncertainty quantification quality on the unseen test data and also on the unlabeled pool so that it can improve the AF effectiveness of querying data points in AL, we next formally show the expected calibration error bound on the unlabeled pool and on the unseen test data of our AF by the theorem as follows:

###### Theorem 4.2 .

For every round $t\in(0,T]$, given $m_{t}=m_{t-1}-k_{t}$ samples in the unlabeled pool $U_{t}$, $n_{t}=n_{t-1}+k_{t}$ samples in the queried dataset $S_{t}$, where $k_{t}$ is the number of selected samples from our AF, suppose the calibration error function is bounded by $L$, the calibration error estimator is unbiased, and the trained model is assumed to have $\frac{1}{k_{t}}\sum_{j\in[k_{t}]}\left\|\mathbb{E}\left[Y|h(x_{j})\right]-h(x_{j})\right\|_{p}^{p}\leq\epsilon$ over queried training points. Then, the expected calibration error on the unlabeled pool of Alg. [1](https://arxiv.org/html/2510.03162v1#alg1) is also bounded by

$$ $\displaystyle\frac{1}{m_{t}}\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\leq\epsilon.$ $$

Furthermore, for any $\delta\in(0,1)$, with probability at least $1-\delta$, the expected calibration error on the unseen test data of Alg. [1](https://arxiv.org/html/2510.03162v1#alg1) is bounded by

$$ $\displaystyle\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}\leq\epsilon+\sqrt{\frac{L^{2}\log(2/\delta)}{2(m_{t}+n_{t})}}.$ $$

The proof is in Apd. [A.2](https://arxiv.org/html/2510.03162v1#A1.SS2). Thm [4.2](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem2) suggests that when we query samples with the highest calibration error before leveraging DNN uncertainty, then train our model on these queried data points with a small enough calibration error loss, we can guarantee that the calibration error on the unlabeled pool and unseen data is also small. Furthermore, the more labeled samples $n_{t}$ and unlabeled samples $m_{t}$ we have, the tighter calibration error upper bound we can guarantee. We confirm this result in Fig. [6](https://arxiv.org/html/2510.03162v1#S5.F6), where the ECE of our proposed method on the unlabeled pool is smaller than other baselines.

###### Theorem 4.1 .

###### Theorem 4.2 .

## 5 Experiments

### 5.1 Experimental settings

Baselines. We follow Lüth et al. (2023) to compare with 6 main AF baselines, including random sampling, least-confidence from softmax, Margin (Wang & Shang, 2014), BALD (Gal et al., 2017), Coreset (Sener & Savarese, 2018), and BADGE (Ash et al., 2020). We also extend to compare with other AL settings, including data-augmentation (CAMPAL (Yang et al., 2023)), two-stage-based (Rand-Entropy, Cluster-Margin (Citovsky et al., 2021)), and batch-based (BatchBALD (Kirsch et al., 2019)). Since our method only modifies the AF, we maintain all baselines that share the same training strategy with the cross-entropy loss. Notably, we focus on the inductive AL (MacKay, 1992; Hübotter et al., 2024), do not use any semi, self-supervised (Gao et al., 2020; Bengar et al., 2021), or post-hoc recalibration techniques in training, though we still add them to compare in our ablation studies (e.g., CALICO (Querol et al., 2024), DDU (Mukhoti et al., 2023), etc.).

Other settings. We deploy models across 6 standard datasets and backbones, including MNIST with MNIST-Net, Fashion-MNIST with GarmentClassifier, SVHN, balanced CIFAR-10, and imbalanced CIFAR-10-LT with ResNet-18, and ImageNet with ResNet-50. We use the train set as a warmup and an unlabeled pool, and the test set as an unseen test dataset. For all baselines, we use the same AL hyperparameters and other training hyperparameters. We run the results across 10 different random seeds, and these seeds are shared across baselines for a fair comparison. We evaluate models with accuracy and Expected Calibration Error (ECE) (Naeini et al., 2015). (Details are in Apd. [B.1](https://arxiv.org/html/2510.03162v1#A2.SS1)).

### 5.2 Main results

Tab. [1](https://arxiv.org/html/2510.03162v1#S5.T1) shows results at particular time steps $t\in\{T/4,T/2,3T/4,T\}$ and Fig. [7](https://arxiv.org/html/2510.03162v1#A2.F7) summarizes results across $T$ time horizons. It can be seen from these results that our CUSAL method consistently outperforms other baselines at almost every time step $t\in[T]$ across different settings by lower ECE. Accompanied by the accuracy performance, even in a challenging dataset like CIFAR-10, our method is the closest point to the best performance point in Fig. [4](https://arxiv.org/html/2510.03162v1#S5.F4). This implies that our AF can query more informative samples and needs fewer query costs to achieve the desired behavior in AL.

**Table 1: Calibration error (lower is better) and accuracy (higher is better) comparison with different baselines across query times $t\in\{T/4,T/2,3T/4,T\}$, where $T=40$ on MNIST and $T=100$ on other datasets. Scores are reported by mean $\pm$ standard deviation from 10 runs. Gray rows indicate our proposed method (i.e., Our CUSAL in Alg. [1](https://arxiv.org/html/2510.03162v1#alg1)). Best scores with the significant test are marked in bold. Figure details are in Fig. [7](https://arxiv.org/html/2510.03162v1#A2.F7).**
| Dataset | Method | Expected Calibration Error ($\downarrow$) | Accuracy ($\uparrow$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ | $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ |  |  |
| MNIST | Random | 0.121 $\pm$ 0.025 | 0.079 $\pm$ 0.005 | 0.064 $\pm$ 0.002 | 0.058 $\pm$ 0.003 | 82.7 $\pm$ 3.1 | 89.8 $\pm$ 0.8 | 92.2 $\pm$ 0.2 | 93.2 $\pm$ 0.1 |
| Least-conf | 0.123 $\pm$ 0.014 | 0.069 $\pm$ 0.011 | 0.039 $\pm$ 0.004 | 0.033 $\pm$ 0.006 | 83.1 $\pm$ 1.4 | 90.7 $\pm$ 1.2 | 94.4 $\pm$ 0.3 | 95.5 $\pm$ 0.6 |  |
| Margin | 0.116 $\pm$ 0.018 | 0.068 $\pm$ 0.010 | 0.042 $\pm$ 0.003 | 0.038 $\pm$ 0.004 | 84.1 $\pm$ 1.5 | 90.9 $\pm$ 1.0 | 94.2 $\pm$ 0.2 | 95.1 $\pm$ 0.5 |  |
| BALD | 0.109 $\pm$ 0.012 | 0.058 $\pm$ 0.007 | 0.036 $\pm$ 0.002 | 0.031 $\pm$ 0.003 | 83.3 $\pm$ 1.4 | 90.9 $\pm$ 1.1 | 94.5 $\pm$ 0.3 | 95.5 $\pm$ 0.6 |  |
| Coreset | 0.122 $\pm$ 0.017 | 0.072 $\pm$ 0.008 | 0.047 $\pm$ 0.003 | 0.041 $\pm$ 0.004 | 82.9 $\pm$ 1.9 | 90.4 $\pm$ 1.0 | 93.7 $\pm$ 0.2 | 94.7 $\pm$ 0.4 |  |
| BADGE | 0.111 $\pm$ 0.012 | 0.061 $\pm$ 0.008 | 0.038 $\pm$ 0.003 | 0.032 $\pm$ 0.005 | 85.3 $\pm$ 1.3 | 92.3 $\pm$ 0.7 | 94.8 $\pm$ 0.2 | 95.7 $\pm$ 0.4 |  |
|  | Ours | 0.089 $\pm$ 0.014 | 0.046 $\pm$ 0.007 | 0.035 $\pm$ 0.003 | 0.030 $\pm$ 0.002 | 86.7 $\pm$ 1.6 | 93.3 $\pm$ 0.8 | 95.1 $\pm$ 0.2 | 95.9 $\pm$ 0.3 |
| SVHN | Random | 0.139 $\pm$ 0.004 | 0.119 $\pm$ 0.003 | 0.110 $\pm$ 0.004 | 0.103 $\pm$ 0.004 | 81.7 $\pm$ 0.3 | 84.8 $\pm$ 0.4 | 86.2 $\pm$ 0.4 | 87.6 $\pm$ 0.4 |
| Least-conf | 0.134 $\pm$ 0.009 | 0.113 $\pm$ 0.006 | 0.099 $\pm$ 0.004 | 0.087 $\pm$ 0.003 | 82.6 $\pm$ 1.0 | 86.1 $\pm$ 0.7 | 88.0 $\pm$ 0.4 | 89.5 $\pm$ 0.3 |  |
| Margin | 0.131 $\pm$ 0.004 | 0.110 $\pm$ 0.003 | 0.098 $\pm$ 0.002 | 0.089 $\pm$ 0.001 | 82.9 $\pm$ 0.4 | 86.3 $\pm$ 0.4 | 88.0 $\pm$ 0.2 | 89.4 $\pm$ 0.2 |  |
| BALD | 0.132 $\pm$ 0.003 | 0.111 $\pm$ 0.003 | 0.100 $\pm$ 0.002 | 0.092 $\pm$ 0.002 | 82.6 $\pm$ 0.3 | 86.0 $\pm$ 0.4 | 87.6 $\pm$ 0.2 | 88.9 $\pm$ 0.2 |  |
| Coreset | 0.133 $\pm$ 0.005 | 0.112 $\pm$ 0.003 | 0.100 $\pm$ 0.002 | 0.090 $\pm$ 0.002 | 82.6 $\pm$ 0.5 | 86.1 $\pm$ 0.4 | 87.8 $\pm$ 0.2 | 89.1 $\pm$ 0.2 |  |
| BADGE | 0.129 $\pm$ 0.003 | 0.108 $\pm$ 0.003 | 0.096 $\pm$ 0.001 | 0.088 $\pm$ 0.001 | 83.2 $\pm$ 0.3 | 86.6 $\pm$ 0.4 | 88.3 $\pm$ 0.1 | 89.6 $\pm$ 0.1 |  |
|  | Ours | 0.123 $\pm$ 0.002 | 0.103 $\pm$ 0.003 | 0.090 $\pm$ 0.001 | 0.081 $\pm$ 0.001 | 83.6 $\pm$ 0.4 | 86.9 $\pm$ 0.4 | 88.7 $\pm$ 0.2 | 90.1 $\pm$ 0.1 |
| F-MNIST | Random | 0.238 $\pm$ 0.007 | 0.210 $\pm$ 0.006 | 0.200 $\pm$ 0.006 | 0.200 $\pm$ 0.005 | 71.4 $\pm$ 0.6 | 75.6 $\pm$ 0.7 | 77.3 $\pm$ 0.6 | 77.9 $\pm$ 0.5 |
| Least-conf | 0.248 $\pm$ 0.017 | 0.217 $\pm$ 0.017 | 0.203 $\pm$ 0.020 | 0.188 $\pm$ 0.016 | 71.6 $\pm$ 1.5 | 76.0 $\pm$ 1.6 | 77.9 $\pm$ 2.0 | 79.6 $\pm$ 1.6 |  |
| BALD | 0.252 $\pm$ 0.019 | 0.203 $\pm$ 0.010 | 0.189 $\pm$ 0.012 | 0.175 $\pm$ 0.010 | 71.4 $\pm$ 1.8 | 77.7 $\pm$ 1.0 | 79.6 $\pm$ 1.2 | 80.8 $\pm$ 1.1 |  |
| Margin | 0.238 $\pm$ 0.006 | 0.208 $\pm$ 0.006 | 0.196 $\pm$ 0.009 | 0.190 $\pm$ 0.005 | 72.2 $\pm$ 0.6 | 76.7 $\pm$ 0.6 | 78.5 $\pm$ 0.8 | 79.5 $\pm$ 0.5 |  |
| Coreset | 0.244 $\pm$ 0.010 | 0.215 $\pm$ 0.010 | 0.202 $\pm$ 0.013 | 0.194 $\pm$ 0.009 | 71.6 $\pm$ 0.9 | 75.8 $\pm$ 1.0 | 77.7 $\pm$ 1.3 | 78.8 $\pm$ 0.9 |  |
| BADGE | 0.233 $\pm$ 0.006 | 0.203 $\pm$ 0.004 | 0.191 $\pm$ 0.005 | 0.186 $\pm$ 0.004 | 72.8 $\pm$ 0.7 | 77.5 $\pm$ 0.4 | 79.3 $\pm$ 0.5 | 80.3 $\pm$ 0.4 |  |
|  | Ours | 0.225 $\pm$ 0.008 | 0.190 $\pm$ 0.004 | 0.175 $\pm$ 0.004 | 0.165 $\pm$ 0.003 | 73.6 $\pm$ 1.1 | 78.7 $\pm$ 0.5 | 80.6 $\pm$ 0.4 | 81.9 $\pm$ 0.3 |
| CIFAR-10 | Random | 0.338 $\pm$ 0.003 | 0.312 $\pm$ 0.001 | 0.288 $\pm$ 0.003 | 0.287 $\pm$ 0.014 | 54.5 $\pm$ 0.6 | 59.7 $\pm$ 0.3 | 63.4 $\pm$ 0.2 | 63.8 $\pm$ 2.0 |
| Least-conf | 0.345 $\pm$ 0.009 | 0.313 $\pm$ 0.007 | 0.285 $\pm$ 0.006 | 0.278 $\pm$ 0.007 | 54.9 $\pm$ 0.4 | 60.8 $\pm$ 0.5 | 64.7 $\pm$ 0.9 | 66.1 $\pm$ 0.6 |  |
| Margin | 0.338 $\pm$ 0.002 | 0.309 $\pm$ 0.002 | 0.285 $\pm$ 0.002 | 0.276 $\pm$ 0.008 | 54.9 $\pm$ 0.2 | 60.8 $\pm$ 0.3 | 64.8 $\pm$ 0.7 | 66.0 $\pm$ 0.8 |  |
| BALD | 0.338 $\pm$ 0.010 | 0.311 $\pm$ 0.013 | 0.283 $\pm$ 0.004 | 0.272 $\pm$ 0.007 | 54.8 $\pm$ 1.0 | 60.5 $\pm$ 0.5 | 64.7 $\pm$ 0.5 | 65.9 $\pm$ 0.5 |  |
| Coreset | 0.343 $\pm$ 0.002 | 0.313 $\pm$ 0.003 | 0.287 $\pm$ 0.003 | 0.282 $\pm$ 0.008 | 54.7 $\pm$ 0.1 | 60.4 $\pm$ 0.3 | 64.1 $\pm$ 0.5 | 65.0 $\pm$ 1.1 |  |
| BADGE | 0.334 $\pm$ 0.006 | 0.304 $\pm$ 0.003 | 0.282 $\pm$ 0.003 | 0.270 $\pm$ 0.008 | 55.2 $\pm$ 0.5 | 61.1 $\pm$ 0.2 | 65.4 $\pm$ 0.9 | 67.0 $\pm$ 0.7 |  |
|  | Ours | 0.329 $\pm$ 0.009 | 0.300 $\pm$ 0.004 | 0.279 $\pm$ 0.004 | 0.261 $\pm$ 0.006 | 55.6 $\pm$ 0.9 | 61.5 $\pm$ 0.5 | 65.1 $\pm$ 0.4 | 67.0 $\pm$ 0.4 |

Figure: Figure 4: 2-D visualizations of AL performance regarding ECE (x-axis) and Accuracy (y-axis) on CIFAR-10 from Tab. [1](https://arxiv.org/html/2510.03162v1#S5.T1). Our method is closest to the Best performance point (i.e., $100\%$ accuracy and $0.0$ ECE).
Refer to caption: x6.png

In particular, on Fashion (F)-MNIST, we observe a remarkably lower ECE and higher accuracy from our AF when compared to other baselines. For example, ours is at around $0.190$, $0.175$, and $0165$ in ECE, lower than others by more than $0.013$, $0.014$ and $0.010$ at time step $t=\{50,75,100\}$ respectively. Similarly, it has a $78.7\%$, $80.6\%$, and $81.9\%$ accuracy, higher than others by more than $1\%$ at each $t=\{50,75,100\}$ correspondingly. It is also worth noting that the least-confident method is uncalibrated, leading to uninformative query samples and resulting in a similarly poor result to random sampling. Meanwhile, methods that aim to improve uncertainty quality, like BALD and BADGE, can improve the calibration and generalization. This confirms the correlation between high-quality uncertainty estimation and generalization in AL. Another grayscale image dataset is MNIST, at $t=10$, our method also achieves around $0.089$ ECE, significantly better than other baselines by more than $0.02$. This demonstrates that our Alg. [1](https://arxiv.org/html/2510.03162v1#alg1) can help improve the uncertainty estimation quality of the DNN in the AL task. In addition to uncertainty estimation, our method also achieves the highest accuracy with $86.7\%$, illustrating that a calibrated uncertainty sampling method can help to query more informative samples to enhance generalization.

Since MNIST is class-balanced, we next evaluate on SVHN to test performance on an imbalanced class distribution (more results with CIFAR-10-LT and data-augmentation are in Apd. [B.3.2](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS2)). We also observe similar behaviors on this real-world dataset, where our method is notably better calibrated and more accurate than other baselines. For instance, our proposed method achieves $0.090$ and $0.081$ ECE, lower than others by more than $0.006$ at $t=\{75,100\}$. At the same time, it also reaches $88.7\%$ and $90.1\%$, higher than others by more than $0.5\%$.

Regarding a more complex setting like CIFAR-10, although Fig. [4](https://arxiv.org/html/2510.03162v1#S5.F4) shows that all of the methods have difficulty improving in this setting, our proposed AF has a competitive result with BADGE in accuracy and still brings out better performance with $61.5\%$, $65.1\%$, and $67\%$, higher than others by more than $0.7\%$, $0.4\%$, and $0.9\%$ at time $t=\{50,75,100\}$ respectively. Notably, at the same time, it still outperforms other baselines in uncertainty estimation quality with $0.300$, $0.279$, and $0.261$ ECE, lower than all others by more than $0.004$, $0.003$, and $0.009$ correspondingly. This suggests that even when models cannot perfectly classify samples, our proposed AF still can help improve the calibration quality, signaling when it is likely to be true or wrong in the real world.

Finally, we provide results on a larger-scale setting with ImageNet. From Fig. [6](https://arxiv.org/html/2510.03162v1#S5.F6) and Tab. [2](https://arxiv.org/html/2510.03162v1#A2.T2), we observe that our method achieves lower ECE with $0.2868$, $0.2523$, $0.2301$, and $0.2228$ and higher accuracy with $56.16\%$, $59.04\%$, $60.28\%$, and $61.00\%$ at $n_{t}=\{40k,60k,80k,100k\}$ (i.e., $t=\{2,4,6,8\}$) than other query-based baselines. Notably, we also compare with CAMPAL (Yang et al., 2023). It is worth noting that CAMPAL modifies the training strategy by using data augmentation techniques, while our setting does not modify the training strategy. Therefore, we add a new setting where our proposed AF is combined with the data-augmentation strategy in CAMPAL. We observe that this combination also has a better performance in both accuracy and uncertainty estimation quality than CAMPAL${}_{\text{BADGE}}^{\text{CHAMBER}}$, confirming the effectiveness of our AF for trustworthiness in AL.

### 5.3 Ablation studies

Figure: Figure 5: Calibration quality on the unlabeled pool with MNIST. More results are in Tab. [4](https://arxiv.org/html/2510.03162v1#A2.T4) and Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10).
Refer to caption: x7.png

The quality of our calibration error estimator.
Recall our Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) estimates the calibration error per sample on the unlabeled pool. To evaluate this estimator’s quality, we compare it to the estimator where we know the true label on the unlabeled pool of Popordanoska et al. (2022) in Eq. [7](https://arxiv.org/html/2510.03162v1#S3.E7). From Fig. [3](https://arxiv.org/html/2510.03162v1#S3.F3), we observe that when the number of cumulative queried samples increases, the gap between the two estimators decreases correspondingly. This shows that the quality of our estimator in Eq. [9](https://arxiv.org/html/2510.03162v1#S3.E9) improves, and confirms the dependence on the number of queried samples in Thm. [4.1](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem1).

Our uncertainty estimation quality on the unlabeled pool. A calibrated DNN on the unlabeled pool is a necessary condition for a high-quality uncertainty sampling AF in AL. Therefore, to better understand why our proposed AF can bring out a better generalization performance, we compare the calibration on the unlabeled pool. Fig. [6](https://arxiv.org/html/2510.03162v1#S5.F6) shows that our method consistently outperforms other baselines across every $t\in[T]$. This means that our AF helps the model improve its predictive certainty, and we can leverage DNN uncertainty to better query data points to enhance AL performance. Furthermore, this result also confirms that, by selecting the least calibrated samples to query, our designed strategy can help improve calibration on the unlabeled pool as seen in Thm. [4.2](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem2).

Comparison with querying by least-calibrated only. Recall our Alg. [1](https://arxiv.org/html/2510.03162v1#alg1) query by the lexicographic order, prioritizing the least calibrated, then the least certain samples. To test the effectiveness of this lexicographic ordering, we also compare our AF with a method of only querying the least calibrated samples. Fig. [12](https://arxiv.org/html/2510.03162v1#A2.F12) shows that our AF results in a better performance with a lower ECE and higher accuracy, confirming the effectiveness of the lexicographic order during query time.

Frequency of samples selected based on calibration versus uncertainty. Our lexicographic order enables our method to focus on querying samples with calibration errors in early rounds, then shift to uncertainty sampling as calibration improves in later rounds. Specifically, from Tab. [5](https://arxiv.org/html/2510.03162v1#A2.T5), the number of selected samples based on calibration error is high at early rounds. The model can then improve the calibration performance. As a result, in later rounds, the calibration error is lower and more uniform across samples (Fig. [3](https://arxiv.org/html/2510.03162v1#S3.F3), Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10)), resulting in more samples being selected by uncertainty-sampling.

Comparison with post-hoc recalibration. To improve predictive performance with calibration in AL, a simple solution may be to use a post-hoc recalibration step during training after each query time. Hence, we additionally compare with this setting in Fig. [6](https://arxiv.org/html/2510.03162v1#S5.F6) and Fig. [11](https://arxiv.org/html/2510.03162v1#A2.F11). Specifically, Least-confident-TS is the result of a setting where we split $80\%$ of the cumulative data to train and $20\%$ to do post-hoc temperature scaling to recalibrate the trained model (Balanya et al., 2024). Because of the need to split the dataset, this post-hoc method sacrifices samples in training, resulting in a significantly lower accuracy. Meanwhile, the calibration performance depends on model confidence and accuracy. As a result, the ECE of this method on both the unlabeled pool and unseen test datasets is also significantly worse than our method across AL time horizons.

## 6 Related work

The literature on the pool-based AL can be categorized into innovating AF and innovating training loss functions (Werner et al., 2024). In particular, innovating AF focuses on designing a new AF and fixing the training strategy. On the other hand, innovating on the training loss function additionally tries to change the training strategy, such as using data augmentation (Kim et al., 2021; Bengar et al., 2021) or leveraging samples from the unlabeled pool (Lüth et al., 2023; Gao et al., 2020).

In the scope of this paper, we consider the innovating AF setting. The literature on this domain could be sub-categorized into two main approaches: uncertainty-based sampling and diversity-based sampling. Regarding uncertainty-based sampling, its main idea is querying the most uncertain samples from the model, including querying samples that lie closest to the linear decision boundary (Schohn & Cohn, 2000; Balcan et al., 2006) or using the DNN predictive model uncertainty, e.g., softmax layer (least-confident), using its predictive entropy, and using the smallest separation of the top two class
predictions (margin sampling) (Wang & Shang, 2014). Later on, Gal et al. (2017) proposes BALD by using MC-Dropout to improve the DNN uncertainty estimation quality on the unlabeled pool, resulting in generalization improvement. Regarding diversity-based sampling, its main idea is to query the most diverse samples. Common approaches include clustering and selecting unlabeled samples that are the furthest away from all cluster centers (Coreset) (Sener & Savarese, 2018; Geifman & El-Yaniv, 2017), selecting unlabeled samples that are maximally indistinguishable (Gissin & Shalev-Shwartz, 2019). Following this direction, Ash et al. (2020) introduces BADGE by sampling groups of points that are disparate and the high magnitudes when represented in a hallucinated gradient space. This can be seen as a hybrid version of the uncertainty-based and diversity-based.

Although the aforementioned AL techniques have shown promising generalization results, their performance regarding the quality of uncertainty estimation remains in question. Meanwhile, this uncertainty quality is an important aspect of a reliable ML model, especially in the AL setting. Hence, there has been a growing interest in studying the correlation between calibration and AL recently  (Sürer & Wild, 2024; Thomas-Mitchell et al., 2023; Rožanec et al., 2023). Closest to our work is CALICO (Querol et al., 2024), a calibrated framework for AL that jointly trains cross-entropy loss with an energy-based function. Yet, its improvement over Least-conf is minor, as the AF stays the same and the joint training is in a semi-supervised way (see Fig. [12](https://arxiv.org/html/2510.03162v1#A2.F12)). DDU (Mukhoti et al., 2023) also evaluates calibration, but requires an additional hold-out dataset to do post-hoc recalibration techniques. Hence, its performances are close to Least-conf-TS (see Fig. [11](https://arxiv.org/html/2510.03162v1#A2.F11)). In contrast, our method enhances uncertainty estimation and generalization performance without requiring modifications to the loss function or using any hold-out recalibration in training.

Regarding consistent calibration error estimators, Zhang et al. (2020) and Popordanoska et al. (2022) propose estimators but require the true label under the IID assumption. Lately, (Popordanoska et al., 2024) tackles this limitation by introducing a consistent estimator without labels under label-shift. There also exists a covariate-shift version in the domain adaptation (Popordanoska et al., 2023), but without theoretical verification for the proposed estimator. In contrast, our work proposes a provable consistent estimator on the unlabeled pool under the setting of covariate shift in the AL framework.

## 7 Conclusion

AF based on uncertainty sampling has become a standard approach in pool-based AL. Yet, the uncertainty quantification quality derived from the DNN model is often uncalibrated. This leads to the model not only being unreliable on unseen data but also to selecting non-informative samples from the unlabeled pool. Hence, we propose CUSAL, a reliable AF, aiming to enhance both calibration and generalization. Our novel AF uses the lexicographic order for calibration and uncertainty sampling, prioritizing fixing calibration error first by using a kernel calibration estimator to choose the least calibrated samples in the unlabeled pool. Theoretically, we provide the calibration estimator error, expected calibration error bound on the unlabeled pool, and on the unseen test set of our AF. Our proposed method empirically surpasses other baselines by having a lower calibration and generalization error across pool-based AL. With this result, we hope our work can open a new direction for improving trustworthiness in the AL setting. Future work includes tackling the computational efficiency limitation of our kernel estimator and extending it to Large language Models.

## Acknowledgement

This work is supported by an Amazon Research Award, a student fellowship from JHU Applied Physics Laboratory, and a seed grant from JHU Institute of Assured Autonomy.

## References

- Ash et al. (2020)
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.
Deep batch active learning by diverse, uncertain gradient lower bounds.
In *International Conference on Learning Representations*, 2020.
- Balanya et al. (2024)
Sergio A. Balanya, Juan Maroñas, and Daniel Ramos.
Adaptive temperature scaling for robust calibration of deep neural networks.
*Neural Computing and Applications*, 2024.
- Balcan et al. (2006)
Maria-Florina Balcan, Alina Beygelzimer, and John Langford.
Agnostic active learning.
In *Proceedings of the 23rd International Conference on Machine Learning*, 2006.
- Bengar et al. (2021)
Javad Zolfaghari Bengar, Joost van de Weijer, Bartłomiej Twardowski, and Bogdan Raducanu.
Reducing label effort: Self-supervised meets active learning.
*2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)*, 2021.
- Bui & Liu (2024)
Ha Manh Bui and Anqi Liu.
Density-softmax: Efficient test-time model for uncertainty estimation and robustness under distribution shifts.
In *Proceedings of the 41st International Conference on Machine Learning*, 2024.
- Citovsky et al. (2021)
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar.
Batch active learning at scale.
In *Advances in Neural Information Processing Systems*, 2021.
- Cui et al. (2019)
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.
Class-balanced loss based on effective number of samples.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019.
- Dasgupta (2011)
Sanjoy Dasgupta.
Two faces of active learning.
*Theor. Comput. Sci.*, 2011.
- Dawid (1982)
A. P. Dawid.
The well-calibrated bayesian.
*Journal of the American Statistical Association*, 1982.
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *2009 IEEE Conference on Computer Vision and Pattern Recognition*, 2009.
- Duffus (2006)
Dwight Duffus.
Ordered sets by egbert harzheim.
*SIAM Review*, 48:160–163, 01 2006.
doi: 10.2307/20453764.
- Fannjiang et al. (2022)
Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos, Jennifer Listgarten, and Michael I. Jordan.
Conformal prediction under feedback covariate shift for biomolecular design.
*Proceedings of the National Academy of Sciences*, 2022.
- Gal & Ghahramani (2016)
Yarin Gal and Zoubin Ghahramani.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning.
In *Proceedings of The 33rd International Conference on Machine Learning*, 2016.
- Gal et al. (2017)
Yarin Gal, Riashat Islam, and Zoubin Ghahramani.
Deep Bayesian active learning with image data.
In *Proceedings of the 34th International Conference on Machine Learning*, 2017.
- Gao et al. (2020)
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö. Arık, Larry S. Davis, and Tomas Pfister.
Consistency-based semi-supervised active learning: Towards minimizing labeling cost.
In *Computer Vision – ECCV 2020*, 2020.
- Geifman & El-Yaniv (2017)
Yonatan Geifman and Ran El-Yaniv.
Deep active learning over the long tail, 2017.
- Gissin & Shalev-Shwartz (2019)
Daniel Gissin and Shai Shalev-Shwartz.
Discriminative active learning, 2019.
- Guo et al. (2017)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.
On calibration of modern neural networks.
In *Proceedings of the 34th International Conference on Machine Learning*, 2017.
- He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016.
- Hübotter et al. (2024)
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause.
Transductive active learning: Theory and applications.
In *Advances in Neural Information Processing Systems*, 2024.
- Kim et al. (2020)
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun.
Task-aware variational adversarial active learning.
*2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020.
- Kim et al. (2021)
Yoon-Yeong Kim, Kyungwoo Song, JoonHo Jang, and Il-chul Moon.
Lada: Look-ahead data acquisition via augmentation for deep active learning.
In *Advances in Neural Information Processing Systems*, 2021.
- Kirsch et al. (2019)
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal.
Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.
In *Advances in Neural Information Processing Systems*, 2019.
- Krizhevsky & Hinton (2009)
Alex Krizhevsky and Geoffrey Hinton.
Learning multiple layers of features from tiny images, 2009.
- Kuleshov et al. (2018)
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.
Accurate uncertainties for deep learning using calibrated regression.
In *Proceedings of the 35th International Conference on Machine Learning*, 2018.
- Lecun et al. (1998)
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.
Gradient-based learning applied to document recognition.
*Proceedings of the IEEE*, 1998.
- Liu et al. (2015)
Anqi Liu, Lev Reyzin, and Brian Ziebart.
Shift-pessimistic active learning using robust bias-aware prediction.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 2015.
- Liu et al. (2020)
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan.
Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.
In *Advances in Neural Information Processing Systems*, 2020.
- Lüth et al. (2023)
Carsten Tim Lüth, Till J. Bungert, Lukas Klein, and Paul F Jaeger.
Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
- MacKay (1992)
David J. C. MacKay.
Information-based objective functions for active data selection.
*Neural Computation*, 1992.
- Mukhoti et al. (2023)
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal.
Deep deterministic uncertainty: A new simple baseline.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2023.
- Murphy (2023)
Kevin P. Murphy.
*Probabilistic Machine Learning: Advanced Topics*.
MIT Press, 2023.
- Nado et al. (2021)
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran.
Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning.
*arXiv preprint arXiv:2106.04015*, 2021.
- Naeini et al. (2015)
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht.
Obtaining well calibrated probabilities using bayesian binning.
In *Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence*, 2015.
- Nalisnick et al. (2019)
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.
Do deep generative models know what they don’t know?
In *International Conference on Learning Representations*, 2019.
- Netzer et al. (2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.
Reading digits in natural images with unsupervised feature learning.
In *NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011*, 2011.
- Ouimet & Tolosana-Delgado (2022)
Frédéric Ouimet and Raimon Tolosana-Delgado.
Asymptotic properties of dirichlet kernel density estimators.
*Journal of Multivariate Analysis*, 2022.
- Popordanoska et al. (2022)
Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko.
A consistent and differentiable lp canonical calibration error estimator.
In *Advances in Neural Information Processing Systems*, 2022.
- Popordanoska et al. (2023)
Teodora Popordanoska, Aleksei Tiulpin, and Matthew Blaschko.
To trust or not to trust: Assessing calibration error under covariate shift without labels, 2023.
- Popordanoska et al. (2024)
Teodora Popordanoska, Gorjan Radevski, Tinne Tuytelaars, and Matthew B. Blaschko.
LaSCal: Label-shift calibration without target labels.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
- Prinster et al. (2024)
Drew Prinster, Samuel Don Stanton, Anqi Liu, and Suchi Saria.
Conformal validity guarantees exist for any data distribution (and how to find them).
In *Forty-first International Conference on Machine Learning*, 2024.
- Querol et al. (2024)
Lorenzo S. Querol, Hajime Nagahara, and Hideaki Hayashi.
Calico: Confident active learning with integrated calibration.
In *Artificial Neural Networks and Machine Learning – ICANN 2024: 33rd International Conference on Artificial Neural Networks, Lugano, Switzerland, September 17–20, 2024, Proceedings, Part I*, 2024.
- Roth & Small (2006)
Dan Roth and Kevin Small.
Margin-based active learning for structured output spaces.
In *Machine Learning: ECML 2006*, 2006.
- Roy & McCallum (2001)
Nicholas Roy and Andrew McCallum.
Toward optimal active learning through sampling estimation of error reduction.
In *International Conference on Machine Learning*, 2001.
- Rožanec et al. (2023)
Jože Rožanec, Luka Bizjak, Elena Trajkova, Patrik Zajec, Jelle Keizer, Blaž Fortuna, and Dunja Mladenić.
Active learning and novel model calibration measurements for automated visual inspection in manufacturing.
*Journal of Intelligent Manufacturing*, 2023.
- Schohn & Cohn (2000)
Greg Schohn and David Cohn.
Less is more: Active learning with support vector machines.
In *Proceedings of the Seventeenth International Conference on Machine Learning*, 2000.
- Sener & Savarese (2018)
Ozan Sener and Silvio Savarese.
Active learning for convolutional neural networks: A core-set approach.
In *International Conference on Learning Representations*, 2018.
- Settles (2009)
Burr Settles.
Active learning literature survey.
Computer sciences technical report, University of Wisconsin–Madison, 2009.
- Shannon (1948)
Claude Elwood Shannon.
A mathematical theory of communication.
*The Bell System Technical Journal*, 1948.
- Song et al. (2019)
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach.
Distribution calibration for regression.
In *Proceedings of the 36th International Conference on Machine Learning*, 2019.
- Sürer & Wild (2024)
Özge Sürer and Stefan M. Wild.
An active learning performance model for parallel bayesian calibration of expensive simulations.
In *NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty*, 2024.
- Thomas-Mitchell et al. (2023)
Adam Thomas-Mitchell, Glenn Hawe, and Paul L A Popelier.
Calibration of uncertainty in the active learning of machine learning force fields.
*Machine Learning: Science and Technology*, 2023.
- Tran et al. (2022)
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan.
Plex: Towards reliability using pretrained large model extensions, 2022.
- Wang & Shang (2014)
Dan Wang and Yi Shang.
A new active labeling method for deep learning.
In *2014 International Joint Conference on Neural Networks (IJCNN)*, 2014.
- Werner et al. (2024)
Thorben Werner, Johannes Burchert, Maximilian Stubbemann, and Lars Schmidt-Thieme.
A cross-domain benchmark for active learning.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.
- Xiao et al. (2017)
Han Xiao, Kashif Rasul, and Roland Vollgraf.
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
- Yang et al. (2023)
Jianan Yang, Haobo Wang, Sai Wu, Gang Chen, and Junbo Zhao.
Towards controlled data augmentations for active learning.
In *Proceedings of the 40th International Conference on Machine Learning*, 2023.
- Zhang et al. (2020)
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han.
Mix-n-match : Ensemble and compositional methods for uncertainty calibration in deep learning.
In *Proceedings of the 37th International Conference on Machine Learning*, 2020.

## Calibrated Uncertainty Sampling for Active Learning (Supplementary Material)

In this supplementary material, we collect proofs and remaining materials deferred from the main paper. In Appendix [A](https://arxiv.org/html/2510.03162v1#A1), we provide the proofs for all our theoretical results, including: proof of Theorem [4.1](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem1) in Appendix [A.1](https://arxiv.org/html/2510.03162v1#A1.SS1); proof of Theorem [4.2](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem2) in Appendix [A.2](https://arxiv.org/html/2510.03162v1#A1.SS2). In Appendix [B](https://arxiv.org/html/2510.03162v1#A2), we provide additional information about our experiments, including: sufficient details about experimental settings in Appendix [B.1](https://arxiv.org/html/2510.03162v1#A2.SS1); demo code in Appendix [B.2](https://arxiv.org/html/2510.03162v1#A2.SS2); additional results in Appendix [B.3](https://arxiv.org/html/2510.03162v1#A2.SS3) with detailed comparison to other baselines in Appendix [B.3.1](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS1); evaluation on an imbalanced benchmark in [B.3.2](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS2); calibration error on the unlabeled pool in Appendix [B.3.3](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS3); additional ablation studies in Appendix [B.3.4](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS4). Finally, the source code to reproduce our results is available in the zipped file of this supplementary material.

## Appendix A Proofs

### A.1 Proof of Theorem 4.1

###### Proof.

Recall the covariate shift of AL existing between the cumulative labeled data $S$ and samples from the unlabeled pool $U$ in Section [4](https://arxiv.org/html/2510.03162v1#S4), i.e.,

$$ $\displaystyle S\sim\mathbb{P}(X)\mathbb{P}(Y|X),\quad U\sim\tilde{\mathbb{P}}(X_{S})\mathbb{P}(Y|X).$ (15) $$

For the conditional expectation, we have

$$ $\displaystyle\mathbb{E}_{\pi_{U}}\left[Y|h(X)=h(x)\right]$ $\displaystyle=\sum_{y_{k}\in\mathcal{Y}}y_{k}\frac{p_{u}(Y=y_{k},h(X)=h(x))}{p_{u}(h(X)=h(x))}$ (16) $\displaystyle=\sum_{y_{k}\in\mathcal{Y}}y_{k}\frac{p_{u}(Y=y_{k}|h(X)=h(x))p_{u}(h(X)=h(x))}{p_{u}(h(X)=h(x))}.$ (17) $$

By the covariate shift in Equation [15](https://arxiv.org/html/2510.03162v1#A1.E15), thus we get

$$ $\displaystyle\mathbb{E}_{\pi_{U}}\left[Y|h(X)=h(x)\right]$ $\displaystyle=\sum_{y_{k}\in\mathcal{Y}}y_{k}\frac{p_{s}(Y=y_{k}|h(X)=h(x))p_{s}(h(X)=h(x))}{p_{s}(h(X)=h(x))}$ (18) $\displaystyle=\sum_{y_{k}\in\mathcal{Y}}y_{k}\frac{p_{s}(h(X)=h(x)|Y=y_{k})p_{s}(Y=y_{k})}{p_{s}(h(X)=h(x))}.$ (19) $$

So, applying the result of Popordanoska et al. (2022), we obtain that when $p_{h(x)}h(x)$ is Lipschitz continuous over the interior of the simplex, $\exists$ a kernel $k$ s.t.

$$ $\displaystyle\operatorname*{plim}_{n_{t}\rightarrow\infty}\frac{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)y_{i}}{\sum_{i=1}^{n_{t}}k\left(h(x);h(x_{i})\right)}$ $\displaystyle=\sum_{y_{k}\in\mathcal{Y}}y_{k}\frac{p_{s}(h(X)=h(x)|Y=y_{k})p_{s}(Y=y_{k})}{p_{s}(h(X)=h(x))}$ (20) $\displaystyle=\mathbb{E}_{\pi_{U}}[Y|h(x)],$ (21) $$

i.e., our estimator in Equation [9](https://arxiv.org/html/2510.03162v1#S3.E9) is a point-wise consistent estimator under AL with covariate shift.

On the other hand, let us denote $s:=h(x)$, $\hat{f}_{n_{t},b}(s):=\mathbb{E}_{\pi_{U}}\left[\widehat{Y|h(x)}\right]$, and $f(s):=\mathbb{E}_{\pi_{U}}\left[Y|h(x)\right]$, then, we have

$$ $\displaystyle\mathbb{E}\left[\left|\mathbb{E}_{\pi_{U}}\left[\widehat{Y|h(x)}\right]-\mathbb{E}_{\pi_{U}}\left[Y|h(x)\right]\right|^{2}\right]$ $\displaystyle=\mathbb{E}\left[\left|\hat{f}_{n_{t},b}(s)-f(s)\right|^{2}\right]$ (22) $\displaystyle=Var\left(\hat{f}_{n_{t},b}(s)\right)+\left[Bias\left(\hat{f}_{n_{t},b}(s)\right)\right]^{2}.$ (23) $$

For $K$ classes, bandwidth $b$, with $k(\cdot)$ is a Dirichlet kernel in the form of Equation [10](https://arxiv.org/html/2510.03162v1#S3.E10), following the result of Ouimet & Tolosana-Delgado (2022), we have the bias is as follows

$$ $\displaystyle Bias\left(\hat{f}_{n_{t},b}(s)\right)=bg(s)+\mathcal{O}(b),$ (24) $$

where

$$ $\displaystyle g(s):=\sum_{i\in[K]}\left(1-(K+1)s_{i}\right)\frac{\partial}{\partial s_{i}}f(s)+\frac{1}{2}\sum_{i,j\in[K]}s_{i}\left(\mathbb{I}_{i=j}-s_{j}\right)\frac{\partial^{2}}{\partial s_{i}\partial s_{j}}f(s).$ (25) $$

And the variance is as follows

$$ $\displaystyle Var\left(\hat{f}_{n_{t},b}(s)\right)=n_{t}^{-1}A_{b}(s)\left(f(s)+\mathcal{O}(b^{1/2})\right)-\mathcal{O}(n_{t}^{-1}),$ (26) $$

where

$$ $\displaystyle A_{b}(s):=\begin{cases}b^{-K/2}\psi(s)(1+\mathcal{O}_{s}(b)),&\text{ if }s_{i}/b\rightarrow\infty,\forall i\in[K],\\ &\text{ and }(1-||s||_{1})/b\rightarrow\infty,\\ \\ b^{-(K+|\mathcal{J}|)/2}\psi_{\mathcal{J}}(s)\prod_{i\in\mathcal{J}}\frac{\Gamma(2k_{i}+1)}{2^{2k_{i}+1}\Gamma^{2}(k_{i}+1)}\left(1+\mathcal{O}_{k,s}(b)\right),&\text{ if }s_{i}/b\rightarrow k_{i},\forall i\in\mathcal{J},\\ &s_{i}/b\rightarrow\infty,\forall i\in[K]\setminus\mathcal{J},\\ &\text{ and }(1-||s||_{1})/b\rightarrow\infty,\end{cases}$ (27) $$

where

$$ $\displaystyle\psi(s):=\psi(s)\text{ and }\psi_{\mathcal{J}}(s):=\left[(4\pi)^{K-|\mathcal{J}|}(1-||s||_{1})\prod_{i\in[K]\setminus\mathcal{J}}s_{i}\right]^{-1/2},$ (28) $$

for every subset of indices $\mathcal{J}\in[K]$. Plug this bias and variance result into Equation [22](https://arxiv.org/html/2510.03162v1#A1.E22), we obtain the mean square error of our estimator in Equation [7](https://arxiv.org/html/2510.03162v1#S3.E7) is bounded by

$$ $\displaystyle\mathbb{E}\left[\left|\hat{f}_{n_{t},b}(s)-f(s)\right|^{2}\right]$ $\displaystyle=n_{t}^{-1}A_{b}(s)\left(f(s)+\mathcal{O}(b^{1/2})\right)-\mathcal{O}(n_{t}^{-1})+b^{2}g^{2}(s)+\mathcal{O}(b^{2})$ (29) $\displaystyle=n_{t}^{-1}b^{-K/2}\left(\psi(s)f(s)+\mathcal{O}_{s}(b^{3/2})\right)-\mathcal{O}(n_{t}^{-1})+b^{2}g^{2}(s)+\mathcal{O}(b^{2})$ (30) $\displaystyle=n_{t}^{-1}b^{-K/2}\psi(s)f(s)+b^{2}g^{2}(s)+\mathcal{O}_{s}(n_{t}^{-1}b^{\frac{-K+1}{2}})+\mathcal{O}(b^{2})-\mathcal{O}(n_{t}^{-1})$ (31) $\displaystyle\leq\mathcal{O}(n_{t}^{-1}b^{\frac{-K+1}{2}})+\mathcal{O}(b^{2})$ (32) $$

of Theorem [4.1](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem1).
∎

###### Proof.

### A.2 Proof of Theorem 4.2

###### Proof.

Given our setting in Section [2](https://arxiv.org/html/2510.03162v1#S2), for every round $t\in[T]$, we have the union of $n_{t}$ samples from the queried set and $m_{t}$ samples from the unlabeled pool is fixed. As discussed in Section [4](https://arxiv.org/html/2510.03162v1#S4), this total $m_{t}+n_{t}$ samples, i.e., $\{x_{i},y_{i}\}_{i\in[m_{t}+n_{t}]}$ are also drawn IID from $\mathbb{P}(X,Y)$. So, recall Hoeffding’s inequality, i.e., let $Z_{1},\cdots,Z_{n}$ be independent bounded random variables with $Z_{i}\in[a,b]$ for all $i$, where $-\infty<a<b<\infty$. Then

$$ $\displaystyle\mathbb{P}\left(\frac{1}{n}\sum_{i=1}^{n}\left(Z_{i}-\mathbb{E}[Z_{i}]\right)\geq t\right)\leq\exp\left(\frac{-2nt^{2}}{(b-a)^{2}}\right).$ (33) $$

Firstly, following conditions in the theorem, by the selection of our AF, i.e., selecting $k_{t}$ highest calibration error samples from $m_{t}+k_{t}$ samples in the unlabeled pool, we have

$$ $\displaystyle\frac{1}{m_{t}}\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}$ $\displaystyle\leq\frac{1}{m_{t}+k_{t}}\sum_{i=1}^{m_{t}+k_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}$ (34) $\displaystyle\leq\frac{1}{k_{t}}\sum_{j=1}^{k_{t}}\left\|\mathbb{E}\left[Y|h(x_{j})\right]-h(x_{j})\right\|_{p}^{p}.$ (35) $$

Combining with the assumption $\frac{1}{k_{t}}\sum_{j=1}^{k_{t}}\left\|\mathbb{E}\left[Y|h(x_{j})\right]-h(x_{j})\right\|_{p}^{p}\leq\epsilon$, we obtain the expected calibration error on the unlabeled pool of Algorithm [1](https://arxiv.org/html/2510.03162v1#alg1) is also bounded by

$$ $\displaystyle\frac{1}{m_{t}}\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\leq\frac{1}{m_{t}+k_{t}}\sum_{i=1}^{m_{t}+k_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\leq\epsilon.$ (36) $$

On the other hand, since the empirical calibration error and the expected error on the unseen test data are

$$ $\displaystyle\frac{1}{m_{t}+n_{t}}\sum_{i=1}^{m_{t}+n_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\quad\text{and}\quad\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}\text{, respectively}.$ (37) $$

So, applying Hoeffding’s inequality, we have

$$ $\displaystyle\mathbb{P}\left(\left|\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}-\frac{1}{m_{t}+n_{t}}\sum_{i=1}^{m_{t}+n_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\right|\geq t\right)$ $\displaystyle\leq 2\exp\left(\frac{-2(n_{t}+m_{t})t^{2}}{L^{2}}\right).$ (38) $$

Let $\delta=2\exp\left(\frac{-2(m_{t}+n_{t})\cdot t^{2}}{L^{2}}\right)$, equivalently $t=\sqrt{\frac{L^{2}\log(2/\delta)}{2(m_{t}+n_{t})}}$, i.e.,

$$ $\displaystyle\mathbb{P}\left(\left|\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}-\frac{1}{m_{t}+n_{t}}\sum_{i=1}^{m_{t}+n_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}\right|\leq\sqrt{\frac{L^{2}\log(2/\delta)}{2(m_{t}+n_{t})}}\right)$ $\displaystyle\geq 1-\delta.$ (39) $$

Due to

$$ $\displaystyle\frac{1}{m_{t}+n_{t}}\sum_{i=1}^{m_{t}+n_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}$ (40) $\displaystyle=\frac{1}{m_{t}+n_{t}}\left[\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}+\sum_{j=1}^{n_{t}}\left\|\mathbb{E}\left[Y|h(x_{j})\right]-h(x_{j})\right\|_{p}^{p}\right]$ (41) $\displaystyle=\frac{1}{m_{t}+n_{t}}\left[\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}+\sum_{i=1}^{t}\sum_{j=1}^{k_{t}}\left\|\mathbb{E}\left[Y|h(x_{j})\right]-h(x_{j})\right\|_{p}^{p}\right]$ (42) $\displaystyle\leq\frac{1}{m_{t}+n_{t}}\left[\sum_{i=1}^{m_{t}}\left\|\mathbb{E}\left[Y|h(x_{i})\right]-h(x_{i})\right\|_{p}^{p}+\epsilon\sum_{i=1}^{t}k_{t}\right]$ (43) $\displaystyle\leq\frac{m_{t}\epsilon+n_{t}\epsilon}{m_{t}+n_{t}}\left(\text{by Equation~\ref{proof:thm:error_bound:eq1} and $\sum_{i=1}^{t}k_{t}\leq n_{t}$}\right),$ (44) $$

we obtain

$$ $\displaystyle\mathbb{P}\left(\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}\leq\epsilon+\sqrt{\frac{L^{2}\log(2/\delta)}{2(m_{t}+n_{t})}}\right)\geq 1-\delta,$ (45) $$

i.e., with probability at least $1-\delta$, the expected calibration error on the unseen data of Algorithm [1](https://arxiv.org/html/2510.03162v1#alg1) is bounded by

$$ $\displaystyle\mathbb{E}_{x\sim\mathbb{P}(X)}\left\|\mathbb{E}\left[Y|h(x)\right]-h(x)\right\|_{p}^{p}\leq\epsilon+\sqrt{\frac{L^{2}\log(2/\delta)}{2(m_{t}+n_{t})}}$ (46) $$

of Theorem [4.2](https://arxiv.org/html/2510.03162v1#S4.Thmtheorem2).
∎

###### Proof.

## Appendix B Experimental Details

### B.1 Experimental settings

AL hyper-parameter settings. We set the number of warm-up datapoints $n_{0}=20$ on MNIST & F-MNIST (Gal et al., 2017), $n_{0}=500$ on SVHN, $n_{0}=1000$ on CIFAR-10 & CIFAR-10-LT (Werner et al., 2024), and $n_{0}=30000$ on ImageNet. Following Gal et al. (2017). All warm-up datasets are sampled randomly but balanced (i.e., the number of samples per class is equal), except the imbalanced CIFAR-10-LT experiments in Apd. [B.3.2](https://arxiv.org/html/2510.03162v1#A2.SS3.SSS2). For feasible computation times, we set the number of time horizons (i.e., AL rounds) $T=8$ on ImageNet, $T=40$ on MNIST, and $T=100$ on other datasets. For each round $t\in[T]$, we set the number of queried samples $k=10$ on MNIST & F-MNIST, $k=50$ on SVHN, $k=100$ on CIFAR-10 & CIFAR-10-LT, and $k=10000$ on ImageNet.

Dataset & other hyper-parameters.
We deploy the models on six datasets, including the MNIST (& Fashion-MNIST) dataset with $d=28\times 28$ digit handwriting (& Zalando’s article) images in $K=10$ classes; the SVHN with $d=32\times 32\times 3$ house numbers images in $K=10$ classes; the CIFAR-10 (& CIFAR-10-LT) dataset with $d=32\times 32\times 3$ images in $K=10$ classes; the ImageNet (ILSVRC 2012) dataset with $d=244\times 244\times 3$ image dimensions in $K=1000$ classes. Regarding the model architectures and hyper-parameters settings, we use MNIST-Net for MNIST (Lecun et al., 1998), GarmentClassifer for Fashion-MNIST (Xiao et al., 2017), RestNet-18 (He et al., 2016) for SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky & Hinton, 2009), and CIFAR-10-LT (Cui et al., 2019), and RestNet-50 (He et al., 2016) for ImageNet (Deng et al., 2009). We use the Adam optimizer with the Cross-entropy loss, learning rate equals $0.001$, batch sizes equal $128$, the number of training epochs equals $30$ on MNIST, SVHN, CIFAR-10, CIFAR-10-LT, and ImageNet, $60$ on Fashion-MNIST. We only normalize data in the data processing step, except in the data-augmentation experiments with CAMPAL (Yang et al., 2023). To evaluate models, we use accuracy and ECE (bin size equals $10$) metrics. Regarding our calibration error estimator hyper-parameters in Equation [9](https://arxiv.org/html/2510.03162v1#S3.E9), we use the absolute norm $p=1$ and set the bandwidth of the Dirichlet kernel $b=0.001$.

Baseline details: Regarding our baseline comparison in the main paper, we compare with other AF (i.e., query-based) methods and follow (Lüth et al., 2023) to select the baselines. To the best of our knowledge, Lüth et al. (2023); Werner et al. (2024) have shown that the following baselines are still state-of-the-art across query-based methods in AL, including:

- •
Random: queries unlabeled samples randomly from the unlabeled pool, i.e., $x_{t}^{*}=\operatorname*{arg\,max}_{x\in U_{t}}A(x;\theta_{t-1})=\mathbb{U}$, where $\mathbb{U}$ represents the uniform distribution.
- •
Least-confident (Roy & McCallum, 2001): queries unlabeled samples from the unlabeled pool by the least confident (i.e., most uncertain) samples of the DNN, i.e., $x_{t}^{*}=\operatorname*{arg\,max}_{x\in U_{t}}\left(1-\max_{y\in[K]}[h(x)]_{y}\right)$.
- •
Least-confident-TS (Guo et al., 2017): queries similar to the Least-confident strategy. However, the DNN is calibrated with an additional hold-out validation dataset by the temperature scaling technique.
- •
Rand-Entropy: is a two-stage-based by querying $k_{m}$ samples by using Random, then selecting $k_{t}$ with the highest predictive entropy from $k_{m}$ samples.
- •
Margin (Wang & Shang, 2014): queries unlabeled samples that are closest to the decision boundary (the ”margin”) of the trained classifier according to a distance function between differences of two most confident classes.
- •
Cluster-Margin (Citovsky et al., 2021): is a two-stage-based and an extension of Margin by querying $k_{m}$ samples by using the margin, then selecting $k_{t}$ from
$k_{m}$ samples by a cluster-based approach to improve diversity.
- •
BALD (Gal et al., 2017): queries by DNN uncertainty, but the model uncertainty is obtained by sampling the probabilistic model’s output with Monte-Carlo dropout sampling through the lens of the Bayesian view (Gal & Ghahramani, 2016).
- •
BatchBALD (Kirsch et al., 2019): is an extension of BALD by jointly scoring points by estimating the
mutual information between a joint of data points and the model parameters.
- •
Coreset (Sener & Savarese, 2018): queries by diversity, i.e., unlabeled samples that are the furthest away from all cluster centers. The clusters are obtained by applying K-Means on a semantically meaningful space with encoded data from the DNN classifier.
- •
BADGE (Ash et al., 2020): queries by incorporating both DNN uncertainty and sample diversity of every selected batch. In particular, it employs a variant of K-Means to select disparate groups of points. Additionally, it uses gradient embeddings of unlabeled samples to query samples with the highest magnitude when represented in a hallucinated gradient space, i.e., samples that the model is expected to change much to improve its performance.

Source code and computing systems.
Our source code includes the dataset scripts, setup for the environment, and our provided code (details in README.md). We run our code on a single GPU: NVIDIA RTX A6000-49140MiB with 8-CPUs: AMD Ryzen Threadripper 3960X 24-Core with 8GB RAM per each and require 160GB available disk space for storage.

### B.2 Demo notebook code for Algorithm 1

| [⬇](data:text/plain;base64,CmltcG9ydCB0b3JjaApccGFyI0VzdGltYXRlIHRoZSBjYWxpYnJhdGlvbiBlcnJvciBvZiBlYWNoIHNhbXBsZSBieSBFcS45LgpkZWYgZ2V0X3JhdGlvX2Nhbm9uaWNhbF9wZXJfc2FtcGxlcyhmLCB5LCBiYW5kd2lkdGgsIHAgPSAxKToKbG9nX2tlcm4gPSBnZXRfa2VybmVsKGYsIGJhbmR3aWR0aCwgZGV2aWNlKQprZXJuID0gdG9yY2guZXhwKGxvZ19rZXJuKQp5X29uZWhvdCA9IG5uLmZ1bmN0aW9uYWwub25lX2hvdCh5LCBudW1fY2xhc3Nlcz1mLnNoYXBlWzFdKQprZXJuX3lfc3BsaXRzID0ga2Vyblt5LnNoYXBlWzBdOiwgOnkuc2hhcGVbMF1dCmtlcm5feSA9IHRvcmNoLm1hdG11bChrZXJuX3lfc3BsaXRzLCB5X29uZWhvdCkKZGVuID0gdG9yY2guc3VtKGtlcm5feV9zcGxpdHMsIGRpbT0xKQpkZW4gPSB0b3JjaC5jbGFtcChkZW4sIG1pbj0xZS0xMCkKcmF0aW8gPSBrZXJuX3kgLyBkZW4udW5zcXVlZXplKC0xKQpjZV9wZXJfc2FtcGxlcyA9IHRvcmNoLnN1bSh0b3JjaC5hYnMocmF0aW8gLSBmW3kuc2hhcGVbMF06XSkqKnAsIGRpbT0xKQpyZXR1cm4gY2VfcGVyX3NhbXBsZXMKXHBhciNTZWxlY3QgdG9wLWsgc2FtcGxlcyBieSB0aGUgbGV4aWNvZ3JhcGhpYyBvcmRlciBpbiBFcS4xMi4KZGVmIHNhbXBsaW5nKG1vZGVsLCBwb29sX2xvYWRlciwgc2VsZWN0X3NhbXBsZXMsIHRyYWlubG9hZGVyKToKbW9kZWwuZXZhbCgpCnRyYWluX291dHB1dHMsIHRyYWluX3RhcmdldHMgPSBbXSwgW10Kd2l0aCB0b3JjaC5ub19ncmFkKCk6CmZvciBkYXRhLCB0YXJnZXQgaW4gdHJhaW4gbG9hZGVyOgpvdXRwdXQgPSBtb2RlbChkYXRhKQpvdXRwdXQgPSB0b3JjaC5zb2Z0bWF4KG91dHB1dCwgZGltPTEpCnRyYWluX291dHB1dHMgPSB0b3JjaC5jYXQoKHRyYWluX291dHB1dHMsIG91dHB1dCksIDApCnRyYWluX3RhcmdldHMgPSB0b3JjaC5jYXQoKHRyYWluX3RhcmdldHMsIHRhcmdldCksIDApCm91dHB1dHMgPSBbXQp3aXRoIHRvcmNoLm5vX2dyYWQoKToKZm9yIGRhdGEsIHRhcmdldCBpbiBwb29sX2xvYWRlcjoKb3V0cHV0ID0gbW9kZWwoZGF0YSkKb3V0cHV0ID0gdG9yY2guc29mdG1heChvdXRwdXQsIGRpbT0xKQpvdXRwdXRzID0gdG9yY2guY2F0KChvdXRwdXRzLCBvdXRwdXQpLCAwKQpccGFyaW5wdXRfZiA9IHRvcmNoLmNhdCgodHJhaW5fb3V0cHV0cywgb3V0cHV0c18xKSwgMCkKZWNlID0gZ2V0X3JhdGlvX2Nhbm9uaWNhbF9wZXJfc2FtcGxlcyhpbnB1dF9mLHRyYWluX3RhcmdldHMsYmFuZHdpZHRoPTAuMDAxKQpjb25mID0gb3V0cHV0cy5tYXgoMSkudmFsdWVzLm51bXB5KCkKb3V0ID0gbnAuY29sdW1uX3N0YWNrKChjb25mLCBlY2UpKQpyZXR1cm4gbnAubGV4c29ydCgob3V0WzosMF0sLW91dFs6LDFdKSlbOnNlbGVjdF9zYW1wbGVzXQpccGFyaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIgpuZXQgPSBNTklTVF9OZXQoKQpwb29sX2RhdGEsIHBvb2xfbGFiZWxzID0gbnAubG9hZChkYXRhX3BhdGgpCmlkeHNfdW5sYWJlbGVkID0gc2FtcGxpbmdfYmFsYW5jZShwb29sX2xhYmVscykKZm9yIHQgaW4gcmFuZ2UoVCk6CnNlbGVjdGVkX2RhdGEgPSBwb29sX2RhdGFbaWR4c191bmxhYmVsZWRdCnNlbGVjdGVkX2xhYmVscyA9IHBvb2xfbGFiZWxzW2lkeHNfdW5sYWJlbGVkXQpwb29sX2RhdGEgPSBucC5kZWxldGUocG9vbF9kYXRhLCBpZHhzX3VubGFiZWxlZCkKcG9vbF9sYWJlbHMgPSBucC5kZWxldGUocG9vbF9sYWJlbHMsIGlkeHNfdW5sYWJlbGVkKQp0cmFpbmxvYWRlciA9IERhdGFMb2FkZXIoRGF0YXNldChzZWxlY3RlZF9kYXRhLCBzZWxlY3RlZF9sYWJlbHMpKQpjcml0ZXJpb24gPSBubi5Dcm9zc0VudHJvcHlMb3NzKCkKb3B0aW1pemVyID0gb3B0aW0uQWRhbShuZXQucGFyYW1ldGVycygpKQpmb3IgZXBvY2ggaW4gcmFuZ2UodHJhaW5fZXBvY2hzKToKdHJhaW4obmV0LCB0cmFpbmxvYWRlciwgY3JpdGVyaW9uLCBvcHRpbWl6ZXIpCnBvb2xfbG9hZGVyID0gdG9yY2gudXRpbHMuZGF0YS5EYXRhTG9hZGVyKERhdGFzZXQocG9vbF9kYXRhLCBwb29sX2xhYmVscykpCiNRdWVyeSB1bmxhYmVsZWQgcG9vbCBzYW1wbGVzIGJ5IHVzaW5nIG91ciBjYWxpYnJhdGVkIHVuY2VydGFpbnR5IHNhbXBsaW5nLgppZHhzX3VubGFiZWxlZCA9IHNhbXBsaW5nKG5ldCwgcG9vbF9sb2FkZXIsIHNlbGVjdF9zYW1wbGVzLCB0cmFpbmxvYWRlcikKdGVzdF9hY2MsIHRlc3RfZWNlID0gdGVzdF9tb2RlbChuZXQsIGRldmljZSwgdGVzdGxvYWRlcikK)<br>import torch<br>\par#Estimate the calibration error of each sample by Eq.9.<br>def get_ratio_canonical_per_samples(f, y, bandwidth, p = 1):<br>log_kern = get_kernel(f, bandwidth, device)<br>kern = torch.exp(log_kern)<br>y_onehot = nn.functional.one_hot(y, num_classes=f.shape[1])<br>kern_y_splits = kern[y.shape[0]:, :y.shape[0]]<br>kern_y = torch.matmul(kern_y_splits, y_onehot)<br>den = torch.sum(kern_y_splits, dim=1)<br>den = torch.clamp(den, min=1e-10)<br>ratio = kern_y / den.unsqueeze(-1)<br>ce_per_samples = torch.sum(torch.abs(ratio - f[y.shape[0]:])**p, dim=1)<br>return ce_per_samples<br>\par#Select top-k samples by the lexicographic order in Eq.12.<br>def sampling(model, pool_loader, select_samples, trainloader):<br>model.eval()<br>train_outputs, train_targets = [], []<br>with torch.no_grad():<br>for data, target in train loader:<br>output = model(data)<br>output = torch.softmax(output, dim=1)<br>train_outputs = torch.cat((train_outputs, output), 0)<br>train_targets = torch.cat((train_targets, target), 0)<br>outputs = []<br>with torch.no_grad():<br>for data, target in pool_loader:<br>output = model(data)<br>output = torch.softmax(output, dim=1)<br>outputs = torch.cat((outputs, output), 0)<br>\parinput_f = torch.cat((train_outputs, outputs_1), 0)<br>ece = get_ratio_canonical_per_samples(input_f,train_targets,bandwidth=0.001)<br>conf = outputs.max(1).values.numpy()<br>out = np.column_stack((conf, ece))<br>return np.lexsort((out[:,0],-out[:,1]))[:select_samples]<br>\parif __name__ == ”__main__”<br>net = MNIST_Net()<br>pool_data, pool_labels = np.load(data_path)<br>idxs_unlabeled = sampling_balance(pool_labels)<br>for t in range(T):<br>selected_data = pool_data[idxs_unlabeled]<br>selected_labels = pool_labels[idxs_unlabeled]<br>pool_data = np.delete(pool_data, idxs_unlabeled)<br>pool_labels = np.delete(pool_labels, idxs_unlabeled)<br>trainloader = DataLoader(Dataset(selected_data, selected_labels))<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(net.parameters())<br>for epoch in range(train_epochs):<br>train(net, trainloader, criterion, optimizer)<br>pool_loader = torch.utils.data.DataLoader(Dataset(pool_data, pool_labels))<br>#Query unlabeled pool samples by using our calibrated uncertainty sampling.<br>idxs_unlabeled = sampling(net, pool_loader, select_samples, trainloader)<br>test_acc, test_ece = test_model(net, device, testloader) |
| --- |

### B.3 Additional results

#### B.3.1 Detailed comparison

Figure: Figure 7: Calibration error (lower is better) and accuracy (higher is better) comparison with different baselines across query times $t\in[T]$ on the test set with different datasets and model architectures. Intervals for each line in the graph depict our results across 10 runs. Table details are in Tab. [1](https://arxiv.org/html/2510.03162v1#S5.T1). Difference details are in Fig. [8](https://arxiv.org/html/2510.03162v1#A2.F8).
Refer to caption: x9.png

Figure: Figure 8: Plot the difference between a method and random selection, where each line represents the results value of the method minus the results value of the random selection baseline from Tab. [1](https://arxiv.org/html/2510.03162v1#S5.T1) and Fig. [7](https://arxiv.org/html/2510.03162v1#A2.F7). Our method consistently outperforms other baselines at almost every time step $t\in[T]$ across different settings by lower ECE and higher accuracy.
Refer to caption: x10.png

**Table 2: Calibration error (lower is better) and accuracy (higher is better) comparison with different baselines across query times $t\in\{T/4,T/2,3T/4,T\}$, where $T=8$ on ImageNet. Scores are reported by mean $\pm$ standard deviation from 10 runs. Gray rows indicate our proposed method. Best scores with the significant test are marked in bold. Figure details are in Fig. [6](https://arxiv.org/html/2510.03162v1#S5.F6). Our algorithm consistently outperforms other AF baselines in this large-scale setting. We additionally compare with the CAMPAL (i.e., CAMPAL${}_{\text{BADGE}}^{\text{CHAMBER}}$) (Yang et al., 2023). The combination of data-augmentation techniques with CAMPAL in training and our proposed AF in query time achieves the best performance.**
| Method | Expected Calibration Error ($\downarrow$) | Accuracy ($\uparrow$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ | $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ |  |
| Random | 0.3168 $\pm$ 0.006 | 0.2732 $\pm$ 0.005 | 0.2492 $\pm$ 0.004 | 0.2402 $\pm$ 0.003 | 55.68 $\pm$ 0.3 | 57.80 $\pm$ 0.2 | 59.11 $\pm$ 0.2 | 60.06 $\pm$ 0.2 |
| Least-conf | 0.2995 $\pm$ 0.005 | 0.2620 $\pm$ 0.005 | 0.2412 $\pm$ 0.004 | 0.2348 $\pm$ 0.003 | 55.86 $\pm$ 0.2 | 58.40 $\pm$ 0.2 | 59.78 $\pm$ 0.2 | 60.66 $\pm$ 0.2 |
| Rand-Entropy | 0.2988 $\pm$ 0.006 | 0.2613 $\pm$ 0.005 | 0.2406 $\pm$ 0.004 | 0.2341 $\pm$ 0.003 | 55.87 $\pm$ 0.3 | 58.44 $\pm$ 0.2 | 59.81 $\pm$ 0.2 | 60.68 $\pm$ 0.2 |
| Margin | 0.2969 $\pm$ 0.005 | 0.2599 $\pm$ 0.004 | 0.2379 $\pm$ 0.003 | 0.2297 $\pm$ 0.002 | 55.99 $\pm$ 0.2 | 58.60 $\pm$ 0.2 | 59.90 $\pm$ 0.1 | 60.68 $\pm$ 0.1 |
| BALD | 0.3018 $\pm$ 0.006 | 0.2628 $\pm$ 0.005 | 0.2397 $\pm$ 0.004 | 0.2315 $\pm$ 0.003 | 55.92 $\pm$ 0.3 | 58.42 $\pm$ 0.2 | 59.70 $\pm$ 0.2 | 60.53 $\pm$ 0.2 |
| BatchBALD | 0.3022 $\pm$ 0.005 | 0.2623 $\pm$ 0.005 | 0.2388 $\pm$ 0.003 | 0.2302 $\pm$ 0.003 | 55.95 $\pm$ 0.3 | 58.46 $\pm$ 0.2 | 59.79 $\pm$ 0.2 | 60.66 $\pm$ 0.2 |
| Coreset | 0.3082 $\pm$ 0.005 | 0.2676 $\pm$ 0.004 | 0.2452 $\pm$ 0.003 | 0.2375 $\pm$ 0.003 | 55.77 $\pm$ 0.2 | 58.10 $\pm$ 0.2 | 59.45 $\pm$ 0.2 | 60.36 $\pm$ 0.2 |
| Cluster-Margin | 0.2961 $\pm$ 0.004 | 0.2593 $\pm$ 0.004 | 0.2375 $\pm$ 0.003 | 0.2292 $\pm$ 0.002 | 55.99 $\pm$ 0.2 | 58.67 $\pm$ 0.2 | 59.97 $\pm$ 0.2 | 60.70 $\pm$ 0.1 |
| BADGE | 0.2920 $\pm$ 0.004 | 0.2570 $\pm$ 0.004 | 0.2361 $\pm$ 0.003 | 0.2279 $\pm$ 0.002 | 56.06 $\pm$ 0.2 | 58.77 $\pm$ 0.2 | 60.10 $\pm$ 0.1 | 60.84 $\pm$ 0.1 |
| Ours | 0.2868 $\pm$ 0.004 | 0.2523 $\pm$ 0.003 | 0.2301 $\pm$ 0.002 | 0.2228 $\pm$ 0.001 | 56.16 $\pm$ 0.2 | 59.04 $\pm$ 0.2 | 60.28 $\pm$ 0.1 | 61.00 $\pm$ 0.1 |
| CAMPAL & BADGE | 0.2795 $\pm$ 0.004 | 0.2465 $\pm$ 0.004 | 0.2249 $\pm$ 0.003 | 0.2161 $\pm$ 0.002 | 57.96 $\pm$ 0.2 | 61.00 $\pm$ 0.2 | 62.42 $\pm$ 0.1 | 63.06 $\pm$ 0.1 |
| CAMPAL & Ours | 0.2623 $\pm$ 0.004 | 0.2332 $\pm$ 0.003 | 0.2101 $\pm$ 0.002 | 0.2011 $\pm$ 0.001 | 58.86 $\pm$ 0.2 | 61.86 $\pm$ 0.2 | 63.12 $\pm$ 0.1 | 63.65 $\pm$ 0.1 |

#### B.3.2 Evaluation on an imbalanced benchmark

Figure: Figure 9: Calibration error (lower is better) and accuracy (higher is better) comparison with different baselines across query times $t\in[T]$, where $T=100$ on an imbalanced setting with CIFAR-10-LT. Intervals for each line in the graph depict our results across 10 runs. Table details are in Tab. [3](https://arxiv.org/html/2510.03162v1#A2.T3).
Refer to caption: x11.png

It is worth noticing that our AF aims to improve uncertainty-based sampling methods and not to focus on diversity. The reason is that we aim to improve not only the accuracy but also the uncertainty estimation quality of DNN in AL. This is crucial because it helps determine when an AI model’s predictions can be trusted, especially in safety-critical applications.

That said, one factor that may impact the performance of uncertainty-based methods is the imbalanced class distributions of the dataset. Hence, we next test the robustness of our method on an imbalanced setting and how our method can extend to address the potential lack of diversity in the selected sample. In particular, we provide our results on Long-Tail CIFAR-10 (Cui et al., 2019) with an imbalance factor of $50$ over ten runs. For each run, we initialize with $1000$ randomly selected data points from the training dataset. Tab. [3](https://arxiv.org/html/2510.03162v1#A2.T3) shows results with ResNet-18, data augmentation (Yang et al., 2023), the number of AL rounds $T=100$, and the number of queried samples $k=100$.

From Tab. [3](https://arxiv.org/html/2510.03162v1#A2.T3) and Fig. [9](https://arxiv.org/html/2510.03162v1#A2.F9), firstly, we observe that diversity-based methods (e.g., Coreset, BADGE) have a higher accuracy than uncertainty-based methods (e.g., Least-conf, Rand-Entropy, Margin). This is because the diversity-based method helps query balance class samples and uncalibrated uncertainty-based methods cause biased sampling. Notably, our method is more calibrated and can mitigate the biased sampling problem, leading to higher accuracy than other uncertainty-based baselines.

Secondly, compared to the best method, i.e., BADGE, although our method has only slightly higher accuracy by not focusing on diversity, it still outperforms BADGE in uncertainty estimation quality with a significantly lower ECE. This shows that our AF is still valuable in high-stakes applications, where humans need to know when an AI model’s predictions can be trusted to make a final decision.

Finally, our method can address the potential lack of diversity in the selected samples by combining with other diversity-based approaches. For example, our AF can incorporate two-stage methods to become a hybrid AF (i.e., uncertainty-based & diversity-based) like Cluster-Margin. Specifically, we can extend our framework to Cluster-Ours by replacing Margin Step 8 in Alg.2 (Citovsky et al., 2021) with our proposed AF, i.e., select $k_{m}$ samples by our AF in Alg. [1](https://arxiv.org/html/2510.03162v1#alg1). After that, we can follow the remaining steps in Alg.2 (Citovsky et al., 2021) to select $k_{t}$ from $k_{m}$ samples by a cluster-based approach to improve diversity. Tab. [3](https://arxiv.org/html/2510.03162v1#A2.T3) and Fig. [9](https://arxiv.org/html/2510.03162v1#A2.F9) show that Cluster-Ours can address the potential lack of diversity on this imbalanced dataset and can bring out the best performance with the highest accuracy and the lowest ECE. This once again confirms the benefits of our proposed AF and its potential to extend to improve trustworthiness across AL settings.

**Table 3: Calibration error (lower is better) and accuracy (higher is better) comparison with different baselines across query times $t\in\{T/4,T/2,3T/4,T\}$, where $T=8$ on CIFAR-10-LT. Scores are reported by mean $\pm$ standard deviation from 10 runs. Gray rows indicate our proposed method. Best scores with the significant test are marked in bold. Figure details are in Fig. [9](https://arxiv.org/html/2510.03162v1#A2.F9). Our algorithm has competitive results in accuracy and outperforms other AF baselines in calibration. We additionally extend our AF to a two-stage approach with Cluster-Ours to improve the diversity in the selected samples (i.e., select $k_{m}$ samples by our AF, then select $k_{t}$ from $k_{m}$ samples by a cluster-based approach to improve diversity (Citovsky et al., 2021).). Cluster-Ours achieves the best performance and outperforms other two-stage-based AL (e.g., Rand-Entropy, Cluster-Margin).**
| Method | Expected Calibration Error ($\downarrow$) | Accuracy ($\uparrow$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ | $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ |  |
| Random | 0.346 $\pm$ 0.002 | 0.322 $\pm$ 0.002 | 0.307 $\pm$ 0.002 | 0.299 $\pm$ 0.003 | 53.9 $\pm$ 0.3 | 58.2 $\pm$ 0.3 | 60.4 $\pm$ 0.2 | 61.9 $\pm$ 0.3 |
| Least-conf | 0.351 $\pm$ 0.006 | 0.322 $\pm$ 0.005 | 0.304 $\pm$ 0.008 | 0.289 $\pm$ 0.004 | 54.5 $\pm$ 0.5 | 59.1 $\pm$ 0.5 | 61.6 $\pm$ 0.7 | 63.4 $\pm$ 0.4 |
| Margin | 0.343 $\pm$ 0.002 | 0.316 $\pm$ 0.002 | 0.299 $\pm$ 0.003 | 0.290 $\pm$ 0.002 | 54.8 $\pm$ 0.2 | 59.4 $\pm$ 0.3 | 61.8 $\pm$ 0.4 | 63.4 $\pm$ 0.2 |
| BALD | 0.346 $\pm$ 0.003 | 0.319 $\pm$ 0.003 | 0.303 $\pm$ 0.005 | 0.293 $\pm$ 0.003 | 54.4 $\pm$ 0.3 | 58.8 $\pm$ 0.4 | 61.3 $\pm$ 0.5 | 62.8 $\pm$ 0.3 |
| BatchBALD | 0.342 $\pm$ 0.003 | 0.312 $\pm$ 0.003 | 0.300 $\pm$ 0.006 | 0.285 $\pm$ 0.004 | 54.6 $\pm$ 0.5 | 59.1 $\pm$ 0.4 | 61.6 $\pm$ 0.5 | 63.1 $\pm$ 0.4 |
| Coreset | 0.341 $\pm$ 0.008 | 0.308 $\pm$ 0.005 | 0.291 $\pm$ 0.005 | 0.278 $\pm$ 0.005 | 54.9 $\pm$ 0.8 | 59.8 $\pm$ 0.4 | 62.3 $\pm$ 0.4 | 64.0 $\pm$ 0.5 |
| BADGE | 0.327 $\pm$ 0.003 | 0.300 $\pm$ 0.002 | 0.285 $\pm$ 0.002 | 0.275 $\pm$ 0.002 | 55.8 $\pm$ 0.4 | 60.5 $\pm$ 0.3 | 63.1 $\pm$ 0.4 | 64.6 $\pm$ 0.4 |
| Ours | 0.320 $\pm$ 0.003 | 0.292 $\pm$ 0.001 | 0.276 $\pm$ 0.002 | 0.262 $\pm$ 0.002 | 56.0 $\pm$ 0.4 | 60.8 $\pm$ 0.2 | 63.2 $\pm$ 0.2 | 65.0 $\pm$ 0.3 |
| Rand-Entropy | 0.343 $\pm$ 0.004 | 0.317 $\pm$ 0.003 | 0.296 $\pm$ 0.002 | 0.287 $\pm$ 0.002 | 54.9 $\pm$ 0.4 | 59.3 $\pm$ 0.3 | 61.9 $\pm$ 0.5 | 63.6 $\pm$ 0.5 |
| Cluster-Margin | 0.336 $\pm$ 0.003 | 0.310 $\pm$ 0.003 | 0.294 $\pm$ 0.002 | 0.284 $\pm$ 0.002 | 55.4 $\pm$ 0.3 | 60.1 $\pm$ 0.3 | 62.7 $\pm$ 0.5 | 64.3 $\pm$ 0.4 |
| Cluster-Ours | 0.302 $\pm$ 0.003 | 0.274 $\pm$ 0.001 | 0.258 $\pm$ 0.001 | 0.244 $\pm$ 0.002 | 57.6 $\pm$ 0.4 | 62.4 $\pm$ 0.1 | 64.7 $\pm$ 0.1 | 66.5 $\pm$ 0.2 |

#### B.3.3 Calibration error on the unlabeled pool

To verify the benefits of our AF in terms of improving uncertainty quantification on the unlabeled pool, we additionally compare our proposed method with other baselines. We summarize this result in Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10) and Tab. [4](https://arxiv.org/html/2510.03162v1#A2.T4). Recall that for a fair comparison, we set the random seeds across baselines to be the same. As a result, for every round $t=0$, the performance across methods is exactly similar. From these figures and the tables, we can also see that our method outperforms other baselines in every AL rounds across different settings by having the lowest ECE. For instance, its ECE is lower than others by more than $0.026$ at $t=10$ on MNIST, $0.01$ at $t=100$ on SVHN, $0.016$ at $t=75$ on F-MNIST, and $0.012$ on CIFAR-10. It is also worth noticing that the ECE on the unlabeled pool in Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10) is generally lower than on the unseen data in Fig. [7](https://arxiv.org/html/2510.03162v1#A2.F7). This is because we can observe the samples of the unlabeled pool during our calibration estimation in Equation [9](https://arxiv.org/html/2510.03162v1#S3.E9). Finally, to be summarized, from Fig. [7](https://arxiv.org/html/2510.03162v1#A2.F7) and Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10), we can see that the effectiveness of our proposed AF in improving model uncertainty estimation performance in both the unlabeled pool and the test set. Hence, by this high-uncertainty estimation quality model, we can leverage calibrated uncertainty sampling to query informative samples to enhance generalization performance.

Figure: Figure 10: Calibration error on the unlabeled pool (lower is better) comparison with different baselines across query times $t\in[T]$ with different datasets and model architectures. Intervals for each line in the graph depict our results across 10 runs. Table details are in Tab. [4](https://arxiv.org/html/2510.03162v1#A2.T4). Our method achieves the lowest calibration error on the unlabeled pool across every $t\in[T]$.
Refer to caption: x12.png

**Table 4: Calibration error on the unlabeled pool (lower is better) comparison with different baselines across query times $t\in\{T/4,T/2,3T/4,T\}$, where $T=40$ on MNIST and $T=100$ on other datasets. Scores are reported by mean $\pm$ standard deviation from 10 runs. Gray rows indicate our proposed method. Best scores with the significant test are marked in bold. Figure details are in Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10).**
| Dataset | Method | Expected Calibration Error |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $t=0$ | $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ |  |  |
| MNIST | Random | 0.287 $\pm$ 0.019 | 0.130 $\pm$ 0.031 | 0.084 $\pm$ 0.004 | 0.070 $\pm$ 0.007 | 0.060 $\pm$ 0.002 |
| Least-conf | 0.287 $\pm$ 0.019 | 0.137 $\pm$ 0.013 | 0.068 $\pm$ 0.003 | 0.046 $\pm$ 0.003 | 0.037 $\pm$ 0.001 |  |
| Margin | 0.287 $\pm$ 0.019 | 0.130 $\pm$ 0.014 | 0.069 $\pm$ 0.003 | 0.049 $\pm$ 0.003 | 0.040 $\pm$ 0.001 |  |
| BALD | 0.287 $\pm$ 0.019 | 0.120 $\pm$ 0.013 | 0.061 $\pm$ 0.003 | 0.043 $\pm$ 0.002 | 0.035 $\pm$ 0.001 |  |
| Coreset | 0.287 $\pm$ 0.019 | 0.135 $\pm$ 0.018 | 0.073 $\pm$ 0.003 | 0.054 $\pm$ 0.004 | 0.045 $\pm$ 0.001 |  |
| BADGE | 0.287 $\pm$ 0.019 | 0.125 $\pm$ 0.010 | 0.064 $\pm$ 0.003 | 0.045 $\pm$ 0.002 | 0.036 $\pm$ 0.001 |  |
|  | Ours | 0.287 $\pm$ 0.019 | 0.094 $\pm$ 0.011 | 0.050 $\pm$ 0.003 | 0.035 $\pm$ 0.002 | 0.028 $\pm$ 0.004 |
| SVHN | Random | 0.437 $\pm$ 0.039 | 0.135 $\pm$ 0.005 | 0.116 $\pm$ 0.003 | 0.108 $\pm$ 0.005 | 0.097 $\pm$ 0.003 |
| Least-conf | 0.437 $\pm$ 0.039 | 0.133 $\pm$ 0.009 | 0.111 $\pm$ 0.007 | 0.096 $\pm$ 0.004 | 0.085 $\pm$ 0.004 |  |
| Margin | 0.437 $\pm$ 0.039 | 0.128 $\pm$ 0.004 | 0.106 $\pm$ 0.003 | 0.095 $\pm$ 0.003 | 0.083 $\pm$ 0.003 |  |
| BALD | 0.437 $\pm$ 0.039 | 0.127 $\pm$ 0.004 | 0.107 $\pm$ 0.003 | 0.098 $\pm$ 0.004 | 0.087 $\pm$ 0.005 |  |
| Coreset | 0.437 $\pm$ 0.039 | 0.131 $\pm$ 0.005 | 0.108 $\pm$ 0.003 | 0.097 $\pm$ 0.002 | 0.086 $\pm$ 0.004 |  |
| BADGE | 0.437 $\pm$ 0.039 | 0.125 $\pm$ 0.002 | 0.105 $\pm$ 0.003 | 0.092 $\pm$ 0.003 | 0.084 $\pm$ 0.004 |  |
|  | Ours | 0.437 $\pm$ 0.039 | 0.120 $\pm$ 0.003 | 0.101 $\pm$ 0.004 | 0.085 $\pm$ 0.002 | 0.074 $\pm$ 0.003 |
| F-MNIST | Random | 0.366 $\pm$ 0.034 | 0.230 $\pm$ 0.009 | 0.204 $\pm$ 0.007 | 0.196 $\pm$ 0.010 | 0.194 $\pm$ 0.008 |
| Least-conf | 0.366 $\pm$ 0.034 | 0.246 $\pm$ 0.017 | 0.215 $\pm$ 0.017 | 0.200 $\pm$ 0.021 | 0.185 $\pm$ 0.014 |  |
| Margin | 0.366 $\pm$ 0.034 | 0.234 $\pm$ 0.007 | 0.204 $\pm$ 0.004 | 0.190 $\pm$ 0.009 | 0.182 $\pm$ 0.006 |  |
| BALD | 0.366 $\pm$ 0.034 | 0.249 $\pm$ 0.020 | 0.198 $\pm$ 0.012 | 0.183 $\pm$ 0.013 | 0.169 $\pm$ 0.014 |  |
| Coreset | 0.366 $\pm$ 0.034 | 0.238 $\pm$ 0.011 | 0.209 $\pm$ 0.012 | 0.196 $\pm$ 0.015 | 0.184 $\pm$ 0.009 |  |
| BADGE | 0.366 $\pm$ 0.034 | 0.225 $\pm$ 0.004 | 0.198 $\pm$ 0.003 | 0.186 $\pm$ 0.006 | 0.180 $\pm$ 0.008 |  |
|  | Ours | 0.366 $\pm$ 0.034 | 0.220 $\pm$ 0.010 | 0.185 $\pm$ 0.006 | 0.167 $\pm$ 0.005 | 0.155 $\pm$ 0.003 |
| CIFAR-10 | Random | 0.329 $\pm$ 0.017 | 0.285 $\pm$ 0.004 | 0.260 $\pm$ 0.002 | 0.237 $\pm$ 0.003 | 0.234 $\pm$ 0.013 |
| Least-conf | 0.329 $\pm$ 0.017 | 0.293 $\pm$ 0.007 | 0.260 $\pm$ 0.006 | 0.234 $\pm$ 0.008 | 0.224 $\pm$ 0.008 |  |
| Margin | 0.329 $\pm$ 0.017 | 0.287 $\pm$ 0.003 | 0.259 $\pm$ 0.003 | 0.233 $\pm$ 0.003 | 0.224 $\pm$ 0.008 |  |
| BALD | 0.329 $\pm$ 0.017 | 0.294 $\pm$ 0.010 | 0.278 $\pm$ 0.015 | 0.235 $\pm$ 0.003 | 0.218 $\pm$ 0.005 |  |
| Coreset | 0.329 $\pm$ 0.017 | 0.290 $\pm$ 0.002 | 0.260 $\pm$ 0.003 | 0.235 $\pm$ 0.003 | 0.229 $\pm$ 0.010 |  |
| BADGE | 0.329 $\pm$ 0.017 | 0.283 $\pm$ 0.005 | 0.252 $\pm$ 0.001 | 0.230 $\pm$ 0.002 | 0.219 $\pm$ 0.009 |  |
|  | Ours | 0.329 $\pm$ 0.017 | 0.278 $\pm$ 0.009 | 0.247 $\pm$ 0.004 | 0.227 $\pm$ 0.003 | 0.206 $\pm$ 0.006 |

#### B.3.4 Additional ablation studies

**Table 5: Frequency of samples selected based on calibration error versus uncertainty in our AF across query times $t\in\{T/4,T/2,3T/4,T\}$, where $T=100$, with the number of queried samples is $k=50$ on SVHN. The number of selected samples based on calibration error is high at early rounds. The model can then improve the calibration performance. As a result, in later rounds, the calibration error is lower and more uniform across samples (Fig. [3](https://arxiv.org/html/2510.03162v1#S3.F3), Fig. [10](https://arxiv.org/html/2510.03162v1#A2.F10)), leading to a higher proportion of samples being selected by uncertainty sampling.**
| Time $t$ | $t=T/4$ | $t=T/2$ | $t=3T/4$ | $t=T$ |
| --- | --- | --- | --- | --- |
| $\#$ samples selected only based on calibration error | 48.0 $\pm$ 1.5 | 22.5 $\pm$ 3.2 | 17.2 $\pm$ 3.5 | 6.1 $\pm$ 2.8 |
| $\#$ samples selected additionally based on uncertainty | 2.0 $\pm$ 1.1 | 27.5 $\pm$ 3.6 | 32.8 $\pm$ 3.8 | 43.9 $\pm$ 2.0 |

Figure: Figure 11: Calibration and accuracy comparison with the post-hoc calibrated uncertainty sampling baseline on MNIST. To improve calibration performance, these post-hoc methods sacrifice samples in training, resulting in a significantly lower accuracy. Meanwhile, the calibration performance depends on model confidence and accuracy. As a result, their results are significantly worse than our AF. We also compare with DDU and it is worth noting that DDU (Mukhoti et al., 2023) uses Temperature Scaling, while our setting does not use this post-hoc recalibration technique.
Refer to caption: x13.png

Figure: Figure 12: Calibration and accuracy comparison between query by only using the least calibrated samples, and ours AF on Fashion-MNIST. Our AF results in a better performance with a lower ECE and higher accuracy, confirming the effectiveness of the lexicographic order during query time. We also compare with CALICO and similar observations to Querol et al. (2024), the improvement of CALICO over Least-conf is minor (even worse in some settings), as the acquisition strategy stays the same as Least-conf and the joint training is in a semi-supervised way.
Refer to caption: x14.png

Figure: Figure 13: Our lexicographic order ablation study by comparing with our proposed other weighting combination strategies. The weighted combination ablation study on SVHN, including: (1) a uniform weighted combination, i.e., ranking by $(calibration+uncertainty)/2$; adaptive weighted combination, i.e., (2) ranking by $(\alpha*calibration+(1-\alpha)*uncertainty)$, where $\alpha=\text{mean(calibration error on the unlabeled pool})\in[0,1]$; (3) ranking by $(k*\alpha*calibration+uncertainty)/(k*\alpha+1)$, where $k=\{2,3\}$. With an appropriate weighting hyperparameter (e.g., $k=3$), the adaptive weighting can further improve the performance. However, with inappropriate weighting (e.g., uniform weighting), it can lead to a degradation of performance. This suggests our lexicographic order is a more flexible strategy because it does not depend on $\alpha$ while still bringing out a good performance in practice.
Refer to caption: x15.png

Figure: Figure 14: The choice of kernel bandwidth $b$, where each line represents the results value of the method minus the results value of the least-confident baseline on SVHN. Following (Popordanoska et al., 2022), we compare our kernel bandwidth $b=\{0.0001,0.0005,0.001\}$. We observe that there is no significant difference between our performances. Yet, if the bandwidth is too large, e.g., $b=1$, it can lead to an oversmoothing problem. And, our ranking in calibration error becomes uniform, causing similar results with the least-confident baseline.
Refer to caption: x16.png

Figure: Figure 15: Calibration error (lower is better) and accuracy (higher is better) comparison across query times $t\in[T]$ on the SVHN with different numbers of warm-up samples $n_{0}=\{250,500,1000\}$. Our results are consistent across different numbers of warm-up samples.
Refer to caption: x17.png

Figure: Figure 16: Calibration error (lower is better) and accuracy (higher is better) comparison across query times $t\in[T]$ on the SVHN with different numbers of queried samples $k=\{25,50,100\}$. Our results are consistent across different numbers of queried samples.
Refer to caption: x18.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: I have several questions regarding the lexicographic order for query and the proof of theoretical guarantees. First on the lexicographic order:

- At each query round, the proposed query strategy select first the least calibrated samples, then the least confident ones with zero calibration error. Does it mean that there is a Dirac at zero in the distribution of calibrated errors? If so, how to understand it? 

- As explained by the authors, their method focuses on least-calibrated samples at early query rounds, then moves to the least confident ones later when the trained model is well calibrated. To illustrate the effectiveness of this lexicographic order, the authors compared the performance of their query strategy with querying by least-calibrated only in Figure 12, where the performance of least-calibrated only diverges already from the proposed method at very early rounds. Could the authors explain this observation while the two methods should be nearly identical at early query rounds?

On the proof of theoretical guarantees:

- Could the authors explain why the result of Popordanoska et al. (2022) can be applied to obtain (20), despite that the sample in the cumulative labeled set are not i.i.d. ?

- Could the authors explain in detail the passage from (42) to (43)? From where I see it, the fact that the average calibration error is smaller than $\epsilon$ on the samples queried at the $t$-th round does not imply that it is also the case for the samples queried at earlier rounds.
soundness: 2
strengths: - The proposed method is well motivated by the importance of well estimated uncertainty in active learning and the calibration issue that is common in large models.

- The use of kernel estimation for evaluating calibration errors on unlabelled data is an interesting and reasonable idea.

- The proposed method is tested on several benchmarks, compared to a series of active learning baselines. The experimental results show interesting gains by the proposed method.

- The discussion on related work is carefully developed.
summary: Motivated by the issue of uncalibrated models in uncertainty-based active learning (AL), the article advocates for a query strategy using calibration error. It selects first the least uncalibrated samples,  then the least confident ones when the calibration error is uniformly small in the remaining unlabeled pool. This strategy encourages the training to focus on the calibration error at early query rounds in order to improve the uncertainty estimation. The calibration error is estimated with kernel method, with theoretical results on the consistency of kernel estimation and the bounds of calibration error on unlabeled and unseen test data. The experimentation on large-scale benchmarks show competitive results by the proposed method in comparison to a series of AL baselines.
weaknesses: - Arguments and results on the effectiveness of the lexicographic order during query over the least-calibrated only strategy might lack consistency (see Questions).

- There might be some issues in the proofs (see Questions)

- Some notations are ambiguous or erroneous. According to the setup in the beginning of Section 2, $x_1$ stand for the first data vector in the initial labeled and unlabeled set at the same time. In Lines 221-222, the index $i$ goes from $0$ to $k$, querying $k+1$ samples instead of $k$, and creating repeated indices for samples queried at different rounds. The distribution of the cumulative labeled dataset $S$ is different from the initial label set $S_0$, meaning that we cannot have $S\sim\mathbb{P}(X)\mathbb{P}(Y\vert X)=\mathbb{P}(X,Y)$ and also $S_0\sim\mathbb{P}(X,Y)$, in contrary to the formula in (14). The abbreviation ECE for Expected Calibration Error is used before the specification of its meaning in Line 311.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) The effect of the kernel can be crucial in practice. Do you have any way or guidelines to select the appropriate kernel?
2) Basically, the proposed CE (loss) is based on the estimation of h, which has vagueness, especially in the early stage. Is there any clue to check this problem?
3) The baseline for the post-calibration in the ablation studies looks insufficient. Can you consider 2~3 baselines for the calibration?
soundness: 3
strengths: The algorithm is simple, and the performance is better. The improvement of calibration is based on a well-designed procedure for kernel estimation using the pooling data. Though the improvements are relatively smaller, in active learning, it is often observed that the impressive point is to improve the calibration and prediction simultaneously.
summary: This paper proposes an active learning algorithm that addresses calibrated models over conventional active learning. The authors consider a calibration metric based on classical kernel estimators of statistics and demonstrate better performance in terms of calibration and prediction, as well as acquisition steps. For the query function, selecting the informative instances is designed according to the lexicographical order. The experiments were conducted on MNIST, SVHN, Fashion MNIST, and CIFAR-10 with the baselines of Random Least-conf Margin, BALD Coreset, and BADGE. Various aspects of calibration approaches are examined in the ablation studies.
weaknesses: The dataset is relatively small; in many papers on topics of active learning, usually 6~8 datasets are examined. Also, the calibration is not studied thoroughly. There are many metrics for the calibration, such as the ECE and the KS-metric-based. Also, the calibration algorithms in processing or post-processing can cooperate with the proposed alg. The merit or reason for the use of a specific kernel type is not thoroughly validated. Furthermore, it is not clear to me why the lexicographical order is essential, as pointed out in the ablation study. Since there are two components in the loss function, what’s the dominant part or what’s the effective part in the prediction? In some cases, regularizing the calibration error can cause the degradation of prediction power. Why does the proposed algorithm not suffer from this phenomenon?

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see the Weaknesses.
soundness: 4
strengths: 1. By directly using calibration error as the primary criterion for the active learning acquisition function, the paper breaks away from the conventional framework that focuses solely on uncertainty.
2. The authors provide solid theoretical contributions by proving the pointwise consistency of the kernel calibration estimator under covariate shift and deriving an upper bound on the final model’s expected calibration error.
summary: The paper proposes Calibrated Uncertainty Sampling for Active Learning (CUSAL), a new acquisition function that explicitly targets low calibration error. For every unlabeled instance, a Dirichlet-kernel estimator approximates the expected calibration error under covariate shift; samples are then chosen by lexicographic ordering that first maximizes the estimated calibration error and, on ties, maximizes model uncertainty. The authors prove that the estimator is point-wise consistent and that the resulting classifier enjoys a bounded expected calibration error on both the unlabeled pool and the unseen test set. Extensive experiments on MNIST, SVHN, CIFAR-10, ImageNet, and long-tail CIFAR-10-LT show consistent reductions in ECE and accuracy gains over six strong baselines.
weaknesses: 1. The method may suffer from high computational overhead, since a kernel matrix needs to be computed for all unlabeled samples in each active learning round.
2. In the third subfigure from the left in Figure 1, is the gap at position 0.4 plotted incorrectly?

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Does the proposed calibration-driven sampling actually outperform a standard Active Learning pipeline that uses the same labeled data but applies explicit post-hoc calibration (e.g., temperature scaling) after each training round?
soundness: 2
strengths: 1. The paper addresses AL from a calibration perspective, which I believe is relatively underexplored. the attempt to connect model calibration quality with sampling decisions introduces an interesting conceptual bridge between uncertainty estimation and data selection.
2The bias variance analysis and convergence discussion (Theorems 4.1, 4.2)shows an effort to provide formal grounding for their estimator, which is important in works. While the assumptions are strong, the theoretical treatment gives the work analytical depth.
summary: This paper proposes a calibration aimed AL sampling method that estimates unlabeled sample calibration error using a kernel based local averaging estimator and prioritizes samples with the largest estimated calibration error for labeling.
weaknesses: 1.The proposed problem and its motivation is unclear to me. The authors claim two critical issues arising from uncalibrated uncertainty, yet the first one, namely  the uncertainty quantification quality of existing AL baselines on unseen test data is not verified, is a general evaluation concern, not a problem specific to Active Learning. It does not logically motivate a new acquisition function. Only the second issue (uncalibrated uncertainty causing unreliable sampling) is actually relevant to AL.
2.The demonstrative example lacks key information and renders confusing messages.  Specifically, In figure1, the caption references T = 50 and k = 10 without prior explanation; these variables are not introduced when the figure is first cited in the text. It is therefore impossible to know whether T denotes total query rounds or training epochs, or whether k is the batch size per query or the number of queried samples. What is more, the use of Least-conf as a baseline is not adequately justified. The paper did not define how “confidence” is computed (by the time this demo example is referred to). whether it is the maximum predicted probability, the softmax margin, or another uncertainty metric. Moreover, least-confidence is not a standard or competitive AL method in recent literature; 
3.The theoretical analysis (4.1, 4.2)only guarantees the consistency and boundedness of calibration error under active learning, but not its effect on classification risk or accuracy. While improved calibration is desirable, AL aims to reduce generalization error with limited labeling budget. Without establishing or empirically validating a link between calibration improvement and boundary learning efficiency, the method falls short of fully achieving AL’s goal of enhancing both calibration and accuracy. Furthermore, even if we focus on the claimed calibration consistency only, its theoretical guarantee has limited relevance to AL. Theorem 4.1 explicitly assumes an infinitely large labeled set, to achieve point-wise consistency of the estimator. This assumption contradicts the low-label regime that defines Al. In realistic settings with few labeled samples, the kernel estimator may have high variance and unreliable calibration estimates, weakening the practical value of the theoretical bound.
4.For the methodology, it is straightforward. However, from the mathematical perspective, the Dirichlet kernel appears to be chosen mainly for analytical convenience (please correct me if I was wrong) it simplifies the derivation of bias and variance bounds under the simplex constraint. Experimentally, the justification reduces to a smoothness argument when tuning the bandwidth. However, this rationale does not sufficiently support the choice of this kernel over other possible kernels. The fact that a bound is easier to derive does not imply that alternative kernels are less effective in practice. In the absence of a stronger theoretical or empirical argument, an ablation comparing different kernel types （Gaussian, etc) should be included to validate whether the proposed choice meaningfully contributes to performance.
5.(This is my major concern.)
The proposed method aims to improve Active Learning by selecting samples that indirectly enhance model calibration. However, once labeled data are available, calibration quality can always be improved more directly through post-hoc methods (e.g., temperature scaling or isotonic regression) using the same labeled set. In that case, any calibration-driven sampling provides no fundamental advantage: both approaches use identical supervision, yet explicit calibration guarantees strictly better or equal ECE. Moreover, if the authors’ argument is that better calibration leads to better acquisition and final accuracy, then a model explicitly calibrated after each AL round would necessarily yield more reliable uncertainty and thus stronger sampling performance. From this perspective, their method merely approximates what direct calibration already achieves through an indirect and less efficient route. While earlier issues such as motivation ambiguity and unclear experimental setup are secondary, this major concern lies in the legitimacy of the proposed approach itself. The paper’s central idea offers no demonstrable benefit over existing, theoretically simpler, and empirically stronger alternatives. In short, if labeled data exist, calibration should be achieved by calibration, not by sampling.
To properly validate the claimed contribution, the authors should include a control experiment under an identical labeling regime:
1.Run a standard AL method (e.g., Entropy or BALD) with the same acquisition budget and identical labeled sets St as those used by the proposed method.
2.After each training round, apply a standard calibration procedure (e.g., temperature scaling) using exactly the same labeled data that the proposed approach employs for its kernel-based estimator.
3.Compare both ECE and accuracy across rounds.


Decision: Reject