ConBatch-BAL: Batch Bayesian Active Learning under Budget Constraints

Authors: Pablo G. Morato, Charalampos P. Andriotis, Seyran Khademi
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Bayesian Active Learning, Real-World Datasets, Batch Acquisition, Budget Constraints

Abstract:
Varying annotation costs among data points and budget constraints can hinder the adoption of active learning strategies in real-world applications. This work introduces two Bayesian active learning strategies for batch acquisition under constraints (ConBatch-BAL), one based on dynamic thresholding and one following greedy acquisition. Both select samples using uncertainty metrics computed via Bayesian neural networks. The dynamic thresholding strategy redistributes the budget across the batch, while the greedy one selects the top-ranked sample at each step, limited by the remaining budget. Focusing on scenarios with costly data annotation and geospatial constraints, we also release two new real-world datasets containing geolocated aerial images of buildings, annotated with energy efficiency or typology classes. The ConBatch-BAL strategies are benchmarked against a random acquisition baseline on these datasets under various budget and cost scenarios. The results show that the developed ConBatch-BAL strategies can reduce active learning iterations and data acquisition costs in real-world settings, and even outperform the unconstrained baseline solutions.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729819940765

thread_type:
official_review

<thread_object>
creator: reviewer_pqzh

<content>
summary:
This paper introduces two Bayesian active learning heuristics—greedy acquisition and dynamic thresholding—for batch acquisition under budget or cost constraints. The authors conducted experiments on one artificial dataset derived from MNIST and two real-world datasets, demonstrating improved performance.

strengths:
The authors clearly presented the two heuristics.

weaknesses:
(1) Vague problem setting: The authors do not provide a clear objective or constraints in Section 3.1. For instance, when considering the examples in Figure 2(b), the cost of a batch depends on the order of the batches, which implies that in Equation 2, the constraints should be $c(x_1, ..., x_n) \leq c_{max}, n \leq n_{max}$. The combinatorial nature of the cost is overlooked in the problem setting.

(2) Limited applications: The proposed methods are applied to only two real-world datasets, both of which are aerial imagery, limiting the scope of the applications.

(3) Limited Bayesian Neural Network (BNN) models tested: The experiments focus solely on MC-Dropout, but other BNN models with superior performance—such as SGHMC [1], SG-MCMC [2], and cSG-MCMC [3]—should also be evaluated.


references:

[1] Chen, Tianqi, Emily Fox, and Carlos Guestrin. "Stochastic gradient hamiltonian monte carlo." In International conference on machine learning, pp. 1683-1691. PMLR, 2014.

[2] Welling, Max, and Yee W. Teh. "Bayesian learning via stochastic gradient Langevin dynamics." In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. 2011.

[3] Zhang, Ruqi, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. "Cyclical stochastic gradient MCMC for Bayesian deep learning." arXiv preprint arXiv:1902.03932 (2019).

Minor suggestions:

(1) In line 134, use ${x_1, ..., x_n} \subset D_{pool}$.

(2) Ensure consistency in the use of "Batch-BALD" or "batch-BALD" (see line 149 vs. line 346).

(3) There is no need to use both "return" and "output" with the same content simultaneously in Algorithm 1 and Algorithm 2.

questions:
It appears that the two strategies, which are the main contributions of this paper, can be easily extended to classical active learning algorithms, such as CoreSet or BADGE. Even in a Bayesian setting, average embeddings can still be used for active learning with CoreSet or BADGE. I believe the main technical challenge lies in addressing the combinatorial cost. The authors should avoid restricting their approach to Bayesian Neural Networks (BNNs) without offering additional benefits beyond BNN model accuracy. I am open to discussing this further in the future.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730383632620

thread_type:
official_review

<thread_object>
creator: reviewer_nngv

<content>
summary:
This paper tackles the challenge of batch Bayesian active learning under budget constraints by introducing two novel heuristics: dynamic thresholding and greedy strategies. It also presents two new real-world datasets featuring geolocated aerial images of buildings. Experimental results show that these strategies outperform a random selection baseline across both real-world datasets and a geolocated MNIST dataset.

strengths:
The paper addresses the practical challenge of batch Bayesian active learning under constraints on both batch size and cost. The two real-world datasets introduced offer valuable resources for evaluating active learning solutions with geolocated points.

weaknesses:
The paper would benefit from a clearer and more detailed description of the method and an improvement of the technical contributions. 

1. While the objective is to achieve high accuracy on the test set, the test set itself has not been utilized in the the solution. If the goal is to improve test set accuracy, why not consider incorporating mutual information with the classification of points in the test set, similar to the approach in the well-known work by Andreas Krause, Near-Optimal Sensor Placements in Gaussian Processes (2008)? In that study, the test set can be treated as the entire design space.

2. The main technical contribution, the two active learning strategies, is explained too briefly and lacks essential details. For instance, the description omits the acquisition function selecting each point in the batch sequentially (before showing it in the pseudocode), as well as the expression of mutual information using Dropout samples. Lines 185–188 raise several challenges, but there is no clear explanation of how these issues are addressed. Additionally, there are notation inconsistencies in Algorithms 1 and 2:

    + Algorithm 1: $\\mathbf{x}\_1^*, \\dots, \\mathbf{x}\_n^*$ and $n$ are used (lines 222, 231) before they are defined. Line 222 could be revised to "return $A\_{i-1}$" and Line 231 to "return $A\_{n\_{\\max}}$". Similar issues exist in Algorithm 2.

    + Line 225: argmax can be revised to argmax_x.

    + $c(x)$ depends on the previous selected point so the input to $c$ should include the previous selected point.

    + What is the computational complexity of the approach?

3. The heuristic employed is relatively straightforward: the algorithm divides the budget constraint by the batch size or aims to fully utilize the available budget with each sampled point.

4. The experiments have several weaknesses:

+ Why not plot accuracy over iterations, instead of only showing the number of iterations required to reach selected accuracy levels?
+ The experiment includes only a simple baseline (random selection), while other batch Bayesian active learning approaches could also be evaluated under an unlimited budget. For a finite budget, some selected points could simply be dropped from the batch to stay within budget.
+ There is no clear description of the training set, test set, initial training set (before active learning), or the number of Dropout samples.
+ The impact of batch size on algorithm performance is not explored.

Additional notation issues:
  
+ $y\_1,\\dots,y\_n$ is used to denote the outputs of $\\mathbf{x}\_1, \\dots, \\mathbf{x}\_n$, but in lines $172$ $y\_{c\_1}, \\dots, y\_{c\_k}$ refer to the $k$ classes. 
  
+ Line 318 states that the "cost is inversely proportional to the number of buildings," yet it also notes that "crowded areas are more expensive," which seems contradictory.

questions:
Please review the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662822788

thread_type:
official_review

<thread_object>
creator: reviewer_7kd8

<content>
summary:
This work introduces two Bayesian active learning strategies, called ConBatch-BAL, designed for batch acquisition under budget constraints. One strategy uses dynamic thresholding to redistribute the budget across the batch, while the other follows a greedy approach to select top-ranked samples within the budget. Both strategies leverage uncertainty metrics computed by Bayesian neural networks.

The study focuses on scenarios with high annotation costs and geospatial constraints, releasing two new datasets of geolocated aerial building images annotated with energy efficiency or typology classes. ConBatch-BAL strategies are benchmarked against a random acquisition baseline and demonstrate improvements, reducing active learning iterations and data acquisition costs, while outperforming the baseline in real-world settings.

strengths:
Diverse datasets and interesting applications. This paper's innovations lie in introducing dynamic thresholding for addressing varying annotation costs problems in AL. This problem definition is clear and worth studying. The experiments datasets are diverse, as they are not limited to typical vision datasets (CIFAR10, CIFAR100, tinyimageNet) but use aerial images.

weaknesses:
1. The authors shall provide more hyperparameter tuning/ablation study on Section 4.1 on how to choose dynamic thresholding.

The initial threshold, c_th,1 = c_max / n_max, is determined by the total budget divided by the maximum allowed batch steps. This choice is intuitive, but it may not adapt well to scenarios with highly varying annotation costs or where uncertainty scores vary significantly from step to step. Can you conduct  an ablation study on the impact of different n_max values could help understand how sensitive the method is to this parameter and provide insights into optimal settings for datasets listed in the papers?

2. The threshold is updated by subtracting the cost of the previously selected sample and dividing by the remaining steps. This is straightforward but may lead to inefficient sampling if a high-cost sample is selected early on, reducing the budget for remaining samples.

Can the authors provide theoretical insights or guarantees about the convergence or stability of this threshold adjustment process could work? For instance, under certain assumptions about cost distribution or sample uncertainty, it might be possible to guarantee that the threshold remains within certain bounds or converges to a stable value?

3. In real-world scenarios, annotation costs and uncertainty values can be highly variable. It’s unclear if the dynamic thresholding method can robustly handle such variability or if it might be affected by extreme values (outliers). Can the authors provide experiments on highly extreme values (outliers) results?

questions:
No. See weakness parts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701413590

thread_type:
official_review

<thread_object>
creator: reviewer_pbam

<content>
summary:
This paper considers an active learning strategy with budget constraints. The proposed algorithm is based on dynamic thresholding and batch acquisition. Two real-world datasets are considered in experiments, with high cost and having geospatial constraints, under various budget and cost scenarios. The results show that the developed ConBatch-BAL strategies can reduce active learning iterations and data acquisition costs in real-world settings. The performance was compared concerning an infinite budget.  Therefore, the main study focuses on budge-constrained active learning with real-world two-image datasets.

strengths:
The paper proposed an algorithm addressing the budget-constrained dynamic querying algorithm, which can be applied to the examined datasets. The analytic results show the trade-off between budget and informativeness (measures by mutual information). The appropriate strategy for reducing the labeling costs and gaining a large information under budget constraints is established and shows a potential to be applied in the real-world problem.

weaknesses:
$\bullet$
The main issue is concerning $c(\bf x)$. It seems that there is no general $c(\bf x)$ to be applied for various datasets such as tablet datasets, conventional image datasets, and so on.  In the paper, only geospatial constraints are related to $c(\bf x)$. It weakens the contribution of the paper. When the image datasets can be considered, can the variance of each pixel be $c(\bf x)$? At least, the more general criteria for the $c(\bf x)$ can be required.  

$\bullet$
Second, the range of experiments can be problematic on datasets. Combined with the first issue, there are few datasets to apply to the proposed algorithm. In active learning, there are datasets such as images, natural language, tablet datasets, and so on. The coverage of the proposed algorithm should not be all over the domains. However, the applicability or potential to other domains is important.

questions:
$\bullet$ 
Can you consider any more metrics for $c(x),$ such as articular types of cost metrics that might be relevant for other real-world active learning scenarios beyond geospatial constraints?

$\bullet$
Can you provide any criteria or discussion for deciding max budget? Please let me know if you consider the specific factor for real datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732311990160

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all reviewers for their time and constructive feedback, which has greatly contributed to improving the quality of the manuscript. We are encouraged by your positive remarks regarding the strength of our contributions, which include (i) a new problem, (ii) two new real-world datasets to support it, and (iii) two algorithmic approaches to address it. We are also inspired by your critical remarks. With the provided responses and applied modifications, we hope to reinforce your support for our work. The revised paper has been submitted to OpenReview, with all modifications highlighted in blue for clarity.

In this global response, we address two comments raised by multiple reviewers:

**G1. Datasets tested.** 
This work is motivated by the fact that existing active learning strategies do not consider varying label annotation costs or budget constraints within the (batch) selection process. In real-world applications, such as drone navigation, building energy audits, or infrastructure management planning, label annotation costs vary among candidate samples and budget constraints limit sample annotation. Based on this observation, we formulate the active learning problem under budget constraints (Eq. 2) and release two real-world datasets, in which label annotation is constrained by geospatial constraints.

To address the challenges that arise in the previously mentioned real-world application, we also introduce two active learning strategies for selecting samples for annotation under budget constraints. Keep in mind that we aim to enable the application of active learning methods to large-scale real-world scenarios (e.g. for estimating the energy performance of an entire city like Rotterdam). By releasing two new datasets, an active learning problem formulation, and two general algorithmic strategies therefor, we make a first step towards our goal.

Due to the above characteristics, this work definitely does not close a topic, but rather opens a new one. We look forward to further research involving new real-world datasets across diverse domains and cost models defined based on metrics beyond distance. We encourage the exploration of more advanced optimization approaches in future work.

Based on this discussion, we have included clarifications in the revised manuscript (Lines 185-188).

**G2. Impact of batch size on the proposed strategies.**

Following this comment, we have conducted additional experiments for varying batch sizes and reported the results in Appendix C.5. The experiments show that the performance of the proposed ConBatch-BAL strategies relative to the baseline remains consistent across batch sizes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the problem of batch Bayesian active learning under budget constraints, introducing two heuristic strategies: dynamic thresholding and greedy acquisition. The proposed methods aim to balance annotation costs and uncertainty metrics for sample selection, with applications to real-world scenarios such as geolocated aerial imagery datasets. The authors also contribute two new datasets designed to test active learning strategies under budget constraints.

**Strengths**
* The paper introduces a novel problem formulation for batch Bayesian active learning under budget constraints.
* The inclusion of two new real-world datasets of geolocated aerial imagery adds value to the community.
* The proposed strategies (dynamic thresholding and greedy acquisition) are practically relevant for scenarios with varying annotation costs.

**Weaknesses**
* The cost function is defined for geospatial constraints, limiting applicability to other domains.
* The experiments are limited in scope, tested on only two real-world datasets, which causes generalizability concerns.
* The dynamic thresholding strategy lacks theoretical guarantees and shows sensitivity to hyperparameter choices.
* Comparisons are limited to a random baseline, omitting evaluations against other batch Bayesian active learning methods.

While the paper addresses a relevant and practical problem, the limitations in scope, theoretical grounding, and experimental evaluation outweigh its contributions. The authors are encouraged to broaden the generality of their cost function and provide stronger theoretical insights in future submissions.