Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel acquisition function to improve active learning performance in Bayesian neural networks.
The acquisition function, referred to as Balanced Entropy Acquisition (BalEntAcq), serves as an uncertainty measure that aims to capture the information balance (or lack thereof) between the uncertainty of the underlying softmax probability computed by the Bayesian neural network and the label to be predicted.

strength_and_weaknesses: Strength:

Through extensive simulations and comparison with some of the existing popular active learning (AL) schemes, the paper demonstrates the potential benefits of the proposed uncertainty measure, BalEntAcq.
Especially, BalEntAcq has been shown to lead to the acquisition of diversified labels for data points in the pool that are well spread along the decision boundaries, thereby resulting in more efficient model enhancement through label acquisition.
Furthermore, the computational cost of the proposed strategy seems to be fairly low compared to many AL strategies based on Bayesian experimental design (BED), thanks to the approximation of the marginal distributions based on Beta distributions.
This makes the proposed AL method possibly much more scalable compared to the BED, which is an important potential merit of the proposed scheme. 
Overall, the authors have shown that AL via BalEntAcq generally leads to performance improvement over other popular schemes based on several widely used benchmarks.

Weakness:

While scalability is one of the main advantages of the proposed method, direct performance assessment results demonstrating the scalability of BalEntAcq appears to be missing in the current study.
It would be interesting and important to see the comparison between the proposed method compared to other schemes, in terms of computational cost and scalability, especially with methods that the paper mentions to be too costly or not scalable (e.g., BADGE, CoreMSE, and other BED based schemes).
Furthermore, it would be helpful to provide experimental support for "linear scalability" and "exponential savings" that can be attained via AL using BalEntAcq as claimed in the current study.

The implications of the sign of BalEnt, the proximity of BalEnt to 0, and its relevance to the location of the corresponding data point with respect to the decision boundaries could (and should) be better explained, considering their importance in how BalEntAcq leads to enhancing the AL performance.


clarity,_quality,_novelty_and_reproducibility: This paper provides a comprehensive review of relevant existing methods, providing a good context for the proposed work.
The technical details, motivation, and main contributions of the proposed uncertainty measure, BalEntAcq, are fairly clearly stated, and the expected merits of the resulting AL scheme are sufficiently supported by extensive performance assessment results.
The proposed scheme is reasonable and moderately novel, but its main strength would be its enhanced AL performance in a variety of practical settings and its low computational cost and scalability rather than novelty.


summary_of_the_review: This paper presents a new uncertainty measure, BalEntAcq, that has the potential to improve AL performance in a variety of settings while still being scalable (e.g., compared to other existing Bayesian schemes).
The authors support their claims based on extensive performance evaluation results using widely used benchmarks, which demonstrate the potential advantages of the proposed AL acquisition function.

---------

The scores have been updated after reviewing the authors' clarifications to the review comments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of selecting the batches from unlabelled datasets to be labelled that are the most informative for a classifier trained on the data. This problem is traditionally studied within the paradigm of active learning, where the data to be labelled is collected sequentially in batches. The batches are selected using some notion of information gain to the current classifier. The paper presents a novel acquisition function for batch selection in active learning. The authors first examine the standard Bayesian Neural network paradigm used in active learning, and study the behavior of existing acquisition functions like BALD. The authors then outline the Balanced Entropy acquisition function, which is the ratio between the marginalized joint entropy and the entropy (plus a constant). The authors then study the behavior of this acquisition function theoretically as well as through a toy experiment. Finally the authors present an empirical study of their approach on large realistic datasets. 
strength_and_weaknesses: **Strengths**

- The paper is very clearly written and easy to follow. The systematic analysis of Bayesian NNs and existing approaches in active learning approaches is quite insightful.
- The proposed acquisition function is conceptually simple and can be easily adopted in practice. The toy experiments are also quite helpful in illustrating the effect of the acquisition function - selection of _diverse_ and _informative_ points.
- The acquisition function also enjoys strong empirical performance in realistic large datasets, performing on par with or better than more sophisticated and expensive-to-evaluate acquisition functions. The experimental analysis is also quite thorough and provides useful insights. 

**Weaknesses**

- I believe the key weakness of the approach to me is that the acquisition function is stated directly in a form without much motivation. It would be nice to understand how the particular ratio comes and the motivations behind said choice. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is quite well written and thorough with sufficient details and examples. 

**Quality and Novelty**

To the best of my knowledge the proposed acquisition function appears to be novel. The paper is also well framed and results in significant empirical improvements. 

**Reproducibility**

The authors provide code with the submission along with sufficient details in the paper to reproduce the results. 
summary_of_the_review: In summary, the paper presents an novel acquisition function for active learning resulting in significant improvements on realistic large datasets. The paper is well written and accompanied with thorough analysis. The motivation for the exact form of the acquisition function is somewhat lacking and would be useful to add. I lean towards acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new acquisition function Balanced Entropy Acquisition (BalEntAcq) for active learning, which captures the information balance between the uncertainty of underlying softmax probability and the label distribution. The new acquisition is designed to be computed without relying on other data points, which allows parallelization in computation. With the Beta distribution approximation of the marginal distribution, the authors derive a closed-form expression of the proposed acquisition. The authors run experiments and show the efficacy of the proposed method.
strength_and_weaknesses: Strength:
1. This paper proposes a theoretically-motivated (Thm 4.1) acquisition function BalEntAcq that can select a diverse set of data points near the decision boundary (at least in the toy example studied in Section 4.2). 
2. The acquisition function can be computed without relying on other data points, which allows parallelization in sampling/computation.
3. Empirical results show the efficacy of the proposed acquisition function.

Weaknesses:
1. It seems that the calculation of BalEnt relies on the Beta distribution approximation of the marginal probability. How difficult is it to calculate the acquisition function without the Beta distribution approximation?
2. It would be great if the authors can add further explanations for the choice of BalEnt[x]^{-1} in BalEntAcq. Currently, it seems to me that such a choice is mainly motivated by the empirical results shown in Appendix A.13.2.
3. Since one selling point of the proposed acquisition function is that it can be computed without relying on other data points, many existing methods (e.g., BADGE) do. I think the authors should add a discussion on the computational gain of the proposed method (e.g., what is the computational complexity of your proposed method, and what is the computational complexity of BADGE).
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easy to follow. The proposed acquisition function looks novel to me, and it is theoretically motivated. I suggest the authors address the above-mentioned weaknesses to improve the quality of the paper.
summary_of_the_review: The proposed acquisition function is theoretically motivated and allows parallelization in computation; empirical results also confirm the efficacy of the proposed method. Overall, I am inclined to accept this paper. However, I believe this paper can be made stronger if the authors can address the weaknesses mentioned above. 

====after rebuttal====

I would like to thank the authors for their response. I have read the response and the comments from other reviewers, and I would like to remain my scores.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper proposes a new acquisition function Balanced Entropy Acquisition (BalEntAcq) for active learning, which captures the information balance between the uncertainty of underlying softmax probability and the label distribution.  The new acquisition is designed to be computed without relying on other data points, which allows parallelization in computation. With the Beta distribution approximation of the marginal distribution, the authors derive a closed-form expression of the proposed acquisition. The authors run experiments and show the efficacy of the proposed method.

Strengths:

- Through extensive simulations and comparisons with baselines.
- Comprehensive review of relevant existing methods
- Enhanced AL performance in a variety of practical settings and its low computational cost and scalability.
- Clearly written and easy to follow.
- The acquisition function can be computed without relying on other data points, which allows parallelization in sampling/computation.

Weaknesses:

- Scalability results not clear.
- The proposed acquisition function could be motivated better.

Recommendation:

All the reviewers vote for acceptance. I, therefore, recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.

justification_for_why_not_higher_score: The paper presents a good contribution to the area of active learning. The results obtained are good and promising, but at the moment there is a large number of different acquisition functions presented in this area and most of them only provide marginal gains. This paper is no different.
justification_for_why_not_lower_score: All reviewers vote for acceptance. 2 with strong scores.
Decision: Accept: notable-top-25%