Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Quality and clarity: the paper is well-written with good knowledge of the current state of the research in the subject area.

Significance: active learning is an important practical direction in machine learning. It currently has two main approaches - one is based on the uncertainty and another one on selecting the most representative subsets. The effective combination of these two is a promising direction of the research and it is addressed in the paper.

Originality: The idea to combine the uncertainty and diverse points selection is not exactly novel, but the paper proposes an interesting idea of improving the point selection with the rescaled geodesic search.

Pros:
* The proposed method is well-reasoned
* Good mathematical justification for the method
* Tractable method implementation is presented.

Cons:
* The results in Table 1 are a bit confusing as the provided std values are quite large. The explicit p-value for the proposed method could be relevant.
* The experiments compare the new method only with basic ones like k-centers and BALD. More complex approaches (like batch BALD [Kirsch, 2019] and Sparse Subset Approximation [Pinsler, 2019]) were mentioned, but not presented in the main paper experiments for some reason, while they use a similar idea and show comparable improvement on the image datasets.
* Experiments 5.1, 5.2 - with a small number of initial samples and large batches, the BALD sometimes performs worse than random sampling; it would be good to add random sampling here for reference.
* The experiments are on the relatively small and simple image datasets, but it’s not clear how well the method would work on real-world size images, as the method partly relies on the l2 norm and it’s not a good metric for bigger images.

Reference
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch
acquisition for deep bayesian active learning. In Advances in Neural Information Processing
Systems, pp. 7024–7035, 2019.

Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch
active learning as sparse subset approximation. In Advances in Neural Information Processing
Systems, pp. 6356–6367, 2019.
review_title: Interesting idea and well-designed method, but experimental results and baselines are relatively weak

final_score: rating: 3: Clear rejection
review_content: This paper is basically unreadable. The sentence structure / grammar is strange, and if that was the only issue it could be overlooked. The paper also does not describe or explain the motivation and interpretation of anything, but instead just lists equations. For example, eta is the parameter that projects a spherical geodesic onto an the ellipsoid one, and an ellipsoid geodesic prevents updates of the core-set towards the boundary regions where the characteristics of the distribution cannot be captured. However, what are these characteristics, and how can they motivate how to choose eta?
review_title: Unreadable

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper introduces a new active learning algorithm called Geometric BALD, or GBALD. While the empirical experiments look promising, the writing for this paper is a bit strange.

Comments and questions:
 - It is a bit strange to say the "deep learning community introduced active learning" and cite a 2017 paper, since active learning has been around for over 20 years before that.
 - I'm confused by the line "BALD has two interpretations: model uncertainty estimation and core-set construction" in the introduction. Can the authors provide a reference or argument for this statement? How is core-set construction an interpretation of BALD?
 - What do the authors mean by "Recently, deep Bayesian AL attracted the community."? What community?
 - How is $p(D_0 | \theta)$ a "prior"? Perhaps the authors meant $p(\theta | D_0)$, but that would be a posterior based on the initial seed set?
 - I don't think this paper uses the term "uninformative prior" correctly. For example, the paper states, "For example, unbalanced label initialization on $D_0$ usually leads to an uninformative prior,".
 - Why is the theory (appendix B) done specifically with three-dimensions? Don't most datasets have many more dimensions?
 - In appendix B, what sort of mathematical objects are A and B? They are explicitly said to be "two classes" but there are spheres that "tightly cover" A and B.
review_title: A possibly promising but strangely written paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper the author/s study/ies the relvant problem of acive learning, i.e. the setting in which labeled data are not available. The innovation of the paper consist in constructing the core-set on an ellipsoid, and not on the typical sphere, preventing its updates towards the boundary regions of the distributions.
The paper states the main improvements of GBALD, the proposed method, over the existing method BALD are twofold: a) to relieve the sensitivity analysis to uninformative prior 
b) to reduce redundant information related to model uncertainty.
Improvements of GBALD over existing models are proved through comparison to BALD spherical interpretation.
In particular, the paper shows that a geodesic search with ellipsoid can give a "lower error bound" which is tighter than that in the specialied literature. Furthermore, the paper aims to prove that with high probability the proposed approach achieves zero error. Numerical experiments conducted under different scenarios prove GBALD improves on BALD and BatchBALD which are considered ot be state-of-the-art meyhods.

----------------------------------------------------------------------------------------------------------------------
Reason for Score:
Overall, I vote for rejecting. I like the cleaness of the proposed approach but I think that the paper in the current status is not yet mature for such a relevant venue as ICLR. However, my opinion is better described in the  Cons section.

----------------------------------------------------------------------------------------------------------------------
Pros.
1) the paper tackles a very relevant problem, the one of unlabeled data.
2) I appreciate the clean mathematical approach.
3) the proposed approach is well described.
3) numerical experiments, on the selected data sets, witness in favour of the proposed method, even if I also have concerns on this part of the paper.
----------------------------------------------------------------------------------------------------------------------
Cons.
1) a fundamental aspect of the proposed approach, when compared to alternative methods, is missing, i.e., the computational cost of the proposed approach when compared to other methods.
2) I was not able to find a discussion about the relevance or cost incidence about data labeling. Which is the concrete advantage of labeling 100 in place of 200 cases? In oother words I would like to provide motivations for the problem tackled and a quantitative assessment of the advantage of the proposed approach.
3) I'm increasingly concerned with the fact that only image datasets are taken into account in numerical experiments, images have a highly structured nature, the concept of smoothness, closeness between pixels, etc... I truly would like to see how the proposed method improves on different types of data, health data, biology data, financial data, ...
If the proposed approaches are confined to image datasets then this should be clearly stated in the paper in my humble opinion.
4) the proposed approach, even if it is well desribed and analyzed, is somewhat straightforward. Indeed, the entire contribution builds on using an ellipsoid in place of shpere.
5) In figure 4 I do not appreciate a strong difference between the proposed apporahc and BALD, at leat for digit 1 and 0, which I also would like to know why they were selected and not othe digits.
6) In figure 6, the advantage of the proposed approach over alternative methods it is limited to the very beginning of the left part of the plots, I can non understand which is the effective/practical/economic advantage of such a difference. I would kindly ask the author to help me understand it.
7) In table 1; standard deviations are huge and thus the expeted value of accuracy, achieved by differente methods, are extrelemy un certain, I would like the authors to help me understand how this impact their conclusions.
8) In Table 2, K-medoids seems to be not that bad, what about comparison in terms of computational efforts to the proposed approach?
----------------------------------------------------------------------------------------------------------------------
Questions during rebuttal period: 

Please address and clarify the cons above 
----------------------------------------------------------------------------------------------------------------------
Some typos: 

page 2: I read "which are no longer distributed uniformly", I would like the author/s to better clarify on this
	I read "It is thus the ranking rejects unnecessary redundant acquisitions.", it makes no sense to me
page 5: equation (10), I would suggest the use of DJ and not Di, indeed in previous part you used xi and Dj, while now using xi and Di makes me confuse
----------------------------------------------------------------------------------------------------------------------
review_title: An incremental contribution which seems to provide marginal improvement in performance

Meta-review: All reviewers express concerns, such as about the presentation, the situation of the paper w.r.t. prior work, the experimental evaluation etc., and recommend rejection.
Decision: Reject