Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new principled active learning algorithm called BAIT, which uses the Fisher information matrix to select informative samples. The approach uses a greedy approximation to solve the global batch acquisition problem. Compared to other approaches, it also works for regression.

The paper provides ablation and gives a new perspective on BADGE, another active learning algorithm.

Multiple experiments both for classification and regression are provided.
main_review: ### Originality

The approach seems interesting. It focuses on using Fisher information matrices to select points that will minimize the overall loss on the pool set: $\underset{S \subset U,|S| \leq B}{\operatorname{argmin}} \operatorname{tr}\left(\left(\sum_{x \in S} I\left(x ; \theta_{1}\right)\right)^{-1} I_{U}\left(\theta_{1}\right)\right)$.

The algorithm to select points is novel in particular. The objective is not submodular, so greedy selection does not have guarantees. Instead, the paper greedily selects more samples than required and then prunes the batch again. (I particularly like the forward (expansion) and backward (pruning) approach. Could this be repeated multiple times to improve the candidate batch?) However, there is no theoretical analysis available of this approach yet.

Fisher Information has been applied to Active Learning with DNNs previously, e.g. ["Active Deep Learning with Fisher Information for Patch-wise Semantic Segmentation", Sourati et al, 2018 (https://pubmed.ncbi.nlm.nih.gov/30450490/)](https://pubmed.ncbi.nlm.nih.gov/30450490/).

The main contribution lies in the novel algorithm (BAIT) that does not require semi-definite programming to find batches and in examining BAIT empirically in deep learning.

Moreover, the paper provides a new perspective on BADGE, an earlier active learning algorithm. It shows that BADGE makes stronger approximations than BAIT. An ablation shows that BAIT outperforms BADGE. I like that BAIT does take into account the whole pool set (through the overall information matrix).

### Quality

Overall, the submission seems sound. The experiments in the main paper are convincing and necessary ablations are provided. The method is very well motivated and deduced. 

While Figure 6 in the paper convincingly shows that BAIT outperforms all other methods, the respective plots in the appendix are less clear than the ones in the main paper. In particular, it seems that the proposed method, BAIT, does not really outperform BADGE on SVHN and CIFAR-10 by much at all. It would be good to swap the experiments on MNIST with another higher-dimensional or more challenging dataset. Maybe CINIC-10 might provide more conclusive results (https://paperswithcode.com/dataset/cinic-10).

Results are also presented for OpenML dataset 155. However, I do not know this dataset and it is not introduced. (That said, citations for MNIST, SVHN and CIFAR-10 are also missing.)

Lastly, the regression experiments use a rather contrived setup. Fitting one-hot encodings is a very specific objective. However, I cannot suggest anything better.

### Clarity

The paper is very well written. The introduction and related work section are great. As noticed, a few more citations ought to be added. The introduction is also rather light on citations.

### Significance

The results appear very significant assuming they perform better than previous methods, which I am not fully convinced of yet.
The proposed algorithm appears significant by itself as the idea of selecting and pruning alternatively is novel. However, given the shown results on CIFAR-10 and SVHN, it would not be clear whether this method is truly SOTA, and whether to recommend this method over BADGE as a baseline in the future.

The application to regression is significant and definitely an improvement over other methods (like BADGE) which only work with classification. However, the regression experiments are not as convincing.

### Summary

The paper introduces a new AL method, which is well motivated and clearly communicated. This method is not limited to classification and also works for AL for regression.

The plots in the appendix are not as clear as I'd like, and I would like to verify that BAIT outperforms BADGE manually as well instead of trusting Figure 6.

### Questions

1. Could you provide experimental results on another higher-dimensional dataset? (E.g. CINIC-10)
2. Could you smooth the plots (using a Gaussian window or similar) to make the SVHN and CIFAR-10 results more easily interpretable?
3. Could you provide a less contrived regression experiment?
4. How dependent is this method on the # of latent dimensions?

I would be happy to increase my score if any or all of the above questions are answered.

### Typos

1. [29] and [30] in the bibliography refer to the same paper.
2. (4): it is not clear what $I_U$ exactly is at first.
3. l.184: what is $V_x$? It has not been introduced.
4. Sec 5.1: Figure 2 is never directly referenced in the text.
5. Figure 5's caption says CIFAR-10 while the plot says OML155.
6. l. 316: what is N? Number of trials?

Disclaimer: I did not check the proofs in the appendix in great detail.

limitations_and_societal_impact: There is no mention of societal impact, which given what other papers have written is not a big loss, as this method does not have any specific new societal impact compared to existing AL methods.

The paper does not mention limitations per-se beyond the high computational load. I wonder how dependent this method is on the # of latent dimensions.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces an algorithm (BAIT) for batch active learning, designed for and tested on neural networks. It centres around an objective in eq 4, argmin_select trace(I_select^-1, I_all), with I as the Fisher information matrix of the final layer parameters of a neural network, an objective that has been previously derived/analysed in simpler settings. The contribution here is applying it to deep neural networks. It draws links with a current SOTA method, BADGE, and shows slight performance benefits over it.
main_review: I’m borderline on this paper. I do find the method interesting, and the empirical performance seems to be in line with the current SOTA method BADGE. However, I believe the presentation of the method needs improving, and there are various claims that I am uncomfortable with, including interpretation of results. Overall, I think the paper could be very strong following a sober revision.

__Method Structure/Clarity__

I found the motivating theory (section 4) of the method confusing. It begins by considering the case of Bayesian linear regression to derive eq 3, but then jumps to pulling out eq 4 from Chaudhuri paper, which it says the paper will optimise. What is the use of beginning with eq 3? The setting considered in the paper is neither Bayesian, nor linear, nor generally regression. I think it would be more worthwhile to discuss eq 4 and the Chaudhuri paper in fuller detail. E.g. it’s said this was shown to be ‘near optimal’ under their set up. What was this set up? What assumptions were required? How do these translate to the neural set up?

Several times (including abstract) it’s stated that we’re optimising a bound on the MLE error. But nowhere in the technical sections did I see any bound. Under what conditions is it a valid bound (I'm assuming just the linear setting)?

As a smaller point, the paper gives the impression that viewing the output of a network as an estimated probability distribution is a novel perspective – I’d argue that this is the default position.

In general, I didn’t come away feeling like I have much intuition for why it’s a good idea to choose a set to minimise eq 4.


__Claims and Wording__

Several times it’s claimed that the method addresses problems of non-convexity in neural systems (including the abstract). But actually, the Fisher information matrix is only ever considered for the last layer parameters, with the majority of the network used as a feature extractor. This is ok as a methodology, but it feels like misleading advertising.

The paper claims to achieve SOTA results. This is a bit of a hard sell when one looks over the full learning curves in the appendix, where curves for BAIT vs BADGE for CIFAR10 and SVHN are pretty well indistinguishable. The zoomed regions in figure 5 seem rather cherry-picked. I’m unsure if very slight advantages justify BAIT’s increased computational demands (k times). As I reviewer, I don’t demand SOTA results if the method is interesting enough in itself, which I think is the case here, and would prefer the authors to have a more considered results discussion.

There does seem to be a performance improvement in the linear setting (‘random embeddings’), but this is of less interest given the paper’s title and main objective. In some ways it seems unfair to give these figures (2, 4) such prominent position.


__Experiments__

Being able to apply the method in regression settings is a great advantage, but the experiments tested simply took the classification datasets (MNIST etc) already used and converted them to regression. This feels like an opportunity missed, as there there are plenty of interesting regression tasks, e.g. depth regression, counting problems, rainfall prediction...

To test the linear setting, data is 'randomly projecting data into d dimensions’ (rather than using features extracted by a neural net). I think the use of ‘random’ is a bit misleading here as it seems to have used some dimensionality reduction technique? I didn’t spot how this projection was done, but it seems important to describe it.

The algorithm works by selecting 2B examples, then discarding B of these. I’d be interested to see ablations/rationale for how this number was selected. Could the algorithm be improved with a larger number?


__Minor__
- Seems like section headings have been crushed up, eg line 262
- Algorithm 1, perhaps move ‘forward greedy optimization’ to line 5?
- Subscripts and superscripts missing on eqs on line 184
- Think redundant /2 in line 498

limitations_and_societal_impact: There’s a short paragraph at the end of the paper outlining the additional computation complexity of the method. Considering the addition of an extra page in this year’s conference, I think this should be expanded.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles active learning with deep learning models, focusing primarily on the problem that labels need to be requested in batches due to the large training cost. The authors propose to rely on the Fisher information and introduce a greedy approach to choose each set of points to be labelled efficiently.

main_review: ## Originality
While the specific approach of the Fisher matrix is new and the paper mentions that it is revisiting a "classic, Fisher-based objective" (l42), it is lacking a proper discussion of the prior work of using such approaches, which have been explored before (e.g. back to MacKay, (1992) who relied on it for active learning with neural networks). 

Similarly, the citations and general presentation require a lot of improvements. The main competitor, BADGE, is mentioned first in the introduction and repeatedly throughout the paper. Yet, it is never connected with a reference leaving the reader to guess its source (ref 35 is mentioned, but only independently). The same applies to a second baseline, coreset, which is never connected to reference 8. Similarly, at other locations, e.g. while one could, if need be, argue that MNIST/SVHN/Cifar10/ResNet/Adam have become common knowledge, it would still be cleaner to cite them. However, the same argument does not apply to something like "OpenML dataset 155" (l293), which I would consider to be nonstandard and unknown by the average reader.


## Quality
The method is presented in a somewhat biased way. E.g. with repeated claims in the regression setting, both that other active learning methods are lacking in that direction, and even deep learning usually being only discussed for classification (l335); a consistent improvement, but small in absolute numbers, is sold as "large margin" (l282) and "drastic" (caption of Figure 4). 
Apart from such outliers and the citation problems mentioned above, the presentation is well done. The method is explored on several data sets (synthetic and real), several architectures and batch size settings, with extensive further code being provided. 

The comparisons are primarily against deterministic methods (Badge, Coresets), lacking a comparison against the growing literature of Bayesian Deep Learning. E.g. the Gal et al. (2017), Kirsch et al. (2019) (the latter of which actively targets batch active learning). What do the authors expect performance-wise compared to such approaches that actively include model uncertainty as well as predictive uncertainty, together with query rules adapted for Bayesian Neural Nets (e.g. (Batch)BALD)?


## Clarity
Apart from the details mentioned in the two sections above, the paper is written clearly and well organized.

## Significance
The significance of the proposed approach is difficult to judge. In relative performance, the proposed model shows consistent improvements upon the baselines (especially the extended experiments in the appendix). Yet, in absolute numbers, the improvements upon simple approaches such as randomly picking new points or a vague confidence metric remain small. This leaves the question open as to how much of a practical influence it will have. It would be interesting to see the performance of the proposed approach to a highly redundant setting where most of the unlabeled points are irrelevant, punishing random query strategies and requiring a clever query, such as the proposed, to select the few highly relevant points.



## Further Questions/Minor comments/typos
- l32 claims a lack of principled batch active learning algorithms. Yet, the paper cites both Sener et al. and Kirsch et al., which try to tackle exactly this task in a principled way.
- Can the authors comment on the repeated claim of lack of approaches for active learning in the regression setting (e.g. l49-50)? A lot of Bayesian uncertainty-based methods are directly applicable to the regression setup. Similarly, Active Learning's closely related sister field, Bayesian Optimization, regularly tackles continuous and discrete parameter spaces. 
- l66 claims references [11-15] to be about batch AL but separate from deep learning. Yet [15] explicitly tackles the task of Bayesian Deep Learning.
- Error: References [29], [30] are the same paper
- $V_x$ appears out of thin air in the equation starting in l184 without a proper introduction (and in a typo misses one subscript after the first equality)
- Error: The right plot in Figure 5 mentions OML155 in its title, while the caption claims it to be Cifar-10.
- Section 6.3 claims deep learning to be most commonly discussed within a classification framework, but it is, of course, also applied to a lot of regression tasks. E.g. staying in the image world, the whole area of depth estimation etc. 
- Figure 1 shows a consistent improvement of the proposed forward/backward approach vs the strict forward, yet the text claims that this holds only "sometimes" (l180). Are there setups where it always performs worse or is it only due to random initializations that sometimes cause it to perform worse in some runs?



___
(MacKay, 1992: Information-based objective functions for active data selection)

limitations_and_societal_impact: The main limitation discussed is the increased cost per sample selection. A societal impact is not discussed (chosen as not applicable in the checklist). 

The societal impact follows the general trend of deep learning approaches here. Better active learning strategies allow for more efficient labeling, giving us less wasted human time as a direct societal benefit. To a second degree, it then depends on whether the labeled data is used for beneficial or harmful tasks.
review_title: null

Meta-review: Reviewers agreed that this is a good contribution to NeurIPS, *provided* that the additional experiments and clarifications provided by the authors during the discussion are included to the paper. 
Decision: Accept (Poster)