Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method for pool-based active learning in CNNs, selecting the next (batch of) data from an unlabeled pool to query their labels to expedite the learning process. The method computes the expected reduction in the predictive variance across a representative set of points and selects the next data point to be queried from the same set. In batch settings, the data points are sequentially selected in a batch (in a greedy way), with predictive variance representation updated after each selection. Experiments are performed on MNIST classification, and regression tasks of alternative splicing prediction, and face age prediction.

Pros:

The method is rather simple and (up to some extent) computationally efficient. The paper is generally well-written and puts the proposed method into context. The proposed method performs better than maximum variance acquisition and random acquisition in the regression experiments. For the classification tasks, its performance is comparable with maximum entropy, batch BALD and robust k-Center acquisition.

Concerns:

Overall, the novelty of the paper is limited. It employs dropout-based variational inference and applies the integrated variance minimization idea. The main contribution seems to be considering a joint normal distribution (whose level of validity is not completely clear/discussed). It is not clear if the proposed acquisition is the statistically optimal one that minimizes the expected MSE. It seems to me that requires taking into account the distribution of the (unobserved) label. Also, assuming all predicted class probabilities across all sample points as jointly Gaussian is strange as the probabilities are bounded and should sum to 1. In fact, the performance comparison in the classification setup is limited to one dataset and the results do not seem to confirm that the proposed method outperforms the baselines. 

Figure 2 seems to suggest that with increasing batch size, the performance gap shrinks. Have the authors tried a larger batch size or checked the sensitivity to it? 

Additionally, how sensitive is the performance to the number of randomly sampled data points for the representative set?

My current rating is based on the aforementioned concerns.

Update:
I thank the authors for their response. I read the authors’ responses and the updated paper. As the authors have addressed some of my concerns and questions, I have adjusted my rating. However, I still have my concerns regarding the novelty and methodological contribution, and the classification setup. I also agree with other reviewers that the presentation can be improved.


review_title: The paper combines dropout-based variational inference and integrated variance minimization

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper Summary

The paper considers the problem of active learning for training convolutional neural networks (CNN) in a sample-efficient manner. The proposed approach is built upon the existing idea of selecting points that maximally reduce expected mean squared error (MSE) on a large representative sample of points. MC-dropout is used for obtaining the estimates of model uncertainty. This idea is used for active learning in regression and classification problems with CNNs. A greedy method is proposed to select a batch of points by maximizing the acquisition function score sequentially obtained by updating the covariance matrix on previous points selected in the batch. Experiments are performed on two regression and one classification task. 


Detailed Comments

- The paper tackles an important problem relevant for many practical applications. Although the proposed approach is based on an existing idea, the application to CNNs specifically seems novel. 

- It is a little worrying to see that random acquisition performs equally well with the proposed acquisition function (Figure 2 for instance). Please provide more discussion on this point and/or run one experiment for more than 3 (current) random runs to get a clear picture. Random acquisition is a simple to implement method and if the gains by the proposed acquisition function are not large enough, random acquisition will be a preferable option for a user. 

- There is limited description about how is the ''representative sample chosen'' in the experimental section. Please provide more details on this. 

- Greedy batch selection methods are known to select similar points losing the batch advantage. Please provide more details on the diversity of the points in batch selection.  

- Why MC-dropout specifically is used for uncertainty estimation? The proposed approach should ideally be agnostic to the method for uncertainty estimation. Please provide more discussion on this choice. 

- In my opinion, although it is a minor point, it is better to show the visualization of EI acquisition function on a standard synthetic function like a sinusoidal function, Branin function etc. instead of a random neural network which might seem a little contrived.

- The writing of the paper can be improved. For example, the usage of the term 'Expected improvement' is spread out across the paper without a proper technical description. Please try to provide a clear technical description of all the key concepts

review_title: Official Review #3

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper considers the active leaning problem by proposing a new acquisition function based on Expected improvement (EI). The paper shows that acquiring the point maximizing the expected reduction in predictive uncertainty across all points is equivalent to maximizing the expected improvement (EI).


The paper considers the setting for CNN classification and regression where the authors derive the variance estimations. The paper also extends to the batch mode to select multiple points simultaneously. The uncertainty is estimated using MC dropout.

The papers show that the improvement in the experiment is not significant comparing to the baselines.

=============================
Major concern: although the paper claims to derive the EI-based acquisition function for active learning. The resulting derivation shows that the final acquisition function only depends on the uncertainty of the training data given the select point (x_new) while the relative contribution of the x_new toward the final performance E(y_train | x_new) has been vanished. Note that in the original form of the EI (Jones et al 1998), the acquisition function does not only depend on the uncertainty, but also the expected function value at the x_new.

The reviewer thinks that it may be better for this paper to position the contribution (abstract/introduction…) as the uncertainty based approach for active learning, then claim the minor/secondary contribution as showing the connection to the EI for the active learning setting. 

At the current form, the reviewer thinks the paper is under the acceptance threshold.


Minor concern:

The paper claims that they utilize full joint covariance rather than just point-wise variances used in previous work. However, the reviewer thinks this may be a wrong claim. In particular, the trace of variance considered in Eq 2 is equivalent to point-wise variance.



Other comments:

The intuition in the resulting acquisition function is very much related to the Predictive Variance Reduction Search [1] – it is worth mentioning. Both approaches rely on:
(1) uncertainty estimation of the some targets;
(2) the objective function is defined using the sum of the uncertainty reduction – the trace of variance in Eq (2) and the original form in Eq (1) share this spirit. 
(3) similar in the batch setting where both approaches will sequentially select point to fill in a batch.

The difference is that [1] considers another problem in Bayes Opt while this paper considers the active learning task.



[1]  Nguyen, V., Gupta, S., Rana, S., Thai, M., Li, C., & Venkatesh, S.  Efficient Bayesian Optimization for Uncertainty Reduction Over Perceived Optima Locations. In IEEE International Conference on Data Mining (ICDM) (pp. 1270-1275). 2019.


review_title: Official Blind Review #1

Meta-review: This paper proposes an approach for active learning in CNNs. The method computes the expected reduction in the predictive variance across a representative set of points and selects the next data point to be queried from the same set. 

Pros:
- The method is rather simple and seems practical. 
- The paper is generally well-written.

Cons:
- The novelty of the paper is limited, as it essentially applies a known approach to CNNs.
- The performance gains presented in experiments seem rather mild, and may not justify using this method.
Decision: Reject