Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to perform active learning using pool selection of deep learning mini-batches using an approximation of the bayesian posterior. Several terms are in turn approximated.

The Maximum Likelihood Estimation (MLE) bayesian inference approach to active learning, the various approximations, and more generally the theoretical framework is very interesting but difficult to follow.

The paper is written in poor English and is sometimes a bit painful to read.

Alternative Active learning strategies and techniques do not need to be described with such detail. On the other hand, the proposed approach has a lot of complex approximations which would benefit from a more detailed/structured presentation. Another dataset would be a big plus (both datasets concern gray digits and USPS and are arguably somewhat similar).
review_title: Very interesting but hard to follow.

final_score: rating: null
review_content: In Figures 1 and 2, what do you mean by groundtruth?

review_title: Groundtruth?

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quality:
The paper initiates a framework to incorporate active learning into the deep learning framework, mainly addressing challenges such as scalability that accompanies the training of a deep neural network. 
However, I think the paper is not well polished; there are quite a lot of grammatical and typing errors.

Clarity:
The paper needs major improvements in terms of clarity. The motivations in the introduction, i.e., why it is difficult to do active learning in deep architectures, could be better explained, and tied to the explanation in Section 3 of the paper. For example, the authors motivated the need of (mini)batch label queries, but never mention it again in Section 3, when they describe their main methodology. 
The related work section, although appearing systematic and thorough, is a little detached from the main body of the paper (related work section should not be a survey of the literature, but help readers locate your work in the relevant literature, and highlight the pros and cons. In this perspective, maybe the authors could shorten some explanations over the related work that are not directly related, while spending more time on discussing/comparing with works that are most related to your current work, e.g., that of Graves '11.

Originality & Significance:
The authors proposed an active learning training framework. The idea is to treat the network parameter optimization problem as a Bayesian inference problem (which is proposed previously by Graves) and formulate the active learning problem as that of sampling the most informative data, where the informativeness is defined by the variational free energy, which depends on the Fisher information. To reconcile the computational burden of computing the inverse of Fisher Information matrix, the authors proposed techniques to approximate it (which seems to be novel)

I think that this paper initiates an interesting direction: one that adapts deep learning to label-expensive problems, via active learning. But the paper needs to be improved in terms of presentation.


review_title: Interesting ideas but not well explained

final_score: rating: null
review_content: In the Experiments of figure 2 what criterion did you use to decide to use 5 or 10 runs of experiments ? Does figure 2 correspond to only one active learning test or are the curves averages over several tests ?
review_title: Experiments

final_score: rating: null
review_content: You mention existing work on the simple approach of uncertainty sampling. Did you try this as a baseline?
review_title: Simple active learning baseline?

final_score: rating: null
review_content: Section 5.2 studies the time complexity of your approach -- up to 30s to select the elements of one minibatch. How does this compare to the time required for using that minibatch to update the model by backprop?
review_title: Overhead of active learning?

final_score: rating: null
review_content: Some of the approximations are quite complex; is your code available?
review_title: Code available?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a mechanism for active learning with convolutional neural networks (CNNs). I would not go as far as the authors in calling these "deep", seeing that they seem to have only 2 hidden layers with only 20 filters each. The active learning criterion is a greedy selection scheme based on variational free energy and a series of approximations.

The paper is sometimes hard to read, due to (a) many grammatical errors and (b) sloppy notation in some places (e.g., on page 5, line 1, f is used but never introduced before). Overall, I give an accepting score, but a weak one because of the grammatical errors. If the paper is accepted, these should be fixed for the final version, optimally by a native speaker. 

The paper's topic is interesting, and the paper appears to succeed in its goal of showing a proof of concept for active learning in CNNs (if only on toy datasets). I'm surprised by the new results on uncertainty sampling and curriculum learning the authors added: why do these methods both break for USPS? In particular, uncertainty sampling did very well (in fact, better than the authors' new method) on MNIST, but apparently horribly on USPS; some explanation for this would be useful.

I have one more question: why is it necessary to first sample a larger subset D \subset U, from which we select using active learning? Is this merely done for reasons of computational efficiency, or can it actually somehow improve results? (If so, it would be instrumental to see the worse results when this is not done.)

review_title: Proof of concept for active learning with CNNs

final_score: rating: null
review_content: In the approximation of Q(\beta), which is crucial for the importance sampling of the data in the authors' framework of active learning, they assumed the current MLE estimator w_hat, is already a good approximation to the true parameter \theta_Y^*; how can this be justified? How is the sampling distribution initialized?
review_title: A strong assumption?

Meta-review: The reviewers agree that the paper pursues an interesting direction to explore active example selection for CNN training, but have unanimously raised serious concerns with regards to overall presentation which needs further improvement (I still see spelling/grammatical errors/sloppy notation in the latest draft). Some sections in the paper are hard to follow. With regards to technical motivation, the link between depth and need for active example selection is alluded to, but not properly explained in the paper. The PCs think that this paper has too many areas in need of improvement to be accepted to the conference.
Decision: Reject