Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Active learning for deep learning is an interesting topic and there is few useful tool available in the literature. It is happy to see such paper in the field. This paper proposes a batch mode active learning algorithm for CNN as a core-set problem. The authors provide an upper bound of the core-set loss, which is the gap between the training loss on the whole set and the core-set. By minimizing this upper bound, the problem becomes a K-center problem which can be solved by using a greedy approximation method, 2-OPT. The experiments are performed on image classification problem (CIFAR, CALTECH, SVHN datasets), under either supervised setting or weakly-supervised setting. Results show that the proposed method outperforms the random sampling and uncertainty sampling by a large margin. Moreover, the authors show that 2-OPT can save tractable amount of time in practice with a small accuracy drop.

The proposed algorithm is new and writing is clear. However, the paper is not flawless. The proposed active learning framework is under ERM and cover-set, which are currently not supported by deep learning. To validate such theoretical result, a non-deep-learning model should be adopted. The ERM for active learning has been investigated in the literature, such as "Querying discriminative and representative samples for batch mode active learning" in KDD 2013, which also provided an upper bound loss of the batch mode active learning and seems applicable for the problem in this paper. Another interesting question is most of the competing algorithm is myoptic active learning algorithms. The comparison is not fair enough. The authors should provide more competing algorithms in batch mode active learning.
review_title: Active learning for CNN

final_score: rating: 7: Good paper, accept
review_content: This paper studies active learning for convolutional neural networks. Authors formulate the active learning problem as core-set selection and present a novel strategy.

Experiments are performed on three datasets to validate the effectiveness of the proposed method comparing with some baselines.

Theoretical analysis is presented to show the performance of any selected subset using the geometry of the data points.

Authors are suggested to perform experiments on more datasets to make the results more convincing.

The initialization of the CNN model is not clearly introduced, which however, may affect the performance significantly.

review_title: An interesting work with both theoretical and empirical results

final_score: rating: 7: Good paper, accept
review_content: After reading rebuttals from the authors: The authors have addressed all of my concerns. THe additional experiments are a good addition.

************************
The authors provide an algorithm-agnostic active learning algorithm for multi-class classification. The core technique is to construct a coreset of points whose labels inform the labels of other points.  The coreset construction requires one to construct a set of  points which can cover the entire dataset. While this is NP-hard problem in general, the greedy algorithm is 2-approximate. The authors use a variant of the greedy algorithm along with bisection search to solve a series of feasibility problems to obtain a good cover of the dataset each time.  This cover tells us which points are to be queried. The reason why choosing the cover is a good idea is because under suitable Lipschitz continuity assumption the generalization error can be controlled via an appropriate value of the covering radius in the data space.  The authors use the coreset construction with a CNN to demonstrate an active learning algorithm for multi-class classification. 
The experimental results are convincing enough to show that it outperforms other active learning algorithms. However, I have a few major and minor comments.

Major comments:

1. The proof of Lemma 1 is incomplete. We need the Lipschitz constant of the loss function. The loss function is a function of the CNN function and the true label. The proof of lemma 1 only establishes the Lipschitz constant of the CNN function. Some more extra work is needed to derive the lipschitz constant of the loss function from the CNN function. 

2. The statement of Prop 1 seems a bit confusing to me. the hypothsis says that the loss on the coreset = 0. But the equation in proposition 1 also includes the loss on the coreset. Why is this term included. Is this term not equal to 0?

3. Some important works are missing.  Especially works related to pool based active learning, and landmark results on labell complexity of agnostic active learning.
UPAL: Unbiased Pool based active learning by Ganti & Gray. http://proceedings.mlr.press/v22/ganti12/ganti12.pdf
Efficient active learning of half-spaces by Gonen et al. http://www.jmlr.org/papers/volume14/gonen13a/gonen13a.pdf
A bound on the label complexity of agnostic active learning. http://www.machinelearning.org/proceedings/icml2007/papers/375.pdf

4.  The authors use L_2 loss as their objective function. This is a bit of a weird choice given that they are dealing with multi-class classification and the output layer is a sigmoid layer, making it a natural fit to work with something like a cross-entropy loss function. I guess the theoretical results do not extend to cross-entropy loss, but the authors do not mention these points anywhere in the paper. For example, the ladder network, which is one of the networks used by the authors is a network that uses cross-entropy for training.

Minor-comment: 
1. The feasibility program in (6) is an MILP. However, the way it is written it does not look like an MILP. It would have been great had the authors mentioned that u_j \in {0,1}. 

2. The authors write on page 4, "Moreover, zero training error can be enforced by converting average loss into maximal loss". It is not clear to me what the authors mean here. For example, can I replace the average error in proposition 1, by maximal loss? Why can I do that? Why would that result in zero training error?

On the whole this is interesting work and the results are very nice. But, the proof for Lemma 1 seems incomplete to me, and some choices (such as choice of loss function) are unjustified. Also, important references in active learning literature are missing.
review_title: This is interesting work and the results are very nice. But, the proof for Lemma 1 seems incomplete to me, and some choices (such as choice of loss function) are not properly justified. Also, important references in active learning literature are missing.

Meta-review: The effectiveness of active learning techniques for training modern deep learning pipelines in a label efficient manner is certainly a very well motivated topic. The reviewers unanimously found the contributions of this paper to be of interest, particularly nice empirical gains over several natural baselines.
Decision: Accept (Poster)