Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors study the problem of incorporating unsupervised (representation pre-training) learning and semi-supervised learning into active learning for image classification; specifically, performing pre-training before active learning starts [Caron, et al., 2018] and then applying inductive label propagation [Issen, et al., 2019] (slightly modification in the cost function to look more like importance sampling) before active learning querying occurs for each round (Algorithm 1).  The most novel technical innovation of this submission is the joint label propagation (jLP) querying function (which is a method of ‘spanning’ the learned manifold space).  Experiments are conducted on four (multi-class) image classification datasets (MNIST, SVHN, CIFAR-10, CIFAR-100), showing that unsupervised learning and semi-supervised learning can improve active learning on these datasets — although random selection often works better (as best as I can tell) implying that negative results are also a contribution of this paper. Finally, some active learning experiments are conducted using a per-round label budget of one example per class — also demonstrating mixed results with random sampling performing better in general. 

In my mind, this paper has two primary components: (1) taking the position that semi-supervised and unsupervised learning can improve overall performance and, in principle, help with active learning and (2) propose jLP, which is a learning algorithm agnostic approach to spanning the manifold space. However, jLP doesn’t really seem to work in general. Thus, the main result is the first point — updating previous (pre-deep learning) results on SS/US AL to deep learning. Honestly, I think the primary conclusion is that semi-supervised and unsupervised learning has improved over the past decade (especially semi-supervised learning for image classification). The second result is that active learning in deep learning (at least for this application) hasn’t kept up. Wrt to (1), as the authors have pointed out, many others have applied semi-supervised learning to AL (including more that the authors didn’t include). Additionally, many have used unsupervised learning for AL (which the authors seem less aware of) from pre-clustering (e.g., [Nguyen & Smeulders, Active Learning using Pre-clustering; ICML04]) to one/few-shot learning (e.g., [Woodward & Finn, Active One-Shot Learning; NeurIPS16 workshop]) to using pre-trained embeddings for many ‘real-world tasks’ (e.g., NER [Shen, et al., Deep Active Learning for Named Entity Recognition; ICLR18] using word2vec). Thus, the interesting question would be to compare multiple pre-training techniques and ideally the relative effect on the active learning component (assuming this is the focus of the paper). With respect to semi-supervised learning, they have validated that inductive label propagation [Issen, et al., 2019] works for this task, but haven’t shown that this helps with active learning. Since this is a negative results without a theoretical contribution, I would again expect trying several semi-supervised algorithm and evaluating their relative performance in general and wrt the active learning querying strategy. Accordingly, I don’t think the contribution of this work in its current state is sufficiently well-developed — and would lean toward rejecting in its current form.

Below are some additional detailed comments (some also covered above): 
— Given that this points toward a negative result, a more convincing direction to take would be to consider more combinations of unsupervised and semi-supervised approaches — specifically emphasizing how they affect the active learning component. This might point to more general findings and maybe toward a theory (maybe even consider a second application).
— The empirical emphasis is more around overall performance rather than the interaction between unsupervised representation learning and active learning, which is more toward the stated goal of the paper.
— Wouldn’t the right way to do (deep) representation learning in multiple rounds be to fine-tune at least some fraction of the time?  If the only claim is pre-training or pre-clustering, people certainly do this — just often not as a point of emphasis.
— The ‘first semi-supervised’ claim really only holds in the context of deep learning; however, scope is really more like semi-supervised applied to image classification, which would be a pretty narrow contribution in scope.
— Overall, there is a general overstatement of contributions and results: this is certainly not the first SSAL or USAL and the statement relative to deep learning is subtle; some of the empirical results are interesting, but I am not sure about ‘spectacular gains’ (and these gains aren’t seemingly due to the contribution of the paper).
— I don’t understand the ensemble model analogy in the abstract; is it because it is a ‘meta-algorithm’?

Some more positive notes: 
+ It is interesting that there is some contradictory evidence relative to [Wang, et al., 2017; Gal, et al., 2017]; this is probably worth digging into a bit deeper.
+ The experimental details well-described given space constraints.

In summary, there are some interesting observations that are probably worth pursuing. However, the current contribution is basically that: (1) active learning doesn’t seem to really help, (2) semi-supervised learning and unsupervised learning improve performance for this task. Since (1) was really the point of the paper (as stated) in the title, I don’t think there is enough here to accept in its current form.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper explores the setting where unsupervised/semi-supervised learning is combined with active learning. The results are that active learning doesn't really help. This paper is interesting in that it provides additional experiments for the intersection of active learning and unsupervised/semi-supervised learning. However, I don't really see the point of this paper. Active learning and unsupervised/semi-supervised learning have been combined before and there are other papers submitted to ICLR this year that combine these. The paper does not claim to provide anything new algorithmically (other than jLP which appears to work no better than random and isn't really advertised as the point of this paper). The only conclusion that I can draw is that sometimes unsupervised/semi-supervised learning works better than active learning, but no understanding of when and why this is the case (from other papers, it is not always the case).


Comments:

 - Although the paper claims to yield a general framework, it only does so partially. For instance, the framework in this work is restricted to semi-supervised methods that use pseudo-labels. 

 - It may be the case that active learning doesn't help or even hurts because the batch size is too large and/or the initial seed set size is too small. Although this paper varies the acquisition strategies, these other hyper-parameters are equally, if not more, important.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper argues that active learning (AL) methods shold combine unsupervised and semi-supervised learning during the iterative training process. Combining these complementary is indeed sensible, and this work is therefore a welcome effort. However, the results are quite mixed, and in fact seem to suggest that AL is rather ineffective. Therefore, what one might take from these results is that unsupervised and semi-supervised learning methods can boost predictive performance; but I think this is widely appreciated already. Perhaps a better framing for this work is: AL using standard metrics seems to be comparatively ineffective, especially when one uses pre-training/semi-supervised learning. 

Some specific comments and questions:

- The authors have decided to frame this paper in terms of improving AL using un/semi-supervised learning. But given that, by the authors' own admission, the "random baseline may actually outperform all other acquisition strategies by a large margin", what is the motivation for adopting "AL" at all? I mean, if we are performing random (iid) sampling, this just reduces to vanilla learning with pre-training and semi-supervision; the 'active' component becomes irrelevant.

- I think the characterization of AL is not quite right on page 2. The authors write that AL is focuses on the "least certain" instances. This is often true -- namely under the popular uncertainty sampling regime -- but not all acquisition strategies use this heuristic. Indeed, even the geometry method the authors use explicitly ignores classifier confidence. 

- The use of sampling in the SSL component is interesting, although an ablation here investigating this specific choice (as opposed to, say, naive sampling with uniform probability over unlabeled instances).

- I would not characterize the gains brought by unlabeled data here as "spectacular".

- As is often the case in work on AL, there is no real notion of a 'test set' here; instead the authors repeat experiments using different seed label sets. It is not entirely clear how much hyperparameter/architecture fine tuning was performed informally, but there is a lot going on here, so I would assume at least some. Therefore there is a risk that all results reported are in some sense optimistic, potentially being "overfit" to these datasets. It would be best to provide additional comparisons of approaches on completely unseen datasets. 
review_title: Official Blind Review #3

Meta-review: This paper argues that incorporating unsupervised/semi-supervised learning into the training process can dramatically increase the performance of models. In particular, its incorporation can result in performance gains that dwarf the gains obtained by collecting data actively alone. The experiments effectively demonstrate this phenomenon. 

The paper is written with a tone that implicitly assumes that "active learning for deep learning is effective" and therefore it is a surprise and a challenge to the status quo that using unlabelled data in intelligent ways alone gets such a boost. On the contrary, reviewers found that active learning not working very well for deep learning is a well-known state of affairs. This is not surprising because the most effective theoretically justifiable active learning algorithms rely on finite capacity assumptions about the model class, which deep learning disobeys. 

Thus, the reviewers found the conclusions to lack novelty as the power of semi-supervised and unsupervised learning is well known. Reject. 
Decision: Reject