Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new combination method for active learning and semi-supervised learning, where the objective is to make predictions that are robust to perturbations (for SSL) and select points for labeling with labels that differ under perturbations. This technique achieves 2x label efficiency over SSL with uniform-random sampling. Additionally, the authors assess (at least for CIFAR-10 with batch size 50) the best starting random seed set as 100 labels, known as K_0 in this work. This work yields pretty good empirical results and has a conceptually unified approach to SSL and active learning building off of recent works. 


Comments:

 - This paper compares in Table 1 the difference between just active learning vs active learning + SSL. I'm not sure this is a fair comparison. I think the better comparison is shown in Table 2. 

 - The authors write that "when only 100 samples are labeled, our method outperforms kcenter by 39.24% accuracy". Do the authors mean after 100 additional labels are acquired (so 200 labels) or is this number off?

 - Can the authors clarify what is meant by "or some labels correspond to rare cases, as in self-driving cars"? Why are such datasets more costly to acquire? Is it because of the size of the self-driving car datasets?

 - Although the method is more unified than some other AL + SSL approaches, I wonder if the L_u(x,M) can be made to look more like the C(B,M) = \sum E(x,M). In particular, L_u(x,M) uses just a single perturbation and a different "distance" function than E(x,M) which uses N perturbations.

 - The authors state that they lose 1.26% accuracy to the fully supervised model. However, this is very much not within the margin of measurement error and 1.26% accuracy is rather significant for accuracies around 95%. Another way of putting it is that the method in the paper with 4K examples has 30% more error compared to the fully supervised method. Can the authors either change this claim or provide a number of labels where their method achieves the fully-supervised accuracy?



review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a semi-supervised active learning method to reduce the labeling cost. In the proposed method, a selection criterion to better integrate AL selection mechanism in SSL training framework is designed. The simple metric that aims to measure the inconsistency across a certain number of meaningful perturbations. It considers N perturbed samples of the original input data x, which can be obtained by standard augmentation operations (e.g. random crops and horizontal flips for image data). Then the variance is adopted to quantify consistency.  In this way, the proposed method prefers data samples with high values, which may possess varying level of difficulty for the model to classify. To verify the effectiveness of the proposed method, several baseline methods are compared on several benchmark data sets, and the proposed method has achieved better performance. Meanwhile, to deal with the “cold start” problem, a measure that is found to be empirically correlated with the AL target loss is proposed, and this measure can be used to assist in determining the proper start size. However, there are some minor concerns:
[1] The consistency of a sample is measured based on the perturbed samples. How to generate these perturbed samples may have a great influence on the query results. In the paper, it said that these samples are generated by standard augmentation operations (e.g. random crops and horizontal flips for image data). This representation is hard to follow in the experiments. If possible, it is better to show in details.
[2] In the uncertainty of active learning, the samples are selected from different distributions in the unlabeled data, for example, the marginal sampling selects the samples around the classification hyperlanes (Settles, Burr. "Active learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 6.1 (2012): 1-114.). Can you show which samples may be selected in the unlabeled data. In this way, the proposed criterion can be followed more easily.
[3] In the experiments, whether the proposed method can select a batch of samples at each iteration. How about the influence of the batch size. 

review_title: Official Blind Review #2

Meta-review: The authors leverage advances in semi-supervised learning and data augmentation to propose a method for active learning. The AL method is based on the principle that a model should consistently label across perturbation/augmentations of examples, and thus propose to choose samples for active learning based on how much the estimated label distribution changes based on different perturbations of a given example. The method is intuitive and the experiments provide some evidence of efficacy. However, during discussion there was a lingering question of novelty that eventually swayed the group to reject this paper. 
Decision: Reject