Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to combine active learning techniques with MixMatch for semi-supervised learning. First, they review active learning and semi-supervised learning, especially MixMatch. Instead of traditional semi-supervised learning with a fixed set of labeled examples, they incrementally grow the labeled set as the training process goes on. They consider several different choices in active learning strategies: uncertainty measure and diversification. Diversification methods are used to balance the samples in different classes and ensure diversity. The cost analysis of adding labeled vs unlabeled data looks interesting. They perform an empirical evaluation on image benchmarks and improve over MixMatch.

Overall, the paper is clearly written and easy to follow. However,  I cannot recommend acceptance because

1. Novelty concern. The combination of two existing techniques seems not novel enough.

2. Missing important baselines in related work and experiments. In the related work on semi-supervised learning (Section 3), the authors only review MixMatch but neglect other literature, e.g.[1,2,3,4]. And semi-supervised learning has a long history and it is not restricted to recent deep learning-based approaches. A thorough review can make the approach well-placed in the literature. In experiments, the authors only compare with MixMatch. I suggest that the authors include the missing literature in the next version.

3. The cost analysis is the most interesting to me. However, Figure 2(b) in Section 5.3 is weird. How can the ratio less than 0? According to the definition in Section 4.3, $L_i \subset L_{i+1}$ and $|L_{i+1}| > |L_i|$ and similar case for $U_{i,j}$, the cost ratio should not be less than 0. I'm also confused by the explanation in Section 5.3.

4. From Figure 1(b) and Table 2, we can see that on CIFAR-100, the improvement of the proposed MMA is not statistically significant, especially when the label budget is low. But on a simpler dataset CIFAR-10, MMA performs better. How does MMA perform on a more challenging task with more classes, e.g. ImageNet?

5. Training time comparison. At the expense of spending more time on selecting uncertain examples and techniques like k-means clustering, MMA is slightly better than MixMatch. A comparison of training time and complexity would be better to convince me.



***
Minor:
page 4 “Starting with from a fixed pool of n unlabeled sample”
page 4 “A corollary question is how do various accuracy targets relate to each other?” 
page 5 “While there are there additional active learning”
page 5 “ Let’s define cl and cu as the cost of respectively obtaining a new labeled sample and a new unlabeled sample.” --> costs



***
References
[1] Temporal Ensembling for Semi-Supervised Learning, ICLR 2017.
[2] Smooth Neighbors on Teacher Graphs for Semi-supervised Learning, CVPR 2018.
[3] Realistic Evaluation of Semi-supervised Learning Algorithms, NeurIPS 2018.
[4] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average, ICLR 2019.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summarize the paper:

This paper proposes a method that can deal with an active-learning scenario for the recently proposed semi-supervised learning method: MixMatch.  More specifically, the proposed method considers uncertainty measures to choose samples and a diversification step to ensure diversity within the sampled batch.  For uncertainty measures, the paper considers the simple maximum confidence and the gap between two most likely classes.  Additional augmentation techniques inspired from MixMatch are used.  For diversification, a clustering method and an information density method are considered.  Furthermore, the paper proposes a cost analysis model to compare labeled and unlabeled samples.  Experiments demonstrate the behavior of the proposed method.


Pros of the paper:

- The experimental results seem to be strong and encouraging.
- The discussions on the cost of labeled and unlabeled samples seems to be an important contribution for semi-supervised active learning.
- The motivation and direction of the paper is simple and easy to follow: Take the state-of-the-art semi-supervised learning algorithm and propose an active-learning version of it.
- It is not a straightforward combination of MixMatch and active learning, and there are some specialized techniques such as “aug” used in the design of the proposed algorithm.


Cons of the paper:

- Only uncertainty based sampling methods are considered, but is this enough?  There seems to be no other papers that deal with active semi-supervised learning for a deep learning context, so it might be important to really explore the many sampling methods (e.g., from survey of Settles 2009).

- A more minor comment: The same issue goes for the semi-supervised learning side.  MixMatch is the state of the art in terms of accuracy for image domains, but it is an ensemble of several semi-supervised learning methods, and have strong assumptions, e.g., smoothness assumption, small distribution overlap, etc.  This will mean the proposed method will also have those strong assumptions and limits the method’s applicability.

- In experiments, it would be better to have figures that are usually used in active learning experiments, where the x-axis is the remaining budget and y-axis is the performance measure.


Additional comments:

Active learning methods gives labels to unlabeled samples in different epochs until the budget is used up, but it would be interesting to give the final labeled and unlabeled dataset after budget is used up as a fixed dataset, and then train the traditional passive MixMatch with this.  Then we can really compare the original MixMatch and active MixMatch.  If the proposed method still works better,  then the proposed method might be meaningful not only as an active learning method but also as a curriculum learning method.

********************
I would like to thank the authors for answering my questions and updating the paper, but would like to keep my score due to the 2nd point of the cons.  A minor comment on the second point of the author response:  The sharpening step in MixMatch can be regarded as an entropy minimization procedure, which I think is based on the assumption of low distribution overlap.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper takes a look at using active learning techniques instead of random sampling for the 
"state-of-the-art" semi-supervised learning (SSL) method MixMatch. At least for this one SSL algorithm, the authors give a strong argument that active learning helps MixMatch (4x label efficiency in some cases) and highlight the active learning algorithms that work best. An additionally interesting point is the value of labeled vs unlabeled data in this setting. For these reasons, I argue for acceptance of this paper. However, I have some reservations which are given below, that perhaps the authors can clarify.  


Things that would have improved my score:

 - This paper relies on MixMatch very heavily as the sole semi-supervised technique. It would be nice to see more of an argument for this choice of a relatively recent paper that hasn't stood the test of time.

 - I am confused why the authors "report the median of the last 20 checkpoints' accuracy where a checkpoint is computed every 65,536 training iterations". As we see later, the authors train after each batch of selected examples for 32,768 iterations which is half of the time between checkpoints. Can the authors comment on this choice?


Minor comments:

 - The end of the abstract makes it sound like like the conclusions are universal ("quickly diminishes to less than 3x once more than 2000 labeled example are observed"). I would be surprised if the authors meant this as a universal statement since no theory is provided and the experiments are on similar datasets.

 - I don't understand why section 3 is not simply titled "MixMatch" since the paper doesn't really touch any other "modern SSL" methods.

 - In the experiments section, 262144 and 32768 iterations seem to come from nowhere. Only later did I realize that these were powers of 2. Can this be clarified?

 - What's the difference between MixMatch and MMA with random selection? Shouldn't these perform the same (which they seem to anyways)?

 - I really like that this paper assesses the label efficiency of their algorithm, rather than merely reporting raw accuracy numbers which aren't as meaningful.

 - I wonder if MMA seems to not give as big gains on CIFAR-100 because the batch size is 10x larger. Generally, I've found that active learning (especially uncertainty sampling methods) work best with smaller batch sizes and I'm not sure I agree with the reasoning that more classes mean one should select larger batch sizes.







review_title: Official Blind Review #3

Meta-review: This paper extends state of the art semi-supervised learning techniques (i.e., MixMatch) to collect new data adaptively and studies the benefit of getting new labels versus adding more unlabeled data. Active learning is incorporated in a natural and simple (albeit, unsurprising) way and the experiments are convincing that this approach has merit.

While the approach works, reviewers were concerned about the novelty of the combination given that its somewhat obvious and straightforward to accomplish. Reviewers were also concerned that the space of both semi-supervised learning algorithms and active learning algorithms was not sufficiently exhaustively studied. As one reviewer points out: neither of these ideas are new or particular to deep learning.

Due to lack of novelty, this paper is not suited for a top tier conference. 
Decision: Reject