Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: This paper focuses on the semi-supervised learning problem, and proposes a way to improve previous pseudo-labeling methods.  In pseudo-labeling, there is an issue called confirmation bias, which accumulates the early errors of wrong pseudo labels.  By adding some simple tricks such as adding mixup augmentation and setting a minimum number of labeled samples per mini-batch, the confirmation bias is shown to be reduced, leading to an improvement in accuracy.  Experiments demonstrate that the additional tricks are meaningful and makes pseudo-labeling better than many baseline methods for semi-superivsed learning, including state-of-the-art consistency regularization methods.


Pros: This is an interesting paper with a clear motivation, which is to fix the so-called confirmation bias that appears in pseudo-labeling methods for semi-supervised learning.  Although the tricks introduced in the paper (mixup and changing the mini-batch selection rules) themselves are not novel, they make the proposed method simple.  It is also shown to be meaningful in reducing the confirmation bias in Table 1 and Figure 2, achieving the original goal of the paper.


Cons: The weakness of the paper is that the intuition or the motivation behind the design of the proposed method is not so clear.  Using mixup is justified by the reason that mixup gives better confidence calibration.  This is important for pseudo-labeling methods, because soft-label output predictions are used as pseudo labels.  On the other hand, however, it was not so obvious why a minimum number of labeled samples per mini-batch was considered.  Can we consider further extensions such as minimum number of labeled samples per mini-batch & per class?  (Perhaps the discussions about mixup and soft labels in the last paragraph of Section 3 should be more emphasized, for example in the last paragraph of the Introduction section.)

Related to the weakness above, it is hard to see how far the regularization effects of adding mixup and mini-batch sampling rules are contributing to add synergy to the pseudo-labeling methods.  This is partially answered with Figure 2, but it would make this easier to see if the experiments included stronger baselines, e.g., by adding the same regularization tricks to consistency regularization methods, perhaps in Table 3.

Finally, since future work on pseudo labels will follow this paper’s setup, hyperparameters such as lamba_A, lambda_H, and alpha should be chosen carefully instead of fixing them.


Other minor comments (that did not impact the score):

- In reference section, "Z. MaXiaoyu Tao" seems to combine two authors.

- Table 3 never appears in the text.  In Section 4.4, "The table" in the second sentence can be changed to "Table 3".

- "architecture plays and important role" --> "architecture plays an important role"

- In Table 2, "+" signs make it look like an equation.  I suggest using commas instead.

-  "ResNet arquitectures" --> "ResNet architectures"

~~~~~
Thank you for the response and for the additional discussions that were included in the updated paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to combine pseudo-labelling with MixUp to tackle the semi-supervised classification problem. My problem is that "MixMatch: A Holistic Approach to Semi-Supervised Learning" by Berthelot et al. is very similar with just a few differences on the pseudo-labelling part. Could you stress more the difference between your paper and their paper ? Because I might be wrong about it.

Pros:
* Good results on C10
* A clear related work section that divides the existing works in pseudo labelling vs consistency
* Interesting results about the effects of using different architectures. I also like the ablation study.

Weaknesses:
* Usually, SVHN is also among the tested datasets
* The pseudo labelling part is a bit unclear.For example, do you just refresh the pseudo-labels at the end of each epoch ?
* minor: a typo with "and important role"

If there was not an existing paper already using MixUp, I would have leaned towards acceptance. You can still motivate the differences with the MixMatch paper.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: OVERALL:
I think this paper is worth accepting.
All modern semi-supervised learning techniques use consistency regularization somehow,
and this paper shows that you can get away with just using pseudo-labeling combined with some
engineering to route around the main issue with pseudo labeling (which is apparently called confirmation bias,
though I hadn't heard that, and I don't like it as a name because it's confusing).

Neither MixUp nor the idea of fixing some number of labeled elements in a minibatch is new,
but that's not the point - we thought one thing, and this paper suggests that
we were wrong about that thing - to me this is exactly the sort of paper it's good to have at conferences.

I would change the framing slightly.
You're not showing that pseudo-labeling can be useful, because many techniques already incorporate a form of pseudo-labeling.
Instead, you're showing you can get away without consistency regularization.

A potential improvement:
If you add up this techique with some of the most recent SLL techniques based on consistency regularization somehow,
does it do better, or are they both acting via the same mechanism?

DETAILED COMMENTS:
> , contrary to previous evidences on pseudo-labeling capabilities (Oliver et al., 2018),
It's not really contrary to the findings of that paper, since you've totally changed the
technique compared to what's evaluated in that paper.

> n (Berthelo et al., 2019) 
It's Berthelot

> and are the mechanisms proposed in Subsection 3.1
Doesn't quite parse

> Network predictions are, of course, sometimes incorrect.
This is a great line.

> We use three image classification datasets...
Why not use SVHN, which is by now super standard for SSL papers?

> , we add the 5K samples back to the training set for comparison
with the state-of-the-art in Subsection 4.4,
This is *allowed* from the perspective of reporting a valid test accuracy,
but if other papers don't do that, it kind of mucks up the comparison, no?

Fig 1 is nice, but why does the effect not seem to be symmetric about the
blue and the red blobs?

> architecture plays and important role


> However, it is already interesting that...  and that future work should take this into account.
This sentence doesn't quite make sense

Re table 4:
I'm curious how e.g. MixMatch would fare w/ the 13-CNN network.
I am surprised that the change from WRN -> 13-CNN matters so much.

review_title: Official Blind Review #1

Meta-review: The paper focuses on semi-supervised learning and presents a pseudo labeling-based approach with i) mixup (Zhang et al. 2018); ii) keeping $k$ labelled examples in each minibatch.

The paper is clear and well-written; it presents a simple and empirically effective idea. Reviewers appreciate the nice proof of concept on the two-moons dataset, the fact that the approach is validated with different architectures. Some details would need to be clarified, e.g. about the dropout control.

A main contribution of the paper is to show that pseudo-labelling plus the combination of mixup and certainty (keeping $k$ labelled examples in each minibatch) can outperform the state of the art based on consistency regularization methods, while being simpler and computationally much less demanding. 

While the paper does a good job of showing that "it works", the reader however misses some discussion about "why it works". It is most interesting that the performances are not improving with $k$ (Table 1). An in-depth analysis of the trade-off between the uncertainty (through mix-up and the entropy of the pseudo-labels) and certainty, and how it impacts the performance, would be appreciated. You might consider monitoring how this trade-off evolves along learning; I suspect that evolving $k$ along the epochs might make sense;  the question is to find a simple way to control online this hyper-parameter.  

The area chair encourages the authors to continue this very promising path of research, and dig a little bit deeper, considering the question of optimizing the trade-off between certainty and uncertainty along the training trajectory.
Decision: Reject