Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper analyses the consistency-based SSL methods in settings  where  data lie a manifold of much lower dimension than the input space and obtains tractable results. The paper relates the analysis with Manifold Tangent Classifiers and shows that the quality of the perturbations plays a key role  to achieve a promising results in this set of SSL methods. Finally, the paper extends the Hidden Manifold Model by incorporating data-augmentation techniques and proposes a framework  to provide a direction for analyzing consistency-based SSL methods.

+This work might be useful for those who want to work in the theoretical part of SSL. 

+The paper analytically discusses that the type of data augmentation plays a significant role in the performance  of the SSL models based on consistency regularization.  

-While many points discussed in the paper are natural to me and well-discussed in the SSL literature (e.g.,  considering geometry of the data to develop an SSL algorithm or effect of the perturbation quality on the performance), relating and analyzing them with Manifold Tangent Classifiers (MTC) is interesting and new to me.  However, I still think that the theoretical part of the work is not strong enough and can be improved. The quality of the paper will be  improved if it uses MTC and provides some new results which do not exist in the SSL literature. This is because  MTC may not be the only approach to analyze these points.  For example, there are many other approaches based on optimal transport (e.g., Wasserstein Distances) that consider the geometry of the data and can be used to analyze the effect of perturbation on the SSL performance.  Then, someone may ask what is special in your approach to analysis consistency-based SSL methods in contrast to other tools/techniques?

-While this paper is mostly an analytical paper, providing more experiments on some claims discussed in the paper  (e.g., Mean Teacher method and Π-model approach share the same solutions in the cases where the data-augmentations are small)  is necessary and can improve the quality of the paper. Furthermore, authors may  use the exact same underlying model, training set-up,  and then try different types of data-augmentation methods on several recent and effective consistency-based SSL methods to show the differences on the performance experimentally. This can better contextualize the paper as we will know how much is the difference in terms of accuracy between different consistency-based SSL methods  with respect to perturbation, or other data-augmentation methods. 

Generally,  I think this  paper provides a good direction for understanding the consistency based-SSL methods.

review_title: On Data-Augmentation and Consistency-Based Semi-Supervised Learning 

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary

This paper proposes a theoretical framework for understanding consistency-based semi-supervised learning. While establishing this framework based on the Hidden Manifold Model, this paper frames the SSL in the context of Manifold Tangent Classifiers.

# Pros

1. Formal understanding of SSL is indeed currently limited and theoretical works are needed.
2. The formalization of minimizers as harmonic function leads to the non-obvious prediction that SSL methods are insensitive, or at least very robust, to the weighting of the consistency loss $\lambda$. To me, this is the most important result of the paper.
3. The bibliography is well documented.

# Cons
1. The experimental verification of insensitivity to the weighting of consistency loss is unconvincing: it is done on a trivial low-dimensional toy dataset. It would be more convincing to take the GitHub code of one or more, possibly recent, SSL methods and vary $\lambda$ on real datasets.
2. I fail to see any other takeaway from the theoretical framework than the predicted insensitivity to $\lambda$.
3. Technically, if I understand correctly, the insensitivity to $\lambda$ is not a takeaway of the Hidden Manifold Model but of the Minimizers are Harmonic Functions.
4. I didn’t see where the claim “... demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances” has indeed been demonstrated.

# Questions and nits
1. “Several extensions such as the Mean Teacher (TV17) and the Virtual Adversarial Training (VAT) (MMIK18) schemes have been recently proposed and have been shown to lead to state-of-the-art results in many SSL tasks.” They are far from the current state of the art, especially so for few labels, this would have been true before the advent of MixMatch but the state of the art changed drastically with its introduction.
2. “... consider as well the tangent plane Tx to M at x.” Here I have a hard time visualizing it. Say the manifold M is a 2D Gaussian point cloud for example, what would be the tangent plane for a point x in that cloud?
3. Following on question 2, how is the tangent plane property exploited other than by defining an orthonormal basis on it? And why can’t an orthonormal basis of the Manifold space itself be enough if instead we assume it is dense?
4. “Enriching the set of data-augmentation degrees of freedom with transformations such as elastic deformation or non-linear pixel intensity shifts is crucial to obtaining a high-dimensional local exploration manifold.” This seems in direct contradiction with MixMatch results which does not use any sophisticated augmentation: just pixel shift, mirroring and random pixel-wise linear interpolation between samples and labels (MixUp).
5. Proposition 4.1: Here I was hoping for a prediction for your method. You mention the “sequence of processes converges weakly” and I was hoping it would explain why SSL techniques are much slower than fully supervised to converge. But then, it seems this statement is not exploited in any way other than saying that it does indeed converge to the solution of the ODE.
6. “This indicates that the improved performances of the Mean Teacher approach sometimes reported in the literature are either not statistically meaningful, or due to poorly executed comparisons, or due to mechanisms not captured by the η → 0 asymptotic.” This is vague, considering Mean Teacher outperforms VAT, would this mean that the last option holds (due to mechanisms not captured by the η → 0 asymptotic)? Just to clarify what the last option actually means: does it mean the proposed analysis relies on assumptions that don’t capture the reality of the phenomenon?
7. “These results are achieved by leveraging more sophisticated data augmentation schemes such as ... Mixup”. It seems odd to see MixUp being referred to as sophisticated (as in domain specific). MixUp is domain agnostic, it simply linearly combines a pair of samples and their labels. So in fact, it seems even simpler than a pixel shift since it’s linear.
8. “... with a neural network with a single hidden layer with N = 100 neurons”. What non-linearity was used? I didn’t seem to find it, or I may have missed it.
9. “Figure 3 (Left) shows that, contrary to several other types of regularizations such as weight-decay, this method is relatively insensitive to the parameter λ”. I didn’t see it. It indeed shows that (a) the method is relatively insensitive to the parameter λ in the context of the toy task but it doesn’t show that (b) the method is sensitive to other types of regularization such as weight-decay.

Note: Ultimately I must admit a lot of the maths are above my head and I have no idea whether they are correct or not, therefore I didn't comment on them and only focused on the parts that I understand. On the other hand, I'm pretty confident in my understanding of SSL techniques and MixUp.

=====POST-REBUTTAL COMMENTS======== 
I thank the authors for the response and the efforts in the updated draft. Most of my queries were clarified and I raised my rating accordingly. However, unfortunately, I still think a more realistic validation (e.g. on non-toy dataset) would benefit the paper. 


review_title: Insufficient experimental confirmation and there are some apparent contradictions in the proposed explanation, notably with MixUp

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors analyze consistency-based models in specific settings where analytically tractable results can be obtained. They establish that leveraging more sophisticated data augmentation schemes is crucial to obtain huge gains when using consistency based models. Finally, they propose an extension of Hidden Manifold Model that incorporates data augmentation for understanding and experimenting with SSL methods.

Pros:
The paper is theoretically well grounded and the authors do a good job of connecting to previous works.

The paper is interesting and well written. It tries to explain the why consistency based method achieve good performance compared to other semi-supervised models. I find this relevant and helpful in understanding what makes these models work the way they do.

Cons:
The experiments are a bit weak only using simple synthetic settings.
Some of the claims in the paper are not backed up by experimental results.

Comments:
More experiments should be provided to establish some of the claims in the paper. Additionally, it will be good to see an experiment on a real dataset albeit small rather than simple synthetic experiments.

One of the claims of the paper is that mean teacher and simple Π-model approach share the same solutions in the regime where the data augmentations are small but advanced data augmentations performs better. It will be nice to demonstrate this with an experiment for the reader to better understand why this happens.
review_title: ON DATA-AUGMENTATION AND CONSISTENCYBASED SEMI-SUPERVISED LEARNING

Meta-review: This paper provides some theoretical perspective on the use of data augmentation in consistency regularization-based semi-supervised learning. The framework used in the paper argues that high-quality data augmentation should move along the data manifold. This generic view allows the paper's ideas to be applied across datasets (as opposed to image-specific data augmentation used in state-of-the-art semi-supervised learning algorithms). I am not aware of any other work raising these points, and indeed this paper is significant in that it provides a new and potentially useful perspective on the most performative semi-supervised learning approach. Reviewers agreed that the paper was clear and useful. The main concern was that the paper only included experiments in toy settings. Indeed, it would have been much more impactful to apply these ideas to state-of-the-art semi-supervised learning methods, but I think it can be excused given the theoretical focus of the work.
Decision: Accept (Poster)