On Data-Augmentation and Consistency-Based Semi-Supervised Learning
Authors: Atin Ghosh
Alexandre H. Thiery
Venue: ICLR 2021 Poster (2021)
Keywords: Semi-Supervised Learning
Regularization
Data augmentation

Abstract: Recently proposed consistency-based Semi-Supervised Learning (SSL) methods such as the Pi-model, temporal ensembling, the mean teacher, or the virtual adversarial training, achieve the state of the art results in several SSL tasks. These methods can typically reach performances that are comparable to their fully supervised counterparts while using only a fraction of labelled examples. Despite these methodological advances, the understanding of these methods is still relatively limited. To make progress, we analyse (variations of) the Pi-model in settings where analytically tractable results can be obtained. We establish links with Manifold Tangent Classifiers and demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances. Furthermore, we propose a simple extension of the Hidden Manifold Model that naturally incorporates data-augmentation schemes and offers a tractable framework for understanding SSL methods.

Published as a conference paper at ICLR 2021
ON DATA- BASED SEMI-SUPERVISED LEARNING
AND
CONSISTENCY-
Atin Ghosh & Alexandre H. Thiery
Department of Statistics and Applied Probability National University of Singapore atin, ghosh@u. nus edu h. thweryenesuus... edu. sg
ABSTRACT
Recently proposed consistency-bart Semi-Supervised Learning (SSL) methods such as the Il-model, temporal ensembling. the mean teacher, or the virtual ad- versarial training. have advanced the state of the art in several SSL tasks. These methods can typically reach performances that are comparable to their fully super- vised counterparts while using only a fraction of labelled examples. Despite these methodological advances, the understanding of these methods is still relatively limited. In this text, we analyse (variations of) the II-model in settings where analytically tractable results can be obtained. We establish links with Manifold Tangent Classifiers and demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances. Importantly, we propose simple exten- sion the Hidden Manifold Model that naturally incorporates tata-augmentttion schemes and offers framework for understanding and experimenting with SSL methods.
INTRODUCTION
Consider dataset Dr Dr that is comprised of labelled samples Dr {xis Jiezz well as unlabelled samples Dy = { hiele. Semi-Supervised Learning (SSL) is concerned with the use of both the labelled and unlabeled data for training. In many scenarios, collecting labelled data is difficult or time consuming expensive so that the amount of labelled data can be relatively small when compared to the amount of unlabelled data. The main challenge of SSI. in the design of methods that can exploit the information contained in the distribution of the unlabelled data (Zhu05; CSZ09).
In modern high-dimensional settings that are common to computer vision, signal processing, Natural Language Processing (NLP) genomics, standard graph/distance based methods (BCO1; ZG02; ZGL03; BNS06; DSSTI9) that are successful in low-dimensional scenarios are difficult to implement. Indeed, in nigh-dimensional spaces, it is often difficult design sensible notions of distances that can be exploited within these methods. We refer the interested reader to the book-length treatments (Zhu05; CSZ09) for discussion of other approaches.
The manifold assumption is the fundamental structural property that exploited in most modern approaches SSL: high-ddmeensooal data samples lie in small neighbourhood of low-dimensional manifold (TP91; BJ03; Pey09; Cay05; RDV+11). In computer vision, the presence of this low- dimensional structure is instrumental the success of (variational) autoencoder and generative adversarial networks: large datasets of images can often be parametrized by a relatively small number of degrees of freedom. Exploiting the unlabelled data uncover this ciw-dimensional structure is crucial to the design of efficient SSL methods. A recent and independent evaluation of several modern methods for SSL. can be found in (OOR+ 18). methods (BAP14; SJT16; LA16; TV17; MMIK 8; 1 GSA+20), the topic of this paper, achieve ofate-of-the art performances in many realistic scenarios.
found there that consistency-based
consistency-based semi-supervised learning methods have recently been shown to achieve state-of the results. Despite these methodological advances, the understanding of these methods is still relatively limited when compared to the fully-supervised setting (SMG13;
Contributions:

---

Published as a conference paper at ICLR 2021
AS17; SBD* 18; TZ15; SZT17). In this article, we do not propose new SSL method. Instead, analyse consistency-bed based methods in settings where analytically tractable results can be obtained, when the data-samples lie in the neighbourhood of well-defined and tractable low-dimensional manifolds, and simple and controlled experiments can be carried out. We establish links with Manifold Tangent Classifiers and demonstrate that consistency-basee SSL methods in general more powerful since they can better exploit the local geometry of the data-manifold if efficient attuuummntttooo/prrrrballte
schemes are used. Furthermore, in section 4. we show that the popular Mean Teacher method and the conceptually more simple II-model approach share the same solutions the regime when the aata-augmenttiions are small; this confirms often reported claim that the data-augmennation schemes leveraged by the recent SSL,, as well as fully unsupervised algorithms, are instrumental to their success. Finally, in section 4.3 we propose an extension of the Hidden Manifold Model (GMKZ19; GI.K 20). This generative model allows us to investigate the properties of consistency-bor based SSL methods, taking into account the autaa-ugmentation process and the underlying low- tww-iieensionality of the data, in simple and principled manner, and without relying on a specific dataset. For gaining understanding of SSL, as well as self-suuervised learning methods, believe it to be important to develop framework that (i) can take into account the geometry of the data (ii) allows the study of the influence of the quality of the data-augmentation schemes (iii) does not rely on any particular dataset. While the understanding of fully-supervised methods have largely been driven by the analysis of simplified model architectures (eg. linear and two-layered models, large dimension asymptotic such the Neural Tangent Kernel,, these analytical tools alone are unlikely to be enough to explain the mechanisms responsible for the success of SSL and self-supervised learning methods (CKNH20; GSA + 20), since they do not, and cannot easily be extended to, account for the geometry of the data and data-augmentation schemes. Our proposed framework offers a small step in that direction.
CONSISTENCY-BASED SEMI-SUPERVISED LEARNING
For concreteness and clarity of exposition, we focus the discussion on classification problems. The arguments described the remaining of this article can be adapted without any difficulty to other situations such regression or image segmentation. Assume that the samples x; E X C RD can be represented as D-dimensional vectors and that the labels belong to C 2 possible classes, Yi E V = (1,.... C). Consider a mapping Fo : TR. + RCC parametrized by 0 θ C R. 1l This can be a neural network, although that is not necessary. For X, the quantity Fo(x) can represent probabilistic output of the classifier, or for example, the pre-softmax activations. Empirical risk minimization consists in minimizing the function
(2(0) IDL|
Σ ((Fo(rs). SETL
for a loss function e: RC x X R. Maximum likelihood estimation corresponds to choosing the loss function as the cross entropy. The optimal parameter found by a variant stochastic gradient descent (RM51) with estimated gradient
Vo
Σ SEB
((Fo()))
B
for a mini-batch B2 of labelled samples. Consistency-based SSL algorithms regularize the learning by enforeing that the learned function r + Fo(2) (z) respects local derivative and invariance constraints. For simplicity. assume that the mapping æ + Fo(x) is deterministic, although the use of drop- out (SHK+1 14) and other sources of stochasticity are popular in practice. The I--mdeel (LA166, SJT16) makes use of stochastic mapping 8 X' 52 - A' that maps a sample x E X'' and source of randomness w Ω C gda to another sample S_, (x) E X. The mapping S describes a stochastic data augmentation process. In computer vision, popular data-augmentation schemes include random translations, rotations, dilatations, croppings, flippings, elastic deformations, color jittering, addition of speckle noise, and many domain speecff variants. In NLP, synonym replacements, insertions and deletions, tack-trraslattinss are often used although is often more difficult to implement these data-augmentation strategies. In purely supervised setting, angetttigentttion can be used
2

---

Published as a conference paper at ICLR 2021
regularizer. Instead of directly minimizing LL. one minimize instead
θ
+
Σ DL ET
In practice, data-augmentatiin regularization, although simple strategy, often crucial to obtaining good generalization properties (PW17, 17; C7M+18; LBC17; PCZ++19 The idea of regularizing by enforcing robustness to the injection of noise can be traced back at least to (Bis95). the I1-model, the aata-tugmentttion mapping S is used to define consistency regularization term,
R(0)= = D
Σ iETZUl
E (11o(( - Fo. (2s))l²}
(1)
The notation 0, designates a copy of the parameter 0, i.e. 0, differentiating the consistency regularization term 0 - R(0), one does not differentiate through 0,. In practice, a stochastic estimate of (R(@) is obtained as follows. For mini-batch B of samples {z, hes, the current value 0, E θ of the parameter and the current predictions f: Fe. (I,). the quantity
and emphasizes that when
[e]s fill?
B
IEB
an approximation of (R) There are indeed many variants (eg. use of different norms, different manners to inject noise), but the general idea is to force the learned function + Fz(z) to be locally invariant to the fata-augmentation scheme Several extensions such as the Mean Teacher (TV17) and the VAT (MMIK18) schemes have been recently proposed and have been shown lead to good results in many SSL tasks. The recently proposed and oftte-of-the-ar B BYO approach (GSA+ 220) is relying on mechanisms that very close the consistency regularization methods discussed on this text.
If one recalls the manifold assumption, this approach is natural: since the samples corresponding to different classes lie on separate manifolds, the function Fe X' - RRC should be constant on each one of these manifolds. Since the correct value of Fo is typically well approximated or known for labelled samples (za, Vs) E DL, the consistency regularization term equation helps propagating these known values across these manifolds. This mechanism is indeed similar to standard SSL graph-based approaches such as label propagation (ZG02). Graph-based methods are difficult to directly implement in computer vision, or NLP, when meaningful notion of distance is not available. This interpretation reveals that it is crucial to include the labelled samples in the regularization term equation in order to help propagating the information contained in the labelled samples to the unlabelled samples. Our numerical experiments suggest that, in the standard setting when the number of labelled samples is much lower than the number of unlabeled samples, i.e. [Dz] |Durl, the formulation equation of the consistency regularization leads to sub-optimal results and convergence issues: the information contained in the labelled data swamped by the number of unlabelled samples. In all our experiments, we have adopted instead the following regularization term
R(0)
1
Σ
Em
{ (x.))
|
Fo. Fo. (x;)
Dz
ET
+
Σ E.
(z)) Fo. (z))))
Puul
jETG
that balances the labelled and unlabelled data samples more efficiently. Furthermore, it clear that the quality and variety of the data-augmenntttion scheme S : X x Ω X' is pivotal to the success of consistency-baat
SSL methods. We argue in this article that the dominant factor contributing to the success of this class of methods. Effort spent on building efficient local data-augmentation schemes will be rewarded in terms of generalization performances. Designing good data-augmentation schemes is an efficient manner of injecting expert/prior knowledge into the learning process. It is done by leveraging the understanding of the local geometry of the data manifold. As usual and not surprisingly (NGP98; MHF 12), in data-scare settings, any type of domain-knowledie knowledge needs to be exploited and we argue that consistency regularization approaches to SSL are instances this general principle.
3

---

Published as a conference paper at ICLR 2021
3 APPROXIMATE MANIFOLD TANGENT CLASSIFIER
has long been known (SLDV98) that exploiting the knowledge of derivatives, or more gener- ally enforcing local invariance properties, can greatly enhance the performance of standard clas- sifiers/regressors (HK02; CS02). In the context of deep-learning, the Manifold Tangent Classi- fier (RDV+11) is yet another illustration of this idea. Consider the data manifold M C X C RD and assume that the data samples lie on a neighbourhood of it. For x E M, consider as well the tangent plane T. to M at x. Assuming that the manifold M is of dimension 1 < d < D, the tangent plane Te is also of dimension d with an orthonormal basis ef...., e E R This informally means that, for suitably small coefficients wi....,a Wa E. the transformed sample e X defined as
x = x
W3 oj
1=1
also lies, or is very close to, the data manifold M. A possible stochastic data Vata-augmentation scheme can therefore be defined as (x) = Vu where Ve = = W3 w. If is a multivariate d- dimensional centred Gaussian random vector with suitably small covariance matrix, the perturbation vector V. is also centred and normally distributed. To enforce that the function x + Fe(z)) is locally approximately constant along the manifold M, one can thus penalize the derivatives of Fe at æ in the directions V.. Denoting by J E RC the Jacobian with respect to I E RD of Fg at x E M, this can be implemented by adding penalization term of the type E. [I V/ Vo
= Tr (T & J.), where I' E R D. is the covariance matrix of the random vector w + Vu. This type of regularization of the Jacobian along the mant-ma is for example used in (BNS06). More generally, if one assumes that for any 2, w E X' Q we have (z) = x + a ((z, mapping D X Q X, it follows that
z w) + O(=?). for some derivative
lim Eww [I 8-0 g2
Fe. w (+) [o(2))1 E., [| D( w) ] = Tr a & J Jz)
where I s is the covariance matrix of the X'vulueed random vector w + D(,, w) X. This shows that consistency-baa methods can be understood as approximated Jacobian regularization methods, as proposed in (SLDV98: RDV 11).
3.1 LIMITATIONS
In practice, even if many local dimension reduction techniques have been proposed, it is still relatively difficult to obtain a good parametrization of the data manifold. The Manifold Tangent Classifier (MTC) (RDV+11) implements this idea by first extracting in an unsupervised manner a good representation of the dataset D by using a AuurractvveAutt-EEnoodee (CAB) (RVM* 11). This CAE can subsequently be leveraged to obtain an approximate basis of each tangent plane Tz, for x; D, which can then be used for penalizing the Jacobian of the mapping x + Fo(x) in the direction of the tangent plane to M at x. The above discussion shows that the somewhat simplistic approach consisting in adding an isotropic Gaussian noise to the data samples is unlikely to deliver satisfying results. It is equivalent
penalizing the Frobenius norm ||J.. j of the Jacobian of the mapping + (z):: in a linear model, that is equivalent to the standard ridge regularization. This mechanism does not take all into account the local-geonne -eomeeyymtrry of the data-manifold Nevertheless, in medical imaging applications where scans are often contaminated by speckle noise, this class of approaches which can be thought off adding artificial speckle noise, can help mitigate over-fitting (DRS+ 18).
There are many situations where, because of data scarcity or the sheer difficulty of unsupervised representation learning in general, domain--pecific data-augmentatton schemes lead to much better regularization than Jacobian penalization. Furthermore, as schematically illustrated in Figure 1, Jacobian penalization techniques are not efficient at learning highly non-linear manifolds that are common, for example, in computer vision. For example, in "pixel space", simple image translation is a highly non-linear transformation only well approximated by a first order approximation for very small translations. In other words, if x E X' represents image and g(æ, v) is its translated version by a vector v, the approximation g(r, v) Ri + Veg(x), with V.g(z) = lim (g(2, e v) g(z)/e. becomes poor soon as the translation vector not extremely small.
In computer vision, translations, rotations and dilatations are often used as sole data- aatta-uggmenattion schemes: this leads poor local exploration of the data-manifold since this type transformations
4

---

Published as a conference paper at ICLR 2021
M
labuted samples wolabeled samppes sameegeen local data sugmenttaion
Figure 1: Left: Jacobian (i.e. first order) Penalization method are short-sighted and do not exploit fully the data-manifold Right: Data-Augmentation respecting the geometry of the data-manifold.
only generate a very low dimensional exploration manifold. More precisely, the exploration manifold emanating from sample 2o E X, i.e. (8(*o, w) 2 is very low dimensional: its dimension is much lower than the dimension d of the data-manifold M. Enriching the set of data-augmentation degrees of freedom with transformations such as elastic deformation or non-linear pixel intensity shifts is crucial to obtaining high-de sigh-dimensional local exploration manifold that can help propagating the information on the data-manifol efficiently (CZM* 19; PCZ 19).
ASYMPTOTIC PROPERTIES
4.1 FLUID LIMIT
Consider the standard [1-model trained with standard Stochastic Gradient Descent (SGD). Denote by Or E the current value of the parameter and 1) > the learning rate. We have
1
A
0k+1
02
V
Σ KEBL
((Fo (2x), 9:
Fr,, (8w[2;)) fz
B
(3)
(Sw - fn 1|)
Σ
+
Bu
kEBU
for parameter À > O that controls the trade-off between supervised and consistency losses, as well as subsets B2 and By of labelled and unlabelled data samples, and f, Fe. (zz) for 8, 0g as discussed Section 2. The right-hand-sid an unbiased estimate of 1 V (zx))) + X ((x))) with variance order O17?). where the regularization term R(0x) is described equation It follows from standard fluid limit approximations (EK09)[Section 4.8] for Markov processes that, under mild regularity and growth assumptions and as 1) - 0, the appropriately time-rescaled trajectory (0x) )2x) can be approximated by the trajectory of the Ordinary Differential Equation (ODE).
Proposition 4.1 Let D( (()) be the usual space of càdlàg I 1llluue functions bounded time interval [0, TT endowed with the standard Skorohod topology. Consider the update equation 3 with learning rate > O and define the continuous time process (l) = Ou/n) The sequence of processes 0" D( TJ, Rl converges weakly in D( (1) and as n + 0 to the solution of the ordinary differential equation
0. = (C(00
The article (TV17) proposes the mean teacher model, averaging approach related to the standard averaging scheme (Po190; PJ92), which modifies the consistency regularization term equation by replacing the parameter 0, by exponential moving average (EMA). In practical
5

---

Published as a conference paper at ICLR 2021
terms, this simply means that, instead of defining f, Fo. (zz), with e. 0g in equation 3, one sets f; = Fore, (F2) where the EMA process (6avg.k k20 is defined through the recursion Barg. = (1-a a)) favg.k-1 + a 0g where the coefficient a > controls the time-scale of the averaging process. The use of the EMA process {Omgg, $220 helps smoothing out the stochasticity of the process lz. Similarly to Proposition 4.1, as 17 - O, the joint process ("g.)) = (f//o) 2 (rg.t/m))
converges as - 0 to the solution of the following ordinary differential equation
0, = -- -----(C (C(0.) + A AR,0,, Owwg.e)
(5)
Aug. = (Davg.t B,)
where the notation R, Bawg.t) designates the same quantity as the one described in equation 2, but with an emphasis on the dependency on the EMA process. At convergence (0e. Bavg.s) - (Bo. Dav.oor),) one must necessarily have that 0_x = Davg,oo confirming that, in the regime of small learning rate 7j - O, the Mean Teacher method converges, albeit often more rapidly, towards the same solution as the more standard Il-model. This indicates that the improved performances of the Mean Teacher approach sometimes reported in the literature are either not statistically meaningful, or due to poorly executed comparisons, or due to mechanisms not captured by the 1 0 asymptotic. Indeed, several recently proposed consistency hased SSL algorithms (BCG- 19; SBI++20; XDH 19) achieve state-of-the-art performance across diverse datasets without employing any exponential averaging processes. These results are achieved by leveraging more sophisticated data augmentation schemes such Rand-Augment (CZSL19), Back Translation (ALAC17) or Mixup (ZCDLPI7).
4.2 MINIMIZERS ARE HARMONIC FUNCTIONS
To understand better the properties of the solutions, we consider a simplified setting further exploited in Section 4.3. Assume that J X = RD + IR and y = I and that, for every y! E y R. the loss function f + {ff, 31) is uniquely minimized at f y/- We further assume that the data-manifold M C IR 2 can be globally parametrized by a smooth and bijective mapping
IR + M C
tata-auumentttion scheme that can be described as Szw (z) = 4(2 + EW) for 2 = $ (xx) and a sample w from a -v-luee centred and isotropic Gaussian distribution. We consider finite set of labelled samples {#i, y/ Jiet, with x; = (2:) and E Rd for i E I,,. We choose to model the large number of unlabelled data samples as a continuum distributed on the data manifold M as the push-forward measure (, of a probability distribution ((zz) whose support is Edd through the mapping Φ This means that an empirical average of the type (1/lDul) Eis,, ((x;) can be replaced by I (=)( p(dz). We investigate the regime e + and, similarly to Section 2, the minimization of the musistenny-rrgullriin objective
RD. Similarly to the Section 2, we consider
/ E. Fl Fe( p p(dz).
(0) + 22
(6)
Rd
For notational convenience, set Sa = Fo o $. Since Sew. (=)) $ + sw), as e + the quantity 1 E.w [|
Fo Fo (o)))) converges to 1o and the objective function equation 6 approaches the quantity
Σ (ff(zz)))) + / m(d))
G(fe) = DDL
KET
A minimizer f: Rd + R of the functional G that is consistent with the labelled data, i.e. f(z;) = y: for i E It, is a minimizer of the energy functional f + fe* [/==)(=)] p(dz) subject to the constraints f(z)) = yt. is the variational formulation of the Poisson equation
{ [af(2)= = for 2 E Rd fzhee f(x)) = us for i Iz.
(8)
Note that the solution does not depend on the regularization parameter À in the regime of + 0: this indicates, will be discussed in Section 4.3 in detail, that the generalization properties of consistency-be based SSL methods will typically be insensitive to this parameter, in the regime of small data-augmentation at least. Furthermore, equation 8 shows that consistency-based SSL methods

---

Published as a conference paper at ICLR 2021
indeed based on the same principles Propagation (ZG02): solutions are gradien//aplaciann penalized interpolating functions. In Figure 2, consider the where D = d = 2 with trivial mapping ((x) We consider labelled data situated on the right (resp. left) boundary of the unit square and corresponding to the label 0 choose the loss function {(f.
standard graph-based approaches such Label
(resp. y 1). For simplicity, Fg = fe with neural network with a single hidden layer with N = 100 neurons. As expected, the 11-model converges to the solution to the Poisson equation 8 in the unit square with boundary condition f(u, v) = 0 for u = 0 and fu, v) for 1.
(f y)² and parametrize
Random Initialization
Solution
1.0
0.8
0.8
0.6
A6
0.2
0.2
0.0
0.2 0.8 0.0
0.0
0.6
0.4 1.0
Figure Labelled data samples with class == (green triangle) and y= +1 (red dot) are placed on the Left/Right boundary of the unit square. Unlabelled data samples (blue stars) are uniformly placed within the unit square. We consider a simple regression setting with loss function E(J: y) = 3 ( y)-
+1
Left: Randomly initialized neural network. Middle: labetled/unlabelled data Right: Solution of obtained by training standard ITmodel. the harmonic function [u, v) u, as described by equation 8.
4.3 GENERATIVE MODEL FOR SUPE-IIEER LEARNING
As has been made clear throughout this text, SSL methods crucially rely the dependence structure of the data. The existence and exploitation of much lower-dimensional manifold M supporting the data-samples is instrumental this class of methods. Furthermore, the performance of consistency- based SSL approaches is intimately related to the data- aatta-aggmetttion schemes they are based upon. SSL
Consequently, in order to understand the mechanisms that are play when consistency- methods are used to uncover the structures present in real datasets, it is important to build simplified and tractable generative models of data that (1) respect these low-dimensional structures and (2) allow the design of efficient data-augmentation schemes. Several articles have investigated the influence of the dependence structures that are present in the data on the learning algorithm (BM13; Mos Here, we follow the Hidden Manifold Model (HMM) framework proposed in (GMKZ19; GLK 20) where the authors describe a model of synthetic data concentrating near fow-dimensional structures and analyze the learning curve associated to class of two-layered neural networks.
0.9
- 1.0
10.0
0.7
100.0
0.5
Unregul arized
=-0.03
TestMLl
0.4
0.5
0030
0.3
0.2
www
20
40
60
100
25
30
75
100
125
130
175
200
Figure 3: Left: For a fixed augmenttgtnt scheme, generalization properties for spanning two orders of magnitude. Right: Influence of the quantity of the data-augmentattin of the generalization properties.
7

---

Published as a conference paper at ICLR 2021
Low-dimensional structure: Similarly Section 4.2, assume that the D-dimensional data samales I; E A' can be expressed as I2 = ((2) E RR' for a fixed smooth mapping & F - >R", .. In other words, the data-manifold M is d-dimensional and the mapping & can be used to parametrize it. The mapping & is chosen to be a neural network with a single hidden layer with H neurons. although
other choices are indeed possible. For = (2l .....; = E IRd set ((2) Al- (<1001 b*) for matrices A0 E IR Hidd and A E D. bias vector b² E R and non-linearity y R + + applied element-wise. In all our experiments, we use the ELU non-linearity. We adopt the standard (i(k) drawn i.i.d from a centred Gaussian distribution with unit variance; this ensures that, if the coordinate of the input vector 2 R are all of order (1), so are the coordinates of x = $ (z).
normalization 1 = Wi. (2) and 111² = w /vH for weights Wis
augaeennmmtt consider data sample 2, E .M on the data-manifoll It can also be ex- pressed as 22 = (22)). We consider the natural data-augmentation process which consists in setting (x,) (2 Ew) for sample E Ied from isotropic Gaussian distribution with unit covariance and & > O. Crucially, the data-augmennattin scheme respect the low-dimensional structure of the data: the perturbed sample (2;) belongs to the data-manifold M for any perturbation vector sw. Note that, for any value of e, the data-augmentation preserves the low-dimensional manifold: perturbed samples (xx) exactly lie on the Kata-manifold The larger e, the more efficient the data-augmentation scheme; this property is important since it allows to study the influence of the amount of aata-augmentatio..
Classification: we consider balanced binary classification problem with |Dz] 2 2 labelled training examples {a,, Vi He., where x; = =22;, and y; E y/ = [-1,+1). The sample 2, E Rdd corre- sponding to the positive (resp. negative) class are assumed to have been drawn i.i.d from a Gaussian distribution with identity covariance matrix and mean μ4+ E Rdd (resp. mean μ E Rd). The distance ||x4 |-||| quantifies the hardness of the classification task.
Neural architecture and optimization: Consider fitting a two-layered neural network Fo : R.D R by minimising the negative log- lilkeiiihood L (0) (1/(Dz) Σ, (z;) vil where {(f. y) log(1 + exp[-y fl). We assume that there are [Pz] = 10 labelled data pairs {z., 3/ Ji as well as [Du] = 1000 unlabelled data samples, that the ambient space has dimension D = 100 and the data manifold .M has dimension d = 10. The function &b uses H = 30 neurons in its hidden layer. In all our experiments, we use standard Stochastic Gradient Descent (SGD) method with constant learning rate and momentum 3 = 0.9. For minimizing the consistency- -bsed SSL objective (0) + À (()). with regularization (0) given in equation 2, we use the standard strategy (TV17) consisting in first minimizing the un- regularrez objective alone L1 for a few epochs in order for the function Fo be learned in the neighbourhood of the few labelled data-samples before switching on the consistency-basee regularization whose role is to propagate the information contained in the labelled samples along the data manifold M.
Insensitivity to A: Figure 3 (Left) shows that this method is relatively insensitive to the parameter A, as long as it is within reasonable bounds. This phenomenon can be read from equation 8 that does not depend on X. Much larger or smaller values (not shown in Figure 3) of X do lead, unsurprisingly, to convergence and stability issues.
Amount of Auggeennmettt:ion: As is reported in many tasks (CZM+ 18; ZCG 19; KYF20), tuning the amount augmentgmenn in deep-learning applications is often a delicate exercise that can greatly influence the resulting performances. Figure 3 (Right) reports the generalization properties of the method for different amount of data- autta-agmmettiion. Too low an amount of data- aata-augmentttion (i.e. 8 0.03) and the final performance is equivalent the un-regularized method. Too large an amount of data-augmentation (i.e. & = 1.0) also leads to poor generalization properties. This is because the choice of 1.0 corresponds to augmented samples that very different from the distribution of the training dataset (i.c. distributional shift), although these samples are still supported by the -ata-manifold.
Quality of the Pata-Augmentatin scheme, we consider a perturbation process implemented as Szw])s(==;) = [b[2++w[k]] for x; = ((x) where the noise term w is defined as follows. For a dala-aaggeentation dimension parameter 1 < k < d we have w[k]] = (f....., ..l........ .x, O,..., 0) for i.i.d standard Gaussian samples fi...., & e R.
to study the influence of the quality of the lata-augmentation
This data-aaggeentation scheme only explores the first & dimensions of the d-dimensional data- manifold: the lower k. the poorer the exploration of the mata-oonif As demonstrated on Figure
8

---

Published as a conference paper at ICLR 2021
Generalization at Epoch 200
1.0
k-10
Texd
0.6
0.6
0.2
02
Augmoniation
Figure 4: Learning curve test (NLL) of the 11-model with À 10 for different "quality" of data-
augmentation. The data manifold is of dimension d 10 in ambient space of dimension D 100. For 2; = (22) and Σ I d, the data-augmentation scheme is implemented as
(z, (2 + w [k]) where [K] is sample from Gaussian distribution whose last (d k) coordinates are zero. In other words, the data-augmentation scheme only explores k dimensions out of the d dimensions of the data-manifold. We
0.3 in all the experiments. Left: Learning
curves (Test NLL) for aeta-auummntttion dimension [5, 10] Right: Test NLL at epoch N 200 (see left plot) for data-augmentation dimension & [5, 10].
0.8
MT: Ber
0.7
MT: fer= 990
MT: #.y=0.995
r1-madel
0.5
0.3
50
100
150
200
250
300
Epochs
Figure 5: Mean-Teacher (MT) learning curves (Test NLL) for different values of the exponential smoothing parameter ÅNT E (0, 1). For ÅMTT {0.9, 0.95, 0.99, 0.995), the final test NLL obtained through the MT approach is identical to the test NLL obtained through the IImmddel all the experiments, we used X = 10 and used SGD with momentum 3 = 0.9.
lower quality auttenttmmtttiionn schemes (i.e. lower values of & E [0, d) hurt the generalization performance of the I1-model.
Hean-Teecher versus 11-model: we implemented the Mean-Teacher (MT) approach with an expo- nential moving average (EMA) process 0agg.k k Bar O + Par) 6g for the MT parameter lagg.k with different scales far E {0.9, 0.95, 0.99, 0.995), as well as a II-model approach, with 10 and 0.3. Figure shows, in accordance with Section 4.1, that the different EMA schemes lead to generalization performances similar to a standard ITmmdel.
CONCLUSION
Consistency-ben based SSL methods rely subtle trade-off between the exploitation of the labelled samples and the discovery of the low-dimensional data-manifold. The results presented in this article highlight the connections with more standard methods such Jacobian penalization and graph- based approaches and emphasize the crucial role the data- iittaaaaummetaatoon scheme. The analysis of consistency- based SSL methods is still in its infancy and our numerical simulations suggest that the variant of the Hidden Manifold Model described in this text is a natural framework to make progress in this direction.

---

Published as a conference paper at ICLR 2021
REFERENCES
[ALAC17] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint arXiv:1710.1104/ 2017.
[AS17] Madhu Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03667, 2017.
[BAP14] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pscudo-ensembles In Advances in Neural Information Processing Systems, pages 3365-3373 2014.
[BCO1] Avrim Blum and Shuchi Chawla. Learning from labeled and unlabeled data using graph mincuts. In Proceedings of the Fighteenth International Conference on Machine Learning, pages 19-26. Morgan Kaufmann Publishers Inc., 2001.
[BCG*19 19] David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: holistic approach to cuni-super learning. In Advances in Neural Information Processing Systems, pages 5050-5060, 2019.
[Bis95] Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-14 116008811161 1995.
[BJ03] Ronen Basri and David W Jacobs. Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis & Machine Intelligence, (2):218-233, 2003.
[BM13] Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1886, 2013.
[BNS06] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research, 7(Nov):2399-2433 2006.
[Cay05] Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
[CKNH20] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A sim- ple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
[CS02] Olivier Chapelle and Bernhard Schölkopf. Incorporating invariances non-linear support vector machines. In Advances in neural information processing systems, pages 609-616, 2002.
[CSZ09] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
[CZM* + 18] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Au- toaugment: Learning augmentation policies from data, arXiv preprint 105.511805.09501, 2018.
[CZM+19) 19] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019.
[CZSL19] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv: 2019.
[DRS+18 18] Sripad Krishna Devalla, Prajwal K Renukanand, Bharathwaj K Sreedhar, Giridhar Subramanian, Liang Zhang, Shamira Perera, Jean-Martial Mari, Khai Sing Chin, Tin A Tun, Nicholas G Strouthidis, al. Drunet: dilated-residual U-net deep learning network to segment optic nerve head tissues in optical coherence tomography images. Biomedical optics express, 9(7):34443328 2018.
10

---

Published as a conference paper at ICLR 2021
[DSST19] Matthew M Dunlop, Dejan Slepčev, Andrew M Stuart, and Matthew Thorpe. Large data and zero noise limits of graph- sasph-baeed semi- sumi--seeeiieed learning algorithms. Applied and Computational Harmonic Analysis, 2019.
[EK09] Stewart N Ethier and Thomas G Kurtz. Markov processes: characterization and convergence, volume 282. John Wiley & Sons, 2009.
[GLK+ Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zde- borová. Generalisation error in learning with random features and the hidden manifold model. arXiv preprint (rXXi:2002.09339 2020.
[GMKZI9] Sebastian Goldt, Mare Mézard, Florent Krzakala, and Lenka Zdeborová. Mod- elling the influence of data structure on learning in neural networks. arXiv preprint arXi:: /500, 2019.
[GSA+20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mo- hammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
[HK02] Bernard Haasdonk and Daniel Keysers. Tangent distance kernels for support vector machines. In Object recognition supported by user interaction for service robots, volume 2, pages 864-868. IEEE, 2002.
[KYF20] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprinl arXiv:2004. 13649, 2020.
[LA16] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, arXiv preprint 1rX1v:1110.02222, 2016.
[LBC17] Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. IEEE Access, 5:5858 5869, 2017.
[17IL+18] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8896-8905, 2018.
[MHF+12] Grégoire Montavon, Katja Hansen, Siamac Fazli, Matthias Rupp, Franziska Biegler, Andreas Ziche, Alexandre Tkatchenko, Anatole Lilienfeld, and Klaus-Robert Müller. Learning invariant representations of molecules for atomization energy prediction. In Advances in Neural Information Processing Systems, pages 440-448, 2012.
[MMIK18] Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
[Mos16] Elchanan Mossel. Deep learning and hierarchal generative models, arXiv preprint 1612.0:61..0003 2016.
[NGP98] Partha Niyogi, Federico Girosi, and Tomaso Poggio. Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE, 86(1 1):219) 2209, 1998.
[0OR+18] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, pages 3235-3246, 2018
[PCZ + 19] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1900:00779999 2019.
[Pey09] Gabriel Peyré. Manifold models for signals and images. Computer Vision and Image Understanding, 113(2)2249-26) 2009.

---

Published as a conference paper at ICLR 2021
[P]92] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 855, 1992.
[Po190] Boris Polyak. New stochastic approximation type procedures. Automat. Telemekh, 7(98-107):2, 1990.
[PW17] Luis Perez and Jason Wang. The effectiveness of data augmentation in image classifica- tion using deep learning, arXiv preprint arXiv: 2017.
[RDV+11] Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In Advances Neural Information Processing Systems, pages 2294-2302, 2011.
[RM51] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[RVM+11] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Con- tractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning. pages 833-840. Omnipress, 2011.
[SBD+18] 18] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchin- sky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. 2018.
[SBL*20] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. arXiv preprint arXiv:2001.0766 2020.
[SHK* 14] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15(1):1929 1958, 2014.
[STT16] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pages 163-1171, 2016.
[SLDV98] Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition-tangeeeee tricks of the trade, pages 239-274. Springer, 1998.
distance and tangent propagation. In Neural
networks:
[SMG13] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlin- ear dynamics of learning in deep linear neural networks. arXiv preprint 2013.
[SZT17] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1rXiv:1703.008/ 2017.
[TP91] Matthew Turk and Alex Pentland. Eigenfaces for recognition. Journal of cognitive neuroscience, 3(1):71-86, 1991.
[TV17] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. In Ad- vances neural information processing systems, pages 1194-1204, 2017.
[ZZ15] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
[XDH 19] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsuper- vised data augmentation for consistency training. 2019.
[ZCDLP17] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17 2017.
12

---

Published as a conference paper at ICLR 2021
[ZCG 19] Barret Zoph. Ekin D Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe Le. Learning data augmentation strategies for object detection. arXiv preprint arXiv: 11172, 2019.
[ZG02] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical report, Citeseer, 2002.
[ZGL03] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pages 912-919, 2003.
[Zhu05] Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical report, Univer- sity of Wisconsin-Madisat Department Computer Sciences, 2005.
13


Reviews:
[{"review_id": "m_P572d9-gk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors analyze consistency-based models in specific settings where analytically tractable results can be obtained. They establish that leveraging more sophisticated data augmentation schemes is crucial to obtain huge gains when using consistency based models. Finally, they propose an extension of Hidden Manifold Model that incorporates data augmentation for understanding and experimenting with SSL methods.\n\nPros:\nThe paper is theoretically well grounded and the authors do a good job of connecting to previous works.\n\nThe paper is interesting and well written. It tries to explain the why consistency based method achieve good performance compared to other semi-supervised models. I find this relevant and helpful in understanding what makes these models work the way they do.\n\nCons:\nThe experiments are a bit weak only using simple synthetic settings.\nSome of the claims in the paper are not backed up by experimental results.\n\nComments:\nMore experiments should be provided to establish some of the claims in the paper. Additionally, it will be good to see an experiment on a real dataset albeit small rather than simple synthetic experiments.\n\nOne of the claims of the paper is that mean teacher and simple Π-model approach share the same solutions in the regime where the data augmentations are small but advanced data augmentations performs better. It will be nice to demonstrate this with an experiment for the reader to better understand why this happens.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qd3y0k6NgTv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the consistency-based SSL methods in settings  where  data lie a manifold of much lower dimension than the input space and obtains tractable results. The paper relates the analysis with Manifold Tangent Classifiers and shows that the quality of the perturbations plays a key role  to achieve a promising results in this set of SSL methods. Finally, the paper extends the Hidden Manifold Model by incorporating data-augmentation techniques and proposes a framework  to provide a direction for analyzing consistency-based SSL methods.\n\n+This work might be useful for those who want to work in the theoretical part of SSL. \n\n+The paper analytically discusses that the type of data augmentation plays a significant role in the performance  of the SSL models based on consistency regularization.  \n\n-While many points discussed in the paper are natural to me and well-discussed in the SSL literature (e.g.,  considering geometry of the data to develop an SSL algorithm or effect of the perturbation quality on the performance), relating and analyzing them with Manifold Tangent Classifiers (MTC) is interesting and new to me.  However, I still think that the theoretical part of the work is not strong enough and can be improved. The quality of the paper will be  improved if it uses MTC and provides some new results which do not exist in the SSL literature. This is because  MTC may not be the only approach to analyze these points.  For example, there are many other approaches based on optimal transport (e.g., Wasserstein Distances) that consider the geometry of the data and can be used to analyze the effect of perturbation on the SSL performance.  Then, someone may ask what is special in your approach to analysis consistency-based SSL methods in contrast to other tools/techniques?\n\n-While this paper is mostly an analytical paper, providing more experiments on some claims discussed in the paper  (e.g., Mean Teacher method and Π-model approach share the same solutions in the cases where the data-augmentations are small)  is necessary and can improve the quality of the paper. Furthermore, authors may  use the exact same underlying model, training set-up,  and then try different types of data-augmentation methods on several recent and effective consistency-based SSL methods to show the differences on the performance experimentally. This can better contextualize the paper as we will know how much is the difference in terms of accuracy between different consistency-based SSL methods  with respect to perturbation, or other data-augmentation methods. \n\nGenerally,  I think this  paper provides a good direction for understanding the consistency based-SSL methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MFoll9c0_VQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper proposes a theoretical framework for understanding consistency-based semi-supervised learning. While establishing this framework based on the Hidden Manifold Model, this paper frames the SSL in the context of Manifold Tangent Classifiers.\n\n# Pros\n\n1. Formal understanding of SSL is indeed currently limited and theoretical works are needed.\n2. The formalization of minimizers as harmonic function leads to the non-obvious prediction that SSL methods are insensitive, or at least very robust, to the weighting of the consistency loss $\\lambda$. To me, this is the most important result of the paper.\n3. The bibliography is well documented.\n\n# Cons\n1. The experimental verification of insensitivity to the weighting of consistency loss is unconvincing: it is done on a trivial low-dimensional toy dataset. It would be more convincing to take the GitHub code of one or more, possibly recent, SSL methods and vary $\\lambda$ on real datasets.\n2. I fail to see any other takeaway from the theoretical framework than the predicted insensitivity to $\\lambda$.\n3. Technically, if I understand correctly, the insensitivity to $\\lambda$ is not a takeaway of the Hidden Manifold Model but of the Minimizers are Harmonic Functions.\n4. I didn’t see where the claim “... demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances” has indeed been demonstrated.\n\n# Questions and nits\n1. “Several extensions such as the Mean Teacher (TV17) and the Virtual Adversarial Training (VAT) (MMIK18) schemes have been recently proposed and have been shown to lead to state-of-the-art results in many SSL tasks.” They are far from the current state of the art, especially so for few labels, this would have been true before the advent of MixMatch but the state of the art changed drastically with its introduction.\n2. “... consider as well the tangent plane Tx to M at x.” Here I have a hard time visualizing it. Say the manifold M is a 2D Gaussian point cloud for example, what would be the tangent plane for a point x in that cloud?\n3. Following on question 2, how is the tangent plane property exploited other than by defining an orthonormal basis on it? And why can’t an orthonormal basis of the Manifold space itself be enough if instead we assume it is dense?\n4. “Enriching the set of data-augmentation degrees of freedom with transformations such as elastic deformation or non-linear pixel intensity shifts is crucial to obtaining a high-dimensional local exploration manifold.” This seems in direct contradiction with MixMatch results which does not use any sophisticated augmentation: just pixel shift, mirroring and random pixel-wise linear interpolation between samples and labels (MixUp).\n5. Proposition 4.1: Here I was hoping for a prediction for your method. You mention the “sequence of processes converges weakly” and I was hoping it would explain why SSL techniques are much slower than fully supervised to converge. But then, it seems this statement is not exploited in any way other than saying that it does indeed converge to the solution of the ODE.\n6. “This indicates that the improved performances of the Mean Teacher approach sometimes reported in the literature are either not statistically meaningful, or due to poorly executed comparisons, or due to mechanisms not captured by the η → 0 asymptotic.” This is vague, considering Mean Teacher outperforms VAT, would this mean that the last option holds (due to mechanisms not captured by the η → 0 asymptotic)? Just to clarify what the last option actually means: does it mean the proposed analysis relies on assumptions that don’t capture the reality of the phenomenon?\n7. “These results are achieved by leveraging more sophisticated data augmentation schemes such as ... Mixup”. It seems odd to see MixUp being referred to as sophisticated (as in domain specific). MixUp is domain agnostic, it simply linearly combines a pair of samples and their labels. So in fact, it seems even simpler than a pixel shift since it’s linear.\n8. “... with a neural network with a single hidden layer with N = 100 neurons”. What non-linearity was used? I didn’t seem to find it, or I may have missed it.\n9. “Figure 3 (Left) shows that, contrary to several other types of regularizations such as weight-decay, this method is relatively insensitive to the parameter λ”. I didn’t see it. It indeed shows that (a) the method is relatively insensitive to the parameter λ in the context of the toy task but it doesn’t show that (b) the method is sensitive to other types of regularization such as weight-decay.\n\nNote: Ultimately I must admit a lot of the maths are above my head and I have no idea whether they are correct or not, therefore I didn't comment on them and only focused on the parts that I understand. On the other hand, I'm pretty confident in my understanding of SSL techniques and MixUp.\n\n=====POST-REBUTTAL COMMENTS======== \nI thank the authors for the response and the efforts in the updated draft. Most of my queries were clarified and I raised my rating accordingly. However, unfortunately, I still think a more realistic validation (e.g. on non-toy dataset) would benefit the paper. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides some theoretical perspective on the use of data augmentation in consistency regularization-based semi-supervised learning. The framework used in the paper argues that high-quality data augmentation should move along the data manifold. This generic view allows the paper's ideas to be applied across datasets (as opposed to image-specific data augmentation used in state-of-the-art semi-supervised learning algorithms). I am not aware of any other work raising these points, and indeed this paper is significant in that it provides a new and potentially useful perspective on the most performative semi-supervised learning approach. Reviewers agreed that the paper was clear and useful. The main concern was that the paper only included experiments in toy settings. Indeed, it would have been much more impactful to apply these ideas to state-of-the-art semi-supervised learning methods, but I think it can be excused given the theoretical focus of the work.
Decision: Accept (Poster)