On Data-Augmentation and Consistency-based
Semi-Supervised Learning
Atin Ghosh, Alexandre H. Thiery
Department of Statistics and Applied Probability National University of Singapore

06/08/2021

Semi-supervised Learning(SSL)
What is SSL? SSL is concerned with the use of both the labelled
and unlabeled data for training.
Challenge: The design of methods that can exploit the information
contained in the distribution of the unlabeled data.
Assumption Cluster assumption, Low-density separation andManifold assumption.

Figure 1: The positive and negative signs show labeled examples from two
different classes. The circles depict the unlabeled examples. Thesedecision boundaries are moved to regions with lower density (solid line)
using unlabeled data.

Manifold learning

I Manifold learning assumes that the observed data lie on a
low-dimensional manifold embedded in a higher-dimensional
space.
I The manifold assumption states that the space of natural
images has the differential-geometric structure of alow-dimensional manifold embedded in the high-dimensional
pixel space.
I It should be emphasized that manifold learning refers to a setof methods based on the manifold assumption.

The taxonomy Deep Semi-supervised learning method.

I DSSL studies how to effectively utilize both labeled andunlabeled data by deep neural networks.
I DSSL can be classified into five categories: generative methods,
consistency regularization methods, graph-based methods,
pseudo-labeling methods, and hybrid methods.

Consistency-based SSL
I Assume that the samples xi ∈ X ⊂ RD and the labels
yi ∈ Y ≡ {1, . . . , C }. We have empirical risk minimization
L(θ) =1 X`(Fθ (xi ), yi ) + R(θ)|DL |(1)i∈IL

I Consistency-based SSL algorithms regularize the learning byenforcing that the learned function x 7→ Fθ (x) respects local
derivative and invariance constraints. In the Π-model, theconsistency regularization term can be written as follows:
R(θ) =no1 X
2Eω Fθ [Sω (xi )] − Fθ? (xi ).
|D|i∈IL ∪IU

Where θ? denotes a copy of the parameter θ, i.e. θ? = θ.

(2)

Consistency-based SSL
I To help propagating the information of labeled samples tounlabeled samples, in the experiment, we have adopted instead
the following regularization termo
1 X n
2Eω Fθ [Sω (xi )] − Fθ? (xi )|DL |i∈ILo
1 X n
2+Eω Fθ [Sω (xj )] − Fθ? (xj )|DU |R(θ) =(3)j∈IU

I The quality and variety of the data-augmentation scheme
S : X × Ω → X is pivotal to the success of consistency-based
SSL methods
I Designing good data-augmentation schemes is an efficientmanner of injecting expert/prior knowledge into the learningprocess.

The contribution of paper

One sentence summary: This paper propose a simple and natural
framework leveraging the Hidden Manifold Model to study modern
SSL methods.
I We analyse consistency-based methods in setting whereanalytically tractable low-dimensional can be obtained.
I We establish links with Manifold Tangent Classifiers anddemonstrate the consistency-based are more powerful.
I We propose an extension of the Hidden Manifold Model toinvestigate the properties of consistency-based SSL methods.

Approximate Manifold Tangent Classifier
I Consider the data manifold M ⊂ X ⊂ RD and the dimensionof M is 1 ≤ d ≤ D. For x ∈ M, the tangent plane Tx is alsoof dimension d with an orthonormal basis ex1 , . . . , exd ∈ RD .
Given small coefficients ω1 , . . . , ωd ∈ R, the transformed
sample x ∈ X also lies, or is very close to, the data manifold
M. x is defined asx = x+dX

ωj exj

j=1

I A possible stochastic data-augmentation scheme can thereforeP
be defined as Sω (x) = x + Vω where Vω = dj=1 ωj exj .
I To enforce that the function x → Fθ (x) is locallyapproximately constant along the manifold M, one can thus
penalize the derivatives of Fθ at x in the directions Vω .

Approximate Manifold Tangent Classifier
I Denoting by Jx ∈ RC ,D the Jacobian with respect to x ∈ RDof Fθ at x ∈ M, this can be implemented by adding apenalization term of the type Eω [kJx Vω k2 ] = Tr Γ ⊗ JT
x Jx ,where Γ ∈ RD,D is the covariance matrix of the random vector
ω → Vω .
I For any x, ω ∈ X × Ω, Sε ω (x) = x + ε , D(x, ω) + O(ε2 ), forsome derivative mapping D : X × Ω → X , it follows that122EkF[S(x)]−F(x)k=EkJD(x,ω)kωεωωxθθ
ε→0 ε2lim= Tr Γx,S ⊗ JTx Jx

where Γx,S is the covariance matrix of the X -valued random
vector ω 7→ D(x, ω) ∈ X . This shows that consistency-based
methods can be understood as approximated Jacobianregularization.

Manifold Tangent Classifier
I The MTC uses a Contractive-Auto-Encoder (CAE) to extract inan unsupervised manner a good representation of the dataset.
I The CAE can subsequently be leveraged to obtain an
approximate basis of each tangent plane Txi for xi ∈ D, which
can then be used for penalizing the Jacobian of the mapping
x 7→ Fθ (x) in the direction of the tangent plane to M at x.
How to build and train a deep network with MTC method.
I Train a stack of K CAE+H layers. Each is trained in turn on
the representation learned by the previous layer.
I For each xi ∈ D compute the Jacobian of the last layer(k)representation ∂h∂x (xi ) and its SVD.Store the leading dMsingular vectors in set.
I On top of the K pre-trained layers, stack an output layer ofsize the number of classes. Fine-tune the whole network, usingfor each xi and tangent directions Bxi

Limitations

Somewhat simplistic approach consisting in adding an isotropic
Gaussian noise to the data samples is unlikely to deliver satisfying
results, Why?
I This mechanism does not take at all into account thelocal-geometry of the data-manifold.
I It is equivalent to penalizing the Frobenius norm kJx k2F of theJacobian of the mapping x 7→ Fθ (x); in a linear model, that isequivalent to the standard ridge regularization.
I Jacobian penalization techniques are not efficient at learning
highly non-linear manifolds that are common.

Limitations
In general, domain-specific data-augmentation schemes lead tomuch better regularization than Jacobian penalization.
copfIµlabelledsamples

oooo

unlabeledsamples

localdata augmentation

Figure 2: Left: Jacobian (i.e. first order) Penalization method areshort-sighted and do not exploit fully the data-manifold Right:
Data-Augmentation respecting the geometry of the data-manifold.

Limitations

I Jacobian penalization techniques are not efficient at learning
highly non-linear manifolds.
I For example, in “pixel space”, a simple image translation is ahighly non-linear transformation only well approximated by afirst order approximation for very small translations.
I In other words, if x ∈ X represents an image and g (x, v ) is its
translated version by a vector v , the approximationg (x, v ) ≈ x + ∇v g (x), with∇v g (x) ≡ limε→0 (g (x, ε v ) − g (x)/ε, becomes poor as soon
as the translation vector v is not extremely small.

Limitations

I In computer vision, translations, rotations and dilatations areoften used as sole data-augmentation schemes: this leads to apoor local exploration of the data-manifold since this typetransformations only generate a very low dimensional
exploration manifold.
I Enriching the set of data-augmentation degrees of freedom
with transformations such as elastic deformation or non-linear
pixel intensity shifts is crucial to obtaining a high-dimensional
local exploration manifold that can help propagating theinformation on the data-manifold efficiently.

4. Asymptotic Properties

4.1 Fluid limit
I Consider the standard Π-model trained with a standard
Stochastic Gradient Descent (SGD). Denote by θt ∈ Θ thecurrent value of the parameter and η > 0 the learning rate. Wehave1 Xθk+1 = θk − η ∇θ`( Fθk (xi ), yi )|BL |i∈BL

2λ X
Fθk (Sω [xj ]) − fj+
|BL |

(4)

j∈BL

2
λ X+
Fθk (Sω [xk ]) − fk|BU |k∈BU

for a parameter λ > 0 that controls the trade-off between
supervised and consistency losses, as well as subsets BL and BUof labelled and unlabelled data samples, and fj ≡ Fθ? (xj ) forθ? ≡ θk .

4.2 Minimizers are harmonic functions
I Assume that M ⊂ RD can be globally parametrized byΦ : Rd → M ⊂ RD . Consider a data-augmentationSεω (x) = Φ(z + εω) for z = Φ−1 (x) and a sample ω from aRd -valued centred and isotropic Gaussian distribution. We
investigate the regime ε → 0 and, the minimization of theconsistency-regularized objectiveZnoλ2Eω Fθ [Sεω (Φ(z))] − Fθ (Φ(z))µ(dz).
LL (θ) + 2ε Rd(5)I As ε → 0,the objective function approaches the quantity
Z1 XG(fθ ) ≡`(fθ (zi ), yi ) + λk∇z fθ (z)k2 µ(dz). (6)|DL |Rdi∈IL4.2 Minimizers are harmonic functions

I The generalization properties of consistency-based SSL
methods will typically be insensitive to this parameter, in theregime of small data-augmentation at least.
I consistency-based SSL methods are indeed based on the same
principles as more standard graph-based approaches.

4.2 Minimizers are harmonic functions

Figure 3: Labelled data samples with class y = 0 (green triangle) andy = +1 (red dot) are placed on the Left/Right boundary of the unitsquare. Unlabelled data samples (blue stars) are uniformly placed within
the unit square. We consider a simple regression setting with loss function`(f , y ) = 12 (f − y )2 . Left: Randomly initialized neural network. Middle:
labelled/unlabelled data Right: Solution of f obtained by training astandard Π-model. It is the harmonic function f (u, v ) = u.

4.3 Generative model for Semi-Supervised LearningI The goal of this paper is to understand the mechanisms thatare at play when consistency-based SSL methods are used touncover the structures present in real datasets.
I It is important to build simplified and tractable generativemodels of data that (1) respect these low-dimensional
structures and (2) allow the design of efficientdata-augmentation schemes.
I Follow the Hidden Manifold Model framework, the author
introduce a model of synthetic data concentrating near
low-dimensional structures and analyze the learning curve
associated to a class of two-layered neural networks.

Low-dimensional structure

I The mapping Φ is chosen to be a neural network with a single
hidden layer with H neurons.
I For z = (z 1 , . . . , z d ) ∈ Rd , set Φ(z) = A1→2 ϕ(A0→1 z + b 1 )for matrices A0→1 ∈ RH,d and A1→2 ∈ RD,H , bias vectorb 1 ∈ RH and non-linearity ϕ : R → R applied element-wise.
(1) √I We adopt the standard normalization A0→1
= wi,j / d andi,j√
1→2 = w (2) / H for weights w (k) drawn i.i.d from a centred
Ai,ji,ji,jGaussian distribution with unit variance.

Data augmentation

I consider the natural data-augmentation Sεω (xi ) = Φ(zi + εω),where the sample ω ∈ Rd samples from an isotropic Gaussian
distribution with unit covariance and ε > 0.
I Sεω (xi ) belongs to the data-manifold M for any perturbation
vector ε ω. For any value of ε, the data-augmentation preserves
the low-dimensional manifold: perturbed samples Sεω (xi )exactly lie on the data-manifold.
I The larger ε, the more efficient the data-augmentation scheme.Neural architecture and optimization
I Fitting a two-layered neural network Fθ : RD → R byPminimising LL (θ) ≡ (1/|DL |) i `[Fθ (xi ), yi ] where`(f , y ) = log(1 + exp[−y f ]).
I Assume that |DL | = 10 labeled data pairs {xi , yi }i=IL , as well
as |DU | = 1000 unlabeled data samples. The ambient space
has dimension D = 100 and the data manifold M hasdimension d = 10I For minimizing the consistency-based SSL objective
LL (θ) + λ R(θ), we use the standard strategy consisting in firstminimizing the un-regularized objective alone LL for a few
epochs in order for the function Fθ to be learned in theneighbourhood of the few labeled data-samples before
switching on the consistency-based regularization.

= 1.0= 10.0= 100.0
Unregularized

0.6
0.5
Test NLL

Test NLL

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2

= 0.03= 0.10= 0.30= 1.00

0.4
0.3
0.2

02040

Epoch6080

1000255075 100 125 150 175 200
EpochFigure 4: Left: For a fixed data-augmentation scheme, generalization
properties for λ spanning two orders of magnitude. Right: Influence of thequantity of the data-augmentation of the generalization properties.

I Much larger or smaller values of λ do lead to convergence andstability issues.
I Too low an amount of data-augmentation (i.e. ε = 0.03) andthe final performance is equivalent to the un-regularized
method. Too large an amount of data-augmentation (i.e.
ε = 1.0) also leads to poor generalization properties.

Quality of the Data-augmentation
Generalization at Epoch 200k=5k=6k=7k=8k=9k=10

Test NLL

0.8

1.00.8Test NLL

1.00.6

0.6
0.4

0.4
0.2

0.2
0255075

100 125 150 175 200
Epoch56789k: Data Augmentation Dimension

10Figure 5: Left: Learning curves (Test NLL) for data-augmentation
dimension k ∈ [5, 10] Right: Test NLL at epoch N = 200 (see left plot)
for data-augmentation dimension k ∈ [5, 10].

Consider a perturbation Sεω[k] (xi ) = Φ(zi + ω[k]) for xi = Φ(zi )where the noise term ω[k] is defined as follows. For adata-augmentation dimension parameter 1 ≤ k ≤ d we have
ω[k] = (ξ1 , . . . , ξk , 0, . . . , 0) for i.i.d standard Gaussian samples
ξ1 , . . . , ξk ∈ R.

Mean-Teacher versus Π-model
I MT parameter: θavg,k = βMT θavg,k−1 + (1 − βMT )θk for theθavg,k with different scales βMT ∈ {0.9, 0.95, 0.99, 0.995}.
MT: MT=0.900
MT: MT=0.950
MT: MT=0.990
MT: MT=0.995-model0.8
Test NLL

0.7
0.6
0.5
0.4
0.3
050

100150Epochs200

250300Figure 6: Mean-Teacher (MT) learning curves (Test NLL) for different
values of the exponential smoothing parameter βMT ∈ (0, 1). ForβMT ∈ {0.9, 0.95, 0.99, 0.995}, the final test NLL obtained through theMT approach is identical to the test NLL obtained through the Π-model.
In all the experiments, we used λ = 10 and used SGD with momentum
β = 0.9.

Comment of reviewer

I This paper provides some theoretical perspective on the use ofdata augmentation in consistency regularization-based
semi-supervised learning.
I The framework used in the paper argues that high-quality dataaugmentation should move along the data manifold.
I This generic view allows the paper’s ideas to be applied acrossdatasets (as opposed to image-specific data augmentation usedin state-of-the-art semi-supervised learning algorithms).