On Data-Augmentation and Consistency-Based Semi-Supervised Learning

1 INTRODUCTION . Consider a datasetD = DL∪DU that is comprised of labelled samplesDL = { xi , yi } i∈IL as well as unlabelled samples DU = { xi } i∈IU . Semi-Supervised Learning ( SSL ) is concerned with the use of both the labelled and unlabeled data for training . In many scenarios , collecting labelled data is difficult or time consuming or expensive so that the amount of labelled data can be relatively small when compared to the amount of unlabelled data . The main challenge of SSL is in the design of methods that can exploit the information contained in the distribution of the unlabelled data ( Zhu05 ; CSZ09 ) . In modern high-dimensional settings that are common to computer vision , signal processing , Natural Language Processing ( NLP ) or genomics , standard graph/distance based methods ( BC01 ; ZG02 ; ZGL03 ; BNS06 ; DSST19 ) that are successful in low-dimensional scenarios are difficult to implement . Indeed , in high-dimensional spaces , it is often difficult to design sensible notions of distances that can be exploited within these methods . We refer the interested reader to the book-length treatments ( Zhu05 ; CSZ09 ) for discussion of other approaches . The manifold assumption is the fundamental structural property that is exploited in most modern approaches to SSL : high-dimensional data samples lie in a small neighbourhood of a low-dimensional manifold ( TP91 ; BJ03 ; Pey09 ; Cay05 ; RDV+11 ) . In computer vision , the presence of this lowdimensional structure is instrumental to the success of ( variational ) autoencoder and generative adversarial networks : large datasets of images can often be parametrized by a relatively small number of degrees of freedom . Exploiting the unlabelled data to uncover this low-dimensional structure is crucial to the design of efficient SSL methods . A recent and independent evaluation of several modern methods for SSL can be found in ( OOR+18 ) . It is found there that consistency-based methods ( BAP14 ; SJT16 ; LA16 ; TV17 ; MMIK18 ; LZL+18 ; GSA+20 ) , the topic of this paper , achieve state-of-the art performances in many realistic scenarios . Contributions : consistency-based semi-supervised learning methods have recently been shown to achieve state-of-the-art results . Despite these methodological advances , the understanding of these methods is still relatively limited when compared to the fully-supervised setting ( SMG13 ; AS17 ; SBD+18 ; TZ15 ; SZT17 ) . In this article , we do not propose a new SSL method . Instead , we analyse consistency-based methods in settings where analytically tractable results can be obtained , when the data-samples lie in the neighbourhood of well-defined and tractable low-dimensional manifolds , and simple and controlled experiments can be carried out . We establish links with Manifold Tangent Classifiers and demonstrate that consistency-based SSL methods are in general more powerful since they can better exploit the local geometry of the data-manifold if efficient data-augmentation/perturbation schemes are used . Furthermore , in section 4.1 we show that the popular Mean Teacher method and the conceptually more simple Π-model approach share the same solutions in the regime when the data-augmentations are small ; this confirms often reported claim that the data-augmentation schemes leveraged by the recent SSL , as well as fully unsupervised algorithms , are instrumental to their success . Finally , in section 4.3 we propose an extension of the Hidden Manifold Model ( GMKZ19 ; GLK+20 ) . This generative model allows us to investigate the properties of consistency-based SSL methods , taking into account the data-augmentation process and the underlying low-dimensionality of the data , in a simple and principled manner , and without relying on a specific dataset . For gaining understanding of SSL , as well as self-supervised learning methods , we believe it to be important to develop a framework that ( i ) can take into account the geometry of the data ( ii ) allows the study of the influence of the quality of the data-augmentation schemes ( iii ) does not rely on any particular dataset . While the understanding of fully-supervised methods have largely been driven by the analysis of simplified model architectures ( eg . linear and two-layered models , large dimension asymptotic such as the Neural Tangent Kernel ) , these analytical tools alone are unlikely to be enough to explain the mechanisms responsible for the success of SSL and self-supervised learning methods ( CKNH20 ; GSA+20 ) , since they do not , and can not easily be extended to , account for the geometry of the data and data-augmentation schemes . Our proposed framework offers a small step in that direction . 2 CONSISTENCY-BASED SEMI-SUPERVISED LEARNING . For concreteness and clarity of exposition , we focus the discussion on classification problems . The arguments described in the remaining of this article can be adapted without any difficulty to other situations such as regression or image segmentation . Assume that the samples xi ∈ X ⊂ RD can be represented as D-dimensional vectors and that the labels belong to C ≥ 2 possible classes , yi ∈ Y ≡ { 1 , . . . , C } . Consider a mapping Fθ : RD → RC parametrized by θ ∈ Θ ⊂ R|Θ| . This can be a neural network , although that is not necessary . For x ∈ X , the quantity Fθ ( x ) can represent probabilistic output of the classifier , or , for example , the pre-softmax activations . Empirical risk minimization consists in minimizing the function LL ( θ ) = 1 |DL| ∑ i∈IL ` ( Fθ ( xi ) , yi ) for a loss function ` : RC × Y 7→ R. Maximum likelihood estimation corresponds to choosing the loss function as the cross entropy . The optimal parameter θ ∈ Θ is found by a variant of stochastic gradient descent ( RM51 ) with estimated gradient ∇θ { 1 |BL| ∑ i∈BL ` ( Fθ ( xi ) , yi ) } for a mini-batch BL of labelled samples . Consistency-based SSL algorithms regularize the learning by enforcing that the learned function x 7→ Fθ ( x ) respects local derivative and invariance constraints . For simplicity , assume that the mapping x 7→ Fθ ( x ) is deterministic , although the use of dropout ( SHK+14 ) and other sources of stochasticity are popular in practice . The Π-model ( LA16 ; SJT16 ) makes use of a stochastic mapping S : X × Ω → X that maps a sample x ∈ X and a source of randomness ω ∈ Ω ⊂ RdΩ to another sample Sω ( x ) ∈ X . The mapping S describes a stochastic data augmentation process . In computer vision , popular data-augmentation schemes include random translations , rotations , dilatations , croppings , flippings , elastic deformations , color jittering , addition of speckle noise , and many more domain-specific variants . In NLP , synonym replacements , insertions and deletions , back-translations are often used although it is often more difficult to implement these data-augmentation strategies . In a purely supervised setting , data-augmentation can be used as a regularizer . Instead of directly minimizing LL , one can minimize instead θ 7→ 1 |DL| ∑ i∈IL Eω [ ` ( Fθ [ Sω ( xi ) ] , yi ) ] . In practice , data-augmentation regularization , although a simple strategy , is often crucial to obtaining good generalization properties ( PW17 ; CZM+18 ; LBC17 ; PCZ+19 ) . The idea of regularizing by enforcing robustness to the injection of noise can be traced back at least to ( Bis95 ) . In the Π-model , the data-augmentation mapping S is used to define a consistency regularization term , R ( θ ) = 1 |D| ∑ i∈IL∪IU Eω { ∥∥Fθ [ Sω ( xi ) ] −Fθ ? ( xi ) ∥∥2 } . ( 1 ) The notation θ ? designates a copy of the parameter θ , i.e . θ ? = θ , and emphasizes that when differentiating the consistency regularization term θ 7→ R ( θ ) , one does not differentiate through θ ? . In practice , a stochastic estimate of ∇R ( θ ) is obtained as follows . For a mini-batch B of samples { xi } i∈B , the current value θ ? ∈ Θ of the parameter and the current predictions fi ≡ Fθ ? ( xi ) , the quantity ∇ { 1 |B| ∑ i∈B ∥∥Fθ [ Sω ( xi ) ] − fi∥∥2 } is an approximation of∇R ( θ ) . There are indeed many variants ( eg . use of different norms , different manners to inject noise ) , but the general idea is to force the learned function x 7→ Fθ ( x ) to be locally invariant to the data-augmentation scheme S. Several extensions such as the Mean Teacher ( TV17 ) and the VAT ( MMIK18 ) schemes have been recently proposed and have been shown to lead to good results in many SSL tasks . The recently proposed and state-of-the-art BYOL approach ( GSA+20 ) is relying on mechanisms that are very close to the consistency regularization methods discussed on this text . If one recalls the manifold assumption , this approach is natural : since the samples corresponding to different classes lie on separate manifolds , the function Fθ : X → RC should be constant on each one of these manifolds . Since the correct value of Fθ is typically well approximated or known for labelled samples ( xi , yi ) ∈ DL , the consistency regularization term equation 1 helps propagating these known values across these manifolds . This mechanism is indeed similar to standard SSL graph-based approaches such as label propagation ( ZG02 ) . Graph-based methods are difficult to directly implement in computer vision , or NLP , when a meaningful notion of distance is not available . This interpretation reveals that it is crucial to include the labelled samples in the regularization term equation 1 in order to help propagating the information contained in the labelled samples to the unlabelled samples . Our numerical experiments suggest that , in the standard setting when the number of labelled samples is much lower than the number of unlabeled samples , i.e . |DL| |DU | , the formulation equation 1 of the consistency regularization leads to sub-optimal results and convergence issues : the information contained in the labelled data is swamped by the number of unlabelled samples . In all our experiments , we have adopted instead the following regularization term R ( θ ) = 1 |DL| ∑ i∈IL Eω { ∥∥Fθ [ Sω ( xi ) ] −Fθ ? ( xi ) ∥∥2 } + 1 |DU | ∑ j∈IU Eω { ∥∥Fθ [ Sω ( xj ) ] −Fθ ? ( xj ) ∥∥2 } ( 2 ) that balances the labelled and unlabelled data samples more efficiently . Furthermore , it is clear that the quality and variety of the data-augmentation scheme S : X × Ω→ X is pivotal to the success of consistency-based SSL methods . We argue in this article that it is the dominant factor contributing to the success of this class of methods . Effort spent on building efficient local data-augmentation schemes will be rewarded in terms of generalization performances . Designing good data-augmentation schemes is an efficient manner of injecting expert/prior knowledge into the learning process . It is done by leveraging the understanding of the local geometry of the data manifold . As usual and not surprisingly ( NGP98 ; MHF+12 ) , in data-scarce settings , any type of domain-knowledge needs to be exploited and we argue that consistency regularization approaches to SSL are instances of this general principle .

Summary: The authors analyze consistency-based models in specific settings where analytically tractable results can be obtained. They establish that leveraging more sophisticated data augmentation schemes is crucial to obtain huge gains when using consistency based models. Finally, they propose an extension of Hidden Manifold Model that incorporates data augmentation for understanding and experimenting with SSL methods.