X-model: Improving Data Efficiency in Deep Learning with A Minimax Model

1 INTRODUCTION . In the last decade , deep learning has become the de facto choice for numerous machine learning applications in the presence of large-scale labeled datasets . However , collecting sufficient labeled data through manual labeling , especially for deep regression tasks such as keypoint localization and age estimation , is prohibitively time-costly and labor-expensive in real-world scenarios . To mitigate the requirement of labeled data , great effort ( Lee , 2013 ; Laine & Aila , 2017 ; Grandvalet & Bengio , 2005 ; Sohn et al. , 2020 ; Chen et al. , 2020b ) has been made to improve data efficiency in deep learning from the perspectives of simultaneously exploring both labeled and unlabeled data based on the intuitions for classification , e.g . cluster assumptions , pseudo labeling strategies , or consistency regularization under different data augmentations . However , most of the existing methods of improving data efficiency focus on classification setup while rare attention has been paid to the other side of the coin , i.e . deep regression which usually requires more human effort or expert knowledge to labeling . Moreover , due to the intrinsic difference between categorical and continuous label space , the methods based on the cluster or low-density separation assumptions ( Lee , 2013 ; Grandvalet & Bengio , 2005 ) for data-efficient classification tasks can not be directly adapted into deep regression . Meanwhile , existing data-efficient regression methods adopt k-nearest neighbor ( kNN ) ( Zhou & Li , 2005b ; Yu-Feng Li , 2017 ) , decision tree ( Levati et al. , 2017 ) or Gaussian Process ( Srijith et al. , 2013 ) as regressors on a fixed and shallow feature space , causing them difficult to be extended into deep learning problems . To develop a general data-efficient deep learning method for both classification and regression setups , we first delved into the existing methods and found that they can be briefly grouped into two categories : 1 ) Encourage invariance to data stochasticity with consistency regularization to make the predictions of the model invariant to local input perturbations , such as Π-model ( Laine & Aila , 2017 ) , UDA ( Xie et al. , 2020 ) and FixMatch ( Sohn et al. , 2020 ) ; 2 ) Encourage invariance to model stochasticity with difference penalty for predictions of models generated from different dropout ( Laine & Aila , 2017 ) or initialization ( Zhou & Li , 2005b ) , as well as models exponentially averaged from history models , such as Mean Teacher ( Tarvainen & Valpola , 2017 ) . Through the success of these above two strategies of consistency regularization that encourages invariance under data and model transformations respectively , it is intuitive to draw a conclusion that the invariance to stochasticity matters for improving data efficiency in deep learning . To take the power of both worlds , we propose a novel χ-model by simultaneously encouraging the invariance to data stochasticity and model stochasticity . First , instead of the weak augmentations ( e.g. , flip and crop ) adopted in Π-model , we utilize the strong augmentations ( e.g. , cutout and contrast ) adopted in FixMatch ( Sohn et al. , 2020 ) to enhance invariance to data stochasticity , as shown in Figure 1 with some example images tailored from Jung et al . ( 2020 ) . A natural question arises : Can we further enhance the invariance to model stochasticity similar to that of data stochasticity ? This paper gives a positive answer by introducing a minimax game between the feature extractor and task-specific heads . Compared to the manually designed strategy of adding a dropout layer , this novel approach directly optimizes a minimax loss function in the hypothesis space . By maximizing the inconsistency between task-specific heads , more diverse learners are generated to further enhance the invariance to model stochasticity and thus fully explore the intrinsic structure of unlabeled data . In short , our contributions can be summarized as follows : • We propose the χ-model that jointly encourages the invariance to data stochasticity and model stochasticity to improve data efficiency for both classification and regression setups . • We make the χ-model play a minimax game between the feature extractor and task-specific heads to further enhance invariance to model stochasticity . • Extensive experiments verify the superiority of the χ-model among various tasks , from an age estimation task to a dense-value prediction task of keypoint localization , a 2D synthetic and a 3D realistic dataset , as well as a multi-category object recognition task . 2 RELATED WORK . 2.1 DATA-EFFICENT CLASSIFICATION . In absence of abundant labeled data , it is reasonable to further explore the additional unlabeled data . A popular approach among these algorithms is Pseudo Labeling ( Lee , 2013 ) which leverages the model itself to generate labels for unlabeled data and uses generated labels for training . Besides Pseudo Labeling , there is another family of algorithms under the umbrella of “ self-training ” , which has received much attention both empirically ( Sohn et al. , 2020 ) and theoretically ( Wei et al. , 2021 ) . They either enforce stability of predictions under different data augmentations ( Tarvainen & Valpola , 2017 ; Xie et al. , 2020 ; Sohn et al. , 2020 ) ( a.k.a . input consistency regularization ) or fit the unlabeled data on its predictions generated by a previously learned model ( Lee , 2013 ; Chen et al. , 2020b ) . Further , Co-Training ( Blum & Mitchell , 1998b ) , Deep Co-Training Qiao et al . ( 2018 ) and Tri-Training ( Zhou & Li , 2005a ) improve data efficiency from an interesting perspective of different views of classifiers . MixMatch ( Berthelot et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) and UDA ( Xie et al. , 2020 ) reveal the crucial role of noise produced by advanced data augmentation methods . FixMatch ( Sohn et al. , 2020 ) uses predictions from weakly-augmented images to supervise the output of strongly augmented data . Meta Pseudo Labels ( Pham et al. , 2021 ) further improves data efficiency by making the teacher constantly adapted by the feedback of the student ’ s performance on the labeled dataset . SimCLRv2 ( Chen et al. , 2020b ) first fine-tunes the pre-trained model from the labeled data and then distills on the unlabeled data . Self-Tuning ( Wang et al. , 2021 ) introduces a pseudo group contrast ( PGC ) mechanism but is limited on classification setup . Besides of involving unlabeled data from the same distribution , another promising direction for improving data efficiency is introducing a complementary perspective to further improve data efficiency by introducing a related but different domain ( Long et al. , 2015 ; Ganin & Lempitsky , 2015 ; Long et al. , 2017 ; Saito et al. , 2018b ; Lee et al. , 2019 ; Zhang et al. , 2019 ; Saito et al. , 2018a ; 2019 ) . Moreover , various recent methods ( van den Oord et al. , 2018 ; He et al. , 2020 ; Wu et al. , 2018 ; Hadsell et al. , 2006 ; Tian et al. , 2019 ; Chen et al. , 2020a ) improve data efficiency by self-supervised learning . However , most existing data-efficient methods focus on classification setup while rare attention has been paid to deep regression . 2.2 DATA-EFFICIENT REGRESSION . Data-efficient regression methods mainly fall into three categories : Co-Training ( Blum & Mitchell , 1998a ) paradigm , kernel regression paradigm , and graph Laplacian regularization paradigm . COREG ( Zhou & Li , 2005b ) , as well as its following work ( Yu-Feng Li , 2017 ) , is a classic algorithm for regression method for improving data efficiency with two regressors ( kNNs ) learned by different distance metrics . It employs the Co-Training paradigm by predicting unlabeled data and using the most promising predictions to train the rest regressors . Transductive Regression ( Cortes & Mohri , 2007 ) exploits local linear regression for unlabeled data , takes those with relatively close neighbors , and trains a kernel regressor to produce the final prediction . Graph Laplacian regularization is a commonly used manifold regularization technique ( Belkin et al. , 2006 ) . It is reasonable to assume that data points with close input values should have similar output values , thereby regularizing the model output with respect to the unlabeled data . Levati et al . ( 2017 ) and Srijith et al . ( 2013 ) adopt decision tree and Gaussian Process as regressors respectively . However , these existing dataefficient regression methods are mainly designed for shallow regression , which requires closed-form or convex-solvers to solve the problem , causing them not suitable for deep regression problems . The comparison of various methods for improving data efficiency in deep learning is shown in Table 1 . 3 PRELIMINARIES . Denote a labeled dataset L = { ( xLi , y L i ) } nL i=1 with nL samples ( xLi , y L i ) , and an unlabeled dataset U = { ( xUi ) } nU i=1 with nU unlabeled samples . Usually , the size nL of L is much smaller than that nU of U and we define the the label ratio as nL/ ( nL + nU ) . Denote θ the feature generator network , and φ the successive task-specific head network . We aim at improving data efficiency in deep learning by fully exploring the labeled and unlabeled data from the perspective of stochasticity . 3.1 INVARIANCE TO DATA STOCHASTICITY . Data-efficient methods with the insight of invariance to data stochasticity aim at making the predictions of the model invariant to local input perturbations by a consistency regularization term . As shown in Figure 2 ( a ) , Π-model ( Laine & Aila , 2017 ; Sajjadi et al. , 2016 ) first generates two examples with different stochastic data augmentations and then introduces a loss term to minimize the distance between their predictions . With the strategy of invariance to data stochasticity , Π-model is believed to augment the model with information about the intrinsic structure ( “ manifold ” ) of U , avoiding overfitting to the labeled data L. Realizing that training a model on its own predictions ( e.g. , Π-model ) often provides no meaningful information , Data Distillation ( Radosavovic et al. , 2017 ) extends the dual-copies of Π-model to multiple transformations as shown in Figure 2 ( c ) . It first generates an ensembled prediction for each unlabeled input from the predictions of a single model run on different transformations ( e.g. , flipping and scaling ) , and then uses it to guide the training of this input . In summary , the training loss of invariance to data stochasticity can be formalized as min θ , φ Ldata ( x , U ) = Exi∈U ` ( ( φ ◦ θ ) ( aug1 ( xi ) ) , ( φ ◦ θ ) ( aug2 ( xi ) ) ) , ( 1 ) where ` ( · , · ) is a proper loss function for the target task . For clarity , we focus on a particular data example xi here and the superscript U is omitted . Recently , instead of the weak augmentations ( e.g. , flip and crop ) adopted in Π-model , many strong augmentations ( e.g. , cutout and contrast ) are adopted in FixMatch ( Sohn et al. , 2020 ) to further enhance invariance to data stochasticity .

Summary: The paper presents a g data-efficient approach that encourages invariance to both data and model stochasticity that works for both classification and regression tasks. Furthermore, the proposed minimax loss function can specifically enhance invariance to model stochasticity. The extensive experimental results verify that the proposed method is effective.