LEARNING TO IMPUTE: A GENERAL FRAMEWORK FOR SEMI-SUPERVISED LEARNING

1 INTRODUCTION . Semi-supervised learning ( SSL ) ( Chapelle et al. , 2009 ) is one of the approaches to learn not only from labeled samples but also unlabeled ones . Under certain assumptions such as presence of smooth prediction functions that map data to labels , of low-dimensional manifolds that the high-dimensional data lies ( Chapelle et al. , 2009 ) , SSL methods provide a way to leverage the information at unlabeled data and lessens the dependency on labels . Recent work ( Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ; Berthelot et al. , 2019 ; Xie et al. , 2019 ) have shown that semi-supervised learning by using only a small portion of labels can achieve competitive results with the supervised counterparts in image classification tasks ( i.e . CIFAR10 , SVHN ) . They built on a variation of the well-known iterative bootstrapping method ( Yarowsky , 1995 ) where in each iteration a classifier is trained on the current set of labeled data , the learned classifier is used to generate label for unlabeled data . However , the generalization performance of this approach is known to suffer from fitting the model on wrongly labeled samples and overfitting into self-generated labels ( Tarvainen & Valpola , 2017 ) . Thus , they mitigate these issues by various regularization strategies . While there exist several regularization ( Srivastava et al. , 2014 ) and augmentation ( Zhang et al. , 2018 ; DeVries & Taylor , 2017 ) techniques in image recognition problems which are known to increase the generalization performance of deep networks , specific regularization strategies for semi-supervised classification are used to estimate correct labels for the unlabeled data by either encouraging the models to produce confident outputs ( Lee , 2013 ; Grandvalet & Bengio , 2005 ) and/or consistent output distributions when its inputs are perturbed ( Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ; Berthelot et al. , 2019 ) . The assumption here is that if a good regularization strategy exists , it can enable the network to recover the correct labels for the unlabeled data and then the method can obtain similar performance with the supervised counterpart when trained on them . Though this ad-hoc paradigm is shown to be effective , it raises a natural question for a more direct approach : Can we instead encourage the network to label the unlabeled data such that the network achieves better generalization performance when trained with them ? In this paper , we propose a new learning-to-learn method for semi-supervised learning that can be put in a meta-learning framework to address this question . Our method involves learning an update rule to label unlabeled training samples such that training our model using these predicted labels improves its performance not only on itself but also on a meta-validation set . Crucially , our method is highly generic and can easily be incorporated to the state-of-the-art methods ( Lee , 2013 ; Berthelot et al. , 2019 ) and boost their performance , in particular , in the regime where the number of available labels is limited . Moreover , our method is not limited to classification problems , we show that it can be extended to semi-supervised regression tasks where the output space is continuous and achieves significant performance gains . 2 RELATED WORK . Semi-supervised classification . There is a rich body of literature in SSL ( Chapelle et al. , 2009 ) for classification . Most of the recent work ( Lee , 2013 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ; Berthelot et al. , 2019 ; Xie et al. , 2019 ) builds on the idea of the bootstrapping technique of Yarowsky ( 1995 ) that involves iterative optimization of classifier on a set of labeled data and refinement of its labels for the unlabeled data . This paradigm is known to overfit on the noisy self-generated labels and thus suffer from low generalization performance . To alleviate the sensitivity to inaccurate labeling , researchers introduce various regularization strategies . Grandvalet & Bengio ( 2005 ) propose a minimum entropy regularizer that encourages each unlabeled sample to be assigned to only one of the classes with high probability . Lee ( 2013 ) instead follow a more direct approach , use the predicted label with the maximum probability for each unlabeled sample as true-label which is called “ pseudolabel ” . An orthogonal regularization strategy is to encourage a classifier to be invariant to various stochastic factors and produce consistent predictions for unlabeled data when the noise is added to intermediate representations ( Srivastava et al. , 2014 ) and to input in an adversarial manner ( Miyato et al. , 2018 ) . In the same vein , Laine & Aila ( 2017 ) ; Tarvainen & Valpola ( 2017 ) introduce the Π-model and Mean-Teacher that are regularized to be consistent over previous training iterations by using a temporal ensembling and teacher/student networks respectively . Recently Berthelot et al . ( 2019 ) introduced MixMatch algorithm that further extended the previous work by unifying the idea of the consistency regularization and augmentation strategies ( Zhang et al. , 2018 ) . While the recent techniques are shown to be effective in several classification benchmarks , the idea of consistency regularization is still implicit in terms of generalization performance . Here we take an orthogonal and more direct approach to this approach and learn to impute the labels of the unlabeled samples that improve the generalization performance of a classifier . We also show that our method can be used with the recent SSL techniques . Semi-supervised regression . Some of the recent techniques in semi-supervised classification are shown to be applicable to regression problems . Jean et al . ( 2018 ) have adapted various existing SSL methods such as label propagation , VAT ( Miyato et al. , 2018 ) and Mean Teacher ( Tarvainen & Valpola , 2017 ) and studied their performance in regression . The same authors also proposed a Bayesian SSL approach for SSL regression problems which is based on the recent deep kernel learning method ( Wilson et al. , 2016 ) based approach for semi-supervised learning that aims at minimizing predictive variance of unlabeled data . As in the classification task , these methods are typically ad-hoc and does not aim to generate labels for the unlabeled data that are optimized to improve the generalization in regression . Meta-learning . Our method is also related to meta-learning ( Schmidhuber , 1987 ; Bengio et al. , 1992 ) and inspired from the recent work ( Andrychowicz et al. , 2016 ; Finn et al. , 2017 ) where the goal is typically to quickly learn a new task from a small amount of new data . Andrychowicz et al . ( 2016 ) ; Finn et al . ( 2017 ) propose a learn gradient through gradient approach to train a model which has a strong generalization ability to unseen test data and can be easily adapted to new/unseen tasks . Sung et al . ( 2018 ) introduce the relation network to learn an additional metric such that the learned model ’ s feature embedding can be generalized for unseen data and unseen tasks . Ren et al . ( 2018b ) adopt the meta learning to learn the weight of samples to tackle the sample imbalance problem . Lorraine & Duvenaud ( 2018 ) ; MacKay et al . ( 2019 ) employ meta learning for automatically learning the hyper-parameters of the deep neural networks . Meta-learning has recently been applied to unsupervised learning ( Hsu et al. , 2019 ) and SSL for few shot learning ( Ren et al. , 2018a ) . Ren et al . ( 2018a ) adapts the prototypical network to use unlabeled examples when producing prototypes , enabling the prototypical network to exploit those unlabeled data to assist few shot classification . Antoniou & Storkey ( 2019 ) propose to learn a label-free loss function , parameterized as a neural network that enables the classifier to leverage the information from a validation set and achieves better performance in few-shot learning . ( Li et al. , 2019 ) propose a meta-learning technique to initialize a self-training model and to filter out noisy pseudo labels for semi-supervised few-shot learning . Similarly , our work also builds on the ideas of optimizing for improving the generalization for unseen samples . However , in contrast to the existing meta-learning methods that is proposed for few-shot learning problems , we focus on semi-supervised learning in general classification and regression problems where the number of samples are not limited to few . 3 METHOD . Consider a dataset D that consists of |L| labeled samples L = { ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( x|L| , y|L| ) } which is further split into a training T and meta-validation set V with |T | and |V| samples respectively . We also have a set of unlabeled samples U = { xu } i=1 , ... , |U| . Further , we let Φθ denote a model ( a function parameterized for instance , as a deep neural network ) with parameters θ , that is trained to predict labels y from given samples x as y = Φθ ( x ) . We are interested in imputing missing labels z of the unlabeled samples that are not only accurate but also improve performance at inference time , when included in our training set for optimizing model parameters θ . A straightforward approach to this problem is to first train a model to optimize the following cost function on the training set T : arg min θ ∑ x∈T ` ( Φθ ( x ) , y ) ( 1 ) where ` is a task specific loss , such as a soft-max followed by a cross-entropy loss for classification or squared loss for regression tasks . One can then use the trained model Φθ to impute labels for samples in the unlabeled set U as z = Φθ ( x̃ u ) as in Tarvainen & Valpola ( 2017 ) ; Berthelot et al . ( 2019 ) , where x̃u denotes randomly perturbed input image xu ( e.g . random crops ) . Note that the label imputation procedure can be replaced with other pseudo-label prediction techniques . Now we can expand T by adding the unlabeled samples with their imputed pseudo-labels i.e . A = T ∪ U and further train θ by using Eq . ( 1 ) . However , as many pseudo-labels z will likely be noisy , there is no guarantee that training on the augmented train set will improve the performance of the model on the meta-validation set V : CV = ∑ x∈V ` ( Φθ ( x ) , y ) . ( 2 ) Note that in all our experiments V is randomly sampled from T which ensures that our method is not trained using any additional data . Our goal is also to minimize the expected value of the loss in Eq . ( 1 ) with respect to the model parameters θ via an algorithm such as stochastic gradient descent ( SGD ) . Here , we go a step further and consider how the label imputation affects the ability of SGD to optimize the loss in Eq . ( 2 ) . We pose this as a meta-learning problem , which is derived next . We start by considering the loss value of the model Φθ on the augmented training set A : CA = ∑ x∈T ` ( Φθ ( x ) , y ) + ∑ xu∈U ` ( Φθ ( x u ) , z ) . ( 3 ) We first simulate a step of SGD using the loss in Eq . ( 3 ) to drive the update of the model parameters θ . At step t , SGD updates the parameters for minimizing Eq . ( 3 ) as follows : θ̂t+1 = θt − η∇θ ( ∑ x∈T ` ( Φθ ( x ) , y ) + ∑ xu∈U ` ( Φθ ( x u ) , z ) ) , ( 4 ) where∇θ is the gradient operator . We then wish to find z that minimizes the meta-validation objective CV ( θ̂t+1 ) in Eq . ( 2 ) with θ̂t+1 . This corresponds to the following bilevel optimization problem min z ( ∑ x∈V ` ( Φθ̂t+1 ( x ) , y ) ) , subject to θ̂t+1 = θt − η∇θ ( ∑ x∈T ` ( Φθ ( x ) , y ) + ∑ xu∈U ` ( Φθ ( x u ) , z ) ) . ( 5 ) Our goal is to learn predicted pseudo-labels that minimize the meta-validation loss in Eq . ( 5 ) . To this end , we propose two options , denoted as Option 1 and Option 2 . In the former , we treat z as latent ( or learnable ) parameters and compute the gradients of the meta-validation loss w.r.t z to updatez , which is used to update model parameters θ . In the latter , z is considered as the output of network Φθ and the gradients are thus computed w.r.t the model parameters θ for updating θ. Optimization . The gradients of CV w.r.t z for Option 1 and w.r.t θ for Option 2 are respectively be written as : ( Option 1 ) ∂C V ∂z = ∑ x∈V ∂ ` ( Φθ̂t+1 ( x ) , y ) ∂θ̂t+1 · ( ∂θt ∂z − η∇z∇θCA ) ( 6 ) ( Option 2 ) ∂C V ∂θt = ∂CV ∂z ∂z ∂θt = ∑ x∈V ∂ ` ( Φθ̂t+1 ( x ) , y ) ∂θ̂t+1 · ( I − η∇θ∇θCA ) ( 7 ) where I is identity matrix and ∇θ∇θ indicates a second order derivative which is supported by standard deep learning libraries such as Pytorch Paszke et al . ( 2017 ) , Tensorflow Abadi et al . ( 2015 ) . We then use the gradients in Eq . ( 6 ) or Eq . ( 7 ) to update the model parameters such that the updated model can predict pseudo-labels that minimize the meta-validation loss . In Option 1 , we update the pseudo-labels as ẑ = z − ηz dC V dz and the updated pseudo-labels are then used to update the model parameters θ by minimizing the loss CU = ∑ xu∈U ` ( Φθt ( x u i ) , ẑi ) . In Option 2 , the gradient in Eq . ( 7 ) is used to update the model parameters as θt+1 = θt − η ∂C V ∂θt We further depict the optimization details of Option 1 and 2 in Alg . 1 . We first estimate the z with the current model θt , update the model to optimize the loss CA and then re-estimate z with θ̂t . This part is so far similar to the self-labeling method of Lee ( 2013 ) . The loss function CA can trivially be replaced by those used in recent work such as Mean-Teacher ( Tarvainen & Valpola , 2017 ) or MixMatch ( Berthelot et al. , 2019 ) ; we applied our method in conjunction with all three loss functions and report results in Section 4 . Next we apply the updated model to a mini-batch of unlabeled samples , compute the loss and simulate a SGD step w.r.t this loss . In Option 1 , we initialize z with the output of Φθ ( x̃u ) and update it by using the gradient from Eq . ( 6 ) . In Option 2 , the meta-update is computed w.r.t θ and used to update the model parameters by using the gradient from Eq . ( 7 ) . Algorithm 1 : Pseudo-code of our method for two variants . Input : T , U , V . training set , unlabeled and meta-validation set resp . Required : model function Φ and its initialization parameters θ0 , a batch of training data BTt = { xi , yi } i=1 , ··· , |BTt | , a batch of unlabeled data B U t = { xui } i=1 , ··· , |BUt | and another batch of training data ( treated as meta-validation data during training ) BVt = { xi , yi } i=1 , ··· , |BVt | , learning rate α , η , ηz , weight λ. for t = 0 , · · · , K − 1 do z = { zi } i=1 , ··· , |BUt | = { Φθt ( x̃ u i ) } i=1 , ··· , |BUt | . estimate pseudo labels for unlabeled samples , x̃ui is randomly perturbed input x u i CA = ∑|BTt | i=1 ` ( Φθt ( xi ) , yi ) + λ ∑|BUt | i=1 ` ( Φθt ( x u i ) , zi ) θ̂t = θt − η dC A dθt . update the model and move to the meta update z = { zi } i=1 , ··· , |BUt | = { Φθ̂t ( x̃ u i ) } i=1 , ··· , |BUt | . estimate pseudo labels using the updated model C = ∑|BUt | i=1 ` ( Φθ̂t ( x u i ) , zi ) θ̂t+1 = θ̂t − α ∂C ∂θ̂t . simulate a SGD step CV ( θ̂t+1 ) = ∑|BVt | i=1 ` ( Φθ̂t+1 ( xi ) , yi ) . evaluate on validation data if option 1 then ẑ = z − ηz dC V dz . correct pseudo labels CU = ∑|BUt | i=1 ` ( Φθt ( x u i ) , ẑi ) θt+1 = θ̂t − η ∂C U ∂θ̂t . update the model with corrected pseudo labels else if option 2 then θt+1 = θ̂t − η ∂C V ∂θ̂t . update the model directly with meta-gradient Output : θK end

Summary: This paper uses a meta-learning approach to solve semi-supervised learning. The main idea is to simulate an SGD step on the loss of the meta-validation data and see how the model will perform if the pseudo-labels of unlabelled data are perturbed. Experiments on classification and regression problems show that the proposed method can improve over existing methods. The idea itself is intriguing but the derivation and some design choice are not very well-explained.