Semi-Supervised Learning with Normalizing Flows

1 INTRODUCTION . In many domains unlabeled data is plentiful , while labeled data may be scarce . Semi-supervised learning framework leverages both labeled and unlabeled data reducing the need for expensive manual annotation . Recently , consistency-based methods have shown outstanding performance in semisupervised image classification ( Laine & Aila , 2016 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Athiwaratkun et al. , 2019 ; Verma et al. , 2019 ; Berthelot et al. , 2019 ) and are currently state-of-the-art on challenging datasets like CIFAR-10 , CIFAR-100 , and ImageNet . However , these methods have not seen much application on domains other than images , where a suitable set of data perturbations to which the classifier should be invariant is not known a priori . In this paper , we employ a simple and principled probabilistic framework for semi-supervised learning . We introduce a mixture distribution for modeling the data , where different components correspond to different classes . We can maximize the joint likelihood of unlabeled ( using the mixture ) and labeled ( using individual mixture components ) data . At test time , we classify the input to belong to the class corresponding to mixture component with the highest likelihood . In order to apply this framework to complex data , we need to choose a sufficiently flexible family of distributions in the mixture , preserving the ability to compute exact likelihoods . We propose Flow Gaussian Mixture Model ( FlowGMM ) , a mixture model based on normalizing flows ( Dinh et al. , 2014 ) . Each component of the mixture is modeled as a trainable invertible transformation ( normalizing flow ) of the corresponding component of a Gaussian mixture in the latent space . The transformation is shared among all the mixture components . Due to invertibility , we can compute exact likelihood of the data using the change of variable formula . We illustrate FlowGMM on a toy problem in Figure 1 . We are solving a binary semi-supervised classification problem on the dataset shown in panel ( a ) : the labeled data is shown with triangles colored according to their class , and unlabeled data is shown with blue circles . We introduce a Gaussian mixture with two components corresponding to each of the classes , shown in panel ( c ) in the latent space Z and an invertible transformation f . The transformation f is then trained to map the data distribution in the data space X to the latent Gaussian mixture in the Z space , mapping the labeled data to the corresponding mixture component . We visualize the learned transformation in panel ( b ) , showing the positions of the images f ( x ) for all of the training data points . The inverse f−1 of this mapping serves as a class-conditional generative model , that we visualize in panel ( d ) . To classify a data point x in the input space we compute its image f ( x ) in the latent space , and pick the class corresponding to the Gaussian that is closest to f ( x ) . We visualize the decision boundary of the learned classifier with a dashed line in panel ( a ) . Many of the previous attempts of using generative models for classification ( Salimans et al. , 2016 ; Nalisnick et al. , 2019 ; Chen et al. , 2019 ) have relied upon multitask learning , where a shared latent representation is learned for the generative model and the classifier . With the method of Chen et al . ( 2019 ) , hybrid modeling is observed to reduce performance for both tasks in the supervised case . For GANs , Dai et al . ( 2017 ) have pointed out that classification performance and generative performance are in direct conflict : a perfect generator yields no benefit to classification performance . FlowGMM takes a different approach , where the generative model is used directly as a Bayes classifier , and in the limit of a perfect generative model the Bayes classifier achieves provably optimal misclassification rate ( see e.g . Mohri et al. , 2018 ) . FlowGMM naturally encodes the clustering principle : the decision boundary between classes must lie in the low-density region in the data space . Indeed , in the latent space the decision boundary between two classes coincides with the hyperplane perpendicular to the line segment connecting means of the corresponding mixture components and passing through the midpoint of this line segment ( assuming the components are normal distributions with identity covariance matrices ) ; in panel ( b ) of Figure 1 we show the decision boundary in the latent space with a dashed line . The density of the latent distribution near the decision boundary is low . As the flow is trained to represent data as a transformation of this latent distribution , the density near the decision boundary should also be low . In panel ( a ) of Figure 1 the decision boundary indeed lies in the low-density region . The contributions of this work are as follows : • We propose FlowGMM , a new probabilistic classification model based on normalizing flows , that can be naturally applied to semi-supervised learning . We evaluate FlowGMM on a range of semi-supervised classification benchmarks including text , tabular and image data , and it works well . FlowGMM outperforms alternative approaches on non-image data and shows promise for image data . We propose modified consistency regularization for FlowGMM and empirically demonstrate that it substantially improves performance of the method on image classification problems . • We conduct a thorough empirical analysis of FlowGMM for supervised and semisupervised classification . One of the important features of FlowGMM is its interpretability . To demonstrate it , we visualize the learned latent space representations for the proposed semi-supervised model and show that interpolations between data points from different classes pass through low-density regions . We show how our classification model can be used for optimization free feature visualization . We also study the predictive uncertainties produced by the method and show that they can be calibrated by scaling the variances of mixture components . 2 BACKGROUND : NORMALIZING FLOWS . The normalizing flow ( Dinh et al. , 2016 ) is an unsupervised model for density estimation defined as an invertible mapping f : X → Z from the data space X to the latent space Z . We can model the data distribution as a transformation f−1 : Z → X applied to a random variable from the latent distribution z ∼ pZ often chosen to be Gaussian . The density of the transformed random variable x = f−1 ( z ) is given by the change of variables formula pX ( x ) = pZ ( f ( x ) ) · ∣∣∣∣det ( ∂f∂x ) ∣∣∣∣ . ( 1 ) The mapping f is implemented as a sequence of invertible functions , parametrized by a neural network with architecture that is designed to ensure invertibility and efficient computation of logdeterminants , and a set of parameters θ that can be optimized . The model can be trained by maximizing the likelihood ( equation 1 ) of the training data with respect to the parameters θ . 3 FLOW GAUSSIAN MIXTURE MODEL ( FLOWGMM ) . In FlowGMM , we introduce a discrete latent variable y for the class label , y ∈ { 1 . . . C } . Our latent space distribution , conditioned on a given label k , is Gaussian with mean µk and covariance Σk : pZ ( z|y = k ) = N ( z|µk , Σk ) . ( 2 ) The marginal distribution of z is then a Gaussian mixture . When the classes are balanced , this distribution is pZ ( z ) = 1 C C∑ k=1 N ( z|µk , Σk ) . ( 3 ) Thus , the likelihood for labeled data is pX ( x|y = k ) = N ( f ( x ) |µk , Σk ) · ∣∣∣∣det ( ∂f∂x ) ∣∣∣∣ , and the likelihood for data with unknown label is pX ( x ) = ∑ k pX ( x|y = k ) p ( y = k ) . If we have access to both a labeled dataset D ` and an unlabeled dataset Du , then we can train our model in a semi-supervised way to maximize the joint likelihood of the labeled and unlabeled data pX ( D ` , Du ) = ∏ ( xi , yi ) ∈D ` pX ( xi , yi ) ∏ xj∈Du pX ( xj ) , ( 4 ) over the parameters θ of the bijective function f , which learns a density model with a Bayes classifier : given a test point x , the model predictive distribution is given by pX ( y|x ) = pX ( x|y ) p ( y ) /p ( x ) = N ( f ( x ) |µy , Σy ) ∑C k=1N ( f ( x ) |µk , Σk ) . ( 5 ) We can then make predictions for a test point x with the Bayes decision rule y = arg max i∈ { 1 , ... , C } pX ( y = i|x ) . Alternatively to direct likelihood maximization , we can adapt Expectation Maximization algorithm for model training ( discussed in Appendix A ) . 3.1 CONSISTENCY REGULARIZATION . Most of the existing state-of-the-art approaches to semi-supervised learning on image data are based on consistency regularization ( Laine & Aila , 2016 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Athiwaratkun et al. , 2019 ; Verma et al. , 2019 ) . These methods penalize changes in network predictions with respect to input perturbations , such as random translations and horizontal flips , with an additional loss term that can be computed on unlabeled data , ` cons ( x ) = ‖g ( x′ ) − g ( x′′ ) ‖2 , ( 6 ) where x′ , x′′ are random perturbations of x , and g is the vector of probabilities over the classes . Motivated by these methods , we introduce a simple consistency regularization term for FlowGMM . Let y′′ be the label predicted on image x′′ by FlowGMM according to equation 5 . We then define the consistency loss term as the negative log likelihood of the input x′ given the label y′′ : Lcons ( x ′ , x′′ ) = − log p ( x′|y′′ ) = − logN ( f ( x′ ) |µy′′ , Σy′′ ) − log ∣∣∣∣det ( ∂f∂x′ ) ∣∣∣∣ . ( 7 ) This loss term encourages the model to map small perturbations of the same unlabeled inputs to the same components of the Gaussian mixture distribution in the latent space . Unlike the standard consistency loss of equation 6 , the proposed loss in equation 7 takes values on the same scale as the data log likelihood ( equation 4 ) , and empirically we found it to perform better . We refer to FlowGMM with the consistency term as FlowGMM-cons . The final loss for FlowGMM-cons is then the weighted sum of the consistency loss ( equation 7 ) and the negative log likelihood of both labeled and unlabeled data ( equation 4 ) . 4 RELATED WORK . Generative models have been used extensively in semi-supervised learning . In the work of Kingma et al . ( 2014 ) , it was shown how the likelihood model of Variational Autoencoder ( Kingma & Welling , 2013 ) could be used for semi-supervised image classification on datasets like MNIST and SVHN . Xu et al . ( 2017 ) later extended this framework to semi-supervised text classification . Generative Adversarial Networks ( GANs ) have been employed for semi-supervised learning through multitask objective where the model learns to simultaneously discriminate generated images from real ( labeled and unlabeled ) images and classify labeled data . Along with GANs and VAEs , Normalizing Flows ( NF ) ( Dinh et al. , 2014 ) present another major class of deep generative models . Unlike GANs and VAEs , normalizing flows can be trained using exact likelihood . NFs admit controllable latent representations and can be sampled efficiently unlike auto-regressive models ( Papamakarios et al. , 2017 ; Oord et al. , 2016 ) . Recent work ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ; Behrmann et al. , 2018 ) demonstrated that normalizing flows can produce high-fidelity samples for natural image datasets . Some normalizing flow papers ( such as RealNVP ( Dinh et al. , 2016 ) ) have used class-conditional sampling , where the transformation is conditioned on the class label . To do so , they pass the class label as an input to coupling layers , conditioning the output of the flow on the class . Deep Invertible Generalized Linear Model ( DIGLM , Nalisnick et al. , 2019 ) , most closely related to our work , trains a classifier on the latent representation of a normalizing flow to perform supervised or semi-supervised image classification . Our approach is principally different , as we use a mixture of Gaussians in the latent space Z and perform classification based on class-conditional likelihoods ( see equation 5 ) , rather than training a separate classifier . One of the key advantages of our approach is the explicit encoding of clustering principle in the method and a more natural probabilistic interpretation .

Summary: The paper describes a normalising flow with the prior distribution represented by a Gaussian mixture model (GMM). The method, FlowGMM, maps each class of the dataset to a Gaussian distribution in the latent space by optimising the joint likelihood of both labelled and unlabelled data, thus making the method useful for semi-supervised problems. Predictions are made using the maximum a posteriori estimate of the class label. To make the method robust to small perturbations of the inputs, the authors introduce a novel consistency regularisation term to the total loss function, which maximises the likelihood of predicting the same class after a perturbation.