Semi-Supervised Learning with Normalizing Flows
Authors: Pavel Izmailov
Polina Kirichenko
Marc Finzi
Andrew Wilson
Venue:  (2020)
Keywords: Semi-Supervised Learning
Normalizing Flows

Abstract: We propose Flow Gaussian Mixture Model (FlowGMM), a general-purpose method for semi-supervised learning based on a simple and principled probabilistic framework. We approximate the joint distribution of the labeled and unlabeled data with a flexible mixture model implemented as a Gaussian mixture transformed by a normalizing flow. We train the model by maximizing the exact joint likelihood of the labeled and unlabeled data. We evaluate FlowGMM on a wide range of semi-supervised classification  problems across different data types: AG-News and Yahoo Answers text data, MNIST, SVHN and CIFAR-10 image classification problems as well as tabular UCI datasets. FlowGMM achieves promising results on image classification problems and outperforms the competing methods on other types of data. FlowGMM learns an interpretable latent repesentation space and allows hyper-parameter free feature visualization at real time rates. Finally, we show that FlowGMM can be calibrated to produce meaningful uncertainty estimates for its predictions. 

Under review as a conference paper at ICLR 2020
SEMI-SUPERVISED LEARNING WITH NORMALIZING FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose Flow Gaussian Mixture Model (FlowGMM), general-purpose method for semi-supervised learning based on a simple and principled proba- bilistic framework. We approximate the joint distribution of the labeled and un- labeled data with flexible mixture model implemented as a Gaussian mixture transformed by normalizing flow. We train the model by maximizing the ex- act joint likelihood of the labeled and unlabeled data. We evaluate FM on a wide range of semi-supervised classification problems across different data types: AG-News and Yahoo Answers text data, MNIST, SVHN and CIFAR-10 10 im- age classification problems as well as tabular UCI datasets. FlowGMM achieves promising results image classification problems and outperforms the competing methods on other types of data. FlowGMM learns an interpretable latent repre- sentation space and allows hyper-paramest free feature visualization real time rates. Finally, we show that FlowGMM can be calibrated to produce meaningful uncertainty estimates for its predictions.
INTRODUCTION
In many domains unlabeled data is plentiful, while labeled data may be scarce. Semi-supervised learning framework leverages both labeled and unlabeled data reducing the need for expensive man- ual annotation. Recently, consistency- -baed methods have shown outstanding performance in semi- supervised image classification (Laine & Aila 2016 Miyato et al. 2018)] Tarvainen & Valpola [017] Athiwaratkun et al. 2019 Verma et al. 2019 Berthelot et 2019) and are currently state-of-the-art on challenging datasets like CIFAR- TO, CIFAR-TOO, and ImageNet. However, these methods have not seen much application domains other than images, where a suitable set of data perturbations to which the classifier should be invariant is not known a priori.
In this paper, employ simple and principled probabilistic framework for semi-supervised learn- ing. We introduce a mixture distribution for modeling the data, where different components corre- spond to different classes. We maximize the joint likelihood of unlabeled (using the mixture) and labeled (using individual mixture components) data. At test time, we classify the input to belong to the class corresponding to mixture component with the highest likelihood.
In order to apply this framework to complex data, we need to choose a sufficiently flexible family of distributions in the mixture, preserving the ability to compute exact likelihoods. We propose Flow Gaussian Mixture Model (FlowGMM), a mixture model based on normalizing flows (Dinh) al. 2014). Each component of the mixture is modeled as a trainable invertible transformation (normalizing flow) of the corresponding component of a Gaussian mixture in the latent space. The transformation shared among all the mixture components. Due to invertibility, we can compute exact likelihood of the data using the change of variable formula.
We illustrate FlowGMM on a toy problem in Figure We are solving binary semi-supervised classification problem on the dataset shown in panel (a): the labeled data is shown with triangles colored according to their class, and unlabeled data shown with blue circles. We introduce a Gaussian mixture with two components corresponding to each of the classes, shown in panel (c) in the latent space Z and an invertible transformation f. The transformation f is then trained to map the data distribution in the data space X' to the latent Gaussian mixture in the Z space, mapping the labeled data to the corresponding mixture component. We visualize the learned transformation in panel (b), showing the positions of the images fxx) for all of the training data points. The inverse

---

Under review as a conference paper at ICLR 2020
X, Data
2, Latent
2, Latent
X, Data
(a)
(b)
(c)
(d)
Figure 1: Illustration of semi-supervised learning with FlowGMM on a binary classification prob- lem. Colors represent the two classes or the corresponding Gaussian mixture components. Labeled data is shown with triangles, colored by the corresponding class label, and blue dots represent un- labeled data. (a): Data distribution and the classifier decision boundary. (b): The learned mapping of the data to the latent space. (c): Samples from the Gaussian mixture in the latent space. (d): Samples from the model in the data space.
f-! of this mapping serves To classify a data point æ in the input space we compute its image f(x) ) in the latent space, and pick the class corresponding to the Gaussian that is closest to f())). We visualize the decision boundary of the learned classifier with dashed line in panel (a).
conditoonii generative model, that visualize in panel (d).
Many of the previous attempts of using generative models for classification (Salimans et al. 2016 Nalisnick et al. 20119] [Chen et al. 2019) have relied upon multitask learning, where shared latent representation learned for the generative model and the classifier. With the method Chen al. 22019, hybrid modeling is observed to reduce performance for both tasks in the supervised case. For GANs, Dai et al. (2017) have pointed out that classification performance and generative per- formance are in direct conffict: a perfect generator yields no benefit to classification performance. FlowGMM takes a different approach, where the generative model used directly as a Bayes clas- sifier, and in the limit of perfect generative model the Bayes classifier achieves provably optimal misclassification rate (see e.g. Mohri et al. 2018).
FlowGMM naturally encodes the clustering principle: the decision boundary between classes must lie in the low-density region in the data space. Indeed, the latent space the decision boundary between two classes coincides with the hyperplane perpendicular to the line segment connecting means of the corresponding mixture components and passing through the midpoint of this line seg- ment (assuming the components normal distributions with identity covariance matrices): in panel (b) of Figure [ we show the decision boundary in the latent space with a dashed line. The density of the latent distribution near the decision boundary is low, As the flow is trained to represent data as transformation of this latent distribution, the density near the decision boundary should also be low. In panel (a) of Figure the decision boundary indeed lies in the low-density region.
The contributions of this work are as follows:
We propose FlowGMM, new probabilistic classification model based on normalizing flows, that can naturally applied to semi-supervised learning. We evaluate FlowGMM range of semi-supervised classification benchmarks including text, tabular and image data, and works well. FlowGMM outperforms alternative approaches on non-image data and shows promise for image data. We propose modified consistency regularization for FlowGMM and empirically demonstrate that it substantially improves performance of the method on image classification problems.
We conduct a thorough empirical analysis of FlowGMM for supervised and semi- supervised classification. One of the important features of FlowGMM is its interpretability. To demonstrate it, we visualize the learned latent space representations for the proposed semi-supervised model and show that interpolations between data points from different classes pass through low-density regions. We show how our classification model can be used for optimization free feature visualization. We also study the predictive uncertainties produced by the method and show that they can be calibrated by scaling the variances of mixture components.
2

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND: NORMALIZING FLOWS
The normalizing flow (Dinh et al. 2016 is an unsupervised model for density estimation defined invertible mapping X' Z from the data space X' to the latent space 2. We can model the data distribution as a transformation f-; 2 + X' applied to a random variable from the latent distribution z pz often chosen to be Gaussian. The density of the transformed random variable (z) is given by the change of variables formula
- -(z))=zz((xx) det
Of Or
(I)
The mapping I' is implemented as sequence of invertible functions, parametrized by neural network with architecture that is designed to ensure invertibility and efficient computation of log- determinants, and set of parameters / that can be optimized. The model can be trained by maxi- mizing the likelihood (equation T the training data with respect to the parameters O.
FLOW GAUSSIAN MIXTURE MODEL (FLOWGMM)
In FlowGMM, we introduce a discrete latent variable for the class label, y E {1...C). Our latent space distribution, conditioned given label k. is Gaussian with mean μ4 and covariance ME::
pz(=|y = k) = =>(zuuu Ex).
The marginal distribution of z is then Gaussian mixture. When the classes are balanced, this distribution is
c
pz E
N(=luu, Ex).
(3)
Thus, the likelihood for labeled data is
[ff Oz
px - =)
Ex) det
and the likelihood for data with unknown label is (x) = En px (xly k)p(y = k). If we have access to both labeled dataset Dj and an unlabeled dataset D.,. then we can train model in semi-supervised way to maximize the joint likelihood of the labeled and unlabeled data
Px DD),
II (E..).)DD
Px (2i, U1)
II Px(2)),
(4)
ayED.
the parameters of the bijective function f. which learns density model with Bayes classi- fier: given a test point the model predictive distribution is given by
.V (f(x)) En)
(x(1)=) px (=()))((w)/p(()
(5)
Σe ((FE))) Ex)
can then make predictions for a test point x with the Bayes decision rule y arg ie (E......
px(v ilır).
Alternatively to direct likelihood maximization, we can adapt Expectation Maximization algorithm for model training (discussed in Appendix A
3.1 CONSISTENCY REGULARIZATION
Most of the existing satte-of-hhear approaches to semi-supervised learning on image data are based consistency regularization (Laine & Aila 2016 Miyato et al.) 2018:) Tarvainen Valpola 2017 Athiwaratkun et al.) 2019 Verma et al.] (219). These methods penalize changes in network predic- tions with respect input perturbations, such random translations and horizontal flips, with an additional loss term that can be computed on unlabeled data,
Econs (x) Ilg(x') (x
3

---

Under review as a conference paper at ICLR 2020
where z', x² are random perturbations of and g is the vector of probabilities the classes.
Motivated by these methods, we introduce a simple consistency regularization term for Flo Let y" be the label predicted image z" by FlowGMM according to equation 5 We then define the consistency loss term as the negative log likelihood of the input x' given the label y":
of Ox
T cons (1', (x',x"))'"""""""))))" - log plo' p(a"'p"")) = log.N(((z")) Pgge)
log det
(7)
This loss term encourages the model to map small perturbations of the same unlabeled inputs to the same components of the Gaussian mixture distribution the latent space. Unlike the standard consistency loss of equation 6 the proposed loss in equation 7 takes values on the same scale as the data log likelihood (equation 4 and empirically we found it to perform better. We refer to FlowGMM with the consistency term FlowGMM-cons. The final loss for FlowGMM-cons is then the weighted sum of the consistency loss (equation 7 and the negative log likelihood both labeled and unlabeled data (equation
RELATED WORK
Generative models have been used extensively in semi cumi-supervieed learning. In the work of Kingma ct al. (2114) it was shown how the likelihood model of Variational Autoencoder (Kingma & Welling 2013) could used for semi-supervised image classification on datasets like MNIST and SVHN. Xu et al. (2017) later extended this framework to remi-supervised text classification. Generative Adversarial Networks (GANs) have been employed for semi-supervised learning through multitask objective where the model learns to simultaneously discriminate generated images from real (labeled and unlabeled) images and classify labeled data.
Along with GANs and VAEs, Normalizing Flows (NF) (Dinh et al. 2014) present another major class of deep generative models. Unlike GANs and VAEs, normalizing flows can be trained using exact likelihood. NFs admit controllable latent representations and can be sampled efficiently unlike auto-regressive models (Papamakarios et al., 2017 Oord et al. 2016). Recent work (Dinh et al. 2016 Kingma & Dhariwal 2018 [ehhrman al. 2018) demonstrated that normalizing flows can produce high-fidelity samples for natural image datasets. Some normalizing flow papers (such as RealNVP (Dinh et al. 2016)) have used class-conditional sampling, where the transformation is conditioned on the class label. To do so, they pass the class label as an input to coupling layers, conditioning the output of the flow on the class.
Deep Invertible Generalized Linear Model (DIGLM, Nalisnick et al. 2019), most closely related to our work, trains classifier on the latent representation of a normalizing flow to perform supervised or semi-supervised image classification. Our approach is principally different, as we use mixture of Gaussians in the latent space Z and perform classification based on class-conditional likelihoods (see equation [). rather than training a separate classifier. One of the key advantages of our ap- proach is the explicit encoding of clustering principle the method and a more natural probabilistic interpretation.
EXPERIMENTS
In experiments, we use the RealNVP normalizing flow architecture. Throughout training, Gaus- sian mixture parameters are fixed: the means are initialized randomly from the standard normal distribution and the covariances are set to I. See Appendix B for further discussion on GMM ini- tialization and training.
5.1 SYNTHETIC DATA
We first apply FlowGMM to range of two-dimensional synthetic datasets, in order to get better visual intuition for the method. We use RealNVP architecture with 5 coupling layers, defined by fully-connected shift and scale networks, each with hidden layer of size 512. In addition to the semi-supereieed setting, we also trained the method only using the labeled data. In Figure 2 we visualize the decision boundaries of the classifier corresponding to FlowGMM for both of these
4

---

Under review as a conference paper at ICLR 2020
(a) Labeled + Unlabeled
(b) Labeled Only
(c) Labeled + Unlabeled
(d) Labeled Only
Figure 2: Illustration of FlowGMM performance on synthetic datasets. Labeled data shown with colored triangles, and unlabeled data is shown with blue circles. Colors represent different classes. We compare classifier decision boundaries when only using labeled data (panels d) and when using both labeled and unlabeled data (panels c) two circles (panels a, b) and pinwheel (panels c, d) datasets. FlowGMM leverages unlabeled data to push the decision boundary to low-density regions of the space.
Accuracy BERT embedded text classification datasets and UCI datasets with small number labeled examples. The kNN baseline, logistic regression, and the 3-Layer NN + Dropout were trained on the labeled data only. Numbers reported for each method are the best of 3 runs (ranked by performance on the validation set). and ny are the number of labeled and unlabeled data points.
Table 1:
Dataset (mu naa classes)
Method
AG-News (200 200k, 4)
Yahoo Answers (800 / 50k, 10)
Hepmass (20/ 140k, 2)
Miniboone (20/ / 65k, 2)
KNN
51.3
28.4
84.6
77.7
Logistic Regression 3-Layer NN + Dropout
78.9
54.9
84.9
75.9
78.1
55.6
84.4
77.3
RBF Label Spreading
54.6
30.4
87.1
78.8
KNN Label Spreading
56.7
25.6
87.2
78.1
11-model FlowGMM
80.6
56.6
87.9
78.3
84.8
57.4
88.8
80.6
settings on the two circles and pinwheel datasets. On both datasets FlowGMM is able to leverage the unlabeled data to push the decision boundary to a low-density region, as expected. On the two circles dataset the method is unable to fit the data perfectly, as it is impossible to represent this dataset an invertible continuous mapping of Gaussians, because they are topologically distinct. FlowGMM still produces reasonable decision boundary and improves over the case when only labeled data is available. We provide additional visualizations Appendix C Figure 4
5.2 TEXT AND TABULAR DATA
We believe that FlowGMM can be especially useful for semi-supervised learning on tabular data. Consistency-basse semi-supervised methods have mostly been developed for image classification, where the predictions of the method are regularized be invariant to random flips and translations of the image. On tabular data, where the structure is less prominent, finding suitable transformations to apply for consistency-basee methods is not-trivial. Similarly, approaches based on GANs have mostly been developed for images. We evaluate FlowGMM the Hepmass and Miniboone UCI classification datasets (previously used in Papamakarios et al. (0177) for density estimation).
Along with standard tabular UCI datasets, also consider text classification AG-News and Yahoo Answers datasets. Using the recent advances in transfer learning for NLP, we construct embeddings for input texts using the BERT transformer model (Devlin al.) 2018) trained corpus of Wikipedia articles, and then train FlowGMM and other baselines on the embeddings.
We compare FlowGMM to the graph based label spreading method from Zhou al. (2004), II-Model (Laine Aila 2016) that uses dropout perturbations, as well as supervised logistic regres- sion, k-nearest neighbors, and neural network that were trained the labeled data only. We report
5

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy of the FlowGMM, VAE model (MI+M2 VAE, Kingma et al. 2014). DIGLM NNllisnick et al. 2019) in supervised and semi-supervised settings on MNIST, SVHN, and CIFAR- 10. FlowGMM Sup (All labels) well as DIGLM Sup (All labels) were trained on full train datasets with all labels to demonstrate general capacity of these models. FlowGMM Sup (nz la- bels) was trained on Ti labeled examples (and no unlabeled data). For reference, at the bottom we list the performance of the II-Model (Laine & Aila 2016) and BadGAN (Dai et al.) (017) as rep- resentative consistency--un based and GAN-based state-of-the- methods. Both of these methods use non-invertible architectures with substantially higher base performance and, thus, are not directly comparable.
Dataset (n4 / (ay)
Method
MNIST
SVHN (1k/72k)
CIFAR-10 (4k/16k)
(1k/59k)
DIGLM Sup (All labels)
99.27
95.74
FlowGMM Sup (All labels)
99.63
95.81
88.44
M1+M2 VAE SSL
97.60
63.98
DIGLM SSL
99.0
FlowGMM Sup (n; labels)
97.36
78.26
73.13
FlowGMM
98.94
82.42
78.24
FlowGMM-cons
99.0
86.44
80,9
BadGAN
95.75
85,59
IT-Model
94.57
87.64
the results in Table 5.1 where FlowGMM beats the competing semi-supervised learning methods on each of the considered datasets. Implementation details for FlowGMM, the baselines, and the dataset preprocessing details are listed in Appendix D
5.3 IMAGE CLASSIFICATION
We next evaluate the proposed method on semi-supervised image classification benchmarks on 10 MNIST and SVHN datasets. For all the datasets, use RealNVP (Dinh et al., (0166) architecture. Exact implementation details are listed in the appendix E The supervised model is trained using the same loss (equation 4), where all the data points are labeled (n., = 0).
We present the results for FlowGMM and FlowGMM-conss in Table 2 We also report results from DIGLM (Nalisnick et al.) 2019) (which only report semi-supervised performance on MNIST and supervised performance on MNIST and SVHN) and the M1+M2 VAE model (Kingma et al. 2014). FlowGMM outperforms M1+M2 model and performs better or on par with DIGLM. Furthermore, FlowGMM-cons improves over FlowGMM on all three datasets, suggesting that consistency regu- larization very beneficial for the proposed model when useful perturbations are available.
Following Oliver et al. (2018), we evaluate FlowGMM-cons varying the number of labeled data Tollow the setup of Kingma et al. (2014) and train GMM-onss on MNIST with 100, 600, 1000 and 3000 labeled data points. We present the results in Table 3 FlowGMM-cons outperforms the M1+M2 model of Kingma et (2014) in all the considered set- tings.
points. Specifically,
We note that the results presented this section are not directly comparable with the state-of-the- art methods using GANs or consistency regularization (see e.g. Laine & Aila 2016 Dai et al.) 2017 Athiwaratkun et al. 2019 Berthelot et al. 2019), as the architecture we employ is much less powerful for classification than the ConvNet and ResNet architectures that have been designed for classification without the constraint of invertibility. We believe that invertible architectures with better inductive biases for classification (possibly like iResNet (Behrmann et al.) 2018)) may help bridge this gap. The space images is challenging to model and while there has been substantial progress using normalizing flows, there is still much ground to cover.

---

Under review as a conference paper at ICLR 2020
Table 3: Semi-supervised classification accuracy for FlowGMM-cons and VAE MI M2 model Kingma et al. 2014) on MNIST for different number of labeled data points Tnp-
Method
my
100
Ty = 600
14
1000
Th = 3000
M1+M2 SSE (ny labels)
96.67
97.41 + 0.05
97.60 + 0.02
97.82 0.04
FlowGMM-cons (ny labels)
98.2
98.7
99
99.2
Table 4: Negative log- log-likelihood and Expected Calibration Error for supervised FlowGMM trained on MNIST (1k train, 1k validation, 10k test) and CIFAR-I0 (50k train, 1K validation, 9k test). FlowGMM-temp stands for tempered FlowGMM where single scalar parameter o² was learned on validation set for variances in all components.
MNIST (test acc 97.3%)
CIFAR-10 (test acc 89.3%)
FlowGMM
FlowGMM-tene
FlowGMM
FlowGMM-tone
NLL + ECE $
0.295 0.024
0.094
2.98 0.108
0.444
0.004
0,038
6 MODEL ANALYSIS
6.1 UNCERTAINTY AND CALIBRATION
In many machine learning applications, it is crucial to understand how confident a model is in its predictions. In classification problems, well-calibrated models are expected to output meaningful probabilities of belonging to a particular class. In Guo et al. (201)). it was observed that modern deep learning models are highly overconfident, however, simple temperature scaling can substantially improve model's calibration. In this section, we analyze the predictive uncertainties produced by the FlowGMM. In Appendix Section F we additionally study the ability of FlowGMM to detect out-of-domain data.
When using FlowGMM for classification, the class predictive probabilities are p(y = k|x) = N( / Ex=lur E..). Since we initialize Gaussian mixture means randomly from the standard normal distribution and do not train them along with the flow parameters (see sec- tion Bb. FlowGMM predictions become inherently overconfident due to the curse of dimensional- ity. Indeed, consider two Gaussians with means sampled independently from the standard normal f1. (O, I) in D-dimensional space. If 81 N(nn 7) is sample from the first Gaus- sian, then its expected squared distances both mixture means are E [||s Mi
D and 3D (for detailed derivation see Appendix Section G In high dimensional spaces, such logits would lead to hard label assignment in FlowGMM (p(y) a:) 1 for exactly one class). In fact, in the experiments we observe that FlowGMM is overconfident and performs hard label assignment: predicted class probabilities are all close to either 1 or O.
||
[||s - 1
We address this problem by learning single scalar parameter o² for all components in the Gaussian mixture (the component & will be N(u, o² f)) by minimizing the negative log likelihood on a vali- dation set. This way we can re-calibrate the variance of the latent GMM in a natural way. This proce- dure is also equivalent to applying temperature scaling (Guo al. 2017) to logits logA (x Σ&). We test FlowGMM calibration on MNIST and CIFAR datasets in the supervised setting. On MNIST we restricted the training set size to 1000 objects, since on the full dataset the model makes too few mistakes which makes evaluating calibration harder. In Table 4 we report negative log likelihood and expected calibration error (ECE, see Guo et al.) (017) for a description of this metric), We can see that re-calibrating variances of the Gaussians in the mixture significantly improves both metrics and mitigates overconfidence. The effectiveness of this simple rescaling procedure suggests that the latent space distances learned by the flow model correlated with the probabilities of belonging to particular class: the closer a datapoint is to the mean a Gaussian in the latent space, the more likely it belongs to the corresponding class.
7

---

Under review as a conference paper at ICLR 2020
0
Ag
N.No,
2
(a)
(b)
(C)
(d)
Figure 3: Visualizations of the latent space representations learned by supervised FlowGMM MNIST. (a): Latent space interpolations between test images from the same class and (b): from different classes. Observe that interpolations between objects from different classes pass through low-density regions. (c): Histogram of distances from unlabeled data to the decision boundary for FlowGMMM trained on 14 labeled and 59k unlabeled data and FlowGMM Sup trained on 1k labeled data only. FlowGMM-cons is able to push the decision boundary away from the data distribution using unlabeled data. (d): Feature visualization for CIFAR10: four test reconstructions are shown an intermediate feature is perturbed. The value of the perturbation a is shown in red vs the distribution the channel activations. Observe that the channel visualized activates on zeroed out pixels to the left of the image mimicking the random translations applied to the training data.
6.2 LEARNED LATENT REPRESENTATIONS
We next analyze the latent representation space learned by FlowGMM. We examine latent interpo- lations between members of the same class in Figure 3 (a) and between different classes in Figure 3 (b) for our MNIST FlowGMM-cons model trained with ny 1k labels. As expected, inter-class interpolations pass through regions of low-density, leading to low quality samples but intra-class interpolations do not. These observations suggest that, as expected, the model learns to put the decision boundary in the low-density region of the data space.
In Appendix section H we present images corresponding to the means of the Gaussian mixture and llss-connitional samples from FlowGMM.
Distance to Decision Boundary To explicitly test this conclusion, compute the distribution of distances from unlabeled data to the decision boundary for FlowGMM-ccon and FlowGMM Sup trained labeled data only. In order to compute this distance exactly for image I, find the two closest means n², μe" to the corresponding latent variable = f(x), and evaluate the
expression d(x) 1| -(x)) 1n" --===())] We visualize the distributions of the distances for 21
the supervised and semi semi-supervised method in Figure 3 (c). While most of the unlabeled data are far from the decision boundary for both methods, the supervised method puts a substantially larger fraction of data close to the decision boundary. For example, the distance to the decision boundary smaller than for 1089 unlabeled data points with supervised model, but only 143 data points with FlowGMM-com This increased separation suggests that FlowGMM-cons indeed pushes the decision boundary away from the data distribution in agreement with the clustering principle.
6.3 FEATURE VISUALIZATION
Feature visualization has become an important tool for increasing the interpretability of neural net- works. The majority of methods rely on maximizing the activations of given neuron, channel, or layer over a parametrization of an input image with different kinds of image regularization (Szegedy) al.) 2013) Olah et al.) 2017]] Mahendran & Vedaldi 2015). These methods, while effective, require optimization and regularization hyper-parameters and iterative optimization too costly for real time interactive exploration.
Since our classification model uses a flow which a sequence of invertible transformations f(z) = f.) fL fi-1 o... fi (xr), intermediate activations can be inverted directly. This means that we can combine the methods of feature inversion and feature maximization directly by feeding in a set of input images, modifying intermediate activations arbitrarily, and inverting the representation.
8

---

Under review as a conference paper at ICLR 2020
Given set of activations in the t layer aele, (,1, fe(())as with channels and spatial extent j, we may perturb a single neuron with
r(a) f.e (f:e(z)) + ag,5c),
where 0, is hot vector at channel c; and o. the standard deviation of the activations in channel c over the the training set and spatial locations. This can be performed at real time rates to explore the activation parametrized by a and the location eij without any optimization or aapee--rraeters.. The feature visualization of intermediate layers on CIFARIO test images are shown in panel (d) of Figure 3 The given channel being visualized appears to activate on the zeroed pixels from random translations well as the green channel, giving us insight into the workings of the model.
7 DISCUSSION
We proposed FlowGMM, natural and interpretable model for semi-supervised learning with nor- malizing flows. FlowGMM outperforms graph-based and consistency-be baselines on tabular data including semi-supervised text classification with BERT embeddings. On image classification. FlowGMM is not yet competitive with the state-of-the-art approaches (Athiwaratkun et al. 2019 Verma et al. 2019). but expect that the results could be greatly improved by using a better flow architecture that has stronger inductive biases for classification.
Flow is greatly extensible and can be easily adapted different settings. For example, con- sider an extreme case of superisueeei learning where some classes may be completely unlabeled, and the model has to identify those extra classes and cluster them separately. An effective model that can operate in this setting could, for example, be used for automated scientific discovery: when applied to classify certain experimental data, it can identify new separate classes that were not sidered by experts. FlowGMM can be applied in this scenario: instead of the standard Gaussian mixture model (GMM) in the latent space we can use the Chinese Restaurant Process GMM (CRP- GMM), which can infer the number of mixture components from data automatically [Rasmussen] 2000
Further, we can easily adapt FlowGMM to few-shot learning setting where the model has to gener- alize classes not seen train time by only using a few examples of that new class. In particular, we can fix the parameters of the flow model and add new Gaussian to the mixture whenever we add new class. We can estimate the parameters of this Gaussian from the few available examples.
We view interpretability as another strong advantage of FlowGMM. The access to latent space repre- sentations and feature visualization technique discussed in Section as well as the ability to sample from the model can be used to get insights into the performance of the model in practical applica- tions.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson There are many consistent explanations of unlabeled data: Why you should average. In International Confer- ence on Learning Representations, 2019. URL htt.:t/ss///eeeeeeeeeeee///////e
EkcKBhA5Y7
Jens Behrmann, David Duvenaud, and preprint /rXiv:/8/ .00995, 2018.
Jacobsen. Invertible residual networks. arXiv
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: holistic approach to semi-supervised learning. arXiv preprint
10Xx::1905500249, 2019.
Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. arXiv preprint (.XX::9906.0273:3 2019.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires bad GAN. In Advances in Neural Information Processing Systems 30, pp. 6510-6520. 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components
estimation. arXiv preprint arXiv:1410.8514 2014.
Laurent Dinh, Jascha Dickssinn,,,,,, and Samy Bengio. Density estimation using Real NVP. arXiv preprint (ot:::8000008033 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. CoRR, abs/1706.04599, 2017. URL/hctp:://rrx...orr///bb wwwpg/g/abb///000/ 04599
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint
1rXiv:1412.6988. 2014.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes.
arXiv preprint
arXiv:1312.6114, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1 xx convolutions. In Advances in Neural Information Processing Systems, pp. 1022541022, 2018
Durk P Kingma, Shakir Mohamed, Danilo Jimenez. Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02222.20000 2016.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions patlern analysis and machine intelligence, 2018.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Eric Nalisnick, Akihiro Matsukawa, Yee whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arxiv: 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. arXiv preprint 1902 2019.
10

---

Under review as a conference paper at ICLR 2020
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 00.23915/distill000077 http:://disill.ppb/2/1 2tte://////eauuu--iisllllll visualization.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: (o01::6:05.055 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp.
2017.
Carl Edward Rasmussen. The infinite gaussian mixture model. In Advances in neural information processing systems, pp. 554-560, 2000.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques training GANs. In Advances neurul information processing systems, pp. 2234-2242, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con- sistency training for semi-supervised learning. arXiv preprint arXiv: 1903.0385,,, 2019.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. Variational autoencoder for semi-supervised text classification. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learn- ing with local and global consistency. In Advances in neural information processing systems, pp- 321-328, 2004.

---

Under review as a conference paper at ICLR 2020
A EXPECTATION MAXIMIZATION
As an alternative to direct optimization of the likelihood (equation 4 we consider Expectation- Maximization algorithm (EM). EM a popular approach for finding maximum likelihood estimates in mixture models. Suppose X = { is the observed dataset, ' = {4 [4=]=11 are corresponding unobserved latent variables (often denoting the component in mixture model) and 0 is a vector of model parameters. EM algorithm consists of the two alternating steps: on E-step, we compute posterior probabilities of latent variables for each data point g(t, [z;) = Ptt z; 8): and on M-step, we fix q and maximize the expected log likelihood of the data and latent variables with respect to 0: E,g log P(X, T + maxg The algorithm can easily adapted to the semi-supervised setting where a subset of data is labeled with 1 then, on E-step we have hard assignment the true mixture component (x, [t. = v/l for labeled data points.
EM algorithm is applicable in our setting which is fitting the transformed mixture of Gaussians. We can perform the exact E-step for unlabeled data in the model since
N(f(x)) 2ie) det
0f 8hr
p(x)(1,0
N(((o))le,E)) F4)
g(t|ar) = p(z)0)
E Lk= N( Mie)
det 81
V Ex)
which coincides with the E-step of EM algorithm on Gaussian mixture model. On M-step, the objective has the following form:
0fa dxp
05e d.y
log N(fo(=))lu Egg)
Σ
Bottaloroy .0) log ((fo(z))) Σ,,
Since the exact solution is not tractable due to complexity of the flow model, we perform a stochastic gradient step to optimize the expected log likelihood with respect flow parameters 0.
Note that unlike regular EM algorithm for mixture models, we have Gaussian mixture parameters {LU,, ) fixed in our experiments, and on M-step the update of 0 induces the change of 2; fo(2;) latent space representations.
Using EM algorithm for optimization in the semi-supervised setting on MNIST dataset with 1000 la- beled images, we obtain 98. 3,.9%% accuracy which is comparable to the result for FlowGMM with reg- ular SGD training. However, our experiments, we observed that on E-step. hard label assignment happens for unlabeled points (g(1)x) P for one of the classes) because of the high dimensionality of the problem (see section 6.1] which affects the M-step objective and hinders training.
B LATENT DISTRIBUTION MEAN AND COVARIANCE CHOICES
Initialization In our experiments, we draw the mean vectors μ4 of Gaussian mixture model ran- domly from the standard normal distribution μi ~ NN00 I), and set the covariance matrices to identity Σ, = for all classes; we fixed GMM parameters throughout training. However, one could potentially benefit from data-dependent placing of means in the latent space. We experimented with different initialization methods, particular, initializing means using the mean point of latent rep-
resentations of labeled data in each class: μ4: (1/nz) Eni f(*))) where z represents labeled data points from class i and nf is the total number of labeled points in that class. In addition, can scale all means by a scalar value A = T44 to increase or decrease distances between them. We observed that such initialization leads much faster convergence of FMMWGMM on semi-supervised classification on MNIST dataset, however, the final performance of the model was worse compared to the one with random mean placing. We hypothesize that it becomes easier for the flow model to warm up faster with data-dependent initialization because Gaussian means are closer to the initial latent representations, but afterwards the model gets stuck in a suboptimal solution.
GMM training FlowGMM would become even more flexible and expressive if could learn Gaussian mixture parameters in principled way. In the current setup where means are sampled from the standard normal distribution, the distances between mixture components are about v2D where D is the dimensionality of the data (see Appendix G). Thus, classes are quite far apart from each
12

---

Under review as a conference paper at ICLR 2020
X, Data
2, Latent
2, Latent
X, Data
Two Circles
8 Gaussians
Pinwheel
(a)
(b)
(c)
(d)
Figure 4: Illustration of FlowGMM synthetic datasets: two circles (top row), eight Gaussians (middle row) and pinwheel (bottom row). (a): Data distribution and classification decision bound- aries. Unlabeled data is shown with blue circles and labeled data is shown with colored triangles, where color represents the class. Background color visualizes the classification decision boundaries of FlowGMM. (b): Mapping of the data the latent space. (c): Gaussian mixture in the latent space. (d): Samples from the learned generative model corresponding to different classes, as shown by their color.
other in the latent space, which, as observed in Section 6.1 leads to model miscalibration. Training GMM parameters can further increase interpretability of the learned latent space representations: we can imagine a scenario in which some of the classes are very similar or even intersecting, and would be useful represent it in the latent space. We could train GMM by directly optimizing likelihood (equation D. or using expectation maximization (see Section A). either jointly with the flow parameters or iteratively switching between training flow parameters with the fixed GMM and training GMM with the fixed flow. In our initial experiments
semi-supervised classification on MNIST, training GMM jointly with the flow parameters did not improve performance lead to substantial change of the latent representations. Further improvements require careful hyper- parameter choice which we leave for future work.
C SYNTHETIC EXPERIMENTS
In Figure we visualize the classification decision boundaries of FlowGMM as well as the learned mapping to the latent space and generated samples for three different synthetic datasets.
D TABULAR DATA PREPARATION AND HYPERPARAMETERS
The AG-News and Yahoo Answers were constructed by applying BERT embeddings to the text input, yielding 768 dimensional vector for each data point. AG-News has classes while Ya-
13

---

Under review as a conference paper at ICLR 2020
Table
Method Learning Rate
AG-News
Yahoo Answers
Hepmass
Miniboone
3-Layer NN Dropout
3e-4
3e-4
3e-4
3e-4
II-model
le-3
le-4
3e-3
le-4
FlowGMM
le-4
le-4
3e-3
3e-4
KNN
k
l: = 18
l: =
k 3
hoo Answers has 10. The UCI datasets Hepmass and Miniboone were constructed using the data preprocessing from Paaaaaaaaiios et al. (2017), but with the inclusion of the removed background process class so that the two problems can be used for binary classification. We then subsample the fraction of background class examples so that the dataset is balanced. For each of the datasets, separate validation set of size 5k was used to tune hyperparameters. All neural network models use the ADAM optimizer (Kingma & Ba 2014).
We tested both using both L2 distance and L2 with inputs normalized to unit norm, (sin" distance), and the latter performed the best. The value & chosen in the method was found sweeping over 1 20, and the optimal values for each of the datasets are shown in
k-Nearest Neighbors:
3 Layer NN + Dropout: The 3-Layer NN Dropout baseline network has three fully connected hidden layers with inner dimension & 512, ReLU nonlinearities, and dropout with = 0.3. We use the learning rate 3 for training the supervised baseline across all datasets.
II-Model:
The II-Model uses the same network architecture, and dropout for the perturbations. The additional consistency loss per unlabeled data point is computed as Tuitthh = llg(=") g 1l1 where g is are the output probabilities after the softmax layer of the neural network and the con- sistency weight = 30 which worked the best across the datasets. The model was trained for 50 epochs with labeled and unlabeled batch size ny for AG-News and Yahoo Answers, and labeled and unlabeled batch sizes ny and 2000 for Hepmass and Miniboone.
Label Spreading: We use the local and global consistency method from Zhou et al. (2004), Y (I -aS) 'Y where in our case Y the matrix of labels for the labeled, unlabeled, and test data but filled with zeros for unlabeled and test. D= D 1/2 computed from the affinity matrix (wr
D-1/2
Wi = exp (-y sin (z., 2f)) where sin? (x.,, == Tee ||| This is equivalent to 1.2 distance on the inputs normalized to unit magnitude. Because the algorithm scales poorly with number of unlabeled points for dense affinity matrices, O(n)), we we subsampled the number unlabeled data points to 10k and test data points to 5k for this graph method. However, we also evaluate the label spreading algorithm with a sparse kNN affinity matrix on using a larger subset 20k of unlabeled data. The two hyperparameters for label spreading (y/k and a) were tuned by separate grid search for each of the datasets. In both cases, we use the inductive variant of the algorithm where the test data is not included in the unlabeled data.
FMowGMM: We train our FlowGMM model with a RealNVP normalizing flow, similar to the ar- chitectures used in PPapamakarios al. (2017). Specifically, the model uses coupling layers, with 1 hidden layer each and 256 hidden units for the UCI datasets but 512 for text classification. UCI models were trained for 50 epochs of unlabeled data and the text datasets of unlabeled data. The labeled and unlabeled batch sizes are the same as in the II-Model.
trained for 30 epochs
The tuned learning rates for each of the models that we used for these experiments are shown in table 5
E IMAGE DATA PREPARATION AND HYPERPARAMETERS
We use the CIFAR-10 10 multi-scale architecture with 2 scales, each containing 3 coupling layers defined by 8 residual blocks with 64 feature maps. We use Adam optimizer (Kingma & Ba 2014) with learning rate 10 for CIFAR-10 and SVHN and 10-4 for MNIST. We train the supervised model for 100 epochs, and semi-supervised models for 1000 passes through the labeled data for CIFAR- 10 and SVHN and 3000 passes for MNIST. We use batch size of 64 and sample 32 labeled and 32 unlabeled data points in each mini-batch For the consistency loss term (equation 7 we linearly increase the weight from ( to for the first 100 epochs following Athiwaratkun al. 2019
14

---

Under review as a conference paper at ICLR 2020
For FlowGMM and FlowGMM we weight the loss labeled data by À 3 (value tuned on validation [Kiggma et al., 2014) on (1). as otherwise, we observed that the method underfits the labeled data.
F OUT-OF-DOMAIN DATA DETECTION
Density models have held promise for being able detect oft-of-domain data, an especially impor- tant task for robust machine learning systems (Nalisnick et al. 2019). Recently, it has been shown that existing flow and autoregressive density models are not as apt at this task as previously thought, yielding high likelihood on images coming from other (simpler) distributions. The conclusion put forward that datasets like SVHN are encompassed by, or have roughly the same mean but lower variance than, more complex datasets like CIFARIO (Nalisnick et al. 2018). We examine this hy- pothesis in the context of our flow model which has a multi-modal latent space distribution unlike methods considered in Nalisnick al. (2018).
0.0020
0.033
MNIST
0.0015
MNIST
7
2
/
Fashion MINIST
FashionMiNSS
0
4
1
0.011
4
9
-
0.00
0.0000
1300
1003
2000
3000
4000
1020
10000
2000
3000
6000
log p(X)MIIST
log FXsshosttMMNI
Figure 5: Left: Log likelihoods on in- and out-of-domain data for our model trained on MNIST. Center: Log likelihoods on in- and out-of-domain data for our model trained on MNShi Right: MNIST digits get mapped onto the sandal mode of the FashionMNIST model 75% of the time, often being assigned higher likelihood than elements of the original sandal class. Representa- tive elements are shown above.
Using a fully supervised model trained on MNIST, we evaluate the log likelihood for data points coming from the NotMNIST dataset, consisting of letters instead of digits, and the FashionM- NIST dataset. We then train supervised model on the more complex dataset FashionMNIST and evaluate on MNIST and NotMNIST. The distribution of the log likelihood l-g
log logpz(f())) + log det (/4) on these datasets is shown in Figure For the model trained on MNIST we see that the data from Fashion MNIST and NotMNIST is assigned lower likelihood, as expected. However, the model trained on FashionMNIST predicts higher likelihoods for MNIST images. The majority (= 75%) of the MNIST data points get mapped into the mode of the Fashion- MNIST model corresponding to sandals, which is the class with the largest fraction of pixels that are zero. Similarly, for the model trained MNIST the image of all zeros has very high likelihood and gets mapped to the mode corresponding to the digit which has the largest fraction of empty space.
G EXPECTED DISTANCES BETWEEN GAUSSIAN SAMPLES
Consider two Gaussians with means sampled independently from the standard normal μr, 122 (0, /) in D-dimensional space. If 81 W(M, 7) is sample from the first Gaussian, then expected squared distances both mixture means are:
E [|s]
41 11
-E [ [||s] p1ll
[a]]
Σ E [(s1, (
D
D
=
E
Σ
[B[s 2n + +
=
E
Σ (1 + ト - -1Â.))
=
15

---

Under review as a conference paper at ICLR 2020
E [|ss|
pall?|
E [ [||s 12ll?|u1, =E Σ
[s11, - p2, p1, u2,1]
D
E
Σ
(1 pI. 2/,, + 12,5)) = 33
For high-dimensional Gaussians the random variables |ss fall and ||sı uell will be concentrated around their expectations. Since the function exp(-x) decreases rapidly zero for positive I, the probability of 81 belonging the first Gaussian exp(-1|si (ex(--))(exp(( + exp(-3D)) 1/(1 exp(-2D)) saturates at with the growth of dimensionality D.
M1 / (exp(-Ilsı (all²)) exp(-llsı x2)l²))
H FLOWGMM AS GENERATIVE MODEL
0
2
G
7
8
P
4
(b)
Figure 6: Visualizations of the latent space representations learned by supervised FlowGMM on MNIST, (a): Images corresponding to means of the Gaussians corresponding to different classes. (b): Class-conditional samples from the model at reduced temperature T 0.25.
In Figure 6alee we show the images (1,,) corresponding to the means of the Gaussians representing each class. We see that the flow correctly learns to map the means to samples from the corresponding Masses.Next, in Figure (b show coass-oonditinall samples from the model. To produce a sample from class i, we first generate trade-off between sample quality and diversity; we then compute the samples as f-1 We set T' 0.25² to produce samples in Figure 6b) As can see, FlowGMM can produce reasonable class-conditional samples simultancously with achieving a high classífication accuracy (99.63%) on the MNIST dataset.
N(,, T7), where T is a temperature parameter that controls
16


Reviews:
[{"review_id": "r1xTxK8CYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a semi-supervised learning model, named as Flow Gaussian Mixture Model (FlowGMM). The model is learnt by maximizing the join likelihood of the labeled and unlabeled data with a consistency regularization.\nThe authors demonstrate that the proposed model outperforms others on text classification; for image classification, the performance can be improved in future.\nAlso the authors demonstrate that the model is interpretable via feature visualizations.\nOverall the paper is fine written.\nYet, The conclusion is not fairly supported and the paper could be much stronger with the issues discussed already but I don’t think its current form is ready yet.\n\nBelow are more detailed comments:\n0) It would be nice to add the definition of the performance metric; without the definitions, none of the numbers in the tables would make sense. \n1) The main result for text classification in Table 1 is reporting the best of 3 runs, which can’t support the conclusion that the proposed method outperforms the other. In general, it’s nice to provide statistical significance comparing two models or reporting the mean and std across multiple runs. \n2) In Table 2, it’s not clear what conclusion could be drawn by comparing the performance of supervised and semi-supervised performance. Are the testing data points the same?\n3) The feature visualization as discussed in Section 6.3 is not explained clear. Specifically, “giving us insight into the workings of the model” is not clear; what exactly insight can we get and what exactly are the workings can we get?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlkOR-cFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes how to use normalising flows for Semi Supervised Learning (SSL). Briefly, the method consists in finding a (bijective) map for transforming a mixture of Gaussians into a density approximating the empirical data-distribution -- as usual for flow methods, the parameters are found through likelihood maximisation. This is a elegant approach that naturally exploits the standard so-called cluster assumption in SSL. The papers also shows how to incorporate a consistency-based regularisation within the method.\n\n\nAlthough it is an elegant and simple approach, and the article is relatively well written, I think that the paper should be rejected because (1) on image classification tasks (and even with consistency regularisation), the performances are well-below the straightforward-to-implement \\Pi-model. (2) for tabular/NLP data, although the performances seems to be good, the comparison with standard methods could have been much better done -- I am still not convinced by the method. \n\nI agree with the authors that there are many situations where it is not possible to find good perturbation (eg. NLP / tabular / genomics / etc...). If the authors could demonstrate more carefully that their approach does lead to state-of-the-art performances in this type of situations, I do believe that the approach would be of great interest. Given that the methods does not work well at all for image classification, I think that he authors should have been much more careful with the comparisons with the standard methods when investigating the performances on NLP/tabular tasks.\n\n\n(1) basic k-NN benchmark?\n(2) basic dimension reduction (PCA / autoencoder / extract lower representation from a NN) associated with either k-NN or label-propagation?\n(3) it is *not* difficult at all to implement label propagation with fast nearest-neighbours (eg. FAISS library) and sparse linear algebra on the full datasets. In the current submission, it has not been done for the NLP datasets.\n(4) There are indeed several ways to compute distance / affinity within label-propagation-type approaches\n(5) Brief description of parameter tuning for label-prop should be added\n\nI think that the method has a lot of potential and the fact that it is not competitive for computer vision task is not important. I encourage the authors to carry out more convincing numerical comparisons ing tabular/NLP/etc.. settings in order to strengthen the message of the paper. If convincing results can be obtained, I believe that the method has a lot of potential.\n\n[Edit after rebuttal]\nI would like to thank the authors to have provided additional label propagation experiments and details -- the proposed method appears to be quite much better than this baseline approach, which is very reassuring and proves that it is worth exploring further this line of work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygksbcRtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a normalising flow with the prior distribution represented by a Gaussian mixture model (GMM). The method, FlowGMM, maps each class of the dataset to a Gaussian distribution in the latent space by optimising the joint likelihood of both labelled and unlabelled data, thus making the method useful for semi-supervised problems. Predictions are made using the maximum a posteriori estimate of the class label. To make the method robust to small perturbations of the inputs, the authors introduce a novel consistency regularisation term to the total loss function, which maximises the likelihood of predicting the same class after a perturbation.\nThe authors further examine the learnt latent space by considering two simple, synthetic datasets that can be easily visualised, showing that the latent space behaves in a way one would intuitively expect.\nThe method is evaluated on both tabular and image data, showing promising results in terms of accuracy (presumably, see below). As the model is found to be overconfident in its predictions, the authors introduce a calibration scheme and empirically verify that it improves the uncertainty estimates. Lastly, the authors introduce a feature visualisation scheme and use it to illustrate the effect of perturbing the activations of the invertible transformations.\n\nI generally like the proposed method, which seems useful and intuitive. I am particularly happy with the discussion on uncertainty calibration, where the authors suggest an elegant addition to the model to increase the variance of the mixture components. I do, however, have significant concerns about the novelty of the paper as well as its structure and clarity, as detailed below. I do, therefore, not recommend it for acceptance.\n\nThe paper reads well, although I feel that it lacks some details and explanations. For example, in table 2, it is never mentioned what \"FlowGMM Sup\" refers to and if it is different from \"FlowGMM Supervised\". It is also not clear what \"(All labels)\" refers to - does it mean that labels were provided for the entire dataset or that the models were trained only on the small subset with labels? Or something else? Which performance metric is used in the tables? The accuracy, presumably, but this is never specifically stated. Similarly, the number of datapoints and ratio of labelled to unlabelled data for the synthetic datasets are not reported. They are not crucial to know but should be included for completeness.\n\nWhile the first half of the paper is informative and well-structured, the second half appears a bit less so. From experimentally verifying that the method works, the paper goes on to discuss uncertainty calibration, examine the latent space representations, and visualising the effect of feature perturbations. While I greatly appreciate the focus on interpreting the trained model, I think it appears somewhat chaotic, as if the authors tried to squeeze too much into the paper. For example, the feature visualisation technique is quite neat, but it works for any flow and is not really used for anything in the paper. I would suggest saving it for a dedicated paper.\n\nI am not convinced by the novelty of this paper. The authors list two contributions: 1) the model itself, 2) an empirical analysis with much focus on the interpretability of the model. While the model is, to my knowledge, indeed novel, the analysis is quite standard, and the interpretability even appears to be oversold. GMMs are nice and intuitive, but not novel in any way, yet the authors seem to be describing the properties of GMMs as specific to their method.\nIn particular, the authors go to great lengths to show that the latent space representations cluster around the means of the mixture components and that the decision boundary lies in low-density regions of the latent space. I do not see why these properties should be so surprising since the method directly optimises the likelihood of the data under the mixture distribution. That this is also empirically observed is, of course, reassuring, but these observations are better suited for the appendix, in particular given that the paper went over the recommended page limit.\nI think that much of the claimed second contribution follows directly from the GMM aspect of the model. Instead of claiming the standard GMM properties as contributions, I think the proposed consistency loss term should be highlighted as a contribution on its own. I find it elegant, and I guess it would be particularly useful for NLP tasks where sentences can be phrased in different ways but still mean the same.\n\nInstead of discussing the latent space, I would have preferred to see extra evaluations of the method, like convergence rates of both FlowGMM and FlowGMM-cons compared to the competing models. Furthermore, a major limitation of the model is that knowledge of the correct number of classes in the data - even in the unsupervised setting. The authors hint at extensions to mitigate this in the discussion (using a Chinese Restaurant Process GMM or by adding extra Gaussians to the mixture during training), but these should have been investigated in the current paper.\n\nIn conclusion, I think that the paper lacks novelty and that it spends far too much space on \"trivial\" properties of the model instead of addressing shortcomings, like the prior specification of the number of classes, which the authors even point out in the discussion.\n\nMinor comments:\n- p 4, bottom: \"each with 1 hidden layers\" -> \"each with 1 hidden layer\"\n- p 5, middle: \"FlowGMM is able to leveraged\" -> \"FlowGMM is able to leverage\"\n- p 5, bottom: \"Table 5.1\" -> \"Table 1\"\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a novel method for semi-supervised learning using GMMs.  Unfortunately the novelty of the contribution is unclear, and the majority of the reviewers find the paper is not acceptable in present form.  The AC concurs.
Decision: Reject