Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper describes a normalising flow with the prior distribution represented by a Gaussian mixture model (GMM). The method, FlowGMM, maps each class of the dataset to a Gaussian distribution in the latent space by optimising the joint likelihood of both labelled and unlabelled data, thus making the method useful for semi-supervised problems. Predictions are made using the maximum a posteriori estimate of the class label. To make the method robust to small perturbations of the inputs, the authors introduce a novel consistency regularisation term to the total loss function, which maximises the likelihood of predicting the same class after a perturbation.
The authors further examine the learnt latent space by considering two simple, synthetic datasets that can be easily visualised, showing that the latent space behaves in a way one would intuitively expect.
The method is evaluated on both tabular and image data, showing promising results in terms of accuracy (presumably, see below). As the model is found to be overconfident in its predictions, the authors introduce a calibration scheme and empirically verify that it improves the uncertainty estimates. Lastly, the authors introduce a feature visualisation scheme and use it to illustrate the effect of perturbing the activations of the invertible transformations.

I generally like the proposed method, which seems useful and intuitive. I am particularly happy with the discussion on uncertainty calibration, where the authors suggest an elegant addition to the model to increase the variance of the mixture components. I do, however, have significant concerns about the novelty of the paper as well as its structure and clarity, as detailed below. I do, therefore, not recommend it for acceptance.

The paper reads well, although I feel that it lacks some details and explanations. For example, in table 2, it is never mentioned what "FlowGMM Sup" refers to and if it is different from "FlowGMM Supervised". It is also not clear what "(All labels)" refers to - does it mean that labels were provided for the entire dataset or that the models were trained only on the small subset with labels? Or something else? Which performance metric is used in the tables? The accuracy, presumably, but this is never specifically stated. Similarly, the number of datapoints and ratio of labelled to unlabelled data for the synthetic datasets are not reported. They are not crucial to know but should be included for completeness.

While the first half of the paper is informative and well-structured, the second half appears a bit less so. From experimentally verifying that the method works, the paper goes on to discuss uncertainty calibration, examine the latent space representations, and visualising the effect of feature perturbations. While I greatly appreciate the focus on interpreting the trained model, I think it appears somewhat chaotic, as if the authors tried to squeeze too much into the paper. For example, the feature visualisation technique is quite neat, but it works for any flow and is not really used for anything in the paper. I would suggest saving it for a dedicated paper.

I am not convinced by the novelty of this paper. The authors list two contributions: 1) the model itself, 2) an empirical analysis with much focus on the interpretability of the model. While the model is, to my knowledge, indeed novel, the analysis is quite standard, and the interpretability even appears to be oversold. GMMs are nice and intuitive, but not novel in any way, yet the authors seem to be describing the properties of GMMs as specific to their method.
In particular, the authors go to great lengths to show that the latent space representations cluster around the means of the mixture components and that the decision boundary lies in low-density regions of the latent space. I do not see why these properties should be so surprising since the method directly optimises the likelihood of the data under the mixture distribution. That this is also empirically observed is, of course, reassuring, but these observations are better suited for the appendix, in particular given that the paper went over the recommended page limit.
I think that much of the claimed second contribution follows directly from the GMM aspect of the model. Instead of claiming the standard GMM properties as contributions, I think the proposed consistency loss term should be highlighted as a contribution on its own. I find it elegant, and I guess it would be particularly useful for NLP tasks where sentences can be phrased in different ways but still mean the same.

Instead of discussing the latent space, I would have preferred to see extra evaluations of the method, like convergence rates of both FlowGMM and FlowGMM-cons compared to the competing models. Furthermore, a major limitation of the model is that knowledge of the correct number of classes in the data - even in the unsupervised setting. The authors hint at extensions to mitigate this in the discussion (using a Chinese Restaurant Process GMM or by adding extra Gaussians to the mixture during training), but these should have been investigated in the current paper.

In conclusion, I think that the paper lacks novelty and that it spends far too much space on "trivial" properties of the model instead of addressing shortcomings, like the prior specification of the number of classes, which the authors even point out in the discussion.

Minor comments:
- p 4, bottom: "each with 1 hidden layers" -> "each with 1 hidden layer"
- p 5, middle: "FlowGMM is able to leveraged" -> "FlowGMM is able to leverage"
- p 5, bottom: "Table 5.1" -> "Table 1"


review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors propose a semi-supervised learning model, named as Flow Gaussian Mixture Model (FlowGMM). The model is learnt by maximizing the join likelihood of the labeled and unlabeled data with a consistency regularization.
The authors demonstrate that the proposed model outperforms others on text classification; for image classification, the performance can be improved in future.
Also the authors demonstrate that the model is interpretable via feature visualizations.
Overall the paper is fine written.
Yet, The conclusion is not fairly supported and the paper could be much stronger with the issues discussed already but I don’t think its current form is ready yet.

Below are more detailed comments:
0) It would be nice to add the definition of the performance metric; without the definitions, none of the numbers in the tables would make sense. 
1) The main result for text classification in Table 1 is reporting the best of 3 runs, which can’t support the conclusion that the proposed method outperforms the other. In general, it’s nice to provide statistical significance comparing two models or reporting the mean and std across multiple runs. 
2) In Table 2, it’s not clear what conclusion could be drawn by comparing the performance of supervised and semi-supervised performance. Are the testing data points the same?
3) The feature visualization as discussed in Section 6.3 is not explained clear. Specifically, “giving us insight into the workings of the model” is not clear; what exactly insight can we get and what exactly are the workings can we get?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper describes how to use normalising flows for Semi Supervised Learning (SSL). Briefly, the method consists in finding a (bijective) map for transforming a mixture of Gaussians into a density approximating the empirical data-distribution -- as usual for flow methods, the parameters are found through likelihood maximisation. This is a elegant approach that naturally exploits the standard so-called cluster assumption in SSL. The papers also shows how to incorporate a consistency-based regularisation within the method.


Although it is an elegant and simple approach, and the article is relatively well written, I think that the paper should be rejected because (1) on image classification tasks (and even with consistency regularisation), the performances are well-below the straightforward-to-implement \Pi-model. (2) for tabular/NLP data, although the performances seems to be good, the comparison with standard methods could have been much better done -- I am still not convinced by the method. 

I agree with the authors that there are many situations where it is not possible to find good perturbation (eg. NLP / tabular / genomics / etc...). If the authors could demonstrate more carefully that their approach does lead to state-of-the-art performances in this type of situations, I do believe that the approach would be of great interest. Given that the methods does not work well at all for image classification, I think that he authors should have been much more careful with the comparisons with the standard methods when investigating the performances on NLP/tabular tasks.


(1) basic k-NN benchmark?
(2) basic dimension reduction (PCA / autoencoder / extract lower representation from a NN) associated with either k-NN or label-propagation?
(3) it is *not* difficult at all to implement label propagation with fast nearest-neighbours (eg. FAISS library) and sparse linear algebra on the full datasets. In the current submission, it has not been done for the NLP datasets.
(4) There are indeed several ways to compute distance / affinity within label-propagation-type approaches
(5) Brief description of parameter tuning for label-prop should be added

I think that the method has a lot of potential and the fact that it is not competitive for computer vision task is not important. I encourage the authors to carry out more convincing numerical comparisons ing tabular/NLP/etc.. settings in order to strengthen the message of the paper. If convincing results can be obtained, I believe that the method has a lot of potential.

[Edit after rebuttal]
I would like to thank the authors to have provided additional label propagation experiments and details -- the proposed method appears to be quite much better than this baseline approach, which is very reassuring and proves that it is worth exploring further this line of work.
review_title: Official Blind Review #1

Meta-review: This paper offers a novel method for semi-supervised learning using GMMs.  Unfortunately the novelty of the contribution is unclear, and the majority of the reviewers find the paper is not acceptable in present form.  The AC concurs.
Decision: Reject