Title: TOWARDS UNSUPERVISED CLASSIFICATION WITH DEEP GENERATIVE MODELS

Abstract: Deep generative models have advanced the state-of-the-art in semi-supervised classification, however their capacity for deriving useful discriminative features in a completely unsupervised fashion for classification in difficult real-world data sets, where adequate manifold separation is required has not been adequately explored. Most methods rely on defining a pipeline of deriving features via generative modeling and then applying clustering algorithms, separating the modeling and discriminative processes. We propose a deep hierarchical generative model which uses a mixture of discrete and continuous distributions to learn to effectively separate the different data manifolds and is trainable end-to-end. We show that by specifying the form of the discrete variable distribution we are imposing a specific structure on the model's latent representations. We test our model's discriminative performance on the task of chronic lymphocytic leukemia (CLL) diagnosis against baselines from the field of computational flow cytometry (FC), as well as the Variational Autoencoder literature.

1. INTRODUCTION:

Variational Autoencoders (VAEs) have recently shown remarkable performance in unsupervised generative modeling of high-dimensional data generated by complex distributions (Kingma & Welling, 2013; Rezende et al., 2014) , as well as semi-supervised classification where only a small subset of the data set is labeled (Kingma et al., 2014) . While the interaction between the generative and the classification capabilities of semi-supervised models has been recently explored in literature (Maaløe et al., 2016; 2017) , there has been little investigation of the discriminative capabilities of a purely unsupervised framework with most works focusing on the task of unsupervised clustering (Johnson et al., 2016; Xie et al., 2016; Jiang et al., 2017) . Furthermore, most of these works have been evaluated on mostly benchmark data sets which do not capture the difficulties that are often encountered on real-world data. For instance, there has been no investigation of the performance of these methods on data sets with significant class imbalance. The question that is, then, posed is whether deep generative models can be used effectively as unsupervised classifiers, which can, in essence, be cast into a question of what type of features and architectural choices are required to achieve good classification performance in an unsupervised manner.

To examine the aforementioned questions we propose a deep hierarchical generative model and evaluate its performance on a difficult real world data set. In principle, we train our model in a completely unsupervised fashion, however in our experiments we rely on labeled data to measure our model's performance using suitable metrics for the problem domain, as well as derive a stopping criterion for training. Our model outperforms established state-of-the-art baselines used in the field of the problem domain. Our contributions are summarized in the following:

• A framework which utilizes a hierarchy of continuous representations which conclude in a discrete variable explicitly representing categories, resulting in complex, expressive, invariant and interpretable representations (Bengio et al., 2013) , which are crucial in separating widely overlapping manifolds and achieve good classification results in significantly imbalanced data sets.

• Controllable representation structure through specification of the form of the aforementioned discrete variable which better suits the task at hand given a problem scenario.

2. TECHNICAL BACKGROUND:

In this section we review techniques and frameworks that provide a useful backdrop for the derivation of our own model.

2.1. DEEP GENERATIVE MODELS:

Variational Autoencoders (VAEs) as presented in Kingma & Welling (2013) use at most two layers of stochastic latent variables, however generalizations to multiple layers of latent variables have since been introduced (Rezende et al., 2014; Burda et al., 2015; Sønderby et al., 2016) .

Generation in a deep generative model is achieved by a top-down stochastic pass through the model defined as:

p θ (x) = p θ (z L )p θ (z L-1 |z L )...p θ (x|z 1 ) (1)

where L is the number of stochastic hidden layers and z L denotes the number of latent variables in the layer. As in a standard VAE the dependence of each layer on the previous one is considered to be nonlinear and is modeled by multi-layered perceptrons (MLPs). Similarly, inference is carried out by a bottom-up stochastic pass through the model's layers:

EQUATION

Optimization of a deep generative model is akin to that of a standard VAE. Namely, the reparameterization trick (Kingma & Welling, 2013) is being applied to each layer of stochastic latent variables.

2.2. CONTINUOUS RELAXATIONS OF DISCRETE RANDOM VARIABLES:

Until recently, models trained with the variational objective have been employing mainly Gaussian latent stochastic variables, optimizing indirectly through discrete variables wherever they were used (e.g. Kingma et al. (2014) integrate over the discrete variable). This was due to the inability of backpropagating through discrete variables because of their discontinuous nature. Jang et al. (2016) and Maddison et al. (2016) independently developed a continuous relaxation of discrete random variables. The resulting distribution was presented as the Gumbel-Softmax distribution and the Concrete distribution respectively but essentially has the same functional form. From this point on, for the sake of clarity we will adopt the latter name to refer to this distribution.

A simple way to sample from a discrete distribution is to employ the Gumbel-Max trick (Gumbel, 1954; Papandreou & Yuille, 2011; Maddison et al., 2014) . The Gumbel distribution produces samples according to -log(-log(u)) where u ∼ Uniform(0,1). Given parameters α 1 , ..., α k and samples g i from Gumbel(0,1), samples z from the Categorical distribution can be drawn according to:

z = arg max i (g i + log α i ) (3)

where z is represented as a one-hot vector. Samples from the Concrete distribution are produced by replacing the argmax operation with the softmax function:

y i = exp((log α i + g i )/τ ) k j=1 exp((log α j + g j )/τ ) for i = 1, ..., k

Crucially, the reparameterization trick can now be applied in a similar manner to Gaussian samples. The probability density function of the Concrete distribution is the following:

EQUATION

As the temperature parameter τ approaches 0, samples from the Concrete distribution more accurately approximate one-hot samples from a Categorical distribution and in the limit, the two distributions become identical. We refer the reader to Jang et al. (2016) ; Maddison et al. (2016) for more details.

3. METHOD:

In this section we present the problem setting and our proposed model. Additionally, we motivate our model design choices and modifications to the variational objective.

3.1. PROBLEM:

Chronic lymphocytic leukemia (CLL) is the most common form of leukemia in adults in Western countries (Hallek, 2013) . CLL is usually diagnosed during routine blood tests and Flow Cytometry (FC) is one of the examination procedures used for confirming the diagnosis (Hallek, 2013) . Flow cytometry (FC) is a powerful technique for single cell analysis, which is routinely used for the diagnosis of haematological malignancies (Brown & Wittwer, 2000) , however it is heavily dependent on the experience of the expert performing it, oftentimes resulting in serious discrepancies across experts.

During or after treatment of CLL, the term Minimal Residual Disease (MRD) is used to define the small amount of leukemic cells detected in patients (typically 10s to a few 100s of leukemic cells in samples of 500,000 cells or more). Particularly for CLL, the limit for MRD positive diagnosis is set at 1 leukemic cell per 10,000 white blood cells in blood or bone marrow (Böttcher et al., 2013) . The problem, from a manifold learning and generative modeling point of view is to adequately separate the two data manifolds of healthy and leukemic cells.

This problem is difficult not just because of the sheer size of the healthy cell population, which leads to significant manifold overlap, but also because there are other manifolds present in the data, e.g. representing different types of cell populations in the blood sample, with many different factors of variation, some of which are known (cell granulation, cell size, etc.) and some of which may be unknown. Most clustering algorithms, which are traditionally used in computational FC, particularly those that "infer" the number of clusters automatically, are unable to separate the manifolds of interest given a particular problem, because they act directly on the input/data space without making any assumptions about the latent structure present in a data set. As a result they are sensitive to noise and ultimately one has to resort to merging clusters. Furthermore, for the clustering results to be interpretable, significant amounts of hyperparameter tuning are necessary during the clustering proper, the cluster merging phase or both (e.g. number of nearest neighbors to be taken into account, appropriate distance metric, linkage, etc) resulting in an impractical, computationally expensive overall solution.

3.2. DEEP GENERATIVE MODELS AS FEATURE REPRESENTATION LEARNERS:

An alternative to clustering algorithms would be to learn a low-dimensional feature mapping as in traditional deterministic autoencoders and then perform clustering in feature space, optimizing the two tasks either separately or jointly (Xie et al., 2016) . While this is a viable strategy, possibly alleviating some of the problems discussed above, depending on the problem domain, the deterministic mapping could potentially lack expressiveness due to the low-dimensionality requirement, as it would need to compress information to avoid the curse of dimensionality. Furthermore, these methods are sensitive to noise and so, they overfit the data, being unable to disentangle the different factors of variation (Bengio et al., 2013) .

A practical and principled solution in this problem domain should be able to adequately separate the multiple manifolds underlying the data and model the signal of interest, remaining invariant to factors of variation. Furthermore, the candidate model should be able to learn feature representations that enforce a structure that facilitates the above and as a result be able to correctly classify each cell in an unsupervised fashion. Ideally it should be trained and optimized end-to-end without resorting to clustering algorithms that are separated from the feature representation learning process.

Latent variable models like VAEs and deep generative models seem like an ideal fit to the above description. They address the problem of generation of new samples with the help of a recognition/inference model which learns the global structure of the training data. In theory, by employing this encoder, such a model could learn representations that would prove useful in unsupervised discriminative tasks. In practice, however, this process is a lot more complicated. First of all, recognition models with inadequate expressiveness are known to encode only local information, because they model well only the regions of the posterior distribution that lie near the training examples, being unable to generalize to the full posterior manifold (Yeung et al., 2017) . When one uses powerful and expressive generative models as in Bowman et al. (2015) ; Serban et al. ( 2017); Fraccaro et al. (2016) , this is further exacerbated to the point where generation on the one hand and representation learning on the other, become competing tasks with the model preferring to encode mainly local information using the generative/decoding model p(x|z) and ignore the latent code (Chen et al., 2016) . From a representation learning standpoint which is what we're interested in for this particular problem, the generative model is now perceived as the regularizer and we need to make sure that the recognition model is expressive enough to be able to model our true posterior as closely as possible, as well as provide an interpretable way of making a diagnosis. As such we propose a framework that addresses both these needs jointly.

3.3. MODEL:

To address the issues discussed in 3.2 we introduce a deep generative model composed of L layers of continuous stochastic latent variables. Beyond alleviating the aforementioned issues, we also aim to model a set of latent factors affecting the cell measurements and eventually, the diagnosis. We introduce a hierarchy of continuous latent variables to express these latent factors. The diagnosis is itself represented as a single (relaxed) binary variable after this cascade of layers. For the sake of brevity we will refrain from explicitly stating the continuous relaxation on this discrete variable for the remainder of this text, except when necessary. To highlight the close relationship between the discrete diagnosis and the continuous latent factors and for practical reasons we denote the mixture of the discrete and continuous variables by h. The generative model assumes the following form:

EQUATION

with

EQUATION

where

z k ∼ N (µ θ (z k+1 ), Σ θ (z k+1 )) for k = 1...L -1, z L ∼ N (µ θ (y), Σ θ (y))

Parameters µ θ (•), Σ θ (•) are non-linear functions of the next layer in the hierarchy computed by feedforward neural networks and θ are the generative parameters. The prior on the discrete variable p(y) is set to be the discrete uniform probability. The observation model is defined as:

EQUATION

and describes the observations conditioned on the latent variables. The inference model is described below:

EQUATION

where with

EQUATION

z k ∼ N (µ φ (z k-1 ), Σ φ (z k-1 )) for k = 1...L, y ∼ Concrete(α φ (z L ), τ )

where similar to the generative model, parameters µ φ (z k-1 ), Σ φ (z k-1 ), α φ (z L ) are functions that describe non-linear relationships between stochastic layers and are computed by feedforward neural networks with φ being the variational parameters. The temperature parameter τ governs the degree of relaxation of the discrete variable and can either be constant, annealed during training or learned. In the case of the first stochastic layer z 1 , layer z 0 refers to the observed data x. The function α φ (z L ) models the relationship between the last Gaussian layer and the discrete variable y.

By conditioning on the discrete variable y, we are enforcing a tight coupling between the latent factors of variation (represented by z) and categorical features (represented by y). As a result the activity of the discrete variable y and continuous variables z 1 ...z L is highly correlated. More than simply capturing the modes of continuous variable z L , the discrete variable y imposes a particular structure that is propagated through all the continuous layers z 1 ...z L of the feature hierarchy. This allows us a degree of control with respect to the structure of the latent variables, which we can exploit depending on the task at hand. Figure 1 illustrates such an example. By increasing K we facilitate multiple manifold learning.

Both in the inference and the generative models, covariances assume a diagonal structure. Our model's overall architecture can be seen in Figure 2 .

Employing a relaxed discrete variable allows us to avoid marginalization over all its possible values. At the same time, however, we are using a continuous surrogate loss in place of the original discrete one (Schulman et al., 2015; Maddison et al., 2016) .

EQUATION

To induce more distributed and thus expressive latent codes we adopt deterministic warm-up per Sønderby et al. ( 2016), where we begin training our model with a deterministic autoencoder and then gradually introduce the KL term, to prevent the approximate posterior from collapsing onto the prior early in training and disconnecting latent units. Thus we introduce a λ term in expression 11, which we linearly anneal during training from 0 to 1: The gradient estimations of the lower bound will be biased with respect to the original discrete loss, but unbiased and low-variance with respect to the surrogate loss (Maddison et al., 2016) .

EQUATION

To obtain low variance updates to our parameters through stochastic gradient optimization of the bound, we use the reparameterization trick for both discrete and continuous variables concurrently:

EQUATION

where h is expressed as a deterministic function of the form presented in Kingma & Welling (2013) and section 2.2:

EQUATION

with ∼ N (0, I), g ∼ Gumbel(0, 1)

We note that q φ (h|x) is used to denote the Concrete density and the KL term is computed according to eq. 20 in (Maddison et al., 2016) . Finally, we also make 

4. EXPERIMENTS:

In our experiments we use two real world data sets of deidentified patient data, which correspond to flow cytometric measurements of two different blood samples1 We investigate our model's ability to learn useful representations via its performance as an "unsupervised classifier". Detailed explanations of the data sets can be found in Appendix A.

4.1. METRICS AND EARLY STOPPING CRITERION:

In our experiments we compare our model's predictive capacity with that of popular baselines from the field of computational FC, as well as generative models. Because in this particular problem one population (i.e. class) vastly outnumbers the other, one cannot rely on accuracy to correctly estimate the models' performance, as a model that would predict only the over-represented class would consistently get high accuracy scores, but in reality would have limited predictive power. Instead, we make use of confusion matrix-based metrics, which are popular in medical applications. More specifically, we use true positive rate (TPR -also called sensitivity or recall) and true negative rate (TNR -also called specificity).

Because we found that training the model is difficult, as far as retaining optimal representations is concerned throughout the training procedure, we use early stopping, where we check the model's performance on the test set with a fixed frequency. To derive a useful criterion we also turn to a confusion matrix-based quantity, Matthews correlation coefficient (MCC), introduced by Matthews (1975) . As a correlation coefficient, MCC takes values in [-1,1] with 1 suggesting that predictions and ground truth are in complete accordance, -1 that they are in complete discord and 0 that predictions are random. MCC is regarded as a good measure for estimating classifier performance on imbalanced data sets (Boughorbel et al., 2017) . Formally, it is defined as follows:

M CC = T P × T N -F P × F N (T P + F P )(T P + F N )(T N + F P )(T N + F N ) (15)

We check MCC in the test set during training in fixed intervals and save the model parameters once it has crossed a set threshold. In general, we noticed in our experiments that a threshold of 0.5 yields good results with very small variance in performance.

4.2. RESULTS:

In our experiments we investigate our model's predictive performance, i.e. its capacity to correctly classify cell measurements2 . Our model's predictions are based on hard assignment of the probabilities obtained by the predictive distribution q φ (y|z). I.e. given a minibatch of observations X = {x (i) } N i=1 , probabilities of samples drawn from q φ (y|z) are thresholded at 0.5. Probabilities that lie above this threshold are considered positive cell measurements.

As baselines we choose the 3 best-performing algorithms from Weber & Robinson (2016) who compare a wide set of clustering algorithm implementations used in computational FC in two different tasks -multiple population identification and rare population detection. We are interested in the latter task which is closer to our own. The first baseline, X-shift (Samusik et al., 2016) is based on k-Nearest Neighbors (kNN) density estimation. Local density maxima become centroids and the remaining points are connected to those centroids via ascending density paths. Rclusterpp (Linderman et al., 2013) is the second baseline and is an efficient implementation (with respect to memory requirements) of hierarchical clustering. The last baseline, flowMEANS (Aghaeepour et al., 2011) is based on k-means clustering but can identify concave cell populations using multiple clusters.

We argue that to successfully learn and separate the two manifolds of healthy and pathological cells, more expressive and distributed representations are necessary, with explicit steps taken during training to enforce these characteristics. Additionally, the proposed model must be a mixture of continuous and discrete distributions to appropriately represent different cell attributes and category respectively. To illustrate the merit in these points, we also include 2 methods from the VAE literature, a vanilla VAE (Kingma & Welling, 2013) , followed by a linear support vector machine (denoted by VAE+SVM in tables 1 and 2) and a version of our own model but within the β-VAE framework (Higgins et al., 2016) . The motivation for VAE+SVM is the fact that given adequate manifold separation, a linear classifier such as a support vector machine should, in principle, be able to correctly classify observations. The β-VAE framework introduces a hyperparameter β to the variational objective, similar to λ, which we are using for deterministic warm-up. For β > 1, the model is forced to learn more compressed and disentangled latent representations, which is what Higgins et al. (2016) argue for. Learning disentangled representations is certainly closely related to successfully separating multiple data manifolds and we consider a representation to be efficient and disentangled in this scenario if it leads to good predictive performance, implying adequate manifold separation. In short we are treating "unsupervised classification" as a proxy for manifold separation. Finally to illustrate the merits of generative modeling in the task of separating widely overlapping data manifolds we also include a deterministic 2-layer MLP classifier which is trained in a supervised way, i.e. using cross entropy as a cost function.

The results of our experiments can be shown in tables 1 and 2, where we denote our model by HCDVAE (Hierarchical Continuous-Discrete VAE). The baselines developed for the domain of FC are in essence clustering algorithms and as such they suffer from issues clustering algorithms traditionally suffer from, the most important of which, are sensitivity to parameter configurations and dependence on initialization schemes. Concordantly, they exhibited high variance across multiple runs so we present the best results across 30 runs for each algorithm. Most of the baselines were able to achieve good predictive performance for the healthy cell population across most runs, which is to be expected since it is overrepresented in the data set, but average and erratic performance, for pathological cells. With the exception of Rclusterpp which achieved good overall performance in the second data set, all algorithms in both data sets exhibited a tendency to either be sensitive (high TPR) sacrificing healthy cell predictive accuracy or be specific (high TNR) and sacrifice pathological cell predictive accuracy.

The supervised MLP classifier is clearly unable to separate the two manifolds, "overfitting" the healthy cell population. This suggests that at least for binary problems in which classes are severely imbalanced, training supervised classifiers (i.e. training on log-likelihood objectives with explicit labeled information) results in subpar performance compared to generative modeling. This is not surprising as appropriate generative models incorporate more information about the data in their representations than just its labels, resulting in greater discriminative power. The performance of VAE+SVM suggests that the topmost latent layer of vanilla VAE is not able to separate the healthy/pathological manifolds. Presumably this is the case because the approximate posterior has collapsed early on the prior and rendered most units inactive, making the overall representation uninformative with respect to the global structure of the data. Deep VAE architectures are also known to be difficult to train, with top layers being slow to learn useful representations or even unable to learn at all. This is not the case, however with β-VAE and HCDVAE. The predictive performance of both approaches is similar, however opting for a more distributed representation seems to yield marginally better predictive capacity. Visually, the two approaches seem to result in similar latent representations as can be seen in Figures 3 and 4 , where they collapse the majority of healthy cells into a compact cluster and separate the pathological cell manifold, remaining largely invariant to other factors present in the data. A possible explanation for the marginally better performance of HCDVAE is that its denser representation captures the activity of more explanatory factors in the data set, which β-VAE's representations miss due to excessive compression. We further hypoth- esize that the mixture of discrete and continuous variables, which HCDVAE and β-VAE share in our experiments provides a powerful posterior which benefits both approaches, resulting in similar performance. The discrete variable y at the top of the feature hierarchy representing cell state (or alternatively category) is largely invariant to perturbations in the factors modeled by the lower latent layers. Overall, HCDVAE achieves near human-level performance in both data sets, especially in the second case. We note that that the ground-truth diagnosis was crosschecked by 2 different human experts.

5. RELATED WORK:

Models that use mixtures of discrete and continuous variables and are trained with the variational objective have previously been discussed in literature. Kingma et al. (2014) proposed a model for semi-supervised classification (M2) based on a discrete class variable y and a continuous "style" variable z. Optimization is performed using the reparameterization trick for z and marginalizing over y. This model is similar in structure to our own and Kingma et al. (2014) also raise the point of learning better discriminative features which are more clearly separable, making classification easier (M1 model). Another similarity is that the distribution q φ (y|x) can be used as a classifier, performing classification through inference. A crucial difference is that we are using a relaxed discrete variable, removing the need to marginalize over its possible values. Additionally, we are enforcing dependence between the continuous and discrete variables. Maaløe et al. (2017) presented a method that also employs discrete and continuous variables also in the context of semi-supervised learning where the continuous variables model the formation of natural clusters in the data and the discrete variables representing class information refine this clustering scheme. Johnson et al. (2016) use both discrete and continuous latent variables to construct a structured VAE that uses conjugate priors to create more flexible approximating posteriors in the context of switching linear dynam-ical systems. Works that attempt to alleviate the non-informativeness of the prior have also been presented in literature. Dilokthanakul et al. (2016) present a VAE variant for the task of unsupervised clustering, introducing a Gaussian Mixture prior to enforce multi-modality on the inference distribution while employing the minimum information constraint of Kingma et al. (2016) to avoid cluster degeneracy. Goyal et al. (2017) develop tree-based priors, which they integrate into VAEs and train them jointly for learning representations in videos with the overall model being able to learn a hierarchical representation of the data set. Finally, Rolfe (2016) presented a model which combines a discrete component which consists of a bipartite Boltzmann machine with binary units on the one hand and a continuous component consisting of multiple continuous layers on the other. To perform optimization using the reparameterization trick, the binary variables are marginalized out. While the proposed "discrete VAE" learns the class of objects in images, it is significantly more complex in its architecture and still relies on marginalization of the discrete variables.

6. CONCLUSION:

We have presented a framework for unsupervised feature learning and classification based on modifications of the original VAE framework (Kingma & Welling, 2013) , making use of a mixture of relaxed discrete and continuous distributions, as well as batch normalization and deterministic warm-up (Sønderby et al., 2016) , which we have found to be crucial for our framework's success. We trained our framework in a completely unsupervised manner, however relied on a stopping criterion that depends on labeled data. We have presented state-of-the-art results in a difficult real world problem having tested our approach against state-of-the-art computational FC and VAE models. Future work includes deriving a completely unsupervised early stopping criterion, adaptation to more problem domains and further principled investigation of the discrete variable to control the framework's modeling capacity.

:

Both data sets are publicly available with permission in https://github.com/dimkal89/ unsupervised_classification

Details on the experimental setup and the architectures used can be found in Appendix B

Figures and Tables:

Figure 1: PCA plots of latent representations for different number of classes K of discrete variable y. Increasing K, imposes a different structure on the model, encouraging it to learn and separate different manifolds. E.g. setting K = 4 the model successfully separates the 4 different cell populations present in the data (lymphocytes, monocytes, granulocytes and destroyed red blood cells).

Figure 2: Model architecture. Left: inference model, right: generative model.

use of batch normalization introduced by Ioffe & Szegedy (2015), since it has been shown to both improve convergence time and allow for the training of the upper layers in deep generative models Sønderby et al. (2016).

Figure 3: PCA plots of latent representations for experiments on both patient data sets for HCD-VAE. Red indicates healthy cells. Blue indicates pathological cells. For clarity only minibatches containing pathological cells are shown. Best viewed in color.

Figure 4: PCA plots of latent representations for experiments on both patient data sets for β-VAE. Red indicates healthy cells. Blue indicates pathological cells. For clarity only minibatches containing pathological cells are shown. Best viewed in color.

Patient data set 1

Table content: <table><tr><td>Method</td><td colspan="2">TPR TNR</td></tr><tr><td>Rclusterpp</td><td>0.5</td><td>0.935</td></tr><tr><td colspan="3">flowMEANS 0.904 0.363</td></tr><tr><td>X-shift</td><td colspan="2">0.434 0.901</td></tr><tr><td>MLP</td><td>0.0</td><td>1.0</td></tr><tr><td>VAE+SVM</td><td colspan="2">0.043 1.0</td></tr><tr><td>β-VAE</td><td>1.0</td><td>0.878</td></tr><tr><td>HCDVAE</td><td>1.0</td><td>0.936</td></tr></table>

Patient data set 2

Table content: <table><tr><td>Method</td><td colspan="2">TPR TNR</td></tr><tr><td>Rclusterpp</td><td colspan="2">0.833 0.931</td></tr><tr><td colspan="3">flowMEANS 0.264 0.999</td></tr><tr><td>X-shift</td><td>0.5</td><td>0.889</td></tr><tr><td>MLP</td><td>0.0</td><td>1.0</td></tr><tr><td>VAE+SVM</td><td colspan="2">0.076 1.0</td></tr><tr><td>β-VAE</td><td>1.0</td><td>0.958</td></tr><tr><td>HCDVAE</td><td>1.0</td><td>0.994</td></tr></table>

Peer review (iclr2018):
 Summary The authors propose a hierarchical generative model with both continuous and discrete latent variables. The authors empirically demonstrate that the latent space of their model separates well healthy vs pathological cells in a dataset for Chronic lymphocytic leukemia (CLL) diagnostics. Main Overall the paper is reasonably well written. There are a few clarity issues detailed below. The results seem very promising as the model clearly separates the two types of cells. But more baseline experiments are needed to assess the robustness of the results. Novelty The model introduced is a variant of a deep latent Gaussian model, where the top-most layer is a discrete random variable. Furthermore, the authors employ the Gumbel-trick to avoid having to explicitly marginalize the discrete latent variables. Given the extensive literature on combining discrete and continuous latent variables in VAEs, the novelty factor of the proposed model is quite weak. The authors use the Gumbel-trick in order to avoid explicit marginalization over the discrete variables. However, the number of categories in their problem is small (n=2), so the computational overhead of an explicit marginalization would be negligible. The result would be equivalent to replacing the top of the model p(y) p(z_L|y) by a GMM p_{GMM}(z_L) with two Gaussian components only. Give these observations, it seems that this is an unnecessary complication added to the model as an effort to increase novelty. It would be very informative to compare both approaches. I would perhaps recommend this paper for an applied workshop, but not for publication in a main conference. Details: 1) Variable h was not defined before it appeared in Eq. (5). From the text/equations we can deduce h = (y, z_1, …, z_L), but this should be more clearly stated. 2) It is counter-intuitive to define the inference model before having defined the generative model structure, perhaps the authors should consider changing the presentation order. 3) Was the VAE in VAE+SVM also trained with lambda-annealing? 4) How does a simple MLP classifier compares to the models on Table 1 and 2? 5) It seems that, what is called beta-VAE here is the same model HCDVAE but trained with a lambda that anneals to a value different than one (the value of beta). In this case what is the value it terminates? How was that value chosen? 6) The authors used 3 stochastic layers, how was that decided? Is there a substantial difference in performance compared to 1 and 2 stochastic layers? 7) How do the different models behave in terms train vs test set likelihoods. Was there overfitting detected for some settings? How does the choice of the MCC threshold affects train/test likelihoods? 8) Have the authors compared explicit marginalizing y with using the Gumbel-trick? Other related work: A few other papers that have explored discrete latent variables as a way to build more structured VAEs are worth mentioning/referring to: [1] Dilokthanakul N, Mediano PA, Garnelo M, Lee MC, Salimbeni H, Arulkumaran K, Shanahan M. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648. 2016 Nov 8. [2] Goyal P, Hu Z, Liang X, Wang C, Xing E. Nonparametric Variational Auto-encoders for Hierarchical Representation Learning. arXiv preprint arXiv:1703.07027. 2017 Mar 21.