Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of generative modeling for mixed discrete-continuous data. To this end, it introduces a novel variant of coupled variational autoencoders. Specifically, they reformulate the coupled variational autoencoder paradigm in such a way that, instead of each "arm" of the autoencoder modeling a different modality, the arms model different chunks of the same modality (which comprises a high-dimensional discrete part, in addition to a continuous part). 

The key novelty of the paper is the idea of postulating a finite mixture model as the likelihood (decoder) pertaining to each arm of the autoencoder, and stipulating that all arms essentially infer similar categorical posteriors of mixture component assignment for the different chunks pertaining to the same observation. This consensus constraint is enforced on the grounds of the Aitchison geometry in the probability simplex, which avoids the mode collapse problem. 

main_review: Strengths: The idea is quiet novel and smart. The technical formulation is appropriate, and the constraint enforcing mixture component assignment consensus is sound. The experimental results are convincing, as they include a number of benchmarks, including some challenging ones, comparison to some SOTA methods, and an ablation study that offers good insights into how and why the method works.

Weaknesses: I did not find any commendable weakness in the paper.
summary_of_the_review: The paper is novel enough, methodologically sound, and empirically well-validated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to learn latent variable model with mixed discrete and continuous latent variables. Specifically, the framework utilizes multiple pairwise-coupled autoencoding arms to learn shared categorical variable. The experiments on dSprites, MNIST and RNA-seq datasets verify the effectiveness of the model.  
main_review: Post-rebuttal: 
The authors' feedback clarify some of my concerns. but I'm still not so convinced by the experiments. The authors argue the model is not suitable for SVHN/cifar10 due to their complexity and this makes it unclear whether the model is applicable/generalizable for the real-world datasets that are either large scale or present extremely challenging backgrounds. Achieve SOTA ACC is definitely not necessary, but I would assume the proposed model has competitive results, however, the preliminary results shown in appendix seem quite weak. Therefore, I still tend to keep my original rating. 

For the paper strength: 

-- The latent variable with mixed representation is an important model and worth exploring. 

-- Not familiar with the bio-literature, but the experiments and applications on RNA-data look interesting. 

For weakness: 

-- The motivations and descriptions of the model need to be improved. Some places are unclear and confusing. For example. the paper emphasized that the work is fully unsupervised and proposes the type-preserving data augmentation. However, since each arm should receive samples that share the same underlying categorical factor, does this requirement impose some weak-supervision signal?  i.e., we have to ensure that each arm observes the samples from same category, but for fully unsupervised setting, we don't know that. Could authors further elaborate on this point? 

-- The experiments seems not sufficient. (a) Since more arms potentially means more encoder/decoder pairs (i.e., more parameters involved), its better to also include the model complexity comparison with the baselines. (b) The scalability of the model is not clear since the experiments are only done on synthetic/gray-scale datasets, what about on svhn or cifar10? For these slightly complicated benchmarks, how does the model perform? How to select the number of arms? Does the model capable dealing with large-scale datasets? 
summary_of_the_review: The major concern goes to the experiment setting. The benchmarks are too simple to show the potential of the model and the baselines compared can be more up-to-date. I lean towards reject and may reconsider my rating based on authors' feedbacks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes cpl-mixVAE, which is a novel VAE formulation that attempts to improve clustering models by adapting ideas from consensus clustering.  The paper proposes a new model structure and a novel distance to help encourage improved model representations.
main_review: Update: I have adjusted my score upwards.  The reviewers have address some of my concerns.  I still believe that the interplay between the  VAE-GAN and the clustering method is not explored well-enough, and that complex interplay can drastically impact the results of the approach.

I have issues with the experiments, justifications, and theory in this manuscript.

First, my major issue with this manuscript is that the proposed approach is not truly an unsupervised method, as is claimed.  It uses some supervised information in order to make its decisions.  To see this, consider the usage of “type-preserving data augmentation that generates independent and identically distributed copies of data while preserving its categorical identity.”  The generation process maintains the label; hence, you are telling the neural network that 2 samples from the same class belong in the same cluster.  Thus, because the data augmentation system is using full data distributions, despite not explicitly using the label, I would argue strongly that supervised information is leaking into your system.  The reason that this data augmentation does not help JointVAE$^\dagger$ like it helps cpl-mixVAE is that in the JointVAE the system is not told that these should exactly match, so the supervised information does not leak into the JointVAE system.

As such, if you are going to use existing clusters, I need to understand the benefits of using this approach rather than a full supervised approach before any consideration of acceptance.  This is especially true as the quantitative metrics are all supervised evaluation criteria.  Since your method is getting some supervised information, it is unsurprising that it does better.  At the same time, it does much worse than basic supervised methods, including logistic regression on MNIST.

If you are going to push this as an unsupervised method, you should explain how it could be used in an unsupervised fashion.  For example, how well does it represent the data in an unsupervised fashion.  Can you use it for model selection?

Theory issues:

Both proposition 1 and proposition 2 require that you know the true $m$ to estimate $m$.  This is not unsupervised theory, and it is misleading to call it that.  Second, the interpretation that the expected log posterior increases with the number of arms is misleading.  This method artificially inflates the posterior and isn’t adding information to the system.  If you are given a new sample where you don’t know the cluster, and does not improve the correct assignment rate in the Bayes optimal case.  This theoretical section needs an increased discussion about these issues.

Minor complaints:

It is confusing to introduce c and s as independent in (1) and then immediately couple them in (3) without discussion.  Please refine this transition.

Expectation on the last term in (3) is on the wrong distribution. 

It is unclear how the distance between the distribution on c in (6) is related to the Aitchison distance used later.  Please provide the relationship.

Additional explanation of 3.3 is necessary.  It seems like the primary benefit is to push it away from trivial solutions, and the authors should expand on how they do that.

In the ablation study, I would like to understand the impact of the novel distance better.

summary_of_the_review: I find the idea of including consensus clustering in a VAE an interesting and potentially useful idea to explore, but the theory explicitly suggests that this model only helps in the supervised case, not an unsupervised case.  Additionally, I would argue that supervised information leaks into cpl-mixVAE through the data augmentation, as is the explicit goal, and it is unfair to compare to purely unsupervised methods on only supervised metrics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a consensus model over a base-level VAE model that has categorical, $\mathbf c$, and continuous, $\mathbf s$ latent components. The base-level model is instantiated a number of times (arms) where given a training instance $\mathbf x$ , its arm is fed with perturbations $\mathbf x_i$ of the instance. The different arms are constrained so that they 
assign the same category to the different $\mathbf x_i$ perturbations. The main motivation for such a consensus constraint is, if I am not mistaken, that it avoids the mode collapse
problem. In addition, accroding to the paper, the suggested approach can handle situations in which we have very different number of training instances in the different categories.


main_review: The paper considers the following factorisation of the the generative model $p(\mathbf x, \mathbf c, \mathbf s) =  p(x|s, c)p(s|c)p(c)$ in which the continuous components are a function
of the categorical, where the categorical variable captures different (discrete) classes within the data and the continuous component explains variations within them.

The paper is mostly easy to read, though there are some points that require clarifications. It also comes with a quite extensive, though not complete, set of experiments and analysis. 

My main issue is that the baselines against which the paper compares do not exhibit the generative structure that is given above; both of them, JointVAE and CascadeVAE, assume that
the categorical and discrete components are independent. This makes it difficult to assess whether the merits of the paper come from the factorization of the generative model that
assumes the dependence of the continuous from the categorical component or whether it comes from the concensus mechanism. Such factorizations have been quite extensively discussed
elsewhere, for example in Lavda et al, Data-dependent conditional priors for unsupervised learning of multimodal data, Entropy, 2020, where the authors discuss how such a factorisation 
actually protects from mode collapse, as well as a number of other issues relevant to the discussion in the current paper, such as how the different categories are used through entropy
regularisers that naturally appear within the initial objective. Since the base level factorisation over which the present paper builds the consencus model is the same, it would have
been very easy to include it as a considerably more relevant baseline in order to demonstrate whether the benefits come from the concensus approach as well as include a discussion of
differences from Lavda et al. 

One more point that I have missed concerns the claim of the paper that the multitude of arms allows to deal with cases where the class distribution is not uniform, is this something that is demonstrated in section 3.2? If yes, I did not understand what is it that makes the model robust to non-uniform $p(\mathbf c)$. Related to that are the suggestions to use 2 arms when the distribution of $p(\mathbf c)$ is uniform. 


Details:

In eq~3 which gives the variational loss for arm $\alpha$ the term $\mathbb E_{q(s_a|c_a,x_a}[D_{KL}(q(c_a|x_a)||p(c_a)]$ should probably be simplified to just the KL 
term $D_{KL}(q(c_a|x_a)||p(c_a))$, since the distributions in the latter do not depend on the $s_a$ variable with respect to which the expectation is taken. See for example
the respective derivation in Lavda et al, 2020. 

In figure b) the graphical model describes the generative model $p(x|s, c)p(s|c)p(c)$ however what is given just bellow seems to be the inference model even though $p$ is used instead of $q$.

I have a bit of a problem conceptualising $q(c|x_1,...,x_A)$, how is this posterior instantiated concretetly? or is it only defined implicitly through equation 2? After going through the proof
of proposition 1. in the appendix this is clear.

Proposition 1. says that as the number of arms/experts increases the likelihod of the true category will also increase.  

Proposition 1. :

* what is the difference of $x_i \sim p(x|m)$ and the $q(x|m)$ that appears in the expectation? $p(x|m)$ never appears in the derivation in the appendix. Is $q(x|m)$ meant to denote that
we randomly draw an $x$ and $p(x|m)$ the noisy versions of it, in which case it would probably be more appropriate to have $p(x_i|x)$.  By the way this two level sampling is missing from 
the derivations, though I am not sure it is needed, but I find confusing the fact that there are two types of rvs here the original $x$ and its noisy versions $x_i$, even if I get the point.
For example in eq. 5 we marginalise over $x$ a quantity that does not contain $x$ but a random variable of $x$ the $x_i$.

* appendix, eq 5: the denominator has dissapeared and repappears again in 6. 


Since the method works by operating on perturbed versions of the training instance one needs to define a perturbation model. The paper proposes a generative model based on GAN-VAE where
the main desiredata is to perturbe the instances while not alterning their latent categorical code, the non-observed class.

When computing the distance over the categorical values $c_a$ it seems that the paper considers that $c_a \in \mathcal S^K$, i.e. they belong to the probability simplex. However $c_a$ is not a probability 
vector but a sample from such a categorical distribution, which should normally be a one hot-encoded vector. Which is the case? i.e. are the distances computed over the samples or over the 
respective probabilities? How relevant is this?


Experiments: 
* the paper said that when the distribution of classes is uniform we should use a 2-arm (dSprites and MNIST), how is this motivated? If I understood correctly the theoretical results, these show that when we increase the number of arms we will have a larger log posterior for the true categorical variable, but I am not sure I saw a relation to the uniformicity of the categories. 

* In the real world experiments, figure 3, how are the categories (i.e. the columns) ordered? I presume the ordering of the three different algorithms is not really comparable and what really matters
is how these align with the real cell types. I am just curious is the orderning based on the largest values that one gets in the diagonal of the confusion matrix? 

* In the same figure, it seems that the discrete compoments that the proposed method uncovers are in agreement with the hierarchical clustering of the cells. So what we have here are two 
unsupervised methods the results of which are in agreement. I guess that the hierachical clustering results have been validated by the domain experts in (Tasic et al., 2018; Yao et al., 2021), 
I am just wondering how to approach such an evaluation where there is no real ground truth. Of course one could say it is a good thing that two rather different methods agree on how they 
cluster cells. 

* The evaluation in figure 4 where a given continuous latent variable varies with the discrete component makes intuitive sense. The figure shows that the profiles produced by cpl-mixVAE varies 
smoothly within a given category and look rather different between the two categories. This is not the case for JointVAE, where the categorical component seems to bring no structure, the two different
categories have rather similar profiles. The later is something to be expected, as also noted by the authors, since JointVAE assumes independence of the categorical and discrete component an assumption
that makes it not the most appropriate baseline. On the same time, other than saying that intuitively the results of clp-mixVAE look good it is hard to make an additional comment, because this requires
quite some expertise on the biology side.  

summary_of_the_review: My main issue is that as the paper is it is difficult to assess whether the strength of the method comes from the concensus mechanism or whether it comes from the factorisation. In addition in the discussion about the number of arms being a strength when the distribution of the categorical factor is not uniform I miss the justification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of factorization of the hidden space of a VAE into two separate components Z=(S,C) where S is continuous and C is discrete. They make a particular assumption about the factorization of the encoder function q(S, C|X)=q(S|X)q(C|X) and they also take a mixture of “A” experts that are made to agree with their discrete assignment C.
main_review: The paper compares the proposed method with a few similar methods, using MNIST but also using a dataset for single-cell RNA sequencing data (a domain with which I am not familiar). I am also not familiar with how much efforts are currently being put into this approach of factorizing Z=(S,C) by other members in the field.

My main question for the authors is to ask about the results in table 1 when it comes to MNIST: do the std measurements indicate a spread over the multiple experimental runs, and why is the accuracy so low on MNIST? MNIST is the kind of dataset where a linear classifier can get 90% accuracy, yet all these methods presented fare much worse.

Minor issue: I think it would be appropriate to mention Kingma & Welling in the background section when VAE are first mentioned in the paper instead of in the later Section 2 when they are described. The same paper can be referenced twice, but at least it should be referenced in the introduction in conjunction to the other papers on VAEs that the authors wish to mention.
summary_of_the_review: Hard to say how novel the contribue is. I want to see the author respond to my question about table 1.
review_title: null

Meta-review: This paper proposes cpl-mixVAE, a method for fitting discrete-continuous latent variable models based on mixture representations and a novel consensus clustering constraint. After extensive discussion, no one was willing to argue in favor of acceptance, and a majority of the reviewers felt another round of revision is needed. Ultimately, I concur that while the ideas are novel and potentially interesting, more effort is needed to convincingly demonstrate the efficacy of the method. Valid concerns were also raised regarding the claimed "unsupervised" nature of the proposed method, a claim which at the very least requires some additional context. At this point, these outstanding issues require an additional round of revision.
Decision: Reject