Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a penalty to learn representations that are
conditionally independent of some observed variables. The idea is to
use the contrastive loss that encourage the representations from the
same conditional variable to be spread out and those from different
conditional variable to be close. The paper demonstrates
application to fairness, counterfactual inference, and data
integration, and demonstrate the effectiveness of the penalty.

main_review: The paper proposes a penalty to learn representations that are
conditionally independent of some observed variables. The idea is to
use the contrastive loss that encourage the representations from the
same conditional variable to be spread out and those from different
conditional variable to be close. The paper demonstrates
application to fairness, counterfactual inference, and data
integration, and demonstrate the effectiveness of the penalty.

Both the theoretical and empirical results of the paper make sense. I
have a few major comments.

One comment is the choice of contrastive loss. It is understandable
that using the loss in Eq 3 would encourage tightly clustered
representations and produce suboptimally informative representations.
One thus may think contrastive loss may help avoid trivial solutions
and preserve informativeness. However, it is unclear why the second
penalty is sufficient to ``avoid tight clusters of points.'' Why is
the contrastive loss necessarily the right choice here? I think the
paper could benefit from a discussion about it.


A second comment is on the comparison between the misalignment penalty
and the other MMD penalties in the literature. The paper argues that
MMD suffers when there are many isolated cluster in the data. The
problem of MMD appears to be related to scaling; and the misalignment
penalty has the self-normalized property. I wonder if it implies that
MMD is equally effective if one manually rescale and recenter these
data points to achieve the normalization effect? What if the global
structure does not have many isolated clusters? Will the misalignment
penalty need to pay a price? Are there other differences between the
misalignment penalty and the MMD approaches that made the misalignment
penalty more preferable?

limitations_and_societal_impact: The paper discussed the limitations in sec 7.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a penalty term called Contrastive Mixture of Posteriors (CoMP) for conditional variational autoencoders to learn latent representations invariant of a condition. Their penalty term doesn’t have extraneous requirements and is free from requiring a kernel (as is the case case with MMD) and enforces independence in latent space with respect to a condition (e.g. batch effect, sensitive information etc.). CoMP penalty itself is made up of two components which 1) encourage latent representations to be clustered together in both conditions - under a condition and under complement of said condition, 2) avoid tight clustering using a contrastive learning approach.
 
The work is useful in some computational biology tasks (batch correction, counterfactual inference) which require low-dimensional representations of high-dimensional sequencing data free from batch effect or other conditions, and also in fair classification to learn a classifier free from sensitive demographic information. For each of these tasks, the authors have presented one experiment and have compared their methods with existing methods based on variational and conditional variance autoencoders.
main_review: The paper is very well-written and easy to follow. The applications and tasks considered are well-motivated with Theorem 1. While relatively straightforward, I find the connections drawn between data integration, counterfactual inference and fair classification to be very useful as it brings together references and results from sub-fields that might not be very aware of each other. Finally, the proposed method is very relevant and well-established for both domains considered in the paper - computational biology and fair classification. The limitations mentioned are also well thought out. 
 
My main remarks (roughly in decreasing order of importance) are:
 
1) Assuming independence between z and c is absolutely crucial to the foundation of the studied setting, the correctness of the suggested method and to its theoretical analysis. However, the assumption rarely holds with biological data where different samples might, at the same time, present different phenotypes, technical biases (or batch effects), cell type compositions etc. I would be very curious to see results on the integration of datasets where we know the assumption is broken. For instance, if one cell type is present only in one batch, will their latent codes overlap with cells from a different cell type from the other batch? This issue, usually refered to as "over-correction" is absolutely crucial in practice. At least in the discussion, I think it could be nice to comment on it and give some insights on how this issue could be tackled both in theory and in practice.
 
2) The batch correction experiment considered is for alignment of tumor RNAseq (TCGA) and cell-lines (Cellinger). For a proper evaluation of batch correction, further experiments especially on single-cell data, where batch effects are much more prominent and the task is generally harder, should have been considered.
 
3) The only batch correction method evaluated outside of variational inference based approaches is Cellinger. It would be good to compare the model on single cell data with at least some popular (among single-cell practitioners) batch correction methods in use, e.g. Seurat and Harmony.
 
4) While understandably, as argued in the paper, the goal is to learn meaningful latent representations free from batch effect, it would still be interesting to evaluate batch correction in the original gene space (Especially considering that MSEs of reconstructions for counterfactual inference experiment are already shown in Tables 9-12 in Appendix D). This would also provide more opportunities for downstream analysis and evaluation of corrected data.
 
5) Only prostate cancer clusters have been visually shown in detail in Fig. 7 of Appendix D for trVAE and CoMP. It would be good to have detailed visualizations for more cancer clusters, or at least to include other methods for prostate cancer clusters. This is particularly important as in Section 6.1, it is mentioned that Appendix D includes qualitative results for several clusters.
 
6) In both batch correction and counterfactual inference experiments, the UMAPs of original data are not shown (Only the latent representations in Figures 3 and 4.). While it is clear that original data would be clustered by condition (batch or original and stimulated), it would still be useful to see that.
 
7) For fair classification, the authors mentioned that the paper shows “published results” from trVAE and VFAE, however in the original trVAE, this experiment was not performed. For completeness, it will be good to include the source where those results were published.

-------------
After checking the author's response, I am upgrading my rating from 7 to 8.
limitations_and_societal_impact: While I think the most important limitation of this work (the need for independence between z and c) was not mentioned, some other limitations were briefly (but appropriately) discussed in the conclusion.
The societal impact were not really discussed but I am not sure how relevant it would be for this specific paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the problem of data integration, counter factual inference and fairness concentrating on large data sets where each data point lives in high dimensions. The authors generate an embedding of the data into a latent space in such a way that the distribution of the embeddings is the same for data from all clusters. The authors propose to learn such an embedding through a variation autoencoder (VAE). They come up with a novel penalty called contrastive mixture of posteriors misalignment (CoMP) penalty which they use to with the ELBO loss to train the VAE. They show empirical results chiefly on single-cell RNA-seq data and also on one income data-sets. 
main_review: The authors consider the problems of data integration, counter factual inference and fairness. They primarily concentrate on single cell RNA-seq for the first two points - a new technology in which one can obtain gene expression on a cell by cell basis  giving us data-sets with millions of cells - and tens of thousand measurements of gene expression for each cell. However different experiments carried out by different scientists unfortunately have batch effects. Being able to combine data from experiments run different scientists to do analysis is necessary for scientific integration - making data integration and dealing with batch effects important problems to consider. The counter-factual inference in single cell datasets is less important as there usually are technical and biological replicates available for experiments. There of course is a plethora of literature on fairness. I did think that the connection between the three was nice, even though it is obvious. 

The authors consider the method of embedding data from different batches into the same latent space as a means of getting rid of batch effects. However, this approach is explicitly not suitable for data integration in single cell RNA-seq because the data generated by different scientists is usually not from the exact same tissue from the same individual, and thus one would expect  the distribution of the latent space to be different in different batches even if there was no batch effect (without additioinal information the problem is not even well defined - but that's what makes working with single cell RNA-seq hard and interesting). 

The authors consider a VAE as means of obtaining such an embedding. They do this by coming up with a novel penalty to the ELBO loss. The penalty penalises differences in the distribution of points in the latent space for different batches - and is different from existing VAE based penalties in that it uses the posterior of the model learnt in the penalty rather than a different kernel.

The authors then provide empirical results on Single Cell RNA seq and an income data. One concern I have is that the authors limit their comparisons to VAE based methods for Single Cell RNA-seq when they are not the most popular ways in literature to deal with batch effects and counter-factual inference not comparing with methods like those of [14], and many other papers exploring that problem such as [100], [101], and [102]. Similarly, there is no comparison for non-VAE based methods (for example [103]) for fairness claims. 

The paper is well written and I liked the broad direction of the paper. But the formulation does not match the application of single cell RNA-seq that the paper purports to target. Furthermore, empirical results are found wanting comparing it with non VAE based approaches; which makes its contribution from the application perspective quite small. The only novelty in the paper seems to be using a different penalty in training the VAE - and I dont know the VAE literature enough to know who important or novel a contribution that is.

To me this is a borderline paper leaning towards rejection from an applications perspective. I like the idea and it is well written, but the formulation does not target any problem which analysts would be interested in and empirical results are not thorough.


[100] Korsunsky I, Millard N, Fan J, Slowikowski K, Zhang F, Wei K, Baglaenko Y, Brenner M, Loh P-r, Raychaudhuri S. Fast, sensitive and accurate integration of single-cell data with Harmony. Nature Methods

[101] Welch J, Kozareva V, Ferreira A, Vanderburg C, Martin C, Macosko E. Integrative inference of brain cell similarities and differences from single-cell genomics. bioRxiv. 2018:459891 Available from: http://biorxiv.org/content/early/2018/11/02/459891.abstract

[102]Stuart T, Butler A, Hoffman P, Hafemeister C, Papalexi E, Mauck WM 3rd, et al. Comprehensive integration of single-cell data. Cell. 2019;177:1888–1902.e21.

[103]Agarwal, Alekh, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. "A reductions approach to fair classification." In International Conference on Machine Learning, pp. 60-69. PMLR, 2018.

limitations_and_societal_impact: listed above
review_title: null

Meta-review: [This meta-review was written jointly by the SAC and AC after extensive discussion of the paper.]

This paper has interesting ideas and compelling results, and reviewers were overall supportive of the proposed work. Unfortunately, the paper has two entangled parts, where one of them has quite serious issues. The first part includes a Contrastive Mixture of Posteriors (CoMP) method that leverages a novel misalignment penalty to enforce certain independence relations; the second is related to counterfactual inference, data integration, and fairness.

Regarding the first part, CoMP enforces that a discrete auxiliary variable is independent of the latent variable inside a (conditional)-VAE without resorting to external discrepancies like an MMD. This is achieved by marginalizing out the inference network using empirical samples from the conditional distribution of the observation given settings of the discrete variable. This part of the paper is smooth, and the contributions are pretty substantial. It also connects quite nicely with the idea of variational inference and does not require an extraneous discrepancy measure or a separately tuned kernel, such as in previous approaches. The biggest benefit of this approach is that the scales of the terms in the loss are on the same scale as expectations of log probabilities or bounds of the same quantity. The approach could benefit many places where conditional-VAEs are used such as molecule generation (J. Lim, S. Ryu, J. W. Kim & W. Y. Kim, 2018) or dialog models (T. Zhao, R. Zhao & M. Eskenazi, 2017).

The second part of the paper regarding counterfactual inference, data integration, and fairness is quite concerning, and we feel there are fundamental misunderstandings regarding each one of them. The very claim that the paper introduces a unifying framework for solving these tasks is, from the beginning, completely off.

To understand why this is the case, consider the critical result used in the paper to compute Eq. 1, a quantity known as the "probability of causation," and written as $p(x_{c=c’} | x_i, c_i) = \int p(z|x_i, c_i) p(x | z, c’) dz$. By and large, the understanding of how to evaluate the l.h.s. of this expression is unsound. If one has the true, underlying structural causal model (SCM), say M*, the probability of causation could be evaluated in the manner described in Eq. 1, as introduced in Sec.7.1.1 of Causality (Pearl, 2000). On the other hand, whenever M* is unknown, this is not the case.  In particular, the probability distributions considered in the r.h.s. equation are arbitrary, representing a *learned* SCM M. The true SCM M* and the learned M are not necessarily the same.

The paper then proposes a method such that M will be trained to fit the data and guarantee the independence between Z and X. Still, it can be shown that it’s generically the case that there exists an M that will *almost always* disagree with the actual generating model M*, while agreeing with respect to the data. This means, therefore, that only with observational data, one cannot make statements about counterfactuals in generality. This is a well-known result in the field. For concreteness, please refer to the example discussed in Sec. 1.4.4 (Pearl, 2000) or examples 7-9 in (Bareinboim, Correa, Ibeling, Icard, 2020). This result is called the causal hierarchy theorem and can be found in Thm. 1 of the previous reference or Thm. 3 in https://arxiv.org/abs/2107.08558. Also, this has been shown in the context of autoencoders with continuous variables, for example, Appendix A.4 of https://arxiv.org/pdf/1907.03451.pdf,  has a construction where $c$ and $z$ are independent, but $x$ is constructed in a way that it is also marginally independent of $c$. This means the pair $(z, c)$ are independent and reconstruct $x$ and the pair $(x, c)$ are independent and reconstruct $x$. The pairs are observationally equivalent even with the extra independence constraint between the latent variable and $c$, but can be different under interventions.

Interestingly, the paper has one sentence (around line 124) acknowledging that it is not always possible to recover the correct model, while the rest of the paper seems to ignore this observation.  By and large, the challenge is that for almost any actual model M*, there are many models M that could fit the observed data equally well, but have very different implications in terms of their interventional and counterfactual distributions. This is certainly the most fundamental issue that disallows the paper to claim to be doing “counterfactual inference in generality.”

There are also serious issues regarding the tasks related to fairness and data fusion, and the purported “unification.” For example, many papers here at NeurIPS and other AI venues propose analyzing the fairness problem through causal lenses. The understanding from this work clarifies that enforcing independence among the protected attribute and the outcome variable is orthogonal to the issue at hand; in other, even when the independence is accomplished, biases may be amplified. Here are a few references that look at fairness from the viewpoint of causality (not an exhaustive list):
 
- Counterfactual Fairness, M. Kusner, J. Loftus, C. Russell, & R. Silva, NeurIPS’17.
- A Causal Framework for Discovering and Removing Direct and Indirect Discrimination. L. Zhang, Y. Wu, & X. Wu. IJCAI’ 2017
- Avoiding discrimination through causal reasoning. N. Kilbertus, M. Rojas-Carulla , G. Parascandolo, M. Hardt, & B. Scholkopf. NeurIPS’17.
- Equality of Opportunity in Classification: A Causal Approach. J. Zhang & E. Bareinboim, NeurIPS’18.
- Path-specific counterfactual fairness. S. Chiappa. AAAI’19.

Having said that, we note that it’s possible to stay oblivious to the causal inference literature and claim a purely statistical notion of fairness, which is anchored in conditional independence relations. In its current form, the paper claims to be doing counterfactual/causal inferences, so it’s expected the paper engages with the literature on fairness from the causal perspective, which goes beyond conditional independence relations and purely statistical measures. 

Finally, the data integration problem has also been studied through the causal framework, and I will suggest Ch. 10 of the Book of Why (Pearl & Mackenzie, 2018) or (Bareinboim & Pearl, PNAS’16), which survey the challenges involved in such endeavors. Even though data integration is a different task, most of the issues with the proposed approach come precisely from the misinterpretation of Eq. 1, following the aforementioned discussion.

All in all, we would like to make it clear that the first dimension of the paper, including CoMP, is appreciated, and we believe the contributions are quite significant. On the other hand, the second part suffers from fundamental flaws. Therefore, we don’t know how to accept the paper without re-thinking and re-writing the paper in a major way. One thought would be to demonstrate the value of CoMP on tasks that are more bread-and-butter for conditional-VAEs like generation (see the dialog and molecule generation references above).

Decision: Reject