A Critical Look at the Consistency of Causal Estimation with Deep Latent Variable Models

1 Introduction Causal inference , dealing with the questions of when and how we can make causal statements based on observational data , has been a topic of growing interest in the deep learning community recently . On the one hand , causal inference promises to provide traditional machine learning and AI with methods for explainability , domain adaptation , and causal reasoning capabilities in general [ Pearl , 2019 ] . On the other hand , many deep learning methods for improving causal inference have been proposed . Some of the models have been designed under the assumption of no unobserved confounding [ Shalit et al. , 2017 , Yoon et al. , 2018 , Shi et al. , 2019 ] , while others utilize latent variables in one way or the other to account for unobserved confounders [ Louizos et al. , 2017 , Rakesh et al. , 2018 , Pfohl et al. , 2019 , Madras et al. , 2019 , Mayer et al. , 2020 , Chen et al. , 2020 , Pawlowski et al. , 2020 , Jesson et al. , 2020 ] . Although some simple models with unobserved confounders are known to produce correct results [ Angrist et al. , 1996 , Pearl et al. , 2016 , Miao et al. , 2018 ] , the consistency , i.e. , whether in the limit of large data the correct causal effect is retrieved , is usually left largely open with deep latent variable models . In particular , Louizos et al . [ 2017 ] proposed the causal effect variational autoencoder ( CEVAE ) for performing causal inference in the setting where we have an unobserved confounder , of which only noisy proxy variables are available . The causal graph is shown in Fig.1b , which contrasts to the standard `` no unobserved confounding '' , or direct confounding , graph in Fig.1a . The proxy variable setting has been studied rigorously elsewhere by [ Pearl et al. , 2016 ] , where the authors provided 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . provably correct methods for a few simple types of data with strong assumptions about the data generating process . They dubbed the process of estimating causal effects in this context as `` effect restoration '' . CEVAE was proposed to relax these assumptions significantly , but the question of consistency was left unanswered . To provide insight into the behavior of deep latent variable models as causal effect estimators , we use CEVAE as a case study . CEVAE is a natural choice because it is based on the well-established standard variational autoencoder [ Kingma and Welling , 2014 , Rezende et al. , 2014 ] , and it allows comparison with analytical , provably correct methods . We conduct rigorous experiments with the model using various synthetic and semi-synthetic data sets and also provide theoretical statements in some special cases . Even though generic theoretical results are difficult to get for variational autoencoders , we provide intuitive conclusions on the assumptions under which the model works or does not work . 2 Preliminaries . 2.1 Model description . The objective in the scenario in Fig.1b is to learn the causal effect between the variables t and y. t is a variable on which we can perform an intervention , e.g. , a treatment on a disease . x is the possibly multidimensional proxy providing indirect and noisy information about the unobserved confounder z . The correct interventional distribution p ( y|do ( t ) ) is defined with the formula p ( y|do ( t ) ) = ∫ p ( y|z , t ) p ( z ) dz , ( 1 ) where the integral is replaced with a sum for a discrete confounder . Note that we denote the latent variable of a VAE with z as well , even though they are conceptually separate and might not be distributed in the same way . The distinction should be clear from the context . At its core , the causal effect variational autoencoder is simply a regular variational autoencoder with two additional assumptions . First , we assume that the latent variable z corresponds to the unobserved confounder in some way so that after training , we can get the causal effect p ( y|do ( t ) ) by estimating with the adjustment formula : p ( y|do ( t ) ) ≈ pθ ( y|do ( t ) ) = ∫ pθ ( y|z , t ) p ( z ) dz , ( 2 ) where θ are learned decoder parameters and p ( z ) is the VAE prior . Second , we assert some additional restrictions on the structure of the decoder . The idea is that the quantities reconstructed during training should follow the conditional independencies specified by the causal graph in Fig.1b so that the conditional probability of observed variables given the latent variable factorizes as pθ ( x i , ti , yi|z ) = pθ ( xi|z ) pθ ( ti|z ) pθ ( yi|z , ti ) , ( 3 ) where superscript i refers to the i : th observation . Thus , we can write the ELBO for the model as L ( θ , φ ) = ∑ i [ Eqφ ( z|xi , ti , yi ) [ log pθ ( x i|z ) + log pθ ( ti|z ) + log pθ ( y i|z , ti ) ] − KL [ qφ ( z|xi , ti , yi ) ||p ( z ) ] ] , ( 4 ) where xi , ti and yi are observed quantities . Thus , we have to define at least four neural networks : The encoder network qφ ( z|xi , ti , yi ) and three decoder networks corresponding to the conditionals in Eq.3 . The original paper also suggested composing the encoder of multiple networks chosen based on the value of the treatment , but that is not strictly necessary and was not motivated by the causal graph . Note that the decoder differs from usual VAE decoders in that the observed treatment value ti has to be given as input to the network corresponding to pθ ( y|z , t ) . The original paper seemed to suggest that the input should be sampled from the pθ ( t|z ) distribution during training ( Fig.2b in [ Louizos et al. , 2017 ] ) , but the factorization in Eq.3 suggests that the observed t should be used instead . In terms of the problem statement , CEVAE is perhaps most closely connected to the methods of effect restoration first proposed by Kuroki and Pearl [ 2014 ] . They offered provably correct , analytical solutions to the proxy variable problem in Fig.1b when all variables are jointly normal or categorical and x consists of two variables conditionally independent of each other given z . The setting was further studied in [ Miao et al. , 2018 ] . Note that since CEVAE assumes a different causal graph from the direct confounding graph in Fig.1a , it can ’ t be assumed to give correct results if the true data generating process is the direct confounding graph . For example , Kuroki and Pearl [ 2014 ] showed that an observed p ( x , t , y ) can map to two completely different causal effects by assuming either the direct or the unobserved confounding graphs . As the original CEVAE paper experimented on data with no unobserved confounding in addition to data sets with the unobserved confounding , we suspect that this may have been a point of confusion for some . 2.2 Possible issues with estimation . Theorem 1 in [ Louizos et al. , 2017 ] states that if CEVAE is able to recover the p ( z , x , t , y ) distribution , it is guaranteed to yield correct causal estimates . However , this leaves open the relevant question about when such an assumption can hold , as recovering p ( z , x , t , y ) entirely is , strictly speaking , impossible due to the unidentifiability of VAEs ( e.g . [ Locatello et al. , 2019 ] ) . In contrast , we don ’ t consider the latent variable of CEVAE to strictly correspond to the true confounder . Instead we view the process of training CEVAE and applying the adjustment formula in Equation 2 simply as a statistical estimator for the causal effect , regardless of what the latent variable exactly represents . That is , we are interested in recovering the correct causal effect and not the true hidden confounder . Our goal is then to study when the resulting causal effect estimates are consistent . Consistency means in general that the estimates of the parameters of interest approach their correct values as the amount of data increases , see , e.g. , [ Schervish , 2012 ] . For consistency to be possible , the model also has to be identifiable with respect to causal effect estimates , so that minimizing the loss ( e.g . maximizing the likelihood or the ELBO ) does not map to multiple data generating parameters θ that correspond to different pθ ( y|do ( t ) ) , see , e.g. , [ Murphy , 2012 ] . This is not to be confused with the common usage of the word `` identifiability '' in causal inference literature , where the question is instead whether it is possible in principle to estimate some causal effect from an observed distribution , e.g. , using the techniques of the famous do-calculus [ Pearl , 2009 ] . In the presence of unobserved variables , results on causal identifiability often rely on parametric knowledge about the underlying data generating process . We refer to the identifiability of CEVAE as a statistical estimator as the model identifiability , to distinguish it from the causal identifiability . Note that model identifiability as defined here guarantees only that a unique causal effect estimate is obtained , but not necessarily that it is consistent , for example if the model is misspecified . The original CEVAE paper suggested multiple scenarios as the motivation for the model , including the case where we have very few parametric assumptions about the data generating process and the case where the distribution of the proxies is complex , as is often reasonable to assume with real-world data . Our aim here is to study how well the model works as we reduce the number of assumptions and move further into the territory envisioned in the paper . Based on the motivating scenarios , we can conceptually separate three goals for CEVAE : 1 . It should produce correct estimates with minimal knowledge about the parametric forms of the data generating p ( x|z ) , p ( t|z ) and p ( y|z , t ) distributions . 2 . It should work if we don ’ t know the form of the unobserved confounder ’ s distribution , which could be categorical or Gaussian , for instance . 3 . It should work with an arbitrarily complex distribution of proxies . With the first goal , the hope is that neural networks will estimate the conditionals correctly enough to estimate the causal effects . With the second goal , the hope is that the true confounder is represented well enough with the standard Gaussian prior of the VAE . In practice , correct estimation could be prevented by many factors . For real-world data sets , it is possible that the causal effect is not identifiable at all from the data , even if we know the parametric form of the data generating process . If it is identifiable in principle , CEVAE might still fail due to inherent model nonidentifiability caused by the nonparametric assumptions and because we don ’ t have a guarantee that finding a unique global optimum for the ELBO leads to a correct causal estimate . Local minima or other issues with optimization could cause further practical problems with the correctness of the results . We focus on the estimation of p ( y|do ( t ) ) instead of individual-level causal effects p ( y|do ( t ) , x ) as that makes the analysis more straightforward . Note that while the average treatment effect ( ATE ) , defined as E [ y|do ( t = 1 ) ] − E [ y|do ( t = 0 ) ] , is a common metric , we are interested in p ( y|do ( t ) ) directly because the estimated ATE can have the correct value even when the estimates of p ( y|do ( t = 1 ) ) and p ( y|do ( t = 0 ) ) are not correct , and furthermore our analysis extends to continuous t .

Summary: The paper takes an in-depth look at using latent variable models in causal effect estimation from proxies. The idea behind such methods, specifically CEVAE, is that the proxies provide sufficient information about the confounder (one extreme case is when the proxies determine the confounder) to ensure that the causal effect is identified. As the authors suggest and show, both theoretically and empirically, this idea bears fruit in terms of estimability of causal effects only under potentially strong assumptions. The authors outline multiple failure modes of the CEVAE which I believe extend to other existing deep learning based methods for proxy-based effect estimation. Finally, experiments show in some more semi-synthetic datasets, the estimated effects are not consistent.