SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
E q u a l i t y 1: \mathbb {P} ^ {\hat {I}} (\hat {U} _ {i} \mid \hat {U} _ {\mathrm {p a} _ {\hat {\mathcal {G}}} (i)}) = \mathbb {P} ^ {\hat {I}} (\hat {U} _ {i} \mid \hat {U} _ {\mathrm {a n} _ {\hat {\mathcal {G}}} (i)}),
$$
$$
\text {E q u a l i t y 2 :} \mathbb {P} ^ {\hat {\mathcal {I}}} (\hat {U} _ {i} \mid \hat {U} _ {\mathrm {a n} _ {\hat {\mathcal {G}} (i)}}) = \mathbb {P} ^ {\mathcal {I}} (U _ {i} \mid U _ {\mathrm {a n} _ {\mathcal {G} (i)}}),
$$
$$
\text {E q u a l i t y 3 :} \mathbb {P} ^ {\mathcal {I}} (U _ {i} \mid U _ {\mathrm {a n} _ {\mathcal {G}} (i)}) = \mathbb {P} ^ {\mathcal {I}} (U _ {i} \mid U _ {\mathrm {p a} _ {\mathcal {G}} (i)}).
$$
Proof of Equality 1. This follows by definition of  $\hat{\mathcal{G}}$ , since it is transitively closed, we have  $\mathrm{pa}_{\hat{\mathcal{G}}}(\boldsymbol{i}) = \mathrm{an}_{\hat{\mathcal{G}}}(\boldsymbol{i})$ .
Proof of Equality 2. By similar arguments below Eq. (13), we have  $\hat{U} = U\hat{\Gamma} +\hat{c}$  for an invertible matrix  $\hat{\Gamma}$ , where  $\hat{\Gamma}_{\tau (j),l} = 0$  for any  $\tau (l)\notin \mathrm{de}_{\mathcal{G}}(\tau (j))$ . Therefore we can recover  $U_{\mathrm{an}_{\mathcal{G}}(i)}$  by linear transforming  $U\hat{\Gamma}_{:, \mathrm{an}_{\mathcal{G}}(i)}$  and vise versa. We can also recover  $U_{i}$  by subtracting linear terms of  $U_{\mathrm{an}_{\mathcal{G}}(i)}$  from  $U\hat{\Gamma}_{:,i}$ .
Note also, since  $\mathcal{T}\mathcal{S}(\mathcal{G}) = \mathcal{T}\mathcal{S}(\hat{\mathcal{G}})$ , there must be  $\mathrm{an}_{\hat{\mathcal{G}}}(\boldsymbol {i}) = \mathrm{an}_{\mathcal{G}}(\boldsymbol {i})$ . Thus
$$
\begin{array}{l} \mathbb {P} ^ {\hat {\mathcal {I}}} (\hat {U} _ {i} \mid \hat {U} _ {\mathrm {a n} _ {\hat {\mathcal {G}}} (i)}) = \mathbb {P} ^ {\mathcal {I}} (U \hat {\Gamma} _ {:, i} \mid U \hat {\Gamma} _ {:, \mathrm {a n} _ {\hat {\mathcal {G}}} (i)}) \\ = \mathbb {P} ^ {\mathcal {I}} (U \hat {\Gamma} _ {:, i} \mid U \hat {\Gamma} _ {:, \mathrm {a n} _ {\mathcal {G}} (i)}) \\ = \mathbb {P} ^ {\mathcal {I}} (U \hat {\Gamma} _ {.;, i} \mid U _ {\mathrm {a n} _ {\mathcal {G}} (i)}) = \mathbb {P} ^ {\mathcal {I}} (U _ {i} \mid U _ {\mathrm {a n} _ {\mathcal {G}}} (i)). \\ \end{array}
$$
Proof of Equality 3. Follows from the Markov property on  $U$ .
Combining these equalities, we have  $\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)}) = \mathbb{P}^{\mathcal{I}}(U_i\mid U_{\mathrm{pa}_{\mathcal{G}}(i)})$  for all  $i\in [p]$ . Thus  $\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}) = \prod_{i = 1}^{p}\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)}) = \prod_{i = 1}^{p}\mathbb{P}^{\mathcal{I}}(U_i\mid U_{\mathrm{pa}_{\mathcal{G}}(i)}) = \mathbb{P}^{\mathcal{I}}(U)$ . Therefore the procedure in Section 4.4 generates  $X$  from the same distribution as  $X = f(U), U\sim \mathbb{P}^{\mathcal{I}}$ .
C Details on Discrepancy-based VAE
In previous sections, we have shown that the data-generating process in Section 2 is identifiable up to equivalence classes. However, the proofs (Appendix A, B) do not lend themselves to an algorithmically efficient approach to learning the latent causal variables from data. Therefore, we propose a discrepancy-based VAE in Section 5, which inherits scalable tools of VAEs that can in principle learn flexible deep latent-variable models. In this framework, Eq. (3) can be computed and optimized efficiently using the reparametrization trick [29] and gradient-based optimizers.
C.1 Maximum Mean Discrepancy
We recall the definition of the maximum mean discrepancy measure between two distributions, and its empirical counterpart.
Definition 3. Let  $k$  be a positive definite kernel function and let  $\mathcal{H}$  be the reproducing kernel Hilbert space defined by this kernel. Given distributions  $\mathbb{P}$  and  $\mathbb{P}'$ , we define
$$
\operatorname {M M D} (\mathbb {P}, \mathbb {P} ^ {\prime}) := \sup  _ {f \in \mathcal {H}} \left(\mathbb {E} _ {\mathbb {P}} [ f (X) ] - \mathbb {E} _ {\mathbb {P} ^ {\prime}} [ f (X) ]\right)
$$
The following empirical counterpart is an unbiased estimate of the squared MMD, see Lemma 6 of [18].
Definition 4. Let  $k$  be a positive definite kernel. Let  $\{X_{(i)}\}_{i=1}^{m}$  be samples from  $\mathbb{P}$  and  $\{X_{(i)}'\}_{i=1}^{m}$  be samples from  $\mathbb{P}'$ . We define
$$
\begin{array}{l} \widehat {\mathrm {M M D}} ^ {2} \big (\{X _ {(i)} \} _ {i = 1} ^ {m}, \{X _ {(i)} ^ {\prime} \} _ {i = 1} ^ {m} \big) = \frac {1}{m (m - 1)} \sum_ {i = 1} ^ {m} \sum_ {j \neq i} k (X _ {i}, X _ {j}) + \frac {1}{m (m - 1)} \sum_ {i = 1} ^ {m} \sum_ {j \neq i} k (X _ {i} ^ {\prime}, X _ {j} ^ {\prime}) \\ - \frac {2}{m ^ {2}} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {m} k (X _ {i}, X _ {j} ^ {\prime}) \\ \end{array}
$$
C.2 Discrepancy VAE Details
We walk through the details of this model in this section, where we illustrate it using two types of interventions, namely do interventions and shift interventions.
Noiseless vs. Noisy Measurement Model with General SCMs. Recall that each latent causal variable  $U_{i}$  is a function of its parents in  $\mathcal{G}$  and an exogenous noise term  $Z_{i}$ . All the  $Z_{i}$ 's are mutually independent. The overall model can be defined (recursively) as
$$
U _ {j} = s _ {j} \left(U _ {\operatorname {p a} _ {\mathcal {G}} (j)}, Z _ {j}\right), \tag {16}
$$
$$
X = f (U _ {1}, \dots , U _ {p}).
$$
In particular, there exists a function  $s_{\emptyset}^{\mathrm{full}}$  such that  $U = s_{\emptyset}^{\mathrm{full}}(Z)$ . We model each intervention  $I$  as a set of intervention targets  $T(I)$  and a vector  $a^{I}$ . Under  $I$ , the observations  $X$  are generated by
$$
U _ {j} ^ {I} = \left\{ \begin{array}{l} s _ {j} \left(U _ {\operatorname {p a} _ {\mathcal {G}} (j)} ^ {I}, Z _ {j}\right) \mathbb {1} _ {j \notin T (I)} + a _ {j} ^ {I} \mathbb {1} _ {j \in T (I)}, \text {f o r d o i n t e r v i t i o n ,} \\ s _ {j} \left(U _ {\operatorname {p a} _ {\mathcal {G}} (j)} ^ {I}, Z _ {j}\right) + a _ {j} ^ {I} \mathbb {1} _ {j \in T (I)}, \text {f o r s h i f t i n t e r v i t i o n ,} \end{array} \right. \tag {17}
$$
$$
X ^ {I} = f (U _ {1} ^ {I}, \dots , U _ {p} ^ {I}).
$$
As above, there exists a function  $s_I^{\mathrm{full}}$  such that  $U^{I} = s_{I}^{\mathrm{full}}(Z)$ . Note that here we assume that the measurements (sometimes called "observations" in the literature<sup>11</sup>)  $X$  are noiseless. Our theoretical results are built upon noiseless measurements. In practice, however, one can consider the noisy measurement model in which  $X = f(U) + \epsilon$  (resp.  $X^{I} = f(U^{I}) + \epsilon$ ), where  $\epsilon$  is some measurement noise independent of  $U$ .
We leave as future work to prove consistency under the noisy measurement model. [28] established identifiability results of the noisy measurement model, when the latent variables conditioned on
<sup>11</sup>We use "measurements" to distinguish from the observational distribution defined for  $U$ .
additionally observed variables follow a factorized distribution in an exponential family. Their techniques can be potentially used to generalize our results to the noisy measurement model; however, further assumptions on the mechanisms  $s_i$ 's will be needed.
Discrepancy-based VAE. We use one decoder
$$
p _ {\theta} (X | U)
$$
parameterized by  $\theta$  to approximate both  $X = f(U)$  and  $X^{I} = f(U^{I})$  in the noiseless measurement model (or  $X = f(U) + \epsilon$  and  $X^{I} = f(U^{I}) + \epsilon$  in the noisy measurement model). As for the encoder, we do not directly learn the posteriors  $\mathbb{P}(U|X)$  and  $\mathbb{P}(U^I |X^I)$ . Instead, we approximate one posterior  $\mathbb{P}(Z|X)$  and then use Eq. (16), (17) to transform  $Z$  into  $U$ ,  $U^{I}$  respectively. This is done by two encoders for  $Z$  and  $(T(I),a^{I})$  parameterized by  $\phi$  and denoted as
$$
q _ {\phi} (Z | X), (T _ {\phi} (I), a _ {\phi} (I))
$$
The dimension  $p$  of  $Z$  is set as a hyperparameter. Note that the procedure of learning a posterior  $\mathbb{P}(Z|X)$  in the observational distribution and then mapping to  $U^I$  using Eq. (17) can be regarded as learning the counterfactual posterior of  $\mathbb{P}(U^I |X)$ .
In the following, to better distinguish data from observational and interventional distributions, we use  $X^{\emptyset}, U^{\emptyset}$  instead of  $X, U$  to denote samples generated by Eq. (16). After encoding  $X^{\emptyset}$  and  $I$  into  $Z$  and  $(T(I), a^{I})$  respectively, we parameterize the causal mechanisms  $s_{j}$ 's in Eq. (16), (17) as neural networks (e.g., multi-layer perceptrons or linear layers). We absorb the parameterizations of  $s_{j}$ 's into  $\theta$  and denote
$$
p _ {\theta , \varnothing} (X ^ {\varnothing} | Z) = p _ {\theta} \big (X ^ {\varnothing} \mid U ^ {\varnothing} = s _ {\varnothing} ^ {\text {f u l l}} (Z) \big),
$$
$$
p _ {\theta , I} (X ^ {I} | Z) = p _ {\theta} \big (X ^ {I} \mid U ^ {I} = s _ {I} ^ {\mathrm {f u l l}} (Z) \big).
$$
Note that in implementation, to make sure  $U_{j}$  only depends on its parents  $U_{\mathrm{pa}_{\mathcal{G}}(j)}$ , one can train an adjacency matrix  $A$  that is upper-triangular up to permutations and then apply any layers after individual rows of matrix  $U \otimes A^{12}$ . Since identifiability can be only up to permutations of latent nodes, one can simply use an upper-triangular adjacency matrix  $A$ .
D Lower Bound to Paired Log-Likelihood
In this section, we consider the paired setting, in which we have access to samples from the joint distribution  $\mathbb{P}(X^{\emptyset}, X^{I})$ . To discuss counterfactual pairs, we must introduce structure beyond the structure described in Section 2. In particular, in the observational setting, assume that the latent variables  $U^{\emptyset}$  are generated from a structural causal model with exogenous noise terms  $Z$ . This implies that there is a function  $g_{\emptyset}$  such that  $U^{\emptyset} = g_{\emptyset}(Z)$ . Similarly, under intervention  $I$ , assume there is a function  $g_{I}$  such that  $U^{I} = g_{I}(Z)$ . Then, given a distribution  $\mathbb{P}(Z)$ , the joint distribution  $\mathbb{P}(X^{\emptyset}, X^{I})$  is simply the induced distribution under the maps  $X^{\emptyset} = f(U^{\emptyset})$  and  $X^{I} = f(U^{I})$ .
Since  $X^{\emptyset}$  and  $X^{I}$  are independent conditioned on  $Z$ , we have
$$
\begin{array}{l} \log \mathbb {P} \left(X ^ {\varnothing}, X ^ {I}\right) \geq \mathbb {E} _ {\mathbb {P} \left(X ^ {\varnothing}, X ^ {I}\right)} \left[ \mathbb {E} _ {q _ {\phi} (Z | X ^ {\varnothing})} \log p _ {\theta , \varnothing} \left(X ^ {\varnothing} \mid Z\right) + \mathbb {E} _ {q _ {\phi} (Z | X ^ {\varnothing})} \log p _ {\theta , I} \left(X ^ {I} \mid Z\right) \right. \tag {18} \\ \left. - D _ {\mathrm {K L}} \left(q _ {\phi} (Z | X ^ {\varnothing}) \| p (Z)\right) \right] \\ \end{array}
$$
We have the following result on the loss function in Eq. (3).
Proposition 2. Let  $k$  be a Gaussian kernel with width  $\epsilon$ , i.e.,  $k(x,y) = \exp \left(-\frac{\|x - y\|_2^2}{2\epsilon^2}\right)$ . Let  $p_{\theta,I}(X^I \mid U)$  be Gaussian with mean  $\mu_\theta^I (U)$  and a fixed variance  $\sigma^2$ . Then, for  $\epsilon$  sufficiently large, for  $\alpha$  given in the proof, and for some constant  $c$  depending only on  $\sigma$  and data dimension  $d$ ,
$$
\mathbb {E} _ {\mathbb {P} \left(X ^ {\varnothing}, X ^ {I}\right)} \left[ \mathbb {E} _ {q _ {\phi} (Z | X ^ {\varnothing})} \log p _ {\theta , I} \left(X ^ {I} \mid Z\right) \right] \geq - \alpha \cdot \operatorname {M M D} \left(p _ {\theta , I} \left(X ^ {I}\right), \mathbb {P} ^ {I} \left(X ^ {I}\right)\right) + c.
$$
Thus, up to an additive constant,  $\mathcal{L}_{\theta,\phi}^{\alpha,1,0}$  lower bounds the paired-data ELBO in Eq.(18) and by extension the paired-data log-likelihood  $\log \mathbb{P}(X^{\emptyset}, X^{I})$ .
$^{12}$ Here  $\otimes$  denotes the Kronecker product.
Proof. By the choice of a Gaussian distribution for  $p_{\theta,I}(X^I \mid U)$ , we have
$$
\log p _ {\theta , I} \left(X ^ {I} \mid Z\right) = \log p _ {\theta} \left(X ^ {I} \mid U ^ {I} = s _ {I} ^ {\text {f u l l}} (Z)\right) = c - \frac {1}{2 \sigma^ {2}} \| X ^ {I} - \mu_ {\theta} ^ {I} (U) \| _ {2} ^ {2}, \tag {19}
$$
where  $c$  is a constant depending only on  $\sigma$  and data dimension  $d$ . Let  $\{(X_{(i)}^{\emptyset}, X_{(i)}^{I})\}_{i=1}^{m}$  be independent and identically distributed according to  $\mathbb{P}(X^{\emptyset}, X^{I})$ . Then
$$
\begin{array}{l} \mathbb {E} _ {\mathbb {P} \left(X ^ {\emptyset}, X ^ {I}\right)} \left[ \mathbb {E} _ {q _ {\phi} \left(Z | x ^ {(0)}\right)} \left[ \log p _ {\theta , I} \left(X ^ {I} \mid Z\right) \right] \right] \\ = \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing}, X ^ {I})} \left[ \mathbb {E} _ {q _ {\phi} \left(Z _ {i} \mid X _ {i} ^ {\varnothing}\right)} \left[ \frac {1}{m} \sum_ {i = 1} ^ {m} \log p _ {\theta , I} \left(X _ {i} ^ {I} \mid Z _ {i}\right) \right] \right] \\ = c - \frac {1}{2 \sigma^ {2}} \mathbb {E} _ {\mathbb {P} (X ^ {\emptyset}, X ^ {I})} \left[ \mathbb {E} _ {q _ {\phi} (Z _ {(i)} | X _ {(i)} ^ {\emptyset})} \left[ \frac {1}{m} \sum_ {i = 1} ^ {m} \| X _ {(i)} ^ {I} - \mu_ {\theta} ^ {I} (U _ {(i)}) \| _ {2} ^ {2} \right] \right] \\ \end{array}
$$
Now, for the empirical MMD, we have
$$
\begin{array}{l} \widehat {\mathbf {M M D}} ^ {2} \left(\left\{X _ {(i)} ^ {I} \right\} _ {i = 1} ^ {m}, \left\{\hat {X} _ {(i)} ^ {I} \right\} _ {i = 1} ^ {m}\right) \\ = \frac {1}{m (m - 1)} \sum_ {i = 1} ^ {m} \sum_ {j \neq i} \exp \left(- \frac {\| X _ {(i)} ^ {I} - X _ {(j)} ^ {I} \| _ {2} ^ {2}}{2 \epsilon^ {2}}\right) + \frac {1}{m (m - 1)} \sum_ {i = 1} ^ {m} \sum_ {j \neq i} \exp \left(- \frac {\| \hat {X} _ {(i)} ^ {I} - \hat {X} _ {(j)} ^ {I} \| _ {2} ^ {2}}{2 \epsilon^ {2}}\right) \\ - \frac {2}{m ^ {2}} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {m} \exp \left(- \frac {\| X _ {(i)} ^ {I} - \hat {X} _ {(j)} ^ {I} \| _ {2} ^ {2}}{2 \epsilon^ {2}}\right) \\ \end{array}
$$
$$
\begin{array}{l} \geq - \frac {2}{m ^ {2}} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {m} \exp \left(- \frac {\| X _ {(i)} ^ {I} - \hat {X} _ {(j)} ^ {I} \| _ {2} ^ {2}}{2 \epsilon^ {2}}\right) \\ \geq - 2 + \frac {1}{2 m ^ {2} \epsilon^ {2}} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {m} \| X _ {(i)} ^ {I} - \hat {X} _ {(j)} ^ {I} \| _ {2} ^ {2} \\ \geq - 2 + \frac {1}{2 m ^ {2} \epsilon^ {2}} \sum_ {i = 1} ^ {m} \| X _ {(i)} ^ {I} - \hat {X} _ {(i)} ^ {I} \| _ {2} ^ {2}, \\ \end{array}
$$
where we have used the positivity of the exponential function and for the penultimate inequality used the fact that  $\epsilon$  is large enough and that  $e^{-x} \leq 1 - x/2$  for  $x$  sufficiently small. Substituting into (20) yields the theorem, with  $\alpha = \frac{1}{2m\sigma^2\epsilon^2}$ .
E Consistency of Discrepancy-based VAE
We consider Discrepancy-based VAE described in the last section. Suppose the conditions in Theorem 2 is satisfied by the ground-truth model, i.e., it is possible to identify CD-equivalence class in theory.
E.1 CD-Equivalence Class
Theorem 4. Let  $X^{\emptyset}, X^{I_1}, \ldots, X^{I_K}$  be generated as in Section 2. Suppose that Assumptions 1, 2, and 3 hold. Define
$$
M _ {1} = \operatorname {a r g m i n} _ {\theta , \phi} \mathcal {L} _ {\theta , \phi}
$$
$$
M _ {2} = \operatorname {a r g m i n} _ {\theta , \phi \in M _ {1}} | \mathcal {T S} (\mathcal {G} _ {\theta}) |
$$
$$
\hat {\theta}, \hat {\phi} \in \operatorname {a r g m i n} _ {\theta , \phi \in M _ {2}} | \mathcal {G} _ {\theta} |
$$
for  $\mathcal{L}_{\theta, \phi}$  defined in Equation 3. Further, suppose that the VAE prior  $p(Z)$  is equal to the true distribution over  $Z$ , that  $p_{\theta}(X \mid U)$  and  $q_{\phi}(Z \mid X)$  are Dirac distributions. Let  $\langle \hat{U}, \hat{\mathcal{G}}, \hat{I}_1, \dots, \hat{I}_K \rangle$  be the solution induced by  $\hat{\theta}, \hat{\phi}$ .
Then  $\langle \hat{U},\hat{\mathcal{G}},\hat{I}_1,\dots,\hat{I}_K\rangle$  is  $CD$ -equivalent to  $\langle U,\mathcal{G},I_1,\dots,I_K\rangle$
Proof. Note that the parameterization of  $s_j$ ,  $\mathcal{G}$ , and the induced distributions of  $U$  through prior  $p(Z)$  using Eq. (16), (17) satisfy (2),(3) and (5) in Remark 2.
The first two terms in combined in Eq. (18) satisfy
$$
\begin{array}{l} \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing})} \left[ \mathbb {E} _ {q _ {\phi} (Z | X ^ {\varnothing})} \log p _ {\theta , \varnothing} (X ^ {\varnothing} | Z) - D _ {K L} \left(q _ {\phi} (Z | X ^ {\varnothing}) \| p (Z)\right) \right] \\ = \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing})} \left[ \log p _ {\theta , \varnothing} (X ^ {\varnothing}) - D _ {K L} \left(q _ {\phi} (Z | X ^ {\varnothing}) \| p _ {\theta , \varnothing} (Z | X ^ {\varnothing})\right) \right] \\ \leq \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing})} \log p _ {\theta , \varnothing} (X ^ {\varnothing}) \\ = \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing})} \log \mathbb {P} \left(X ^ {\varnothing}\right) - D _ {K L} \left(p _ {\theta , \varnothing} \left(X ^ {\varnothing}\right) \| \mathbb {P} \left(X ^ {\varnothing}\right)\right) \\ \leq \mathbb {E} _ {\mathbb {P} (X ^ {\varnothing})} \log \mathbb {P} (X ^ {\varnothing}), \\ \end{array}
$$
where the equality holds if and only if  $q_{\phi}(Z|X^{\emptyset}) = p_{\theta, \emptyset}(Z|X^{\emptyset})$  and  $p_{\theta, \emptyset}(X^{\emptyset}) = \mathbb{P}(X^{\emptyset})$ . On the other hand, since  $\mathrm{MMD}(\cdot, \cdot)$  is a valid measure between distributions, we have
$$
- \operatorname {M M D} \left(\mathbb {P} _ {\theta , \phi} \left(\hat {X} ^ {\hat {I} _ {k}}\right), \mathbb {P} \left(X ^ {I _ {k}}\right)\right) \leq 0,
$$
where the inequality is satisfied with equality if and only if  $\hat{X}^{\hat{I}_k}$  and  $X^{I_k}$  are equal in distribution.
Therefore if the learned intervention targets of  $I_{1}, \dots, I_{K}$  cover  $[\hat{p}]$  and the minimum loss function is not larger than for  $\hat{p} = K$ , we have the solution satisfy (1)-(5) in Remark 2. Since  $\mathcal{G}$  has the sparsest transitive closure and  $\mathcal{G}$  is the sparsest with this transitive closure, (6)-(7) in Remark 2 are also satisfied. Therefore Remark 2 guarantees the smallest  $\hat{p} \leq K$  satisfying the conditions recovers the CD-equivalence class.
Note that in practice, it can be hard to ensure that the gradient-based approach returns a DAG  $\mathcal{G}$  that has the sparsest transitive closure and is simultaneously the sparsest DAG with this transitive closure. We instead search for sparser DAGs  $\mathcal{G}$  by penalizing its corresponding adjacency in Eq. (3).
E.2 Consistency for Multi-Node Interventions
Theorem 3 guarantees that in an SCM with additive noises where interventions modify the exogenous noises, if the CD equivalence can be identified, we can extrapolate to unseen combinations of interventions with different intervention targets. In fact, for certain types of interventions, extrapolation to unseen combinations of any interventions is possible. We illustrate this for shift interventions in an SCM with additive Gaussian noises, where an intervention changes the mean of the exogenous noise variable.
For single-node intervention  $I$ , let  $a^I$  denote the corresponding changes in the mean of the exogenous noise variables, i.e.,
$$
a _ {i} ^ {I} = \left\{ \begin{array}{l l} \mathbb {E} (\epsilon_ {i} ^ {I}) - \mathbb {E} (\epsilon_ {i}), & i \in T (I), \\ 0, & i \notin T (I). \end{array} \right.
$$
We encode it as  $\hat{I}$  with  $T(\hat{I})$  containing one element and  $\hat{\mathbf{a}}^{\hat{I}}$  being a one-hot vector, where
$$
\hat {a} _ {i} ^ {\hat {I}} = \left\{ \begin{array}{l l} \hat {a} _ {i}, & i \in T (\hat {I}), \\ 0, & i \notin T (\hat {I}). \end{array} \right.
$$
We extend this notation for  $I$  with potentially multiple intervention targets (i.e., sets  $I, \hat{I}$  that contain multiple elements) where  $\mathbf{a}^I, \hat{\mathbf{a}}^{\hat{I}}$  can be a multi-hot vector.
In the shift intervention case, from Theorem 3, we know that the encoded  $\hat{\mathbf{a}}^{\hat{I}_1},\dots,\hat{\mathbf{a}}^{\hat{I}_K}$  satisfy  $\hat{\mathbf{a}}^{\hat{I}_k} = M(\mathbf{a}^{I_k})$  in the limit of infinite data, where  $M$  is a linear operation with  $M(\mathbf{a})_i = \Upsilon_{\tau (i),i}a_{\tau (i)}$ . Thus for single-node interventions  $I_{t(1)},\ldots ,I_{t(k)}$  amongst  $I_{1},\ldots ,I_{K}$ , the multi-node intervention  $\mathcal{I} = I_{t(1)}\cup \ldots \cup I_{t(k)}$  corresponds multi-hot vector  $\mathbf{a}^{\mathcal{I}}$  that satisfies  $M(\mathbf{a}^{\mathcal{I}}) = M(\mathbf{a}^{I_{t(1)}} + \ldots +$
<sup>13</sup>Note that we allow overlapping intervention targets among  $I_{t(1)}, \ldots, I_{t(k)}$ , where  $I_{t(1)} \cup \ldots \cup I_{t(k)}$  adds up all the shift values for intervention target  $i$ .
$\mathbf{a}^{I_{t(k)}}) = \hat{\mathbf{a}}^{\hat{I}_{t(1)}} + \ldots +\hat{\mathbf{a}}^{\hat{I}_{t(k)}}$  . Thus if we encode  $\mathcal{L}$  as  $\hat{\mathbf{a}}^{\hat{\mathcal{I}}}:= \hat{\mathbf{a}}^{\hat{I}_{t(1)}} + \ldots +\hat{\mathbf{a}}^{\hat{I}_{t(k)}}$  , we can also generate  $\hat{X}^{\hat{\mathcal{I}}}$  from the ground-truth distribution of  $X = f(U)$  where  $U\sim \mathbb{P}_U^\mathcal{I}(U)$  following the encoding-decoding process of Fig. 4.
F Discrepancy-based VAE Implementation Details
We summarize our hyperparameters in Table 2. Below, we describe where they are used in more detail. We use a linear structural equation with shift interventions. In practice, due to the nonlinear encoding from the latent  $U$  to observed  $X$ , not much expressive power is lost. Code for our method is at https://github.com/uhlerlab/discrepancy_vae.
Loss function
Kernel width (MMD)	200
Number of kernels (MMD)	10
λ	0.1
βmax	1
αmax	1
Training
tmax	100
Learning rate	0.001
Batch size	32
Table 2: Hyper-Parameters
VAE Parameterization. As is standard with VAEs, our encoder and decoder are parameterized as neural networks, and the exogenous variables are described via the reparameterization trick. We use a standard isotropic normal prior for  $p(Z)$ . To encode interventions, the function  $T_{\phi}(\cdot)$  is parameterized as a fully connected neural network, where for differentiable training  $T_{\phi}(C)$  is encoded as a one-hot vector via a softmax function, i.e.,  $T_{\phi}(C)_i = \frac{\exp(tT_{\phi}'(C)_i)}{\sum_{j=1}^p \exp(tT_{\phi}'(C)_j)}$  for some fully connected  $T_{\phi}'$  and temperature  $t > 0$ . During training, we adopt an annealing temperature for  $t$ . In particular,  $t = 1$  until half of the epochs elapse, and  $t$  is linearly increased to  $t_{\max}$  over the remaining epochs. At test time, the temperature of the softmax is set to a large value, recovering a close-to-true one-hot encoding.
Loss Functions. We use a mixture of MMD discrepancies, each with a Gaussian kernel with widths that are dyadically spaced [18]. This helps prevent numerical issues and vanishing gradient issues in training. The coefficient  $\alpha$  of the discrepancy loss term  $\mathcal{L}_{\theta,\phi}^{\mathrm{discrep}}$  is given the following schedule:  $\alpha = 0$  for the first 5 epochs, then  $\alpha$  is linearly increased to  $\alpha_{\mathrm{max}}$  until half of the epochs elapse, at which point it remains at  $\alpha_{\mathrm{max}}$  for the rest of training. Similarly, the coefficient  $\beta$  of the KL regularization term is given the following schedule:  $\beta = 0$  for the first 10 epochs, then  $\beta$  is linearly increased to  $\beta_{\mathrm{max}}$  until half of the epochs elapse, at which point it remains at  $\beta_{\mathrm{max}}$  for the rest of training.
Optimization. We train using the Adam optimizer, with the default parameters from PyTorch and a learning rate of 0.001.
Biological Data. For the experiments described in Section 6, the encoder  $q_{\phi}$  was implemented as a 2-layer fully connected network with leaky ReLU activations and 128 hidden units. The intervention encoder  $T_{\phi}$  uses 128 hidden units. To account for interventions with less samples, we use a batch size of 32. We train for 100 epochs in total, which takes less than 45 minutes on a single GPU.
G Extended Results on Biological Dataset