SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If  $\tau(j) \in \mathrm{de}_{\mathcal{G}}(\tau(k_0))$ , then since  $\tau(k) \in \overline{\mathrm{de}}_{\mathcal{G}}(\tau(k_0))$ , we have  $\tau(k_0) \in [p] \setminus \mathrm{de}_{\mathcal{G}}(\tau(k))$  (illustrated in Figure 8B). Then we have  $S \subset [p] \setminus \overline{\mathrm{de}}_{\mathcal{G}}(\tau(k_0)) \subset [p] \setminus \mathrm{de}_{\mathcal{G}}(\tau(k))$ , and thus  $S \cup \{\tau(k_0)\} \subset [p] \setminus \mathrm{de}_{\mathcal{G}}(\tau(k))$ . In fact,  $S \cup \{\tau(k_0)\}$  is a subset of  $[p] \setminus (\mathrm{de}_{\mathcal{G}}(\tau(k)) \cup \{\tau(j)\})$ , since  $\tau(j) \in \mathrm{de}_{\mathcal{G}}(\tau(k))$  by  $\tau(k) \in \mathrm{pa}_{\mathcal{G}}(\tau(i))$  (definition of  $k$ ). Therefore  $\tilde{U}_j$  can be written as a linear combination of  $U_{\tau(j)}$  and  $U_{S'}$  for  $S' \subset [p] \setminus (\mathrm{de}_{\mathcal{G}}(\tau(k)) \cup \{\tau(j)\})$  with nonzero coefficient for  $U_{\tau(j)}$ . Since  $\tau(j) \in \mathrm{mch}_{\mathcal{G}}(\tau(k))$  (definition of  $k$ ), by Assumption 2, we have  $\mathbb{P}(\tilde{U}_j) \neq \mathbb{P}^{I_{\rho_k}}(\tilde{U}_j)$ , as  $I_{\rho_k}$  targets  $U_{\tau(k)}$ .
(Case 2): In this case  $k_0 = i$  (illustrated in Figure 8C). Then for any  $l < j$  such that  $\tau(l) \in \mathrm{de}_{\mathcal{G}}(\tau(k))$ , the coefficient of  $U_{\tau(l)}$  in  $\tilde{U}_j$  is zero. Otherwise since  $\mathrm{de}_{\mathcal{G}}(\tau(k)) \subset \mathrm{de}_{\mathcal{G}}(\tau(k_0)) = \mathrm{de}_{\mathcal{G}}(\tau(i))$ , it holds that  $\tau(l) \in \mathrm{de}_{\mathcal{G}}(\tau(i))$ , which by Eq. (11) implies  $i < l < j$ . Thus  $l$  satisfies Case 1, a contradiction. Therefore, also by Eq. (11),  $\tilde{U}_i$  can be written as a linear combination of  $U_{\tau(j)}$  and  $U_S$  with nonzero coefficient of  $U_{\tau(j)}$ , where  $S \subset [p] \setminus \mathrm{de}_{\mathcal{G}}(\tau(k))$ .
Since  $\tau(j) \in \mathrm{de}_{\mathcal{G}}(\tau(i)) = \mathrm{de}_{\mathcal{G}}(\tau(k_0))$ , by definition of  $k$ , we have  $\tau(j) \in \mathrm{mch}_{\mathcal{G}}(\tau(k))$ . Note that  $\tilde{U}_i$  can be written as a linear combination of  $U_{\tau(j)}$  and  $U_{S'}$  with nonzero coefficient of  $U_{\tau(j)}$ , where  $S' \subset [p] \setminus (\mathrm{de}_{\mathcal{G}}(k) \cup \{\tau(j)\})$ . By Assumption 2,  $\mathbb{P}(\tilde{U}_j) \neq \mathbb{P}^{I_{\rho_k}}(\tilde{U}_j)$ , as  $I_{\rho_k}$  targets  $U_{\tau(k)}$ .
Therefore, in both cases it holds that  $\mathbb{P}(\tilde{U}_j) \neq \mathbb{P}^{I_{\rho_k}}(\tilde{U}_j)$ . Since  $i \leq k < j$  and  $\tau(k) \in \overline{\mathrm{de}}_G(\tau(i))$ , the claim is proven.
Now, we use marginal changes to define a graph associated to any topologically-ordered representation. We use Proposition 1 to show that picking the topologically-ordered representation which yields the sparsest graph will recover the transitive closure of  $\mathcal{G}$ .
Lemma 7. Let  $\tilde{U}$  be a topological representation of  $X$  with intervention targets  $\rho_{1},\ldots,\rho_{p}$ . Let  $\hat{R}^{\prime}\in \mathbb{R}^{p\times p}$  be an invertible upper triangular and let  $\hat{U} = \tilde{U}\hat{R}^{\prime}$ . Define the following:
- Let  $\hat{\mathcal{G}}_0^{\hat{R}'}$  be the DAG such that  $i \to j \in \hat{\mathcal{G}}_0$  if and only if  $i < j \in [p]$  and  $\mathbb{P}(\hat{U}_j) \neq \mathbb{P}^{I_{\rho_i}}(\hat{U}_j)$ .
Let  $\hat{\mathcal{G}}^{\hat{R}'} = \mathcal{T}\mathcal{S}(\hat{\mathcal{G}}_0^{R'})$ .
Let  $\hat{R}$  be such that  $\hat{\mathcal{G}}^{\hat{R}}$  has the fewest edges over any choice of  $\hat{R}'$ . Then  $\hat{\mathcal{G}}^{\hat{R}} = \mathcal{T}S(\mathcal{G}_{\tau})$ . We call  $\hat{U}$  a sparsest topological representation of  $X$ .
Direction 1.
We first show that for any  $\hat{R}^{\prime}$
$$
\mathcal {T} \mathcal {S} \left(\mathcal {G} _ {\tau}\right) \subseteq \hat {\mathcal {G}} ^ {\hat {R} ^ {\prime}}. \tag {10}
$$
Let  $i \to j \in \mathcal{T}S(\mathcal{G}_{\tau})$ , so  $i < j$ . By Proposition 1, we have  $k$  such that  $i \leq k < j$  with  $\tau(k) \in \overline{\operatorname{de}}_{\mathcal{G}}(\tau(i))$ . By definition of  $\hat{\mathcal{G}}_0^{\hat{R}'}$ , we have  $k \to j \in \hat{\mathcal{G}}_0^{\hat{R}'}$ . Repeating this argument iteratively, we obtain a directed path from  $i$  to  $j$  in  $\hat{\mathcal{G}}_0^{\hat{R}'}$ . Thus, by definition of  $\hat{\mathcal{G}}^{\hat{R}'}$ , we have  $\mathcal{T}\mathcal{S}(\mathcal{G}_{\tau}) \subseteq \hat{\mathcal{G}}^{\hat{R}'}$ .
Direction 2.
Now we give an example of  $\hat{R}$  such that the constructed  $\hat{\mathcal{G}}^{\hat{R}}$  satisfies
$$
\hat {\mathcal {G}} ^ {\hat {R}} \subseteq \mathcal {T} \mathcal {S} (\mathcal {G} _ {\tau}).
$$
Denote  $\tilde{\Gamma}\hat{R} = \hat{\Gamma}$  and  $\hat{c} = \tilde{c}\hat{R}$ . Since  $\hat{R}$  is upper-triangular and invertible, by Eq. (9), we have  $\hat{U} = U\hat{\Gamma} + \hat{c}$ , where
$$
i <   j \Rightarrow \hat {\Gamma} _ {\tau (j), i} = 0 \quad \text {a n d} \quad \hat {\Gamma} _ {\tau (j), j} \neq 0, \tag {11}
$$
where  $\tau$  is the topological order in Eq. (9). By Eq. (11), there exists an invertible upper-triangular matrix  $R\in \mathbb{R}^{p\times p}$  such that  $\hat{U} = (U\hat{\Gamma} +\tilde{c})R = (U_{\tau (1)},\dots,U_{\tau (p)}) + c$  for some constant vector  $c$ . Now for  $i < j\in [p]$ , we have  $i\rightarrow j\in \hat{\mathcal{G}}_0^R\Leftrightarrow \mathbb{P}(U_{\tau (j)})\neq \mathbb{P}^{I_{\rho_i}}(U_{\tau (j)})$ . Since  $I_{\rho_i}$  targets  $U_{\tau (i)}$ , this would only be true when  $\tau (j)\in \mathrm{de}_{\mathcal{G}}(\tau (i))$ . Therefore  $i\rightarrow j\in \hat{\mathcal{G}}_0^{\hat{R}}\Rightarrow \tau (j)\in \mathrm{de}_{\mathcal{G}}(\tau (i))$ . Thus  $\hat{\mathcal{G}}_0^{\hat{R}}\subseteq \mathcal{T}\mathcal{S}(\mathcal{G}_\tau)$ . As  $\mathcal{T}\mathcal{S}(\mathcal{G}_\tau)$  is a transitive closure, this means  $\hat{\mathcal{G}}^{\hat{R}} = \mathcal{T}\mathcal{S}(\hat{\mathcal{G}}_0^{\hat{R}})\subseteq \mathcal{T}\mathcal{S}(\mathcal{G}_\tau)$ .
Analogously to Lemma 6, the following result shows that a sparsest topological representation is topologically consistent with  $U$  in a stronger sense than a topological representation.
Lemma 8. Let Assumption 2 hold. For  $\tilde{\Gamma} \in \mathbb{R}^{p \times p}$  and  $\tilde{c} \in \mathbb{R}^p$ , let  $\tilde{U} = U\tilde{\Gamma} + \tilde{c}$  be a sparsest topological representation of  $U$  with intervention targets  $\rho_1, \ldots, \rho_p$ . Let  $\tau$  be a topological ordering of  $\mathcal{G}$  such that
$$
i <   j \Rightarrow \tilde {\Gamma} _ {\tau (j), i} = 0 \quad \text {a n d} \quad \tilde {\Gamma} _ {\tau (j), j} \neq 0. \tag {12}
$$
Then  $\tilde{\Gamma}_{\tau(j),l} = 0$  for  $\tau(l) \notin \mathrm{deg}_{\mathcal{G}}(\tau(j))$ .
Proof. For sake of contradiction, let  $l,j \in [p]$ . Without loss of generality, let  $j$  be the largest value for which  $\tilde{\Gamma}_{\tau(j),l} \neq 0$  and  $\tau(l) \notin \mathrm{de}_{\mathcal{G}}(\tau(j))$ . By transitivity of  $\mathrm{de}_{\mathcal{G}}$  and the choice of  $j$  as the largest value, there is no  $j'$  such that  $\tau(j') \in \mathrm{de}_{\mathcal{G}}(\tau(j))$  and  $\tilde{\Gamma}_{\tau(j'),l} \neq 0$ . Therefore  $\tilde{U}_l$  can be written as a linear combination of  $U_{\tau(j)}$  and  $U_S$  with nonzero coefficient of  $U_{\tau(j)}$ , where  $S \subset [p] \setminus \overline{\mathrm{de}}_{\mathcal{G}}(\tau(j))$ .
By Assumption 2, we have  $\mathbb{P}(\tilde{U}_l) \neq \mathbb{P}^{I_{\rho_j}}(\tilde{U}_l)$ . Since  $\tau(l) \notin \mathrm{de}_{\mathcal{G}}(\tau(j))$  and  $\hat{\mathcal{G}} = \mathcal{T}\mathcal{S}(\mathcal{G}_{\tau})$ , we have  $j \to l \notin \hat{\mathcal{G}}$ , in which case  $\mathbb{P}(\tilde{U}_l) \neq \mathbb{P}^{I_{\rho_{j'}}}(\tilde{U}_l)$  violates Condition 1, a contradiction.
B.3.3 Proof of Theorem 1
Theorem 1. Under Assumption 1 and Assumption 2 for  $I_1, \ldots, I_K$ , we can identify  $\langle \hat{\mathcal{G}}, \hat{I}_1, \dots, \hat{I}_K \rangle$  where  $\hat{\mathcal{G}} = \mathcal{T}S(\mathcal{G}_{\pi})$ , and  $\hat{I}_k = (I_k)_\pi$  for some permutation  $\pi$ .
Here, we combine the results of the previous two sections to show that we can recover  $\mathcal{G}$  up to transitive closure and permutation, and that we recover the intervention targets  $I_{1},\ldots ,I_{K}$  up to the same permutation.
Proof. By Lemma 1 and Remark 1, we can assume, without loss of generality, that  $p$  is known and that  $X = f(U) = U\Lambda + b$  for some non-singular matrix  $\Lambda$ , as this can be identified from observational data  $\mathcal{D}$ .
By Lemma 7, we can identify a topological representation  $\hat{U} = U\hat{\Gamma} +\hat{c}$  with intervention targets  $\rho_{1},\ldots ,\rho_{p}\in [K]$ , where  $\hat{\Gamma}\in \mathbb{R}^{p\times p}$  and  $\hat{c}\in \mathbb{R}^p$ . Further, for some unknown topological ordering  $\tau$  of  $\mathcal{G}$ ,  $\hat{\Gamma}$  satisfies Eq. (11),  $T(I_{\rho_i}) = \tau (i)$  for  $i\in [p]$ , and we identify  $\hat{\mathcal{G}} = \mathcal{T}\mathcal{S}(\mathcal{G}_{\tau})$ .
Identifying additional intervention targets.
So far, we only guarantee that we identify the intervention targets for  $I_{\rho_1}, \ldots, I_{\rho_p}$ . Now, consider any  $k \in [K] \setminus \{\rho_1, \ldots, \rho_p\}$ . Let  $l$  be such that  $T_{\mathcal{G}}(I_k) = \tau(l)$ . We now argue that  $l$  can be identified as the smallest  $l'$  in  $[p]$  such that  $\mathbb{P}(\hat{U}_{l'}) \neq \mathbb{P}^{I_k}(\hat{U}_{l'})$ .
By Assumption 2, we have  $\mathbb{P}(\hat{U}_l) \neq \mathbb{P}^{I_k}(\hat{U}_l)$ , since  $I_k$  targets  $U_{\tau(l)}$  and  $\hat{U}_l$  can be written as a linear combination of  $U_{\tau(1)}, \ldots, U_{\tau(l)}$  with nonzero coefficient  $U_{\tau(l)}$  (note that  $U_{\tau(1)}, \ldots, U_{\tau(l-1)} \in [p] \subset \overline{\mathrm{de}}_{\mathcal{G}}(\tau(l)))$ .
On the other hand, for  $l' < l$ , we have  $\mathbb{P}(\hat{U}_l) = \mathbb{P}^{I_k}(\hat{U}_l)$ , since  $\hat{U}_{l'}$  can be written as a linear combination of  $U_{\tau(1)}, \dots, U_{\tau(l')}$  and  $\tau$  is the topological order.
In this section, we show that by introducing Assumption 3, we can go beyond recovering the transitive closure of  $\mathcal{G}$ , and we instead recover  $\mathcal{G}$ . We begin by establishing a basic fact about conditional independences in our setup.
Claim 1. Under Assumption 1, let  $\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}$  denote (potentially linear combinations of) components of  $U$ , and assume that  $\mathbf{A} \perp \mathbf{B} \mid \mathbf{C}, \mathbf{D}$  and  $\mathbf{A} \perp \mathbf{C} \mid \mathbf{B}, \mathbf{D}$ . Then  $\mathbf{A} \perp \mathbf{B} \mid \mathbf{D}$ .
Proof. By Assumption 1,  $\mathbb{P}_{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}}$  has positive measure on some full-dimensional set. By Proposition 2.1 of [56],  $\mathbb{P}_{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}}$  is a graphoid, i.e., it obeys the intersection property. Invoking this property, we obtain  $\mathbf{A} \perp \mathbf{B} \mid \mathbf{D}$ , as desired.
With this, we are ready to prove Theorem 2, which we recall here.
Note that, from Theorem 1, we have already identified the interventions  $I_1, \ldots, I_K$  up to CD-equivalence for a permutation  $\tau$ . Thus, the only remaining result to show is that we identify  $\mathcal{G}$  up to the same permutation.
In particular, we can again characterize the solution in terms of the sparsest solution.
Theorem 2, Constructive. Let  $\hat{U}$  be a sparsest topological representation of  $X$  with intervention targets  $\rho_{1},\ldots ,\rho_{p}$ . Let  $\bar{R}^{\prime}\in \mathbb{R}^{p\times p}$  be an invertible upper triangular matrix, and let  $\bar{U} = \hat{U}\bar{R}^{\prime}$ . Define the following:
- Let  $\bar{\mathcal{G}}^{\bar{R}'}$  be the DAG such that  $i \to j$  for  $i < j \in [p]$ , if and only if  $\bar{U}_i \not\sqcup \bar{U}_j \mid \bar{U}_1, \ldots, \bar{U}_{i-1}, \bar{U}_{i+1}, \ldots, \bar{U}_{j-1}$
Let  $\bar{R}$  be such that  $\bar{\mathcal{G}}^{\bar{R}}$  has the fewest edges over any choice of  $\bar{R}'$ . Then  $\bar{\mathcal{G}}^{\bar{R}} = \mathcal{G}_{\tau}$  for  $\tau$  satisfying Eq. (12).
Proof. By Lemma 7, we have  $\hat{U} = U\hat{\Gamma}$  for some matrix  $\hat{\Gamma} \in \mathbb{R}^{p \times p}$  satisfying Eq. (9) under some topological order  $\tau$  of  $\mathcal{G}$ . Further, we identify  $\hat{\mathcal{G}} = \mathcal{T}\mathcal{S}(\mathcal{G}_{\tau})$ .
Denoting  $\hat{\Gamma}\bar{R} = \bar{\Gamma}$  and  $\bar{c} = \hat{c}\bar{R}$ , by Lemma 8, we have  $\bar{U} = U\bar{\Gamma} +\bar{c}$  with
$$
i <   j \Rightarrow \bar {\Gamma} _ {\tau (j), i} = 0 \quad \text {a n d} \quad \bar {\Gamma} _ {\tau (j), j} \neq 0, \tag {13}
$$
$$
\tau (l) \notin \mathrm {d e} _ {\mathcal {G}} (\tau (j)) \Rightarrow \bar {\Gamma} _ {\tau (j), l} = 0.
$$
First, we show that
$$
\mathcal {G} _ {\tau} \subseteq \bar {\mathcal {G}} ^ {\bar {R}}.
$$
Assume on the contrary that there exists  $\tau(i) \to \tau(j) \in \mathcal{G}$  such that  $i \to j \notin \bar{\mathcal{G}}^{\bar{R}}$ . By definition, we have  $\bar{U}_i \perp \bar{U}_j \mid \bar{U}_1, \dots, \bar{U}_{i-1}, \bar{U}_{i+1}, \dots, \bar{U}_{j-1}$ . By Eq. (13), we know that we can retrieve  $U_{\tau(1)}, \dots, U_{\tau(i-1)}$  by linearly transforming  $\bar{U}_1, \dots, \bar{U}_{i-1}$ ; this implies  $U_{\tau(i)} \perp \bar{U}_j \mid U_{\tau(1)}, \dots, U_{\tau(i-1)}, \bar{U}_{i+1}, \dots, \bar{U}_{j-1}$ . By subtracting terms in  $U_{\tau(1)}, \dots, U_{\tau(i-1)}$  from  $\bar{U}_{i+1}, \dots, \bar{U}_j$  and then subtracting terms  $\bar{U}_l$  from  $\bar{U}_{l+1}, \dots, \bar{U}_j$  for  $l = i+1, \dots, j-1$ , we have that
$$
U _ {\tau (i)} \perp U _ {\tau (j)} + c _ {j} U _ {\tau (i)} \mid U _ {\tau (1)}, \dots , U _ {\tau (i - 1)}, \tag {14}
$$
$$
U _ {\tau (i + 1)} + c _ {i + 1} U _ {\tau (i)}, \dots , U _ {\tau (j - 1)} + c _ {j - 1} U _ {\tau (i)},.
$$
for some  $c_{i+1}, \ldots, c_j \in \mathbb{R}$ . Since by Eq. (13) there is  $\bar{\Gamma}_{\tau(i),l} = 0$  for any  $\tau(l) \notin \mathrm{de}_{\mathcal{G}}(\tau(i))$ , this subtraction gives us  $c_l = 0$  if  $\tau(l) \notin \mathrm{de}_{\mathcal{G}}(\tau(i))$ .
Therefore let
$$
\mathbf {A} = U _ {\tau (j)} + c _ {j} U _ {\tau (i)},
$$
$$
\mathbf {B} = U _ {\tau (i)}
$$
$$
\mathbf {C} = \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {l \leq j - 1, \tau (l) \notin \mathrm {p a} _ {\mathcal {G}} (\tau (j))}, \text {a n d}
$$
$$
\mathbf {D} = \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {\tau (l) \in \mathrm {p a} _ {\mathcal {G}} (\tau (j)) \backslash \{\tau (i) \}},
$$
where  $c_{1} = \ldots = c_{i - 1} = 0$  . There is  $\mathbf{A}\perp \mathbf{B}\mid \mathbf{C},\mathbf{D}$
On the other hand, since  $\tau(i) \to \tau(j) \in \mathcal{G}$ , i.e.,  $\tau(i) \in \mathrm{pa}_{\mathcal{G}}(\tau(j))$ . We will now show that this implies  $\mathbf{A} \perp \mathbf{C} \mid \mathbf{B}, \mathbf{D}$ . Starting with the local Markov property, we have for any  $c_1, \ldots, c_{i-1}, c_{i+1}, \ldots, c_j$  that
$$
\begin{array}{l} U _ {\tau (j)} \perp \left\{U _ {\tau (l)} \right\} _ {l \leq j - 1, \tau (l) \notin \mathrm {p a} _ {\mathcal {G}} (\tau (j))} \mid \left\{U _ {\tau (l)} \right\} _ {\tau (l) \in \mathrm {p a} _ {\mathcal {G}} (\tau (j))} \\ \Rightarrow \quad U _ {\tau (j)} + c _ {j} U _ {\tau (i)} \perp \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {l \leq j - 1, \tau (l) \notin \mathrm {p a} _ {\mathcal {G}} (\tau (j))} \mid \left\{U _ {\tau (l)} \right\} _ {\tau (l) \in \mathrm {p a} _ {\mathcal {G}} (\tau (j))} \tag {15} \\ \Rightarrow \quad U _ {\tau (j)} + c _ {j} U _ {\tau (i)} \perp \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {l \leq j - 1, \tau (l) \notin \operatorname {p a} _ {\mathcal {G}} (\tau (j))} \\ \mid U _ {\tau (i)}, \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {\tau (l) \in \operatorname {p a} _ {\mathcal {G}} (\tau (j)) \backslash \{\tau (i) \}} \\ \end{array}
$$
where the first implication follows from the definition of conditional independence, and the second implication follows since  $\{U_{\tau(l)}\}_{\tau(l) \in \mathrm{pa}_{\mathcal{G}}(\tau(j))}$  is a deterministic function of  $U_{\tau(i)}, \{U_{\tau(l)} + c_l U_{\tau(i)}\}_{\tau(l) \in \mathrm{pa}_{\mathcal{G}}(\tau(j)) \setminus \{\tau(i)\}}$ .
Thus, by Claim 1, if  $i\to j\notin \bar{\mathcal{G}}^{R}$  and  $\tau (i)\rightarrow \tau (j)\in \mathcal{G}$ , then  $\mathbf{A}\perp \mathbf{B}\mid \mathbf{D}$ , i.e.,
$$
U _ {\tau (i)} \perp U _ {\tau (j)} + c _ {j} U _ {\tau (i)} \mid \left\{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \right\} _ {l \in \operatorname {p a} _ {\mathcal {G}} (\tau (j)) \backslash \{\tau (i) \}}.
$$
Since  $c_{l} = 0$  for any  $\tau (l)\notin \mathrm{de}_{\mathcal{G}}(\tau (i))$  and  $\tau$  is the topological order, this can be further written as
$$
U _ {\tau (i)} \perp U _ {\tau (j)} + c _ {j} U _ {\tau (i)} \mid \{U _ {\tau (l)} \} _ {l \in \operatorname {p a} _ {\mathcal {G}} (\tau (j)) \backslash (S \cup \{\tau (i) \})}, \{U _ {\tau (l)} + c _ {l} U _ {\tau (i)} \} _ {l \in S},
$$
where  $S = \mathrm{pa}_{\mathcal{G}}(\tau(j)) \cap \mathrm{de}_{\mathcal{G}}(\tau(i))$ , which violates Assumption 3. Therefore we must have  $\mathcal{G}_{\tau} \subseteq \bar{\mathcal{G}}^{\bar{R}}$ . Direction 2.
There exists an invertible upper-triangular matrix  $\bar{R} \in \mathbb{R}^{p \times p}$  such that  $\bar{U} = \hat{U}\bar{R} = (U_{\tau(1)}, \dots, U_{\tau(p)}) + \bar{c}$  for some constant vector  $\bar{c}$ . Note that clearly  $\bar{U}$  satisfies Condition 1. Also for  $i < j \in [p]$  such that  $\tau(i) \to \tau(j) \notin \mathcal{G}$ , by the Markov property and  $\tau$  being the topological order, we have  $\bar{U}_i \perp \bar{U}_j \mid \bar{U}_1, \dots, \bar{U}_{i-1}, \bar{U}_{i+1}, \dots, \bar{U}_{j-1}$ . Thus  $\tau(i) \to \tau(j) \notin \mathcal{G} \Rightarrow i \to j \notin \bar{\mathcal{G}}$ , and hence  $\bar{\mathcal{G}} \subseteq \mathcal{G}_{\tau}$ , which completes the proof.
Remark 2. These proofs (Lemma 1, Theorem 1,2) together indicate that under Assumptions 1,2,3, we can identify  $\langle \mathcal{G}, I_1, \dots, I_K \rangle$  up to its CD-equivalence class by solving for the smallest  $\hat{p}$ , an encoder  $\hat{g}: \mathbb{R}^n \to \mathbb{R}^{\hat{p}}$ ,  $\hat{\mathcal{G}}$  and  $\hat{I}_1, \dots, \hat{I}_{\hat{K}}$  that satisfy
(1) there exists a full row rank polynomial decoder  $\hat{f}(\cdot)$  such that  $\hat{f} \circ \hat{g}(X) = X$  for all  $X \in \mathcal{D} \cup \mathcal{D}^{I_1} \cup \ldots \cup \mathcal{D}^{I_K}$ ;
(2) the induced distribution on  $\hat{U} \coloneqq \hat{g}(X)$  by  $X \in \mathcal{D}$  factorizes with respect to  $\hat{\mathcal{G}}$ ;
(3) the induced distribution on  $\hat{U}$  by  $X\in \mathcal{D}^{I_k}$  where  $k\in [K]$  changes the distribution of  $\hat{U}_{T_{\hat{\mathcal{G}}}(\hat{I}_k)}$  but does not change the joint distribution of non-descendants of  $\hat{U}_{T_{\hat{\mathcal{G}}}(\hat{I}_k)}$  in  $\hat{\mathcal{G}}$ ;
(4)  $[\hat{p}]\subseteq T_{\hat{\mathcal{G}}}(\hat{I}_1)\cup \ldots \cup T_{\hat{\mathcal{G}}}(\hat{I}_{\hat{K}});$
(5)  $\hat{\mathcal{G}}$  has topological order  $1, \dots, \hat{p}$ ;
(6) the transitive closure  $\mathcal{T}\mathcal{S}(\hat{\mathcal{G}})$  of the DAG  $\hat{\mathcal{G}}$  is the sparsest amongst all solutions that satisfy (1)-(5);
(7) the DAG  $\hat{\mathcal{G}}$  is the sparsest amongst all solutions that satisfy (1)-(6);
We will use these observations in Appendix E to develop a discrepancy-based VAE and show that it is consistent in the limit of infinite data.
Proof. We first show that there is a solution to (1)-(7). For this, it suffices to show that there is a solution to (1)-(5). Then since  $\hat{p}$  and  $\hat{\mathcal{G}}$  are discrete, one can find the solution to (1)-(7) by searching amongst all solutions to (1)-(5) such that  $\hat{p}$  is the smallest and (6)-(7) are satisfied. Assume without loss of generality that  $\mathcal{G}$  has topological order  $1,\dots,p$ . Then  $\hat{p} = p$ ,  $\hat{g} = f^{-1}$ ,  $\hat{\mathcal{G}} = \mathcal{G}$ , and  $\hat{I}_k = I_k$  for  $k\in [K]$  satisfy (1)-(5).
Next we show that any solution must recover  $\hat{p} = p$  and  $\langle \hat{\mathcal{G}},\hat{I}_1,\dots ,\hat{I}_K\rangle$  that is in the same CD equivalence class as  $\langle \mathcal{G},I_1,\ldots ,I_K\rangle$ . Since we solve for the smallest  $\hat{p}$ , the former paragraph also implies that  $\hat{p}\leq p$ . By the proof of Lemma 1, (1) guarantees that  $\hat{p}\geq p$ . Therefore it must hold that  $\hat{p} = p$ .
Since we solve for the sparsest transitive closure, the first paragraph implies that  $\mathcal{T}S(\hat{\mathcal{G}}) \subset \mathcal{T}S(\mathcal{G})$ . Also by the proof of Lemma 1,  $\hat{U}$  can be written as an invertible linear mixing of  $U$ . Then (3)-(5) guarantee that Condition 1 and Condition 2 in Step 1 in the proof of Theorem 1 are satisfied. Then by the proof of Step 2 in the proof of Theorem 1, we have  $\mathcal{T}S(\mathcal{G}) \subset \mathcal{T}S(\hat{\mathcal{G}})$ . Therefore, it must hold that  $\mathcal{T}S(\hat{\mathcal{G}}) = \mathcal{T}S(\mathcal{G})$ .
Lastly, by (2) and (5), we obtain that  $\hat{\mathcal{G}}$  satisfies Condition 1 and Condition 2 in Theorem 2. Therefore by the proof of Theorem 2, we obtain  $\mathcal{G} \subset \hat{\mathcal{G}}$ . Again, by the first paragraph and the fact that the sparsest transitive closure satisfies  $\mathcal{T}\mathcal{S}(\hat{\mathcal{G}}) = \mathcal{T}\mathcal{S}(\mathcal{G})$ , we obtain that the sparsest  $\hat{\mathcal{G}}$  with this transitive closure must satisfy  $\hat{\mathcal{G}} \subset \mathcal{G}$ , and thus  $\hat{\mathcal{G}} = \mathcal{G}$ . With this result, it is easy to see that  $\hat{I}_k = I_k$  for all  $k \in [K]$ , as  $I_k$  changes the distribution of  $\hat{U}_{T_{\mathcal{G}}(I_k)}$  but does not change the joint distribution of  $\hat{U}_{[p] \setminus \overline{\operatorname{de}}_{\mathcal{G}}(T_{\mathcal{G}}(I_k))}$ .
Therefore we can recover  $p$  and the CD equivalence class of  $\langle \mathcal{G}, I_1, \dots, I_K \rangle$  by solving (1)-(7). Note that this proof assumes the topological order of  $\mathcal{G}$  is 1, ...,  $p$ , and therefore it does not violate the fact that  $\mathcal{G}, I_1, \dots, I_K$  cannot be recovered exactly.
Now, we will show that recovering  $\langle U,\mathcal{G},I_1,\ldots ,I_K\rangle$  up to Theorem 1 is sufficient for predicting the effect of combinatorial interventions..
Theorem 3. Letting  $\langle \hat{U},\hat{\mathcal{G}},\hat{I}_1,\dots,\hat{I}_K\rangle$  be the solution identified in the proof of Theorem 1. Then the interventional distribution  $\mathbb{P}^{\mathcal{I}}$  for any combinatorial intervention  $\mathcal{I}\subset \{I_1,\dots,I_K\}$  is given by Eq. (2), i.e., we can generate samples  $X$  from the distribution  $X = f(U),U\sim \mathbb{P}^{\mathcal{I}}$ .
Proof. Since  $\mathcal{I}$  contains interventions with different intervention targets, for each  $i\in [p]$ , we can define  $\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)})$  as  $\mathbb{P}^{\hat{I}_k}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)})$  if  $i = T_{\hat{\mathcal{G}}}(\hat{I}_k)$  for some  $I_k\in \mathcal{I}$  and otherwise  $\mathbb{P}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)})$ . Using this definition, we define the joint distribution of  $\hat{U}$  as  $\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}) = \prod_{i = 1}^{p}\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}_i\mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}}(i)})$ . In the following we show that  $\mathbb{P}^{\hat{\mathcal{I}}}(\hat{U}) = \mathbb{P}^{\mathcal{I}}(U)$  in the sense that  $\mathbb{P}^{\hat{\mathcal{I}}}\big(\hat{U} = \hat{f}^{-1}(x)\big) = \mathbb{P}^{\mathcal{I}}(U = f^{-1}(x))$  for all  $x\in \mathbb{R}^n$ .
Our proof combines the following equalities. For any  $i \in [p]$ , we have