SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. arXiv preprint arXiv:2110.15796, 2021.
[2] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems, 2022.
[3] K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022.
[4] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[5] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[6] J. Brehmer, P. D. Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[7] S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.02235, 2023.
[8] C. Bunne, S. G. Stark, G. Gut, J. S. Del Castillo, M. Levesque, K.-V. Lehmann, L. Pelkmans, A. Krause, and G. Ratsch. Learning single-cell perturbation responses using neural optimal transport. Nature Methods, pages 1-10, 2023.
[9] R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019.
[10] D. M. Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554, 2002.
[11] P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
[12] J. Cussens. Gobnilp: Learning bayesian network structure with integer programming. In International Conference on Probabilistic Graphical Models, pages 605-608. PMLR, 2020.
[13] A. Dixit, O. Parnas, B. Li, J. Chen, C. P. Fulco, L. Jerby-Arnon, N. D. Marjanovic, D. Dionne, T. Burks, R. Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. cell, 167(7):1853-1866, 2016.
[14] B. Fine and G. Rosenberger. The fundamental theorem of algebra. Springer Science & Business Media, 1997.
[15] C. Frogner, F. Mirzazadeh, and J. Solomon. Learning embeddings into entropic Wasserstein spaces. arXiv preprint arXiv:1905.03329, 2019.
[16] L. A. Gilbert, M. A. Horlbeck, B. Adamson, J. E. Villalta, Y. Chen, E. H. Whitehead, C. Guimaraes, B. Panning, H. L. Ploegh, M. C. Bassik, et al. Genome-scale crispr-mediated control of gene repression and activation. Cell, 159(3):647-661, 2014.
[57] J. Tian and J. Pearl. Causal discovery from changes. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pages 512-521, 2001.
[58] C. Uhler, G. Raskutti, P. Buhlmann, and B. Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics, pages 436-463, 2013.
[59] B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.
[60] J. von Kugelgen, M. Besserve, W. Liang, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. arXiv preprint arXiv:2306.00542, 2023.
[61] M. J. Vowels, N. C. Camgoz, and R. Bowden. D'ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys, 55(4):1-36, 2022.
[62] Y. Wang, L. Solus, K. D. Yang, and C. Uhler. Permutation-based causal inference algorithms with interventions. In Neural Information Processing Systems, volume 31, 2017.
[63] J. Wu, Z. Huang, D. Acharya, W. Li, J. Thoma, D. P. Paudel, and L. V. Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3713-3722, 2019.
[64] F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems, 33:14891-14902, 2020.
[65] F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370-24387. PMLR, 2022.
[66] K. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal dags under interventions. In International Conference on Machine Learning, pages 5541-5550. PMLR, 2018.
[67] K. D. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal DAGs under interventions. Proceedings of Machine Learning Research, 80:5537-5546, 2018.
[68] H. Yu and J. D. Welch. Perturbnet predicts single-cell responses to unseen chemical and genetic perturbations. *BioRxiv*, pages 2022–07, 2022.
[69] J. Zhang, C. Squires, and C. Uhler. Matching a desired causal state via shift interventions. Advances in Neural Information Processing Systems, 34:19923-19934, 2021.
[70] X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
[71] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
A Useful Lemmas 16
A.1 Remarks on Assumption 1 16
A.2 Proof of Lemma 1 16
B Proof of Identifiability with Soft Interventions 17
B.1 Faithfulness Assumptions 17
B.2 Summary of representations 21
B.3 Proof of Theorem 1 21
B.4 Proof of Theorem 2 25
B.5 Proof of Theorem 3 28
C Details on Discrepancy-based VAE 29
C.1 Maximum Mean Discrepancy 29
C.2 Discrepancy VAE Details 29
D Lower Bound to Paired Log-Likelihood 30
E Consistency of Discrepancy-based VAE 31
E.1 CD-Equivalence Class 31
E.2 Consistency for Multi-Node Interventions 32
F Discrepancy-based VAE Implementation Details 33
G Extended Results on Biological Dataset 33
G.1 Single-node interventions 34
G.2 Double-node interventions 35
G.3 Structure Learning 36
H Extended Experiments 36
H.1 Ablation Studies 36
H.2 Simulation 37
I Extended Discussion 38
I.1 Limitations and Future Work 38
I.2 Discussion of Contemporaneous Works. 39
A Useful Lemmas
A.1 Remarks on Assumption 1
Here we show that the assumption on the functional class of  $f$  is satisfied if  $f$  is linear and injective, whenever the support of  $\mathbb{P}_U$  has non-empty interior. Recall Assumption 1.
Denote the support of  $\mathbb{P}_U,\mathbb{P}_X$  as  $\mathbb{U},\mathbb{X}$  respectively. Let  $\mathbb{U}^{\circ}$  be the interior of  $\mathbb{U}$ .
Lemma 2. Suppose  $\mathbb{U}^{\circ}$  is a non-empty subset of  $\mathbb{R}^p$ . If  $f:\mathbb{U}\to \mathbb{X}$  is linear and injective, then it must be a full row rank polynomial.
Proof. Since  $f$  is linear, it can be written as  $f(U) = U H + h$  for some  $H \in \mathbb{R}^{p \times n}$  and  $h \in \mathbb{R}^p$ . If  $H$  is not of full row rank, then there exists a non-zero vector  $V \in \mathbb{R}^p$  such that  $V H = 0$ . Let  $U \in \mathbb{U}^\circ$ , then there exists  $\epsilon > 0$  such that  $U + \epsilon V \in \mathbb{U}$ . We have  $f(U + \epsilon V) = f(U)$ , which violates  $f$  being injective. Therefore  $H$  must have full row rank.
The proof of Lemma 1 follows from [3]. For completeness, we present a concise proof here. Then we state a few remarks. Recall Lemma 1.
Proof. We solve for the smallest integer  $\hat{p}$  such that there exists a full row rank polynomial  $\hat{f}:\mathbb{R}^{\hat{p}}\to \mathbb{R}^n$  where  $\hat{U} := \hat{f}^{-1}(X)$  for  $X\in \mathbb{X}$  has non-empty support  $\hat{\mathbb{U}}^{\circ}\subseteq \mathbb{R}^{\hat{p}}$ . In other words, denote all pairs of  $\mathbb{P}_U,f$  that satisfy Assumption 1 as  $\mathcal{F}_p$ , we solve for
$$
\min  _ {(\mathbb {P} _ {\hat {U}}, \hat {f}) \in \mathcal {F} _ {\hat {p}}} \hat {p} \quad \text {s u b j e c t} \mathbb {P} _ {\hat {f} (\hat {U})} = \mathbb {P} _ {X}. \tag {4}
$$
Note that  $\hat{f}(\hat{U}) = X = f(U)$  for all  $U \in \mathbb{U}$ . Since  $\hat{f}, f$  are full row rank polynomials, there exist full row rank matrices  $\hat{H} \in \mathbb{R}^{(p + \ldots + p^d) \times n}$ ,  $H \in \mathbb{R}^{(p + \ldots + p^d) \times n}$  and vectors  $\hat{h}, h \in \mathbb{R}^n$  such that
$$
(\hat {U}, \bar {\otimes} \hat {U} ^ {2}, \dots , \bar {\otimes} \hat {U} ^ {d}) \hat {H} + \hat {h} = \hat {f} (\hat {U}) = X = f (U) = (U, \bar {\otimes} U ^ {2}, \dots , \bar {\otimes} U ^ {d}) H + h. \tag {5}
$$
Since  $\hat{H}, H$  are of full rank, they have pseudo-inverses  $\hat{H}^{\dagger}, H^{\dagger}$  such that  $\hat{H}\hat{H}^{\dagger} = \mathbf{I}_{p + \ldots + p^{d}}$  and  $HH^{\dagger} = \mathbf{I}_{p + \ldots + p^{d}}$ . Multiplying  $\hat{H}^{\dagger}$  to Eq. (5), we have
$$
(\hat {U}, \bar {\otimes} \hat {U} ^ {2}, \dots , \bar {\otimes} \hat {U} ^ {d}) = (U, \bar {\otimes} U ^ {2}, \dots , \bar {\otimes} U ^ {d}) H \hat {H} ^ {\dagger} + (h - \hat {h}) \hat {H} ^ {\dagger}.
$$
Therefore  $\hat{U}$  can be written as a polynomial of  $U$ , i.e.,  $\hat{U} = poly_1(U)$ . Similarly, we have  $U = poly_2(\hat{U})$ . Therefore  $U = poly_2(poly_1(U))$  for all  $U \in \mathbb{U}$ . Since  $\mathbb{U}^\circ$  is non-empty, we know that  $U = poly_2(poly_1(U))$  on some open set. By the fundamental theorem of algebra [14], we know that  $poly_1$  and  $poly_2$  must have degree 1. Thus  $\hat{U} = U\Lambda + b$  for some full row rank matrix  $\Lambda$  and vector  $b$ . Since  $\Lambda \in \mathbb{R}^{p \times \hat{p}}$  is of full row rank, it indicates that  $p \leq \hat{p}$ . Since  $\mathbb{P}_U, f \in \mathcal{F}_p$  satisfy  $\mathbb{P}_{f(U)} = \mathbb{P}_X$ , by Eq. (4), we must have  $\hat{p} \leq p$ . Thus  $\hat{p} = p$  and  $\hat{U} = U\Lambda + b$  for some non-singular matrix  $\Lambda$  and vector  $b$ .
This proof also shows that we can only identify  $U$  up to such linear transformations with observational data. Since for any non-singular matrix  $\Lambda$  and vector  $b$ , let  $\hat{f}(\hat{U}) = f((\hat{U} - b)\Lambda^{-1})$ . We have  $\mathbb{P}_{\hat{U}}, \hat{f}$  satisfy Assumption 1 and they generate the same observational data.
Remark 1. With observational data  $X = f(U) \in \mathcal{D}$ , we can identify  $\hat{U} = \hat{g}(X)$  such that  $\hat{U} = U\Lambda + b$  for non-singular  $\Lambda$ . Then for any interventional data  $X = f(U) \in \mathcal{D}^I$ , the analytic continuation of  $\hat{g}$  to  $\mathcal{D}^I$  satisfies  $\hat{U} := \hat{g}(X) = U\Lambda + b$  for all  $X \in \mathcal{D}^I$ .
Proof. The proof follows immediately by writing  $\hat{g}, f^{-1}$  as polynomial functions.
Next, we discuss identifiability of the underlying DAG  $\mathcal{G}$ . First, we give an example showing that any causal DAG can explain the observational data.
Example 4. Suppose the ground-truth DAG is an empty graph  $\mathcal{G} = \varnothing$ . With observational data alone, any DAG can explain the data.
Proof. Let  $\hat{\mathcal{G}}$  be an arbitrary DAG with topological order  $\tau(1), \ldots, \tau(p)$ , i.e.,  $\tau(j) \in \mathrm{pa}_{\hat{\mathcal{G}}}(\tau(i))$  only if  $j < i$ . Let  $\Lambda$  be the permutation matrix such that  $\hat{U} = U\Lambda$  satisfies  $\hat{U}_{\tau(i)} = U_i$  for any  $i \in [p]$ . Then  $\hat{U}$  factorizes as  $\mathbb{P}(\hat{U}) = \mathbb{P}(U) = \prod_{i=1}^{p} \mathbb{P}(U_i) = \prod_{i=1}^{p} \mathbb{P}(\hat{U}_{\tau(i)})$ . This implies  $\hat{U}_{\tau(i)} \perp \hat{U}_{\tau(j)}$  for  $j \leq i-1$ . Therefore  $\mathbb{P}(\hat{U}_{\tau(i)}) = \mathbb{P}(\hat{U}_{\tau(i)} \mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}(i)}})$  and  $\mathbb{P}(\hat{U}) = \prod_{i=1}^{p} \mathbb{P}(\hat{U}_{\tau(i)} \mid \hat{U}_{\mathrm{pa}_{\hat{\mathcal{G}}(i)}})$  factorizes with respect to  $\hat{\mathcal{G}}$ . Thus  $\hat{\mathcal{G}}$  can explain the data.
Therefore with observational data alone, we cannot identify the underlying DAG  $\mathcal{G}$  up to any nontrivial equivalence class. In [3], it was shown that with a do intervention<sup>8</sup> per latent node and assuming the interior of the support of the non-targeted variables is non-empty, one can identify  $U$  up to a finer class of linear transformations. Namely, one can identify  $U$  up to CD-equivalence (permutation and element-wise affine transformation); see Definition 1. Then, assuming for example faithfulness and influentiality [57], one can identify  $\mathcal{G}$ .
While several extensions beyond do-interventions are discussed in [3], they all involve manipulating the support of the intervention targets. In the case where the support of the intervention targets remains unchanged (e.g., additive Gaussian SCMs with shift interventions), a completely new approach and theory needs to be developed.
B Proof of Identifiability with Soft Interventions
In this section, we provide the proofs for the results in Section 4. While our main focus is on general types of soft interventions, our results also apply to hard interventions which include do-interventions as a special case.
Notation. We let  $e_i$  denote the indicator vector with the  $i$ -th entry equal to one and all other entries equal to zero. To be consistent with other notation in the paper, let  $e_i \in \mathbb{R}^p$  be a row vector. We call  $j \in \mathrm{ch}_{\mathcal{G}}(i)$  a maximal child of  $i$  if  $\mathrm{pa}_{\mathcal{G}}(j) \cap \mathrm{de}_{\mathcal{G}}(i) = \emptyset$ . Denote the set of all maximal children of  $i$  as  $\mathrm{mch}_{\mathcal{G}}(i)$ . For node  $i$ , define  $\overline{\mathrm{de}}_{\mathcal{G}}(i) := \mathrm{de}_{\mathcal{G}}(i) \cup \{i\}$ . Given a DAG  $\mathcal{G}$ , we denote the transitive closure of  $\mathcal{G}$  by  $\mathcal{T}\mathcal{S}(\mathcal{G})$ , i.e.,  $i \to j \in \mathcal{T}\mathcal{S}(\mathcal{G})$  if and only if there is a directed path from  $i$  to  $j$  in  $\mathcal{G}$ .
B.1 Faithfulness Assumptions
We start by discussing previous interventional faithfulness assumptions. Prior interventional faithfulness assumptions [57, 66, 24] vary by a few technicalities; but they all assume that all causal variables are observed (causal sufficiency), and, more importantly, that intervening on a node will always change the marginal of its descendants. In particular, [57] (Definition 2, called "influentiality") only made this assumption and showed that the causal graph is identifiable up to its transitive closure by detecting marginal changes. [57] showed that their algorithm consistently identifies the full causal graph by assuming additionally that intervening on a node changes the conditional distribution of its direct children giving its neighbors (details can be found in Assumption 4.5 of [66]). A similar notion was also introduced in [24] where they made further assumptions regarding changes in the conditional distributions.
<sup>8</sup>Do interventions are a special type of hard interventions where the intervention target collapses to one specific value.
We now show our linear interventional faithfulness (Assumption 2) is satisfied by a large class of nonlinear SCMs and soft interventions. Recall Assumption 2.
Assumption 2. Intervention  $I$  with target  $i$  satisfies linear interventional faithfulness if for every  $j \in \{i\} \cup \mathrm{ch}_{\mathcal{G}}(i)$  such that  $\mathrm{pa}_{\mathcal{G}}(j) \cap \mathrm{de}_{\mathcal{G}}(i) = \emptyset$ , it holds that  $\mathbb{P}(U_j + U_SC^\top) \neq \mathbb{P}^I(U_j + U_SC^\top)$  for all constant vectors  $C \in \mathbb{R}^{|S|}$ , where  $S = [p] \setminus (\{j\} \cup \mathrm{de}_{\mathcal{G}}(i))$ .
In Example 2, we discussed a 2-node graph where Assumption 2 is satisfied. This example can be extended in the following way, which subsumes the case in Example 3.
Example 5. Consider an SCM with additive noise, where each mechanism  $\mathbb{P}(U_k\mid U_{\mathrm{pa}_{\mathcal{G}}(k)})$  is specified by  $U_{k} = s_{k}(U_{\mathrm{pa}_{\mathcal{G}}(k)}) + \epsilon_{k}$ , where  $\epsilon_{k}$  for  $k\in [p]$  are independent exogenous noise variables. Assumption 2 is satisfied if I only changes the variance of  $\epsilon_{i}$  and  $s_j$  is a quadratic function with nonzero coefficient of  $U_i^2$  for each  $j\in \mathrm{mch}_{\mathcal{G}}(i)$ .
Proof. If  $j = i$  in Assumption 2, then  $S = [p] \setminus \overline{\mathrm{de}}_{\mathcal{G}}(i) \supset \mathrm{pa}_{\mathcal{G}}(i)$ . Since  $U_S \perp \epsilon_i$ , we have
$$
\operatorname {V a r} (U _ {i} + U _ {S} C ^ {\top}) = \operatorname {V a r} (\epsilon_ {i}) + \operatorname {V a r} (s _ {i} (U _ {\mathrm {p a} _ {\mathcal {G}} (i)}) + U _ {S} C ^ {\top}).
$$
Note that  $\mathbb{P}^I$  does not change the joint distribution of  $U_{S}$ , and therefore
$$
\mathrm {V a r} _ {\mathbb {P}} (s _ {i} (U _ {\mathrm {p a} _ {\mathcal {G}} (i)}) + U _ {S} C ^ {\top}) = \mathrm {V a r} _ {\mathbb {P} ^ {I}} (s _ {i} (U _ {\mathrm {p a} _ {\mathcal {G}} (i)}) + U _ {S} C ^ {\top}).
$$
By  $\operatorname{Var}_{\mathbb{P}}(\epsilon_i) \neq \operatorname{Var}_{\mathbb{P}^I}(\epsilon_i)$ , we then know that  $\operatorname{Var}_{\mathbb{P}}(U_i + U_SC^\top) \neq \operatorname{Var}_{\mathbb{P}^I}(U_i + U_SC^\top)$ . Thus  $\mathbb{P}(U_i + U_SC^\top) \neq \mathbb{P}^I(U_i + U_SC^\top)$ .
If  $j \neq i$  in Assumption 2, then by linearity of expectation  $\mathbb{E}(U_j + U_SC^\top) = \mathbb{E}(U_j) + \mathbb{E}(U_S)C^\top$ . Note that  $S = [p] \setminus (\{j\} \cup \mathrm{de}_{\mathcal{G}}(i)) = [p] \setminus \mathrm{de}_{\mathcal{G}}(i)$ , and therefore  $\mathbb{E}_{\mathbb{P}}(U_S) = \mathbb{E}_{\mathbb{P}^I}(U_S)$ . Next we show that  $\mathbb{E}_{\mathbb{P}}(U_j) \neq \mathbb{E}_{\mathbb{P}^I}(U_j)$ . Once this is proven, then we have that  $\mathbb{E}_{\mathbb{P}}(U_j + U_SC^\top) \neq \mathbb{E}_{\mathbb{P}^I}(U_j + U_SC^\top)$ , which concludes the proof for  $\mathbb{P}(U_j + U_SC^\top) \neq \mathbb{P}^I(U_j + U_SC^\top)$ .
Since  $s_j$  is a quadratic function of  $U_i$ , suppose the coefficient of  $U_i^2$  in  $s_j$  is  $\beta \neq 0$ . Then
$$
\begin{array}{l} \mathbb {E} (U _ {j}) - \mathbb {E} (\epsilon_ {j}) = \mathbb {E} (U _ {j} - \epsilon_ {j}) \\ = \mathbb {E} \left(s _ {j, 0} \left(U _ {\operatorname {p a} _ {\mathcal {G}} (j) \backslash \{i \}}\right) + s _ {j, 1} \left(U _ {\operatorname {p a} _ {\mathcal {G}} (j) \backslash \{i \}}\right) \cdot U _ {i} + \beta U _ {i} ^ {2}\right) \\ = \mathbb {E} \left(s _ {j, 0} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}\right) + s _ {j, 1} ^ {\prime} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}\right) \cdot \epsilon_ {i} + \beta \epsilon_ {i} ^ {2}\right) \tag {6} \\ = \mathbb {E} \big (s _ {j, 0} (U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}) \big) + \mathbb {E} \big (s _ {j, 1} ^ {\prime} (U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}) \cdot \epsilon_ {i} \big) + \beta \mathbb {E} (\epsilon_ {i} ^ {2}), \\ \end{array}
$$
for some functions  $s_{j,0}, s_{j,1}$  and  $s_{j,1}'$ . Since  $\mathrm{pa}_{\mathcal{G}}(j) \cap \mathrm{de}_{\mathcal{G}}(i) = \emptyset$ , we know that  $\mathbb{P}^I$  will not change the joint distribution of  $U_{\mathrm{pa}_{\mathcal{G}}(j) \setminus \{i\}}$  and that  $\epsilon_i \perp U_{\mathrm{pa}_{\mathcal{G}}(j) \setminus \{i\}}, U_{\mathrm{pa}_{\mathcal{G}}(i)}$ . Therefore we have
$$
\mathbb {E} _ {\mathbb {P}} \big (s _ {j, 0} \big (U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}} \big) \big) = \mathbb {E} _ {\mathbb {P} ^ {I}} \big (s _ {j, 0} \big (U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}} \big) \big),
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathbb {P}} \left(s _ {j, 1} ^ {\prime} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}\right) \cdot \epsilon_ {i}\right) = \mathbb {E} _ {\mathbb {P}} \left(s _ {j, 1} ^ {\prime} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}\right)\right) \cdot \mathbb {E} _ {\mathbb {P}} (\epsilon_ {i}) \\ = \mathbb {E} _ {\mathbb {P} ^ {I}} \left(s _ {j, 1} ^ {\prime} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}\right)\right) \cdot \mathbb {E} _ {\mathbb {P} ^ {I}} \left(\epsilon_ {i}\right) \\ = \mathbb {E} _ {\mathbb {P} ^ {I}} \left(s _ {j, 1} ^ {\prime} \left(U _ {\mathrm {p a} _ {\mathcal {G}} (j) \backslash \{i \}}, U _ {\mathrm {p a} _ {\mathcal {G}} (i)}\right) \cdot \epsilon_ {i}\right). \\ \end{array}
$$
By  $\mathbb{E}_{\mathbb{P}}(\epsilon_j) = \mathbb{E}_{\mathbb{P}^I}(\epsilon_j), \mathbb{E}_{\mathbb{P}}(\epsilon_i^2) \neq \mathbb{E}_{\mathbb{P}^I}(\epsilon_i^2)$  and Eq. (6), we have  $\mathbb{E}_{\mathbb{P}}(U_j) \neq \mathbb{E}_{\mathbb{P}^I}(U_j)$ , which concludes the proof.
This example shows how we may check  $\mathbb{P}(U_j + U_SC^\top) \neq \mathbb{P}^I (U_j + U_SC^\top)$  by examining the mean and variance of  $U_{j} + U_{S}C^{\top}$ . In general, this can be extended to checking any finite moments of  $U_{j} + U_{S}C^{\top}$  as stated in the following lemma.
Lemma 3. Assumption 2 is satisfied if for each  $i \in [p]$  one of the following conditions holds:
(1) if  $\mathbb{E}_{\mathbb{P}}(U_i \mid U_{\mathrm{pa}_{\mathcal{G}}(i)}) = \mathbb{E}_{\mathbb{P}^I}(U_i \mid U_{\mathrm{pa}_{\mathcal{G}}(i)})$ , then there exists an integer  $m > 1$  such that
$$
\mathbb {E} _ {\mathbb {P}} (U _ {i} ^ {m} \mid U _ {\mathrm {p a} _ {\mathcal {G}} (i)}) \neq \mathbb {E} _ {\mathbb {P} ^ {I}} (U _ {i} ^ {m} \mid U _ {\mathrm {p a} _ {\mathcal {G}} (i)}),
$$
and the smallest  $m$  that satisfies this also satisfies  $\mathbb{E}_{\mathbb{P}}(U_i^m) \neq \mathbb{E}_{\mathbb{P}^I}(U_i^m)$ . In addition, for all  $j \in \mathrm{mch}_{\mathcal{G}}(i)$ , it holds that  $\mathbb{E}_{\mathbb{P}}(U_j) \neq \mathbb{E}_{\mathbb{P}^I}(U_j)$ ;
(2) if  $\mathbb{E}_{\mathbb{P}}(U_i)\neq \mathbb{E}_{\mathbb{P}^I}(U_i)$ , then for all  $j\in \mathrm{mch}_{\mathcal{G}}(i)$ , there exists an integer  $m > 1$  such that
$$
\mathbb {E} _ {\mathbb {P}} \left(\left(U _ {j} + c _ {j} U _ {i}\right) ^ {m} \mid U _ {S \backslash \{i \}}\right) \neq \mathbb {E} _ {\mathbb {P} ^ {I}} \left(\left(U _ {j} + c _ {j} U _ {i}\right) ^ {m} \mid U _ {S \backslash \{i \}}\right),
$$
where  $S$  is as defined in Assumption 2, and the smallest  $m$  that satisfies this also satisfies
$\mathbb{E}_{\mathbb{P}}((U_j + c_j U_i)^m) \neq \mathbb{E}_{\mathbb{P}^I}((U_j + c_j U_i)^m)$ , where
$$
c _ {j} = - \frac {(\mathbb {E} _ {\mathbb {P}} (U _ {j}) - \mathbb {E} _ {\mathbb {P} ^ {I}} (U _ {j}))}{(\mathbb {E} _ {\mathbb {P}} (U _ {i}) - \mathbb {E} _ {\mathbb {P} ^ {I}} (U _ {i}))}.
$$
Proof. Suppose (1) holds true. If  $j = i$  in Assumption 2, then  $\mathbb{P}(U_S) = \mathbb{P}^I (U_S)$  for  $S = [p]\backslash \overline{\mathrm{de}}_{\mathcal{G}}(i)$ , and