SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This motivates the approximate method Algorithm 1. For any target embedding  $v$ , one first sample subspaces spanned by randomly selected embeddings. Embeddings that, on average have high cosine similarities with the target embedding after projecting onto orthogonal complements of previously sampled random subspaces, are considered to be candidates for the generalized Markov boundary. The final selection of generalized Markov boundary searches over these top  $K$  candidates.
Empirically, for text embedding models like CLIP, random projections prove to be advantageous in revealing semantically related concepts. In Section 5.2, we provide several examples where, for a given target embedding, the embeddings that exhibit high correlation after random projections are more semantically meaningful compared to embeddings with merely high cosine similarity with the target embedding before projections.
4 Independence Preserving Embeddings (IPE)
In the previous sections, we discussed the Markov boundary of embeddings under the partial orthogonality independence model. In Section 5, we will test its effectiveness at capturing the "semantic independence structure" through experiments conducted on CLIP text embeddings. The belief is that the linear algebraic structure possesses the capacity to uphold the independence structure of semantic meanings.
A natural question to ask is: Is it always possible to use vector space embeddings to preserve independence structures of interest? In this section, we study the case for random variables. Consider an embedding function  $f$  that maps a random variable  $X$  to  $f(X) \in \mathbb{R}^d$ . Ideally, it is desirable for the partial orthogonalities of embeddings to mirror the conditional independences present in the joint distribution of  $X$ . We call such representations independence preserving embeddings (IPE) (Definition 8). In this section, we delve into the theoretical feasibility of these embeddings by initially demonstrating the construction of IPE and then showing how one can use random projection to reduce the dimension of IPE. We believe that studying IPE lays the theoretical foundation to understand embedding models in general.
Definition 8 (Independence Preserving Embedding Map). Let  $V$  be a finite set of random variables with distribution  $\bar{P}$ . A function  $f: V \to \mathbb{R}^d$  is called an independence preserving embedding map (IPE map) if
$$
\mathcal {I} _ {O} (f (V)) \subseteq \mathcal {I} _ {P} (V).
$$
An IPE map is called a faithful IPE map if
$$
\mathcal {I} _ {O} (f (V)) = \mathcal {I} _ {P} (V).
$$
4.1 Existence and Universality of IPE Maps
We first show that for any distribution  $P$  over random variables  $V$ , we can construct an IPE map.
For any distribution  $P$  over  $V$ , there exists a minimal  $I$ -map  $\mathcal{G} = (V,E)$  such that  $\mathcal{I}_G(V) \subseteq \mathcal{I}_P(V)$  (See Section 2). We will use  $\mathcal{G}_P$  to be a minimal  $I$ -map of  $P$  and  $\mathrm{adj}(\mathcal{G}_P)$  to be the adjacency matrix of  $\mathcal{G}_P$ . We further define  $\mathrm{adj}_{\varepsilon}(\mathcal{G}_P)$  to be an adjusted adjacency matrix with  $\varepsilon \in \mathbb{R}$  where
$$
\operatorname {a d j} _ {\varepsilon} \left(\mathcal {G} _ {P}\right) = \mathbb {1} + \varepsilon \operatorname {a d j} \left(\mathcal {G} _ {P}\right)
$$
and  $\mathbb{1}$  is the identity matrix.
Ideally, this matrix is invertible, however, it turns out that not every  $\varepsilon$  produces an invertible  $\mathrm{adj}_{\varepsilon}(\mathcal{G}_P)$ . We therefore define the following perfect perturbation factor. For any matrix  $A \in \mathbb{R}^{n \times n}$ , define  $A_{\mathcal{I},\mathcal{J}}$  to be the submatrix of  $A$  with row and column indices from  $\mathcal{I}$  and  $\mathcal{J}$ , respectively. If  $\mathcal{I} = \mathcal{J}$ , the submatrix is called a principal submatrix and we denote it simply as  $A_{\mathcal{I}}$ .
Definition 9 (Perfect Perturbation Factor). For a given graph  $\mathcal{G} = (V,E)$  where  $n = |V|$ ,  $\varepsilon$  is called a perfect perturbation factor if (1)  $\mathrm{adj}_{\varepsilon}(\mathcal{G}_P)$  is invertible and (2) for any  $\mathcal{I} \subseteq [n]$ ,  $(\mathrm{adj}_{\varepsilon}(\mathcal{G}_P)_{\mathcal{I}})^{-1} = 0$  if and only if  $v_{\mathcal{I}_i} \perp_{\mathrm{G}} v_{\mathcal{I}_j} \mid \{v_k : k \notin \mathcal{I}\}$  where  $\mathcal{I}_i$  is the  $i$ th element of  $\mathcal{I}$ .
Theorem 10. Let  $V$  be a finite set of random variables with distribution  $P$ .  $\mathcal{G}_P$  is a minimal  $I$ -map of  $P$ . Let  $A$  be equal to  $\mathrm{adj}_{\varepsilon}(\mathcal{G}_P)^{-1}$  with eigen decomposition  $A = U\Sigma U^T$ . If  $\varepsilon$  is a perfect perturbation factor, then the function  $f$  with
$$
f (v _ {i}) = U _ {i} \Sigma^ {1 / 2}
$$
is an IPE map of  $P$  where  $v_{i}$  is a random variable in  $V$  and  $U_{i}$  is the  $i$ -th row of  $U$ . Furthermore, if  $P$  is faithful to  $\mathcal{G}_P$ , then  $f$  is a faithful IPE map for  $\mathcal{P}$ .
Remark 11. One can always normalize these embeddings to have unit norms without changing the partial orthogonality structures.
Finding a perfect perturbation factor might seem daunting, but the following lemma, which is a direct consequence of Theorem 1 in Lneička and Matuš [LM07], shows that almost every  $\varepsilon$  is a perfect perturbation factor.
Lemma 12. For any simple graph  $G$ ,  $\varepsilon$  is perfect for all but finitely many  $\varepsilon \in \mathbb{R}$ .
4.2 Dimension Reduction of IPE
Theorem 10 shows how to learn a perfect IPE but it requires the dimension of embeddings to be the same as the number of variables in  $V$ . In the worst case, this is inevitable for a faithful IPE map: If the random variables in  $V$  are mutually independent, then we need at least  $|V|$  dimensions in the embedding space to contain  $V$  orthogonal vectors.
But this is not practical. Suppose we want to embed millions of random variables (e.g. tokens) in a vector space, having the dimension of each embedding be in the magnitude of millions is less than ideal. Therefore, one needs to do dimension reduction.
In this section, we show that by using random projection, the partial orthogonalities induced by Markov boundaries are preserved approximately. Intuitively, this is guaranteed by the Johnson-Lindenstrauss lemma [Vem05].
Theorem 13. Let  $U$  be a set of vectors in  $\mathbb{R}^n$  where  $n = |U|$  and every vector is a unit vector. Let  $\Sigma$  be a matrix in  $\mathbb{R}^{n\times n}$  where  $\Sigma_{ij} = \langle u_i,u_j\rangle$ . Assume  $\lambda_1 = \lambda_{\min}(\Sigma) > 0$ . Then there exists a mapping  $g:\mathbb{R}^n\to \mathbb{R}^k$  where  $k = \lceil 20\log (2n) / (\varepsilon ')^2\rceil$  with  $\varepsilon^\prime = \min \{1 / 2,\varepsilon /C,\lambda_1 / 2r^2\}$  and  $\varepsilon \in (0,1)$  such that for any  $u_{i}\in U$  with its unique Markov boundary  $M_{i}\subseteq U$  and any  $u_{j}\in U\setminus (\{u_{i}\} \cup M_{i})$ , we have
$$
\left| \left\langle \operatorname {p r o j} _ {g (M _ {i})} ^ {\perp} [ g (u _ {i}) ], \operatorname {p r o j} _ {g (M _ {i})} ^ {\perp} [ g (u _ {j}) ] \right\rangle \right| \leq \varepsilon
$$
where  $r_i = |M_i|$ ,  $r = \max_i |M_i|$  and  $C = (r + 1)^3 \left( \frac{2\lambda_{\max}(\Sigma) + 2(r + 1)^2}{\lambda_{\min}(\Sigma)} \right)^r$ .
Theorem 13 shows that as long as the partial orthogonality structure of embeddings is sparse in the sense that the size of the Markov boundary for each embedding is small. Then one can reduce the dimension of the embedding and the residuals of target and test vectors after projection onto the Markov boundary are almost orthogonal.
Remark 14. The assumption in Theorem 13 is satisfied by the construction of IPE in Section 4.1.
One of the central hypotheses of the paper is that the partial orthogonality of embeddings, and its byproduct generalized Markov boundary, carry semantic information. To verify this claim, we provide both quantitative and qualitative experiments. Throughout this section, we consider the set of normalized embeddings  $\mathbf{E}$  that represent the 49815 words in the Brown corpus [FK79]. For each target embedding of a word, under any experiment setting, we automatically filter words, whose embeddings have 0.9 or above cosine similarities with the target embedding, or words, whose Wu-Palmer similarity measure with the target word is almost 1. The purpose of this filtering step is to prevent the inclusion of synonyms.
5.1 Semantic Structure of Partial Orthogonality
To examine the rule of partial orthogonality, nine categories are chosen, each with 10 words in it (See Table 3 in Appendix B). Specifically, each word within a given category is a hyponym for that category in WordNet [Mil95]. We assess how much, on average, the cosine similarities between words within each category decrease when conditioned on these different nine categories. By conditioning, we use the clip embedding of the category word of interest and project out the subspace of that clip embedding. The results are shown in Figure 2. We normalize reduction values by sampling 10,000 embeddings and calculating the mean and standard deviation of cosine reductions between these embeddings. It is apparent that on average, cosine similarities of intra-category words decrease more than inter-category words. One interesting finding is that when conditioned on the category word "food", the average similarities between word pairs in "beverage" also drop considerably. We suspect this is because one synset of "food" is also a hypernom of "beverage". Although words in the "food" category are chosen to mean solid food, it could also mean nutrient which also encompasses the meaning of "beverage".
Figure 2: Experiments show conditioning on the category word, cosine similarities of intra-category words decrease more than inter-category words. Each row shows the (normalized) average cosine similarities reduction between words within each category when conditioned on the category word of that row.
Figure 3: Experiments show that learned generalized Markov boundaries have on average smaller principal angles with the description embedding compared to subspaces spanned by randomly selected embeddings. The standard errors are over 50 examples.
5.2 Sampling Random Subspaces
The first step of Algorithm 1 is to find embeddings that have high similarities with the target embeddings even after projecting onto orthogonal complements of subspaces spanned by randomly selected embeddings. It turns out that this step can reveal semantic meanings. In this section, we design experiments to show both quantitatively and qualitatively that embeddings of words that remain highly correlated with the target embedding after projection are semantically closer to the target word. In various experimental configurations, we employ 10 sets of 50 randomly chosen embeddings to form random projection subspaces for each target embedding. Qualitatively, Table 4 in Appendix B gives a few examples showing that the words that on average remain highly correlated with the target word tend to possess greater semantic significance. Quantitatively, we calculate the average Wu-Palmer similarities between target words and the top 10 correlated words before and after random projections. We conduct experiments on 1000 random words as well as 300 common nouns provided by ChatGPT. The results are shown in Table 4 verify our claims. This set of experiments also indirectly shows that the embeddings are noisy and that generalized Markov boundaries are indeed needed.
Table 1: Experiments show that the top 10 words that have, on average, high correlations with target words after projecting onto the orthogonal complements of randomly selected linear subspaces have higher Wu-Palmer similarities with the target words than the top 10 highly correlated words without projections. This table contains average Wu-Palmer similarities with standard errors.
Target	Before Projection	After Projection
Random Words	0.223 ± 0.006	0.245 ± 0.007
Common Nouns	0.343 ± 0.008	0.422 ± 0.008
5.3 Generalized Markov Boundaries
We first demonstrate that Algorithm 1 can find generalized Markov boundaries. The experiments are run over 1000 randomly selected words. In particular, Table 2 shows that with a relatively small candidate set, the algorithm can already approximate generalized Markov boundaries well, suggesting that the size of generalized Markov boundaries for CLIP text embeddings should be small.
Semantic Meanings of Markov Boundaries The estimated generalized Markov boundaries returned by Algorithm 1 is a set of embeddings. It is reasonable to anticipate that the linear spans of these embeddings hold semantic meanings. To evaluate this hypothesis, we propose to calculate the smallest principal angles [KA02] between the span of generalized Markov boundaries and the span of selected embeddings that are meaningful to the target word.
We again conducted both quantitative and qualitative experiments. Qualitatively, Figure 4 in Appendix B give a few examples comparing target words' generalized Markov boundaries with the span of selected embeddings. For instance, the generalized Markov boundary of 'car' is more aligned with the subspace spanned by embeddings of 'road' and 'vehicle' than the span of 'sea' and 'boat' and randomly selected subspaces. This suggests that the estimated generalized Markov boundaries hold semantic significance. To verify this quantitatively, we ask ChatGPT to provide a list of common nouns with short descriptions (selected examples are provided in Table 5). We then use CLIP text embedding to convert the description sentence into one vector and compare the smallest angle between the description vector with generalized Markov boundaries and random linear spans. Figure 3 shows that the generalized Markov boundaries are more semantically meaningful than random subspaces.
Table 2: With relatively small number of  $K$ , the average  $\mathrm{S}_{\mathcal{M}}(v, \mathrm{E})$  is small. The standard errors are over 1000 experiments.
K	1	3	5	8	10
AVERAGE SM(v,E)	0.345±0.03	0.128±0.03	0.054±0.002	0.015±0.002	0.008±0.001
This paper studies the role of partial orthogonality in analyzing embeddings. Specifically, we extend the idea of Markov boundaries to embedding space. Unlike Markov boundaries in graphical models, the boundaries for embeddings are not guaranteed to be unique. We propose alternative relaxed definitions of Markov boundaries for practical use. Empirically, these tools prove to be useful in finding the semantic meanings of embeddings. We also introduce the concept of independence preserving embeddings where embeddings use partial orthogonalities to preserve the conditional independence structures of random variables. This opens the door for substantial future work. In particular, one promising theoretical direction is to study if CLIP text embeddings preserve the structures in the training distributions.
This work is supported by ONR grant N00014-23-1-2591, Open Philanthropy, NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business.
[Ali+10] C. F. Aliferis, A. Statnikov, I. Tsamardinos, S. Mani, and X. D. Koutsoukos. "Local causal and markov blanket induction for causal discovery and feature selection for classification part i: algorithms and empirical evaluation." In: Journal of Machine Learning Research 1 (2010).
[AH19] C. Allen and T. Hospedales. “Analogies explained: towards understanding word embeddings”. In: International Conference on Machine Learning. PMLR. 2019.
[AAZ22] A. A. Amini, B. Aragam, and Q. Zhou. “A non-graphical representation of conditional independence via the neighbourhood lattice”. In: arXiv preprint arXiv:2206.05829 (2022).
[Aro+16] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. "A latent variable model approach to pmi-based word embeddings". In: Transactions of the Association for Computational Linguistics (2016).
[BK19] T. Boege and T. Kahle. "Construction methods for gaussoids". In: arXiv preprint arXiv:1902.11260 (2019).
[Bub+23] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. "Sparks of artificial general intelligence: early experiments with gpt-4". In: arXiv preprint arXiv:2303.12712 (2023).
A Additional Proofs
Lemma 15. Let  $v$  be an element in  $\mathbf{E}$ , and  $\mathcal{M}$  a Markov boundary of  $v$ , then  $\mathcal{M}$  is also a generalized Markov boundary.
Proof. By definition, for any  $u \in V \backslash (\{v\} \cup \mathcal{M})$ ,  $\mathrm{S}^{\mathrm{c}}_{\mathcal{M}}(v,u) = 0$ . Therefore,  $\mathcal{M}$  is also a generalized Markov boundary.
$$
p r o j _ {\mathcal {M} _ {1}} [ v _ {i} ] = p r o j _ {\mathcal {M} _ {2}} [ v _ {i} ]
$$
Proof. To slightly abuse notation, we also use  $\mathcal{M}_1$  to be a matrix where each column is an element in  $\mathcal{M}_1$ . We define  $\mathcal{M}_2$  similarly.
Because  $\mathcal{M}_1$  and  $\mathcal{M}_2$  are two distinct Markov boundaries, they must not be empty. Therefore,  $v_i \not\ll_{\mathrm{O}} \mathcal{M}_1$  and  $v_i \not\ll_{\mathrm{O}} \mathcal{M}_2$ . By the definition of Markov boundary, we also have  $v_i \not\ll_{\mathrm{O}} \mathcal{M}_1 | \mathcal{M}_2$  and  $v_i \not\ll_{\mathrm{O}} \mathcal{M}_2 | \mathcal{M}_1$ . Note that  $\mathcal{M}_2$  and  $\mathcal{M}_1$  must have full rank, otherwise, they are not minimal.
$$
\begin{array}{l} \langle \operatorname {p r o j} _ {\mathcal {M} _ {1}} ^ {\perp} [ v _ {i} ], \operatorname {p r o j} _ {\mathcal {M} _ {1}} ^ {\perp} [ v _ {j} ] \rangle = 0, \forall v _ {j} \in \mathcal {M} _ {2} \\ \Longleftrightarrow \langle \operatorname {p r o j} _ {\mathcal {M} _ {1}} ^ {\perp} [ v _ {i} ], v _ {j} \rangle = 0, \forall v _ {j} \in M _ {2} \\ \Longleftrightarrow v _ {i} ^ {T} \mathcal {M} _ {1} \left(\mathcal {M} _ {1} ^ {T} \mathcal {M} _ {1}\right) ^ {- 1} \mathcal {M} _ {1} ^ {T} v _ {j} = v _ {i} ^ {T} v _ {j} \forall v _ {j} \in \mathcal {M} _ {2} \\ \Longleftrightarrow v _ {i} ^ {T} \mathcal {M} _ {1} \left(\mathcal {M} _ {1} ^ {T} \mathcal {M} _ {1}\right) ^ {- 1} \mathcal {M} _ {1} ^ {T} \mathcal {M} _ {2} = v _ {i} ^ {T} \mathcal {M} _ {2} \\ \end{array}
$$
With (compact) singular value decomposition, we have  $\mathcal{M}_1 = U_1\Sigma_1V_1^T$  and  $\mathcal{M}_2 = U_2\Sigma_2V_2^T$ . Then,
$$
\begin{array}{l} v _ {i} ^ {T} \mathcal {M} _ {1} \left(\mathcal {M} _ {1} ^ {T} \mathcal {M} _ {1}\right) ^ {- 1} \mathcal {M} _ {1} ^ {T} \mathcal {M} _ {2} = v _ {i} ^ {T} U _ {1} U _ {1} ^ {T} \mathcal {M} _ {2} = v _ {i} ^ {T} \mathcal {M} _ {2} \\ \Longleftrightarrow v _ {i} ^ {T} U _ {1} U _ {1} ^ {T} U _ {2} = v _ {i} ^ {T} U _ {2} \\ \end{array}
$$
$$
v _ {i} ^ {T} U _ {2} U _ {2} ^ {T} U _ {1} = v _ {i} ^ {T} U _ {1}
$$
$$
v _ {i} ^ {T} U _ {1} U _ {1} ^ {T} U _ {2} U _ {2} ^ {T} = v _ {i} ^ {T} U _ {2} U _ {2} ^ {T}
$$
$$
\operatorname {p r o j} _ {\mathcal {M} _ {2}} \left[ \operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] \right] = \operatorname {p r o j} _ {\mathcal {M} _ {2}} \left[ v _ {i} \right]
$$
On the other hand,  $v_{i}\not\equiv_{\mathrm{O}}\mathcal{M}_{1}$
$$
\operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] = U _ {1} U _ {1} ^ {T} v _ {i} \neq 0
$$
$$
\operatorname {p r o j} _ {\mathcal {M} _ {2}} \left[ v _ {i} \right] = U _ {2} U _ {2} ^ {T} v _ {i} \neq 0
$$
Therefore, we must have,
$$
\operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] \in \operatorname {s p a n} \left(\mathcal {M} _ {2}\right)
$$
which means,
$$
\operatorname {p r o j} _ {\mathcal {M} _ {2}} \left[ \operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] \right] = \operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] = \operatorname {p r o j} _ {\mathcal {M} _ {2}} \left[ v _ {i} \right]
$$
$$
\operatorname {p r o j} _ {\mathcal {M} _ {1}} [ v _ {i} ] = v _ {i}.
$$
Proof. Because  $v_{i} \in \operatorname{span}(\mathrm{E} \setminus \{v_{i}\})$ , then  $v_{i} = \sum_{k=1}^{m} \alpha_{k} v_{k}$  with  $\mathrm{E}' = \{v_{1}, \dots, v_{m}\} \subseteq \mathrm{E}$ .
Since  $\mathcal{M}_1$  is a Markov boundary of  $v_{i}$
$$
v _ {i} ^ {T} \mathcal {M} _ {1} \left(\mathcal {M} _ {1} ^ {T} \mathcal {M} _ {1}\right) ^ {- 1} \mathcal {M} _ {1} ^ {T} v _ {k} = v _ {i} ^ {T} v _ {k} \forall v _ {k} \in \mathrm {E} ^ {\prime}
$$
$$
v _ {i} ^ {T} \mathcal {M} _ {1} \left(\mathcal {M} _ {1} ^ {T} \mathcal {M} _ {1}\right) ^ {- 1} \mathcal {M} _ {1} ^ {T} \sum_ {k = 1} ^ {m} \alpha_ {k} v _ {k} = v _ {i} ^ {T} \sum_ {k = 1} ^ {m} \alpha_ {k} v _ {k}
$$
$$
\left\langle p r o j _ {\mathcal {M} _ {1}} \left[ v _ {i} \right], v _ {i} \right\rangle = \left\langle v _ {i}, v _ {i} \right\rangle
$$
Thus,  $\text{proj}_{\mathcal{M}_1}[v_i] = v_i$ .
A.1 Construction of IPE Map
Proof. Let  $|V| = n$  and, to slightly abuse notation, we use index  $i \in [n]$  to mean the vertex  $v_{i}$  and the  $i$ -th embedding. And we use  $A_{V_1,V_2}$  where  $V_{1}, V_{2} \subseteq V$  to mean the submatrix of  $A_{\{i:v_i \in V_1\}, \{i:v_i \in V_2\}}$ .
Because  $G_{p}$  is a minimal  $I$ -map of  $P$ , we have  $\mathcal{I}_G(V)\subseteq \mathcal{I}_P(V)$ .
We just need to show  $\mathcal{I}_G(V) = \mathcal{I}_O(f(V))$ . And if  $P$  is faithful to  $G_{p}$ , then  $\mathcal{I}_O(f(V)) = \mathcal{I}_G(V) = \mathcal{I}_P(V)$ .
If  $v_{i} \perp_{\mathrm{G}} v_{j}|V'$  where  $V' \subseteq V$ , let  $V^{c} = V \setminus V'$ , then
$$
A = \left( \begin{array}{c} A _ {V ^ {c}}, A _ {V ^ {c}, V ^ {\prime}} \\ A _ {V ^ {\prime}, V ^ {c}}, A _ {V ^ {\prime}} \end{array} \right)
$$
On the other hand, if  $f(v_{i}) \perp_{\mathrm{O}} f(v_{j}) | f(V')$ , then
$$
f \left(v _ {i}\right) ^ {T} f \left(v _ {j}\right) - f \left(v _ {i}\right) ^ {T} f \left(V ^ {\prime}\right) \left(f \left(V ^ {\prime}\right) ^ {T} f \left(V ^ {\prime}\right)\right) ^ {- 1} f \left(V ^ {\prime}\right) ^ {T} f \left(v _ {j}\right) = 0 \tag {A.1}
$$
Note that by our construction,  $(f(V^{\prime})^{T}f(V^{\prime}))^{-1} = A_{V^{\prime}}$  is invertible. We can write (A.1) as follows:
$$
f (v _ {i}) ^ {T} f (v _ {j}) - f (v _ {i}) ^ {T} f (V ^ {\prime}) (f (V ^ {\prime}) ^ {T} f (V ^ {\prime})) ^ {- 1} f (V ^ {\prime}) ^ {T} f (v _ {j}) = A _ {i, j} - A _ {i, V ^ {\prime}} A _ {V ^ {\prime}} ^ {- 1} A _ {V ^ {\prime}, j}
$$
By Schur's complement, we have that
$$
\left(\operatorname {a d j} _ {\varepsilon} \left(G _ {P}\right) _ {V ^ {c}}\right) ^ {- 1} = \left(A ^ {- 1}\right) _ {V ^ {c}} ^ {- 1} = A _ {V ^ {c}} - A _ {V ^ {c}, V ^ {\prime}} A _ {V ^ {\prime}} ^ {- 1} A _ {V ^ {\prime}, V ^ {c}}
$$
Because  $\varepsilon$  is a perfect perturbation factor, by definition,  $f(v_{i})\perp_{\mathrm{O}}f(v_{j})|f(V^{\prime})$  if and only if  $v_{i}\perp_{\mathrm{G}}v_{j}|V^{\prime}$ . By the compositional property, we have that  $\mathcal{I}_G(V) = \mathcal{I}_O(f(V))$ .
Proof. This is a direct consequence of Theorem 1 in [LM07].
A.2 Dimension Reduction of IPE
Theorem 13. Let  $U$  be a set of vectors in  $\mathbb{R}^n$  where  $n = |U|$  and every vector is a unit vector. Let  $\Sigma$  be a matrix in  $\mathbb{R}^{n\times n}$  where  $\Sigma_{ij} = \langle u_i,u_j\rangle$ . Assume  $\lambda_1 = \lambda_{\min}(\Sigma) > 0$ . Then there exists a mapping  $g:\mathbb{R}^n\to \mathbb{R}^k$  where  $k = \lceil 20\log (2n) / (\varepsilon ')^2\rceil$  with  $\varepsilon^{\prime} = \min \{1 / 2,\varepsilon /C,\lambda_{1} / 2r^{2}\}$  and  $\varepsilon \in (0,1)$  such that for any  $u_{i}\in U$  with its unique Markov boundary  $M_{i}\subseteq U$  and any  $u_{j}\in U\setminus (\{u_{i}\} \cup M_{i})$ , we have
where  $r_i = |M_i|$ ,  $r = \max_i|M_i|$  and  $C = (r + 1)^3\left(\frac{2\lambda_{\max}(\Sigma) + 2(r + 1)^2}{\lambda_{\min}(\Sigma)}\right)^r$ .