SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yibo Jiang $^{1}$ , Bryon Aragam $^{2}$ , and Victor Veitch $^{3,4}$
$^{1}$ Department of Computer Science, University of Chicago
$^{2}$ Booth School of Business, University of Chicago
$^{3}$ Department of Statistics, University of Chicago
$^{4}$ Data Science Institute, University of Chicago
Machine learning tools often rely on embedding text as vectors of real numbers. In this paper, we study how the semantic structure of language is encoded in the algebraic structure of such embeddings. Specifically, we look at a notion of "semantic independence" capturing the idea that, e.g., "eggplant" and "tomato" are independent given "vegetable". Although such examples are intuitive, it is difficult to formalize such a notion of semantic independence. The key observation here is that any sensible formalization should obey a set of so-called independence axioms, and thus any algebraic encoding of this structure should also obey these axioms. This leads us naturally to use partial orthogonality as the relevant algebraic structure. We develop theory and methods that allow us to demonstrate that partial orthogonality does indeed capture semantic independence. Complementary to this, we also introduce the concept of independence preserving embeddings where embeddings preserve the conditional independence structures of a distribution, and we prove the existence of such embeddings and approximations to them.
This paper concerns the question of how semantic meaning is encoded in neural embeddings, such as those produced by [Rad+21]. There is strong empirical evidence that these embeddings—vectors of real numbers—capture the semantic meaning of the underlying text. For example, classical results show that word embeddings can be used for analogical reasoning [e.g., Mik+13; PSM14], and such embeddings are the backbone of modern generative AI systems [e.g., Ram+22; Bub+23; Sah+22; Dev+18]. The high-level question we're interested in is: How is the semantic structure of text encoded in the algebraic structure of embeddings? In this paper, we provide evidence that the concept of partial orthogonality plays a key role.
The first step is to identify the semantic structure of interest. Intuitively, words or phrases possess a notion of semantic independence, which does not have to be statistical in nature. For example, the word "eggplant" seems more similar to "tomato" than to "ennui". Yet, if we were to "condition" on the common property of "vegetable", then "eggplant" and "tomato" should be "independent". And, if we condition on both "vegetable" and "purple", then "eggplant" may be "independent" of all other words. However, it is difficult to formalize what is meant by "independent" and "condition on" in these informal statements. Accordingly, it is hard to establish a formal definition of semantic independence, and thus it is challenging to explore how this structure might be encoded algebraically!
The key observation in this paper is to recall that most reasonable concepts of "independence" adhere to a common set of axioms similar to those defining probabilistic conditional independence. Formally, this abstract idea is captured by the axioms of the so-called independence models [Lau96]. Thus, if
(a) 'eggplant'
(b) 'zebra'
Figure 1: For target embedding of "eggplant", the set of embeddings that include "purple" and "vegetable" forms the subspace such that after projection, the residual of 'eggplant' has the lowest cosine similarity with residuals of other test embeddings. This matches our intuition for the meaning of "eggplant". Similarly, for target embedding of "zebra", the set of embeddings that include "striped" and "animal" forms the most suitable subspace.
semantic independence is encoded algebraically, it should be encoded as an algebraic structure that respects these axioms. In this paper, we use a natural candidate independence model in vector spaces known as partial orthogonality [Lau96; AAZ22]. Here, for two vectors  $v_{a}$  and  $v_{b}$  and a conditioning set of vectors  $v_{C}$ , partial orthogonality takes  $v_{a}$  independent  $v_{b}$  given  $v_{C}$  if the residuals of  $v_{a}$  and  $v_{b}$  are orthogonal after projecting onto the span of  $v_{C}$ . We discover that this particular tool is indeed valuable for understanding CLIP embeddings. For instance, Figure 1 shows that after projecting onto the linear subspace spanned by CLIP embeddings of "purple" and "vegetable", the residual of embedding "eggplant" has on average low cosine similarity with the residuals of random test embeddings, which also matches our intuitive understanding of the word.
Since partial orthogonality is an independence model, we can go one step further to define Markov boundaries for embeddings as well. Drawing inspiration from graphical models, it is reasonable to expect that the Markov boundary of any target embedding should constitute a minimal collection of embeddings that encompasses valuable information regarding the target. Unlike classical applications of partial orthogonality in regression and Gaussian models, however, the geometry of embeddings presents several subtle technical challenges to directly adopting the usual notion of Markov boundary. First, the intersection axiom never holds for practical embeddings, which makes the standard Markov boundary non-unique. More importantly, practical embeddings could potentially incorporate distortion, noise and undergo phenomena resembling superposition [Elh+22]. Therefore, in this paper, we introduce generalized Markov boundaries for studying the structure of text embeddings.
Contributions Specifically, we make the following contributions:
1. We adapt ideas from graphical independence models to specify the structure that should be satisfied by semantic independence. We discover that partial orthogonality in the embedding space offers a natural way of encoding semantic independence structure (Section 2).
2. We study the semantic structure of partial orthogonality via Markov boundaries. Due to the unique characteristics of embeddings and noise in learning, exact orthogonality is unlikely to hold. So, we give a distributional relaxation of the Markov boundary and use this to provide a practical algorithm for finding generalized Markov boundaries and measuring the semantic independence induced by generalized Markov boundaries (Section 3.2).
3. We introduce the concept of independence preserving embeddings, which studies how embeddings can be used to maintain the independence structure of distributions. This holds its own intrigue for further research (Section 4).
4. Finally, we design and conduct experimental evaluations on CLIP text embeddings, finding that the partial orthogonality structure and generalized Markov boundary encode semantic structure (Section 5).
Throughout, we use CLIP text embeddings as a running example, though the method and theory presented can be applied more broadly.
Related work There are many papers [e.g., Aro+16; GAM17; AH19; EDH19; Tra+23; Per+23; Lee+23; MEP23; Wan+23] connecting semantic meanings and algebraic structures of popular embeddings like CLIP [Rad+21], Glove [PSM14] and word2vec [Mik+13]. Simple arithmetic on these embeddings reveals that they carry semantic meanings. The most popular arithmetic operation is called linear analogy [EDH19]. There are several papers trying to understand the reasoning behind this phenomenon. Arora et al. [Aro+16] explains this by proposing the latent variable model but it requires the word vectors to be uniformly distributed in the embedding space which generally is not true in practice [MT17]. Alternatively, [GAM17; AH19] adopts the paraphrase model that also does not fit practice. [EDH19], on the other hand, studies the geometry of embeddings that decomposes the shifted pointwise mutual information (PMI) matrix. Trager et al. [Tra+23] and Perera et al. [Per+23] decomposes embeddings into combinations of a smaller set of vectors that are more interpretable. On the other hand, similar to using vector orthogonality to represent (conditional) independence, kernel mean embeddings [Mua+17] are Hilbert space embeddings of distributions that can also be used to represent conditional independences [Son+09; SFG13]. It is a popular method for machine learning, and causal inference [Gre+05; Moo+09; GS20]. But unlike independence preserving embeddings, kernel mean embeddings use the kernel and do not explicitly construct finite-dimensional vector representations.
2 Independence Model and Markov Boundary
Let  $\mathrm{E}$  be a finite set of embeddings with  $|\mathrm{E}| = n$  and each embedding is of size  $d$ . Every embedding is a vector representation of a word. In other words, there exists a function  $f$  that maps words to  $n$  vectors in  $\mathbb{R}^d$ . As explained above, we might expect embeddings to encode "independence structures" between words. These independence structures are difficult to define formally, though the structure is similar to that of probabilistic conditional independence. We will use independence models as an abstract formalization of this structure.
2.1 Independence Model
Throughout this paper, we use many standard definitions and facts about graphical models and more generally, abstract independence models. A detailed overview of this material can be found, for instance, in [Lau96; Stu05].
Suppose  $V$  is a finite set. In the case of embeddings,  $V$  would be a set of vectors. An independence model  $\mathbb{1}_{\sigma}$  is a ternary relation on  $V$ . Let  $A, B, C, D$  be disjoint subsets of  $V$ . Then a semi-graphoid is an independence model that satisfies the following axioms:
(A1) (Symmetry) If  $A \perp_{\sigma} B|C$ , then  $B \perp_{\sigma} A|C$ ;
(A2) (Decomposition) If  $A\perp_{\sigma}(B\cup D)|C$  , then  $A\perp_{\sigma}B|C$  and  $A\perp_{\sigma}D|C$
(A3) (Weak Union) If  $A\perp \mathbb{L}_{\sigma}(B\cup D)|C$  , then  $A\perp \mathbb{L}_{\sigma}B|(C\cup D)$
(A4) (Contraction) If  $A\perp_{\sigma}B|C$  and  $A\perp_{\sigma}D|(B\cup C)$ , then  $A\perp_{\sigma}(B\cup D)|C$
The independence model is a graphoid if it also satisfies
(A5) (Intersection) If  $A\perp_{\sigma}B|(C\cup D)$  and  $A\perp_{\sigma}C|(B\cup D)$ , then  $A\perp_{\sigma}(B\cup C)|D$
And, the graphoid is called a compositional graphoid if it also satisfies
(A6) (Composition) If  $A \perp_{\sigma} B|C$  and  $A \perp_{\sigma} D|C$ , then  $A \perp_{\sigma}(B \cup D)|C$ .
We also use  $\mathcal{I}_{\sigma}(V)$  to be the set of conditional independent tuples under the independence model  $\perp_{\sigma}$ . In other words, if  $(A,B,C) \in \mathcal{I}_{\sigma}(V)$ , then  $A \perp_{\sigma} B|C$  where  $A, B, C$  are disjoint subsets of  $V$ .
Probabilistic Conditional Independence  $(\mathbb{L}_{\mathrm{P}})$  Given a finite set of random variables  $V$ , probabilistic conditional independence over  $V$  defines an independence model that satisfies (A1)-(A4) which means that probabilistic independence models are semi-graphoids. In general, however, they are not compositional graphoids. If the distribution has strictly positive density w.r.t. a product measure, then the intersection axiom is true. In this case, probabilistic independence models are graphoids. Still, in general, the composition axiom is not satisfied because pairwise independence does not imply
joint independence. One notable exception is when the distribution is regular multivariate Gaussian; then the probabilistic independence model is a compositional graphoid.
Undirected Graph Separations  $(\mathbb{1}_{\mathrm{G}})$  For a finite undirected graph  $\mathcal{G} = (V,E)$ . One can easily show that ordinary graph separation in undirected graphs is a compositional graphoid. The relations between probabilistic conditional independences and graph separations are well-studied in the graphical modeling literature [KF09; Lau96]. We recall a few important definitions here for completeness. Consider a natural bijection between graphical nodes and random variables. Then if  $\mathcal{I}_G(V)\subseteq \mathcal{I}_P(V)$ , we say the distribution  $\mathcal{P}$  over  $V$  satisfies the Markov property with respect to  $\mathcal{G}$  and  $\mathcal{G}$  is called an  $I$ -map of  $\mathcal{P}$ . An I-map  $\mathcal{G}$  for  $\mathcal{P}$  is minimal if no subgraph of  $\mathcal{G}$  is also an I-map of  $\mathcal{P}$ . It is not difficult to show that there exists a minimal I-map  $\mathcal{G}$  for any distribution  $\mathcal{P}$ .
Remark 1. Not every compositional graphoid can be represented by an undirected graph. Sadeghi [Sad17] provides sufficient and necessary conditions for this.
Partial Orthogonality  $(\mathbb{L}_{\mathrm{O}})$  Let  $V$  be a finite collection of vectors in  $\mathbb{R}^d$ . If  $a \in V, b \in V$  and  $C \subseteq V$ , then we say that  $a$  and  $b$  are partially orthogonal given  $C$  if
$$
a \perp_ {\mathrm {O}} b | C \Longleftrightarrow \left\langle \operatorname {p r o j} _ {C} ^ {\perp} [ a ], \operatorname {p r o j} _ {C} ^ {\perp} [ b ] \right\rangle = 0,
$$
where  $\mathrm{proj}_{C}^{\perp}[a] = a - \mathrm{proj}_{C}[a]$  is the residual of  $a$  after projection onto the span of  $C$ . It is not hard to verify that  $\perp_{\mathrm{O}}$  is a semi-graphoid that also satisfies the composition axiom (A6). When  $V$  is a set of linearly independent vectors, then  $\perp_{\mathrm{O}}$  satisfies (A5) and thus is a compositional graphoid. Partial orthogonality has been studied under different names in the statistics literature for many decades. For example, if we replace Euclidean space with the  $L^2$  space of random variables, partial orthogonality is equivalent to the well-known concept of partial correlation or second-order independence (Example 2.26 in [Lau20]). The concept of geometric orthogonality (Example 2.27 in [Lau20]) is closely related but does not always satisfy the intersection axiom. More recently, the concept of partial orthogonality in abstract Hilbert spaces was defined and studied extensively in [AAZ22]. Finally, when  $V$  is a linearly independent collection of vectors, partial orthogonality yields a stronger independence model known as a Gaussoid, which is well-studied [e.g. LM07; BK19, and the references therein]. It is worth emphasizing that in the present setting of text embedding, we typically have  $d \ll n$ , and hence  $V$  cannot be linearly independent.
2.2 Markov boundaries
Suppose  $\perp_{\sigma}$  is an independence model over a finite set  $V$ . Let  $v_{i}$  be an element in  $V$ , then the Markov blanket  $\mathcal{M}$  of  $v_{i}$  is any subset of  $V \setminus \{v_{i}\}$  such that
$$
v _ {i} \perp_ {\sigma} V \setminus (\{v _ {i} \} \cup \mathcal {M}) | \mathcal {M}
$$
A Markov boundary is a minimal Markov blanket.
A Markov boundary, by definition, always exists and can be an empty set. However, it might not be unique. It is well-known that the intersection property is a sufficient condition to guarantee Markov boundaries are unique. Thus, the Markov boundary is unique in any graphoid. The proof is presented here for completeness.
Theorem 2. If  $\perp_{\sigma}$  is a graphoid over  $V$ , then the Markov boundary is unique for any element in  $V$ .
Proof. Let  $v_{i} \in V$ . Suppose  $v_{i}$  has two distinct Markov boundaries  $\mathcal{M}_1, \mathcal{M}_2$ . Then they must be non-empty and  $v_{i} \not\perp_{\sigma} \mathcal{M}_{1}, v_{i} \not\perp_{\sigma} \mathcal{M}_{2}, v_{i} \not\perp_{\sigma} \mathcal{M}_{2} | \mathcal{M}_{1}, v_{i} \not\perp_{\sigma} \mathcal{M}_{1} | \mathcal{M}_{2}$ . By the intersection axiom,  $v_{i} \not\perp_{\sigma} \mathcal{M}_{1} \cup \mathcal{M}_{1}$ . Then by the decomposition axiom,  $v_{i} \not\perp_{\sigma} \mathcal{M}_{1}$  and  $v_{i} \not\perp_{\sigma} \mathcal{M}_{2}$  which is a contradiction.
Remark 3. For any semi-graphoid, the intersection property is not a necessary condition for the uniqueness of Markov boundaries. See Remark 1 in [WW20].
The connection between orthogonal projection and graphoid axioms is well-known [Lau96; Daw01; Whi09]. But graphoid axioms find their primary applications in graphical models [Lau96]. In particular, there are many existing papers on Markov boundary discovery for graphical models [Tsa+03; Ali+10; SV16; GA21; TAS03; Pen+07]. They typically assume faithfulness or the
distributions are strictly positive, which are sufficient conditions for the intersection property and thus ensure unique Markov boundaries. As an important axiom for graphoids, the intersection property has also been thoroughly investigated [SMMR05; Pet15; Fin11]. But the intersection property rarely holds for embeddings (See Section 3), which means there could be multiple Markov boundaries. [SLA13; WW20] study this case for graphical models and causal inference.
3 Markov Boundary of Embeddings
As indicated in Section 2, partial orthogonality  $(\mathbb{L}_{\mathrm{O}})$  can be used as an independence model over vectors in Euclidean space and is a compositional semi-graphoid. Thus, one can use partial orthogonality to study embeddings, which are real vectors. When  $n \leq d$  and the vectors in  $\mathbf{E}$  are linearly independent, every vector in  $\mathbf{E}$  has a unique Markov boundary by Theorem 2.
Unfortunately, when  $d < n$ , which happens in practice with embeddings as there are usually more objects to embed than the embedding dimension, there is a possibility of having multiple Markov boundaries. In fact, the main challenge with Markov boundary discovery for embeddings is that the intersection property generally does not hold, as opposed to graphical models where this property is commonly assumed [Tsa+03; Ali+10; SV16].
While the Markov boundary might not be unique, the following theorem says that all Markov boundaries of the target vector capture the same "information" about that vector.
Theorem 4. Let partial orthogonality  $\perp_{\mathrm{O}}$  be the independence model over a finite set of embedding vectors E. Suppose  $\mathcal{M}_1, \mathcal{M}_2 \subseteq \mathrm{E}$  are two distinct Markov boundaries of  $v_i \in \mathrm{E}$ , then,
$$
\operatorname {p r o j} _ {\mathcal {M} _ {1}} [ v _ {i} ] = \operatorname {p r o j} _ {\mathcal {M} _ {2}} [ v _ {i} ]
$$
When  $d \ll n$ , then it is likely that the target embedding  $v_{i}$  lies in the linear span of other embeddings (i.e.,  $v_{i} \in \operatorname{span}(\mathrm{E} \setminus \{v_{i}\})$ ), Corollary 5 below shows that, in this case, the span of any Markov boundary is precisely the subspace that contains  $v_{i}$ :
Corollary 5. Let parital orthogonality  $\mathbb{L}_{\mathrm{O}}$  be the independence model over a finite set of embedding vectors E. Suppose  $\mathcal{M}_1\subseteq \mathrm{E}$  is a Markov boundary of  $v_{i}\in \mathrm{E}$  and  $v_{i}\in \operatorname {span}(\mathrm{E}\setminus \{v_{i}\})$  , then,
$$
\operatorname {p r o j} _ {\mathcal {M} _ {1}} \left[ v _ {i} \right] = v _ {i}.
$$
In other words, to find a Markov boundary of  $v_{i}$ , we need to find some vectors such that their linear combination is exactly  $v_{i}$ . This seems very strict but is necessary because the formal definition of the Markov boundary requires residual orthogonalities between  $v_{i}$  and every other vector. In the sequel, we show how to relax the definition of the Markov boundary.
3.1 From Elementwise Orthogonality to Distributional Orthogonality
Corollary 5 suggests that the span of the Markov boundary for any target vector should contain that target vector. This is a consequence of the elementwise orthogonality constraint because the definition of the Markov boundary requires the residual of a target vector to be orthogonal to the residual of any test vector. The implicit assumption here is that embeddings are distortion-free and every non-zero correlation is meaningful. However, due to the inherent limitation of the embedding dimension—which often restricts the available space for storing all the orthogonal vectors—and noises introduced from training, embeddings are likely prone to distortion when compressed into a relatively small Euclidean space. In fact, we empirically show in Section 5.2 that inner products in embedding space do not necessarily respect semantic meanings faithfully. Therefore, the notion of elementwise orthogonality loses practical significance.
Instead of enforcing elementwise orthogonality, we relax the definition of the Markov boundary of embeddings such that intuitively, after projection, the residual of the target vector and the residuals of test vectors should be orthogonal in a distributional sense where the distribution is the empirical distribution over test vectors. To capture distributional orthogonalities, this paper focuses on the average of cosine similarities.
In particular, we have the following definition of generalized Markov boundary for partial orthogonality.
Algorithm 1: Approximate Algorithm to Find Generalized Markov Boundary
Input:  $v,\mathrm{E}$    
/\* E is the set of all embeddings and  $v\in \mathbb{E}$  is the target embedding \*/   
Input:  $n_r,d_r,K$ $\ast n_{r}$  is the number of sampled random subspaces,  $d_{r}$  is the number of sampled vectors for each random subspace and  $K$  is the number of candidate vectors to construct generalized Markov boundary \*/   
Output:  $\mathcal{M}\subseteq \mathbb{E}$ $\ast \mathcal{M}$  is the estimated generalized Markov boundary for  $v$    
for  $i\gets 1$  to  $n_r$  do randomly sample a set of  $d_{r}$  vectors  $\mathcal{M}_i = \{v_k^i\}_{k = 1}^{dr}\subseteq (\mathrm{E}\setminus \{v\})$  caculate  $\mathrm{S}^{\mathrm{c}}_{\mathcal{M}_i}(v,u)$  for all  $u\in (\mathrm{E}\backslash \{v\})$    
end   
Find the top  $K$  vectors  $\{u_i\}_{i = 1}^K$  with the highest  $\sum_{i}\mathrm{S}^{\mathrm{c}}_{\mathcal{M}_i}(v,u)$    
Find the subset  $\mathcal{M}$  of  $\{u_i\}_{i = 1}^K$  that has the lowest  $\mathrm{S}_{\mathcal{M}}(v,\mathrm{E})$    
Result:  $\mathcal{M}$
Definition 6 (Generalized Markov Boundary for Partial Orthogonality). Given a finite set  $\mathrm{E}$  of embedding vectors. Let  $v$  be an element in  $\mathrm{E}$ , then a generalized Markov boundary  $\mathcal{M}$  of  $v$  is a minimal subset of  $\mathrm{E} \setminus \{v\}$  such that
$$
\mathrm {S} _ {\mathcal {M}} (v, \mathrm {E}) = \frac {1}{| \mathrm {E} _ {\mathcal {M}} ^ {v} |} \sum_ {u \in \mathrm {E} _ {\mathcal {M}} ^ {v}} \mathrm {S} ^ {c} _ {\mathcal {M}} (v, u) = 0
$$
where  $\mathrm{S}^{\mathrm{c}}_{\mathcal{M}}(v,u)$  is the cosine similarity of  $u$  and  $v$  after projection and  $\mathrm{E}_{\mathcal{M}}^{v} = \mathrm{E}\backslash (\{v\} \cup \mathcal{M})$ . Specifically,  $\mathrm{S}^{\mathrm{c}}_{\mathcal{M}}(v,u) = \frac{\langle\mathrm{proj}_{\mathcal{M}}^{\perp}[v],\mathrm{proj}_{\mathcal{M}}^{\perp}[u]\rangle}{||\mathrm{proj}_{\mathcal{M}}^{\perp}[v]||\cdot||\mathrm{proj}_{\mathcal{M}}^{\perp}[u]||}$ .
Intuitively, this suggests that, on average, there is no particular direction of residuals that have nontrivial correlations with the residual of the target embedding.
Remark 7. It is evident that the conventional definition of Markov boundary implies Definition 6 (Lemma 15 in Appendix A).
3.2 Finding Generalized Markov Boundary
With a formal definition of the generalized Markov boundary established, our objective is now to identify this boundary. One can always use brute force by enumerating all possible subsets of  $\mathrm{E}$ , but the algorithm would be infeasible when  $|\mathrm{E}|$  is large.
Suppose  $v \in \mathrm{E}$  is a target vector and  $\mathcal{M}$  is its generalized Markov boundary, then we can write  $v = v_{\perp} + v_{\parallel}$  where  $v_{\perp} = \operatorname{proj}_{\mathcal{M}}^{\perp}[v]$  and  $v_{\parallel} = \operatorname{proj}_{\mathcal{M}}[v]$ . Intuitively, Definition 6 suggests that the residual of test vectors can appear in any direction relative to  $v_{\perp}$ . Therefore, if one samples random test vectors  $\{u_i\}$ , their span is likely to be close to  $v_{\perp}$ . In other words, the residual of  $v$  after projection onto span  $(\{u_i\})$  should contain more information about the generalized Markov boundary direction  $v_{\parallel}$ .