SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $g$  be a linear map of Lemma 16 with error parameter  $\varepsilon' \in (0, \frac{1}{2})$ . For convenience, let  $\tilde{u}_i = g(u_i)$ ,  $\tilde{u}_j = g(u_j)$  and  $\tilde{M}_i = g(M_i)$ . Let  $r_i = |M_i|$ . To slightly abuse notation, we use  $M_i$  and  $\tilde{M}_i$  to also mean matrices where each column is an element in the set. Furthermore, we also define  $\tilde{\Sigma}$  to be  $\tilde{\Sigma}_{ij} = \langle \tilde{u}_i, \tilde{u}_j \rangle$ . We use  $\Sigma_{A,B}$  where  $A, B \subseteq U$  to be a submatrix where the row indices are from  $A$  and the column indices are from  $B$ , and when  $A = B$ , we just use  $\Sigma_A$  for simplicity. In particular, let  $\Sigma_{(i,M_i),(j,M_i)} = \left( \begin{array}{cc} \Sigma_{i,j}, & \Sigma_{i,M_i} \\ \Sigma_{M_i,j}, & \Sigma_{M_i} \end{array} \right)$ . And we can define a similar thing for  $\tilde{\Sigma}$ .
We first want to find  $\varepsilon^{\prime}$  such that  $\tilde{\Sigma}_{M_i}$  is non-singular for all  $i\in |U|$ . Note that for any  $u_{i}\in U$ , we know that by Weyl's inequality for eigenvalues [HJ12],
$$
\left| \lambda_ {\min} (\tilde {\Sigma} _ {M _ {i}}) - \lambda_ {\min} (\Sigma_ {M _ {i}}) \right| \leq \left| \left| \tilde {\Sigma} _ {M _ {i}} - \Sigma_ {M _ {i}} \right| \right| \leq \left| \left| \tilde {\Sigma} _ {M _ {i}} - \Sigma_ {M _ {i}} \right| \right| _ {F} \leq r ^ {2} \varepsilon^ {\prime}
$$
$$
\lambda_ {\min } \left(\tilde {\Sigma} _ {M _ {i}}\right) \geq \lambda_ {\min } \left(\Sigma_ {M _ {i}}\right) - r ^ {2} \varepsilon \geq \lambda_ {\min } (\Sigma) - r ^ {2} \varepsilon = \lambda_ {1} - r ^ {2} \varepsilon^ {\prime}
$$
Therefore, if we want  $\lambda_{\min}(\tilde{\Sigma}_{M_i}) > \frac{\lambda_1}{2}$  we need  $\varepsilon' < \frac{\lambda_1}{2r^2}$ .
On the other hand, because  $M_{i}$  is an Markov boundary for  $u_{i}$ , we have
$$
u _ {i} ^ {T} u _ {j} - u _ {i} ^ {T} M _ {i} \left(M _ {i} ^ {T} M _ {i}\right) ^ {- 1} M _ {i} ^ {T} u _ {j} = 0 \tag {A.2}
$$
Note that  $M_{i}$  must be full rank. Otherwise, we can find a subset of  $M_{i}$  to be the Markov boundary. And there is a different way to write this. Remember that  $\Sigma_{(i,M_i),(j,M_i)} = \left( \begin{array}{cc}\Sigma_{i,j}, & \Sigma_{i,M_i}\\ \Sigma_{M_i,j}, & \Sigma_{M_i} \end{array} \right)$ . Using Schur's complement, we have that
$$
\begin{array}{l} \det  \left(\Sigma_ {\left(i, M _ {i}\right), \left(j, M _ {i}\right)}\right) = \det  \left(\Sigma_ {M _ {i}}\right) \det  \left(\Sigma_ {i, j} - \Sigma_ {i, M _ {i}} ^ {T} \left(\Sigma_ {M _ {i}}\right) ^ {- 1} \Sigma_ {j, M _ {i}}\right) \\ = \det \left(\Sigma_ {M _ {i}}\right) \left(u _ {i} ^ {T} u _ {j} - u _ {i} ^ {T} M _ {i} \left(M _ {i} ^ {T} M _ {i}\right) ^ {- 1} M _ {i} ^ {T} u _ {j}\right) \\ \end{array}
$$
We want to estimate the following:
$$
\begin{array}{l} \left| \left\langle \operatorname {p r o j} _ {g \left(M _ {i}\right)} ^ {\perp} [ g (u _ {i}) ], \operatorname {p r o j} _ {g \left(M _ {i}\right)} ^ {\perp} [ g (u _ {j}) ] \right\rangle \right| = \left| \tilde {u} _ {i} ^ {T} \tilde {u} _ {j} - \tilde {u} _ {i} ^ {T} \tilde {M} _ {i} \left(\tilde {M} _ {i} ^ {T} \tilde {M} _ {i}\right) ^ {- 1} \tilde {M} _ {i} ^ {T} \tilde {u} _ {j} \right| \\ = \left| \frac {\operatorname* {d e t} \left(\tilde {\Sigma} _ {(i , M _ {i}) , (j , M _ {i})}\right)}{\operatorname* {d e t} \left(\tilde {\Sigma} _ {M _ {i}}\right)} \right| \tag {A.3} \\ \end{array}
$$
We already know that  $\operatorname*{det}(\tilde{\Sigma}_{M_i}) > (\frac{\lambda_1}{2})^{r_i}$ . On the other hand, by Theorem 2.12 in [IR08], we have that
$$
\begin{array}{l} \left| \det  \left(\tilde {\Sigma} _ {\left(i, M _ {i}\right), \left(j, M _ {i}\right)}\right) \right| = \left| \det  \left(\tilde {\Sigma} _ {\left(i, M _ {i}\right), \left(j, M _ {i}\right)}\right) - \det  \left(\Sigma_ {\left(i, M _ {i}\right), \left(j, M _ {i}\right)}\right) \right| \\ \leq (r _ {i} + 1) | | \tilde {\Sigma} _ {(i, M _ {i}), (j, M _ {i})} - \Sigma_ {(i, M _ {i}), (j, M _ {i})} | | \max  \{\left| | \Sigma_ {(i, M _ {i}), (j, M _ {i})} | \right|, \left| | \tilde {\Sigma} _ {(i, M _ {i}), (j, M _ {i})} | \right| \} ^ {r _ {i}} \\ \end{array}
$$
By Weyl's inequality for singular values, we have that
$$
\begin{array}{l} \left\| \tilde {\Sigma} _ {(i, M _ {i}), (j, M _ {i})} \right\| = \sigma_ {\max } \left(\tilde {\Sigma} _ {(i, M _ {i}), (j, M _ {i})}\right) \leq \sigma_ {\max } \left(\Sigma_ {(i, M _ {i}), (j, M _ {i})}\right) + (r _ {i} + 1) ^ {2} \varepsilon^ {\prime} \\ \leq \lambda_ {\max } (\Sigma) + (r _ {i} + 1) ^ {2} \varepsilon^ {\prime} \leq \lambda_ {\max } (\Sigma) + (r _ {i} + 1) ^ {2} \\ \end{array}
$$
$$
| \det  (\tilde {\Sigma} _ {(i, M _ {i}), (j, M _ {i})}) | \leq (r _ {i} + 1) (r _ {i} + 1) ^ {2} \varepsilon^ {\prime} (\lambda_ {\max } (\Sigma) + (r _ {i} + 1) ^ {2}) ^ {r _ {i}}
$$
$$
\left| \frac {\det  (\tilde {\Sigma} _ {(i , M _ {i})} , (j , M _ {i}))}{\det  (\tilde {\Sigma} _ {M _ {i}})} \right| \leq \varepsilon^ {\prime} \frac {(r _ {i} + 1) ^ {3} (\lambda_ {\max } (\Sigma) + (r _ {i} + 1) ^ {2}) ^ {r _ {i}}}{(\frac {\lambda_ {1}}{2}) ^ {r _ {i}}}
$$
Let  $C = (r + 1)^3\left(\frac{2\lambda_{\max}(\Sigma) + 2(r + 1)^2}{\lambda_{\min}(\Sigma)}\right)^r$ . Then,
$$
| \frac {\det  (\tilde {\Sigma} _ {(i , M _ {i}) , (j , M _ {i})})}{\det  (\tilde {\Sigma} _ {M _ {i}})} | \leq \varepsilon^ {\prime} C
$$
Let  $\varepsilon' = \min \left\{\frac{1}{2}, \frac{\varepsilon}{C}, \frac{\lambda_1}{2r^2}\right\}$  and  $k = \lceil \frac{20\log(2n)}{(\varepsilon')^2} \rceil$ , we have that
Lemma 16. Let  $\varepsilon \in (0, \frac{1}{2})$ . Let  $V \subseteq \mathbb{R}^d$  be a set of  $n$  points and  $k = \lceil \frac{20\log(2n)}{\varepsilon^2} \rceil$ , there exists a linear mapping  $g: \mathbb{R}^d \to \mathbb{R}^k$  such that for all  $u, v \in V$ :
$$
| \langle g (u), g (v) \rangle - \langle u, v \rangle | \leq \varepsilon
$$
Proof. The proof is an easy extension of the JL lemma [Vem05] by adding all the  $-v_{i}$  for all  $v_{i} \in V$  into the set  $V$ .
B Additional Experiments, Figures and Tables
Table 3: 9 categories of words used to test the semantic meaning of partial orthogonality
Category	Words in Category
‘vehicle’	‘car’, ‘bicycle’, ‘skateboard’, ‘motorcycle’, ‘helicopter’, ‘truck’, ‘boat’, ‘airplane’, ‘submarine’, ‘scooter’
‘animal’	‘lion’, ‘dolphin’, ‘eagle’, ‘dog’, ‘elephant’, ‘cat’, ‘rat’, ‘giraffe’, ‘bird’, ‘tiger’
‘tool’	‘hammer’, ‘screwdriver’, ‘wrench’, ‘pliers’, ‘hacksaw’, ‘drill’, ‘chisel’, ‘plunger’, ‘trowel’, ‘cutter’
‘clothing’	‘shirt’, ‘pants’, ‘dress’, ‘sweater’, ‘jacket’, ‘hat’, ‘socks’, ‘gloves’, ‘scarf’, ‘vest’
‘beverage’	‘coffee’, ‘tea’, ‘soda’, ‘lemonade’, ‘milk’, ‘wine’, ‘beer’, ‘sake’, ‘smoothie’, ‘nectar’
‘science’	‘biology’, ‘ecology’, ‘genetics’, ‘chemistry’, ‘physics’, ‘geology’, ‘mathematics’, ‘linguistics’, ‘psychology’, ‘cryptography’
‘furniture’	‘couch’, ‘bed’, ‘cabinet’, ‘dresser’, ‘hallstand’, ‘lamp’, ‘bench’, ‘chair’, ‘table’, ‘closet’
‘plant’	‘daisy’, ‘pine’, ‘iris’, ‘lily’, ‘oak’, ‘tulip’, ‘fern’, ‘rose’, ‘bamboo’, ‘cactus’
‘food’	‘chocolate’, ‘meat’, ‘steak’, ‘pasta’, ‘fish’, ‘brisket’, ‘sausage’, ‘loaf’, ‘roe’, ‘lobster’
Table 4: Experiments show that top correlated words with target words after projecting onto the orthogonal complements of randomly selected linear subspaces are more semantically meaningful
Target	Top Correlated Words Before Projection	Top Correlated Words After Projection
‘eggplant’	‘potato’, ‘banana’, ‘grape’, ‘vegetable’
‘bananas’, ‘tomato’, ‘espagnol’
‘eternal’, ‘potatoes’, ‘e.g.’	‘grape’, ‘purple-black’, ‘purple’, ‘turnips’
‘plum’, ‘lilac’, ‘vegetable’
‘vegetables’ ‘banana’, ‘ultra-violet’
‘king’	‘mister’, ‘bossman’, ‘thet’, ‘thatt’
‘beast’, ‘killed’, ‘yesiree’
‘bossed’, ‘outdo’, ‘queen’s’	‘royalty’, ‘sport-king’, ‘bossman’, ‘kingan’
‘mister’, ‘prince’s’, ‘princess’
‘princes’ ‘handsomest’, ‘ruling’
‘advise’	‘spoken’, ‘askin’, ‘concur’, ‘applies’
‘said’, ‘accordig’, ‘astute’
‘pertinent’, ‘evident’, ‘preached’	‘guidelines’, ‘guidance’, ‘tips’, ‘motto’
‘motivating’, ‘encourages’, ‘advising’
‘advisory’ ‘self-help’, ‘reminder’
‘work-out’	‘healthy’, ‘weights’, ‘worked’, ‘time-on-the-job’
‘on-the-job’, ‘work-success’, ‘busy-work’
‘out'n’, ‘healthiest’, ‘hardworking’	‘gym’, ‘weights’, ‘footing’, ‘running’
‘jogs’, ‘dumbbells’, ‘conditioning’
‘body-building’ ‘runing’, ‘pumped-up’
‘poem’	‘!’, ‘ya’, ‘eh’, ‘yes’, ‘;’
‘mem’, ‘oh’, ‘)’, ‘poignant’, ‘hee’	‘poems’, ‘poetizing’, ‘poetry’s’, ‘rhyming’
‘sonnet’, ‘lyrics’, ‘recited’
‘poetically’ ‘sonnets’, ‘rhyme’
Table 5: Selected examples provided by ChatGPT when asked "give me a list of 50 common nouns, each with a short description, and the first one is eggplant"
Target Word	Description Sentence
‘eggplant’	‘A purple or dark-colored vegetable with a smooth skin, often used in cooking and known for its mild flavor.’
‘dog’	‘A domesticated mammal often kept as a pet or used for various purposes.’
‘book’	‘A physical or digital publication containing written or printed content.’
‘car’	‘A motorized vehicle used for transportation on roads.’
‘tree’	‘A woody perennial plant with a main trunk and branches, usually producing leaves.’
‘house’	‘A building where people live, providing shelter and accommodation.’
‘computer’	‘An electronic device used for processing and storing data, and performing various tasks.’
‘cat’	‘A small domesticated carnivorous mammal commonly kept as a pet.’
‘chair’	‘A piece of furniture designed for sitting on, often with a backrest and four legs.’
‘phone’	‘A communication device that allows voice calls and text messaging.’
(a) 'lemon'
(b) 'book'
(c) 'car'
(d) 'king'
Figure 4: Linear subspaces spanned by estimated generalized Markov boundaries have smallest subspace angles with linear subspaces spanned by embeddings that best match semantic meanings