SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \frac {\exp \left(\left\langle v _ {- i} ^ {*} (x) , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*}} - \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {j} \right\rangle\right)}{Z} \right| (76) \\ = \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {j} \right\rangle\right)}{Z} \cdot \left| \frac {\exp \left(\left\langle v _ {- i} ^ {*} (x) , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*}} \cdot \frac {Z}{\exp \left(\left\langle v _ {- i} (x) , v _ {j} \right\rangle\right)} - 1 \right| (77) \\ = \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {j} \right\rangle\right)}{Z} \cdot \left| \exp \left(\left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {j} \right\rangle - \log Z\right)\right) - 1 \right|. (78) \\ \end{array}
$$
$$
\begin{array}{l} \log \frac {\exp \left(\langle v _ {- i} (x) , v _ {j} \rangle\right)}{Z} = \langle v _ {- i} (x), v _ {j} \rangle - \log Z (79) \\ = \left\langle v _ {- i} (x), v _ {j} - v _ {0} \right\rangle + \left\langle v _ {- i} (x), v _ {0} \right\rangle - \log Z _ {\mathrm {b u l k}} + \log \frac {Z _ {\mathrm {b u l k}}}{Z} (80) \\ \geq b _ {j} ^ {*} - \varepsilon_ {b} + \log \frac {Z _ {\text {b u l k}} ^ {*}}{Z ^ {*}} + \left(\log \frac {Z _ {\text {b u l k}}}{Z} - \log \frac {Z _ {\text {b u l k}} ^ {*}}{Z ^ {*}}\right) (81) \\ \geq b _ {j} ^ {*} - \varepsilon_ {b} + \log p _ {b} - \varepsilon_ {p}. (82) \\ \end{array}
$$
Moreover, when  $\Delta_j - 2\varepsilon_b - \varepsilon_p > 0$ , we have by (72) that
$$
\begin{array}{l} \left| \exp \left(\left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {j} \right\rangle - \log Z\right)\right) - 1 \right| (83) \\ \geq \exp \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right) - 1. (84) \\ \end{array}
$$
When  $\Delta_j - 2\varepsilon_b - \varepsilon_p\leq 0$  , we have
$$
\left| \exp \left(\left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {j} \right\rangle - \log Z\right)\right) - 1 \right| \geq 0 = e ^ {0} - 1. \tag {85}
$$
$$
\left| \exp \left(\left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {j} \right\rangle - \log Z\right)\right) - 1 \right| \geq e ^ {\sigma \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1. \tag {86}
$$
Therefore, when  $\langle v_{-i}(x),v_j - v_0\rangle \geq b_j^*$  , we must have
$$
\left| p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) - p \left(x _ {i} = j \mid x _ {- i}\right) \right| \geq e ^ {b _ {j} ^ {*} - \varepsilon_ {b} - \varepsilon_ {p}} \cdot p _ {b} \cdot \left(e ^ {\sigma \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right). \tag {87}
$$
In the second case where  $\langle v_{-i}(x),v_j - v_0\rangle < b_j^*$ , we have  $\langle v_{-i}(x),v_j\rangle < \langle v_{-i}(x),v_0\rangle +b_j^*$ . The proof for the second case is very similar to that of the first case, with the main difference being we replace  $\langle v_{-i}(x),v_j\rangle$  in the first case by  $\langle v_{-i}(x),v_0\rangle +b_j^*$  in the second case.
In the second case, by the definition of  $\Delta_j$  we know that  $\Delta_j = (\langle v_{-i}^*(x), v_j^* - v_0^* \rangle - b_j^*)$ .
Similar to the first case, we can get
$$
\begin{array}{l} \left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {0} \right\rangle + b _ {j} ^ {*} - \log Z\right) (88) \\ = \left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z _ {\mathrm {b u l k}} ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {0} \right\rangle + b _ {j} ^ {*} - \log Z _ {\mathrm {b u l k}}\right) + \left(\log \frac {Z _ {\mathrm {b u l k}} ^ {*}}{Z ^ {*}} - \log \frac {Z _ {\mathrm {b u l k}}}{Z}\right) (89) \\ \geq \left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z _ {\text {b u l k}} ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {0} \right\rangle + b _ {j} ^ {*} - \log Z _ {\text {b u l k}}\right) - \varepsilon_ {p} (90) \\ \geq \left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} - v _ {0} ^ {*} \right\rangle - b _ {j} ^ {*} + \left(\left\langle v _ {- i} ^ {*} (x), v _ {0} ^ {*} \right\rangle - \log Z _ {\mathrm {b u l k}} ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {0} \right\rangle - \log Z _ {\mathrm {b u l k}}\right) - \varepsilon_ {p} (91) \\ \geq \Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}. (92) \\ \end{array}
$$
Therefore, for the difference in probabilities, if  $\Delta_j - 2\varepsilon_b - \varepsilon_p > 0$ , we have  $\left(\langle v_{-i}^*(x), v_j^*\rangle - \log Z^*\right) > \left(\langle v_{-i}(x), v_0\rangle + b_j^* - \log Z\right)$ , so
$$
\begin{array}{l} \left| p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) - p \left(x _ {i} = j \mid x _ {- i}\right) \right| (93) \\ = \left| \frac {\exp \left(\left\langle v _ {- i} ^ {*} (x) , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*}} - \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {j} \right\rangle\right)}{Z} \right| (94) \\ \geq \frac {\exp \left(\left\langle v _ {- i} ^ {*} (x) , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*}} - \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {0} \right\rangle + b _ {j} ^ {*}\right)}{Z} (95) \\ = \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {0} \right\rangle + b _ {j} ^ {*}\right)}{Z} \cdot \left(\exp \left(\left(\left\langle v _ {- i} ^ {*} (x), v _ {j} ^ {*} \right\rangle - \log Z ^ {*}\right) - \left(\left\langle v _ {- i} (x), v _ {0} \right\rangle + b _ {j} ^ {*} - \log Z\right)\right) - 1\right) (96) \\ \geq \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {0} \right\rangle + b _ {j} ^ {*}\right)}{Z} \cdot \left(e ^ {\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}} - 1\right). (97) \\ \end{array}
$$
If  $\Delta_j - 2\varepsilon_b - \varepsilon_p \leq 0$ , the absolute value of probability difference can be trivially bounded below by 0. Thus,
$$
\left| p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) - p \left(x _ {i} = j \mid x _ {- i}\right) \right| \geq \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {0} \right\rangle + b _ {j} ^ {*}\right)}{Z} \cdot \left(e ^ {\sigma \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right). \tag {98}
$$
$$
\begin{array}{l} \log \frac {\exp \left(\left\langle v _ {- i} (x) , v _ {0} \right\rangle + b _ {j} ^ {*}\right)}{Z} = b _ {j} ^ {*} + \left\langle v _ {- i} (x), v _ {0} \right\rangle - \log Z _ {\text {b u l k}} + \log \frac {Z _ {\text {b u l k}}}{Z} (99) \\ \geq b _ {j} ^ {*} - \varepsilon_ {b} + \log \frac {Z _ {\text {b u l k}} ^ {*}}{Z ^ {*}} + \left(\log \frac {Z _ {\text {b u l k}}}{Z} - \log \frac {Z _ {\text {b u l k}} ^ {*}}{Z ^ {*}}\right) (100) \\ \geq b _ {j} ^ {*} - \varepsilon_ {b} + \log p _ {b} - \varepsilon_ {p}, (101) \\ \end{array}
$$
we finally get the same bound as the first case:
$$
\left| p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) - p \left(x _ {i} = j \mid x _ {- i}\right) \right| \geq e ^ {b _ {j} ^ {*} - \varepsilon_ {b} - \varepsilon_ {p}} \cdot p _ {b} \cdot \left(e ^ {\sigma \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right). \tag {102}
$$
Merging the two cases, we know that for all  $j \in S(x)$ ,
$$
\left| p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) - p \left(x _ {i} = j \mid x _ {- i}\right) \right| \geq e ^ {b _ {j} ^ {*} - \varepsilon_ {b} - \varepsilon_ {p}} \cdot p _ {b} \cdot \left(e ^ {\sigma \left(\Delta_ {j} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right). \tag {103}
$$
This finishes the proof of Lemma 5.
E Experiment details
Language models. We use various versions of GPT-2 Radford et al. [2019] and OPT Zhang et al. [2022] with number of parameters ranging from 125M to 1.5B. We use the base, medium, large, xl version of GPT-2 which have #parameters 124M / 355M / 774M / 1.5B and hidden dimension 768 / 1024 / 1280 / 1600. For OPT, we use three versions, with #parameters 125M / 350M / 1.3B and hidden dimension 768 / 1024 / 2048. For all these models, the word probability is the softmax of the product of the penultimate layer representation and the dictionary. This is consistent with our theoretical model introduced in Section 2. The parameter settings and performances of these models are shown in Table 1.
Table 1: Parameters and performances of language models
Model	#Param	Hidden dim	Perp
GPT-2	124M	768	25.92
GPT-2 Medium	355M	1024	19.19
GPT-2 Large	774M	1280	17.13
GPT-2 XL	1.5B	1600	15.34
OPT-125M	125M	768	25.01
OPT-350M	350M	1024	19.69
OPT-1.3B	1.3B	2048	13.16
Dataset. We use WikiText-2 Merity et al. [2016] as the text corpus. WikiText-2 has about 280k tokens, and we only use the first 1/4 of it for computational efficiency. The perplexities of the language models on this corpus are shown in Table 1.
F Existence of anchor vector is not trivial
In this section, we will empirically verify that the existence of the anchor vector in large-scale language models does not arise from the trivial reasons that we could think of.
Linear approximation is not accurate for original partition function. Table 2 shows the mean squared approximation error of the log bulk partition function for different models and different  $k$ . For the models whose log bulk partition functions can be well approximated linearly, their log original partition function cannot. This confirms the necessity of removing the words of interest in the partition function.
Table 2: Mean squared approximation error of log bulk partition function for different models and different  $k$  .
Model\k	0	10	100	1000	10000
GPT-2	625.8	651.4	665.2	685.5	727.4
GPT-2 M	0.7647	0.3444	0.3005	0.2683	0.2396
GPT-2 L	0.7296	0.1288	0.0740	0.0317	0.0022
GPT-2 XL	0.7716	0.1219	0.0648	0.0279	0.0019
OPT-125M	0.6688	0.1010	0.0443	0.0117	0.0006
OPT-350M	2.2449	0.9393	0.7587	0.6182	0.4275
OPT-1.3B	0.5234	0.0984	0.0402	0.0117	0.0006
Word embeddings are not uniformly distributed on a sphere. Figure 2 shows the singular values of dictionary matrices from different models. For the four selected models, their word embeddings are close to being low-rank. Therefore, the word embeddings in these models are far from uniformly distributed on a sphere.
(a) GPT-2 Large
(b) GPT-2 XL
(c) OPT-125M
Figure 2: Singular values of dictionary matrices.
(d) OPT-1.3B
Dictionary atoms are not uniformly distributed on sphere. Figure 3 shows the histogram of the  $\ell_2$ -norm of the dictionary atoms from different language models, and Figure 4 shows the distribution of the Cosine similarity between two random atoms. The norms of these atoms are somewhat bounded, but their cosine similarity is strongly biased towards the positive part, and the dictionary matrices are close to low rank, indicating that these vectors are far from uniformly distributed on a sphere. Instead, they may concentrate around a cone-shaped region.
(a) GPT-2
(b) GPT-2 Medium
(e) OPT-125M
Figure 3:  $\ell_{2}$ -norms of atoms
(c) GPT-2 Large
(f) OPT-350M
(d) GPT-2 XL
(g) OPT-1.3B
(Bulk) partition functions have large variations. Figure 5 shows the histogram of log partition function and log bulk partition function for the language models, where the bulk words are defined as all the words except the top 100, in terms of logit values. We can see from the figures that both the partition function and the bulk partition function vary a lot depending on the samples. Therefore, in recent large-scale language models, the partition functions are not stable across samples.
(a) GPT-2
(e) OPT 125M
Figure 4: Cosine similarity between two random atoms
(b) GPT-2 Medium
(c) GPT-2 Large
(f) OPT 350M
(d) GPT-2 XL
(g) OPT 1.3B
(a) GPT-2 Large
Figure 5: Histogram of log original and bulk partition functions
(c) OPT-125M