SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We choose  $\tau \in \mathbb{R}$  such that  $\forall x, i, \tau < \min_{j \in [n]} b_j^* - \max_{j \in [n]} z_j^*(x, i)$ , and  $\forall x, i, \forall j \in [n]$ , we set  $\hat{z}_j(x, i) := z_j^*(x, i) + \tau$ , then
$$
\forall j \in [ n ], \hat {z} _ {j} (x, i) - b _ {j} ^ {*} <   z _ {j} ^ {*} (x, i) + \min  _ {j \in [ n ]} b _ {j} ^ {*} - \max  _ {j \in [ n ]} z _ {j} ^ {*} (x, i) - b _ {j} ^ {*} \leq 0, \tag {6}
$$
which implies that  $\sigma (\hat{z}_j(x,i) - b_j^*) = 0$ . Therefore,  $\forall x,i$ , we have  $\hat{f} (x,i) = 0$
Theorem 2 indicates that without any structure in the representations, the student model is able to shift the logits for any sample and keep the pre-training loss unchanged. In the worst case, it can shift the logits for unseen data drastically, resulting in a bad downstream performance. Therefore, a theoretical guarantee for downstream performance requires structure in the representations learned by the pre-trained model.
4 "Anchor vector" hypothesis and empirical verifications
In Section 3.2, we showed that the shift-invariance of the softmax function can potentially make the student logits useless for the downstream task. Therefore, to understand why the downstream tasks benefit from representations from the pre-trained model, we need to understand the structure of these representations, and this structure must be able to handle the shift-invariance problem.
4.1 "Anchor vector" hypothesis
There are different ways to prevent the shift-invariance of softmax from influencing the performance of the downstream tasks. One way of doing this is to keep the partition function stable. Recall that in (4), the probability of a word is the exponential of the corresponding logit divided by the partition function. If the partition function is constant for different samples, the logits can be uniquely determined by the probabilities, which solves the shift-invariance problem. Arora et al. [2016] showed that when both the word embeddings and the latent representation are uniformly distributed on a sphere, the partition function is close to a constant with high probability. They also empirically verified this uniformity of word embeddings trained using GloVe [Pennington et al., 2014] and word2vec [Mikolov et al., 2013] on English Wikipedia. However, as we will see in later experiments in Sections 4.2 and F, this is not true for recent large-scale pre-trained language models.
Instead of uniformity of word embeddings, in large pre-trained models such as GPT-2 [Radford et al., 2019], we observe that if we remove several most frequent words from the computation of the log partition function, the remaining part can be well approximated by the inner product between the
hidden representation  $v_{-i}(x)$  and a fixed vector. This motivates us to have the following "anchor vector" hypothesis:
Definition 1. For a sample  $x$  and position  $i$ , we could select a set of bulk words  $B(x,i) \subset [n]$  and we define the bulk partition functions as  $Z_{\mathrm{bulk}}^{*}(x,i) \coloneqq \sum_{j \in B(x,i)} \exp(\langle v_{-i}^{*}(x_{-i}), v_{j}^{*} \rangle)$  and  $Z_{\mathrm{bulk}}(x,i) \coloneqq \sum_{j \in B(x,i)} \exp(\langle v_{-i}(x_{-i}), v_{j} \rangle)$ .
The selection of bulk words  $B(x, i)$  can usually be selected manually or by a simple algorithm. For instance, we can construct  $B(x, i)$  by taking out the words corresponding to the largest entries in  $p(x_i | x_{-i})$ . We can also manually select all the words that are irrelevant to the downstream task.
Hypothesis 1 ("Anchor vector" hypothesis). There exists  $v_0 \in \mathbb{R}^d$  such that
$$
\langle v _ {- i} (x _ {- i}), v _ {0} \rangle \approx \log Z _ {\mathrm {b u l k}} (x, i).
$$
If our hypothesis holds, we can use  $v_{0}$  as an anchor for the logits because it can be used to estimate the partition function, and this does not change the form of the downstream task by much. In the next subsection, we will show that Hypothesis 1 holds for most popular language models and provide some discussions. More details about these models are provided in Appendix E.
4.2 Empirical verification of "anchor vector" hypothesis
Figure 1a plots the mean squared approximation error of the log bulk partition function. We use different versions of pre-trained GPT-2 [Radford et al., 2019] and OPT [Zhang et al., 2022], and use the first  $1/4$  of WikiText-2 [Merit et al., 2016] as the input text. The hidden representations we use in this experiment are the last hidden states of these models, i.e., the output of the penultimate layer. The dimension of the hidden representations ranges from 768 to 2048, and the number of tokens is about 70k. We choose the bulk words to be all the words except those having top- $k$  probabilities and compute the optimal anchor vector using the closed-form least-squares solution. In our experiments, we use the mean squared error (MSE) to measure the approximation quality. Formally, the MSE is defined as
$$
\epsilon_ {\mathrm {M S E}} = \min  _ {v _ {0}} \mathbb {E} _ {x, i} [ (\langle v _ {- i} (x _ {- i}), v _ {0} \rangle - \log Z _ {\mathrm {b u l k}} (x, i)) ^ {2} ].
$$
The values of the bulk partition functions are around 10 (with comparable standard deviation), and we can see from Figure 1a that the MSE is usually several orders of magnitude smaller. Therefore, the inner product between the hidden representation and the optimal anchor vector can usually approximate the log bulk partition function well, and the approximation improves as  $k$  increases, i.e., when we ignore more top words. This validates our "anchor vector" hypothesis.
Figure 1: Empirical verifications of "anchor vector" hypothesis
The existence of anchor vector is quite surprising. Although there are some settings where the partition function can become easy to predict, e.g., when the word embeddings are uniformly distributed on a sphere, we have empirically shown that the large-scale language models considered in this paper
do not fall under these settings. Figure 1b shows that the word embeddings are far from uniformly distributed on a sphere, and Figure 1c indicates that the values of log partition functions and log bulk partition functions vary greatly for different samples. More experiments and discussions are provided in Section F.
5 Anchor vector guarantees performance transfer from pre-training to downstream tasks
Based on the counterexamples in Section 3 and the observations in Section 4, we now give sufficient conditions on the downstream tasks so that the downstream classification accuracy provably benefits from pre-trained representations.
5.1 Model and assumptions
For huge language models, the anchor vector hypothesis states that there exists a vector  $v_{0}^{*}$  such that its inner product with the hidden state  $v_{-i}^{*}(x_{-i})$  well approximates the logarithm of the bulk partition function. Therefore, we can use  $v_{0}^{*}$  as an anchor to handle the shift invariance problem of the softmax function, i.e., we want to subtract  $\langle v_{-i}^{*}(x_{-i}), v_{0}^{*} \rangle$  from  $\langle v_{-i}^{*}(x_{-i}), v_{j}^{*} \rangle$ . As a result, we modify our model for the downstream task to be
$$
f ^ {*} (x, i) := \sum_ {j = 1} ^ {k} a _ {j} ^ {*} \sigma \left(\left\langle v _ {- i} ^ {*} \left(x _ {- i}\right), v _ {j} ^ {*} - v _ {0} ^ {*} \right\rangle - b _ {j} ^ {*}\right). \tag {7}
$$
The student model is also modified accordingly:
$$
f (x, i) := \sum_ {j = 1} ^ {k} a _ {j} \sigma \left(\left\langle v _ {- i} (x), v _ {j} - v _ {0} \right\rangle - b _ {j}\right). \tag {8}
$$
Note that in the above model we have already adapted Assumption 1 which was originally stated for the linear model. Therefore, we also update the assumption and restate it below:
Assumption 4 (At most  $k$  words of interest). Assume there are at most  $k$  vectors in  $\{v_j^*\}_{j=1}^n$  whose logits are relevant to the downstream task and WLOG assume these  $k$  vectors are  $\{v_1^*, \ldots, v_k^*\}$ . In other words, we assume there exist coefficients  $\{a_j^*\}_{j=1}^k \in \mathbb{R}^k$  such that  $f^*(x, i) := \sum_{j=1}^k a_j^* \sigma(\langle v_{-i}^*(x_{-i}), v_j^* - v_0^* \rangle - b_j^*)$ .
We are still assuming the teacher-student setting and the log-linear word production model for the word probabilities, as restated below:
$$
p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) = \frac {\exp \left(\langle v _ {- i} ^ {*} (x) , v _ {j} ^ {*} \rangle\right)}{Z ^ {*}}, \quad p \left(x _ {i} = j \mid x _ {- i}\right) = \frac {\exp \left(\langle v _ {- i} (x) , v _ {j} \rangle\right)}{Z}. \tag {9}
$$
Under the modified model defined above, we will make some additional assumptions. As noticed in the experiments, the log bulk partition function (as defined in Definition 1) can be linearly approximated by the hidden state. Normally, the bulk only contains words that are not related to the downstream task, and every single word usually has a small probability, but the total probability of the bulk words is not negligible, as reflected in the following two assumptions. Note that the set of bulk words can be strictly contained in the complement of the set of words of interest. This is useful especially when we are not certain about which words are important for the downstream task.
Assumption 5 (Bulk contains no words of interest). For all  $x$  and  $i$ ,  $B(x, i) \cap \{1, \dots, k\} = \emptyset$ .
Assumption 6 (Lower bound of bulk probability). For all  $x$  and  $i$ ,
$$
\frac {Z _ {\text {b u l k}} ^ {*} (x , i)}{Z ^ {*} (x , i)} \geq p _ {b}.
$$
The "anchor" vector  $v_0^*(i)$  is used to handle the instability of the partition function, and we formalize our anchor vector hypothesis as the following assumption:
Assumption 7 (Linear approximation of log bulk partition function). There exist  $v_0, v_0^* \in \mathbb{R}^d, \varepsilon_b \in \mathbb{R}^+, s.t.$
$$
\forall x, \max \{| \langle v _ {- i} ^ {*} (x), v _ {0} ^ {*} \rangle - \log Z _ {\mathrm {b u l k}} ^ {*} (x, i) |, | \langle v _ {- i} (x), v _ {0} \rangle - \log Z _ {\mathrm {b u l k}} (x, i) | \} \leq \varepsilon_ {b}.
$$
Furthermore, we assume  $\varepsilon_{b} \leq \frac{\gamma}{4k\max_{j \in [k]} |a_{j}^{*}|}$ .
For notational simplicity, we will use all the notations (including  $f^{*}$ ,  $f$ ,  $v_{0}^{*}$ , and  $v_{0}$ ) without  $i$  when the selection of  $i$  is clear from the context.
5.2 Main theorem and interpretations
In our model, the ground-truth function  $f^{*}$  contains  $k$  terms with coefficients  $\{a_j^*\}_{j=1}^k$  and we define the margin  $\gamma$  as the margin for  $f^{*}$ . If we scale  $\{a_j^*\}_{j=1}^k$  or increase  $k$  by adding duplicated terms to  $f^{*}$ , we can scale  $\gamma$  arbitrarily without changing the pre-training performance or the downstream prediction of the student model. To construct a quantity that better indicates the difficulty of the downstream task, we introduce the following definition of the normalized margin that is invariant to the scaling of  $k$  and  $\{a_j^*\}_{j=1}^k$ :
Definition 2. The normalized margin is defined as  $\Gamma := \frac{\gamma}{k \max_{j \in [k]} |a_j^*|}$ .
Then we are ready to state our main result. The proof and further discussions for this theorem will be provided in Section B.
Theorem 3. Let  $\epsilon_{\mathrm{KL}} := \mathbb{E}_{x \sim \mathcal{D}_{pre}}[D_{\mathrm{KL}}(p^*(x) || p(x))]$  be the pre-training loss, and  $\epsilon_{CLS} := \operatorname*{Pr}_{x \sim \mathcal{D}_{DS}}[f(x) \cdot f^*(x) < 0]$  be the downstream classification error rate, where  $\mathcal{D}_{pre}$  and  $\mathcal{D}_{DS}$  are the input distributions of pre-training and downstream data. Under Assumptions 2-7, further assuming  $\min_{j \in [k]} b_j^* \geq \epsilon_b - \log(4k)$  and  $8\epsilon_b < \Gamma < 6$ , there exists a set of parameters  $(a_j)_{j=1}^n, (b_j)_{j=1}^n$  such that
$$
\epsilon_ {C L S} \leq \epsilon_ {\mathrm {K L}} \cdot \frac {2 8 8}{\mu \cdot p _ {b} ^ {2} \cdot \Gamma^ {2}}. \tag {10}
$$
This theorem shows that when an "anchor vector" exists, we can upper bound the downstream classification error by the KL divergence of the student model during pre-training. This upper bound becomes smaller when the distribution of downstream input is close to that of pre-training, the bulk probability is non-negligible, and the normalized margin of the ground-truth classifier is large. Here we discuss these ways to decrease the upper bound and their corresponding intuitions.
Large  $\mu$ .  $\mu$  is larger when the data distributions of pre-training and the downstream task are closer, which helps with the performance transfer.
Large  $p_b$ . The bulk probability  $\frac{Z_{\mathrm{bulk}}^*}{Z^*}$  is usually at least a constant in practice. When the bulk probability becomes larger, the anchor vector plays a more important role in the partition function and the downstream task.
Large  $\Gamma$ . A larger normalized margin makes it harder for the student model to make mistakes in downstream prediction.
Remark 1. For ease of presentation, we are making additional assumptions in Theorem 3, e.g.,  $\min_{j\in [k]}b_j^*\geq \epsilon_b - \log (4k)$  and  $8\epsilon_{b} < \Gamma < 3$ . More general cases are covered in Lemma 1. We also discuss potential ways to improve the bound in Section B.1.
6 Conclusions, limitations and future work
In this paper, we analyzed when and how the representations generated by pre-trained large-scale language models can be used in downstream classification tasks. We found two necessary conditions for guaranteeing the usefulness of these representations: the insensitivity of the downstream task to super-small probability words, and the underlying structure in the representations to handle the shift-invariance of softmax. We also provide a sufficient condition, i.e., the existence of an anchor vector, that can guarantee the representations from pre-trained language model to help with downstream tasks. We verify this existence empirically in various large language models and believe that this is an important reason why recent large-scale language models can adapt to different downstream tasks.
We made some assumptions to provide theoretical understanding on how representations can be helpful for downstream applications, and these assumptions may not always hold in practice. For instance, some downstream tasks require multiple rounds of reasoning and do not rely on a small set of words, and the "anchor vector" hypothesis can be weaker for some models such as OPT-350M and GPT-2 Medium. Relaxing these assumptions is a meaningful future direction.
While our work showed the existence of the anchor vector, it remains unclear why this vector exists in most large language models. It might be related to the initialization, optimization, and structure of the neural networks, especially transformers, and it could also be related to the underlying structure of the training data. Digging deeper into this may reveal more fundamental properties of these models. Our analysis of downstream tasks focuses on classification tasks. Since these large networks perform well in various types of downstream tasks, another future direction would be to analyze other kinds of downstream tasks.
We only consider using the last layer representation without fine-tuning the whole model, which is usually weaker in performance. Furthermore, we model the network before the softmax function as a black box, ignoring its inner structure. Further opening up this black box and consider fine-tuning the whole model are important for deeper understanding of the structures in the learned representations from pre-training.
This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER) and a Sloan Research Fellowship.
S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
S. Arora, Y. Liang, and T. Ma. A simple but tough-to-beat baseline for sentence embeddings. In International conference on learning representations, 2017.
S. Arora, M. Khodak, N. Saunshi, and K. Vodrahalli. A compressed sensing view of unsupervised text embeddings, bag-of-n-grams, and lstms. In International Conference on Learning Representations, 2018.
Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? arXiv preprint arXiv:1704.03471, 2017.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
X. Cai, J. Huang, Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations, 2020.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
K. Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512, 2019.
J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019.
Y. Hao, L. Dong, F. Wei, and K. Xu. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620, 2019.
G. Wiedemann, S. Remus, A. Chawla, and C. Biemann. Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv:1909.10430, 2019.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
T. Zhang and T. Hashimoto. On the inductive bias of masked language modeling: From statistical to syntactic dependencies. arXiv preprint arXiv:2104.05694, 2021.
Theorem 1. Suppose the downstream task performance depends only on a function  $f^{*}(x,i) = \langle v_{-i}^{*}(x_{-i}),u^{*}\rangle = \sum_{t = 1}^{k}c_{t}^{*}\langle v_{-i}^{*}(x_{-i}),v_{t}^{*}\rangle$ . For  $t^{-} \in [k]$ , define  $p^{-} := p^{*}(x_{i} = t^{-}|x_{-i})$  and assume  $p^{-} \leq \frac{1}{2}$ . Then for all  $s \in \mathbb{R}^+$ , there exist functions  $v_{-i}$  and  $\{v_t\}_{t=1}^k$  such that  $D_{\mathrm{KL}}(p^{*}(x_{i}|x_{-i})||p(x_{i}|x_{-i})) \leq 2sp^{-}$  and  $f(x,i) := \sum_{t = 1}^{k}c_{t}^{*}\langle v_{-i}(x_{-i}),v_{t}\rangle \leq f^{*}(x,i) - s \cdot c_{t^{-}}^{*}$ .
Proof. We choose functions  $v_{-i}$  and  $\{v_{k_t}\}_{t=1}^r$  such that  $\forall x, i, \forall t \in [n] \setminus \{t^-\}$ ,  $\langle v_{-i}(x_{-i}), v_t \rangle = \langle v_{-i}^*(x_{-i}), v_t^* \rangle$ . Besides,  $\forall x, i, \langle v_{-i}(x_{-i}), v_{t^-} \rangle = \langle v_{-i}^*(x_{-i}), v_{t^-}^* \rangle - s$ .
From the definition of  $p^{-}$  and  $t^{-}$  we know that
$$
p ^ {-} = p ^ {*} \left(x _ {i} = t ^ {-} \mid x _ {- i}\right) = \frac {\exp \left(\left\langle v _ {- i} ^ {*} , v _ {t ^ {-}} ^ {*} \right\rangle\right)}{Z ^ {*} (x , i)}. \tag {11}
$$
Following our construction of the student model, its partition function satisfies
$$
\begin{array}{l} Z (x, i) = \sum_ {j = 1} ^ {n} \exp \left(\langle v _ {- i}, v _ {j} \rangle\right) > \sum_ {j \neq t ^ {-}} \exp \left(\langle v _ {- i} ^ {*}, v _ {j} ^ {*} \rangle\right) (12) \\ = Z ^ {*} (x, i) - \exp \left(\left\langle v _ {- i} ^ {*}, v _ {t ^ {-}} ^ {*} \right\rangle\right) = \left(1 - p ^ {-}\right) Z ^ {*} (x, i). (13) \\ \end{array}
$$
Besides,
$$
Z (x, i) = \sum_ {j = 1} ^ {n} \exp \left(\langle v _ {- i}, v _ {j} \rangle\right) <   \sum_ {j = 1} ^ {n} \exp \left(\langle v _ {- i} ^ {*}, v _ {j} ^ {*} \rangle\right) = Z ^ {*} (x, i). \tag {14}
$$
$$
\forall j \neq t ^ {-}, \quad \frac {p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right)}{p \left(x _ {i} = j \mid x _ {- i}\right)} = \frac {\frac {\exp \left(\left\langle v _ {- i} ^ {*} , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*} (x , i)}}{\frac {\exp \left(\left\langle v _ {- i} ^ {*} , v _ {j} ^ {*} \right\rangle\right)}{Z (x , i)}} = \frac {Z (x , i)}{Z ^ {*} (x , i)}. \tag {15}
$$
$$
\frac {p ^ {*} \left(x _ {i} = t ^ {-} \mid x _ {- i}\right)}{p \left(x _ {i} = t ^ {-} \mid x _ {- i}\right)} = \frac {\frac {\exp \left(\left\langle v _ {- i} ^ {*} , v _ {j} ^ {*} \right\rangle\right)}{Z ^ {*} (x , i)}}{\frac {\exp \left(\left\langle v _ {- i} ^ {*} , v _ {j} ^ {*} \right\rangle - r\right)}{Z (x , i)}} = \frac {Z (x , i)}{Z ^ {*} (x , i)} \cdot e ^ {s}. \tag {16}
$$
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(p ^ {*} \left(x _ {i} \mid x _ {- i}\right) \mid p \left(x _ {i} \mid x _ {- i}\right)\right) = \sum_ {j = 1} ^ {n} p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) \log \frac {p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right)}{p \left(x _ {i} = j \mid x _ {- i}\right)} (17) \\ = \sum_ {j \neq t ^ {-}} p ^ {*} \left(x _ {i} = j \mid x _ {- i}\right) \log \frac {Z (x , i)}{Z ^ {*} (x , i)} + p ^ {-} \cdot s \log \frac {Z (x , i)}{Z ^ {*} (x , i)} (18) \\ <   s p ^ {-} \log \frac {1}{1 - p ^ {-}} <   2 s p ^ {-} (19) \\ \end{array}
$$
$$
f (x, i) = \sum_ {t = 1} ^ {k} c _ {t} ^ {*} \left\langle v _ {- i} \left(x _ {- i}\right), v _ {t} \right\rangle = \sum_ {t = 1} ^ {k} c _ {t} ^ {*} \left\langle v _ {- i} ^ {*} \left(x _ {- i}\right), v _ {t} ^ {*} \right\rangle - c _ {t} ^ {*} \cdot s = f ^ {*} (x, i) - s \cdot c _ {t -} ^ {*}. \tag {20}
$$
B Proof and discussions for main theorem
Here we provide a proof sketch of Theorem 3.
Theorem 3. Let  $\epsilon_{\mathrm{KL}} := \mathbb{E}_{x \sim \mathcal{D}_{pre}}[D_{\mathrm{KL}}(p^*(x) || p(x))]$  be the pre-training loss, and  $\epsilon_{CLS} := \operatorname*{Pr}_{x \sim \mathcal{D}_{DS}}[f(x) \cdot f^*(x) < 0]$  be the downstream classification error rate, where  $\mathcal{D}_{pre}$  and  $\mathcal{D}_{DS}$
are the input distributions of pre-training and downstream data. Under Assumptions 2-7, further assuming  $\min_{j\in [k]}b_j^*\geq \epsilon_b - \log (4k)$  and  $8\epsilon_{b} < \Gamma < 6$ , there exists a set of parameters  $(a_{j})_{j = 1}^{n},(b_{j})_{j = 1}^{n}$  such that
Proof. We have the following lemma providing a lower bound of the TV distance between  $p^*(x_i | x_{-i})$  and  $p(x_i | x_{-i})$ , which will be proved in Section C.
Lemma 1. There exists a choice of  $(a_j)_{j=1}^k, (b_j)_{j=1}^k$  such that if Assumptions 2-7 hold and there exists  $x \in \mathrm{POS} \cup \mathrm{NEG}$  and  $i$  such that  $f(x, i) \cdot f^*(x, i) < 0$ , then we must have
$$
\left. \operatorname {T V} \left(p ^ {*} \left(x _ {i} \mid x _ {- i}\right), p \left(x _ {i} \mid x _ {- i}\right)\right) \geq p _ {b} \min  _ {\varepsilon_ {p} \geq 0} \left\{\left(1 - e ^ {- \varepsilon_ {p}}\right) + k \cdot \min  _ {j \in [ k ]} e ^ {b _ {j} ^ {*} - \varepsilon_ {b} - \varepsilon_ {p}} \cdot \left(e ^ {\sigma \left(\frac {\Gamma}{2} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right) \right\}. \right. \tag {21}
$$
From Lemma 1 and Pinsker's inequality we can lower bound the KL divergence at any incorrectly classified sample: For all  $x \in \mathrm{POS} \cup \mathrm{NEG}$ ,
$$
\begin{array}{l} \operatorname {K L} \left(p ^ {*} \left(x _ {i} \mid x _ {- i}\right) | | p \left(x _ {i} \mid x _ {- i}\right)\right) (22) \\ \geq 2 \left(\operatorname {T V} \left(p ^ {*} \left(x _ {i} \mid x _ {- i}\right), p \left(x _ {i} \mid x _ {- i}\right)\right)\right) ^ {2} (23) \\ \geq 2 p _ {b} ^ {2} \min  _ {\varepsilon_ {p} \geq 0} \left\{\left(1 - e ^ {- \varepsilon_ {p}}\right) + k \cdot \min  _ {j \in [ k ]} e ^ {b _ {j} ^ {*} - \varepsilon_ {b} - \varepsilon_ {p}} \cdot \left(e ^ {\sigma \left(\frac {\Gamma}{2} - 2 \varepsilon_ {b} - \varepsilon_ {p}\right)} - 1\right) \right\} ^ {2} (24) \\ \end{array}
$$
Since the pre-training loss is the expected KL divergence for pre-training data, it is lower bounded by the KL divergence on the incorrectly classified samples, which is related to the downstream classification error rate: