Connecting Pre-trained Language Model and Downstream Task via Properties of Representation
Authors: Chenwei Wu
Holden Lee
Rong Ge
Venue: NeurIPS 2023 poster (2023)
Keywords: language model representation
downstream performance
deep learning theory
TL;DR: We empirically find a structure in representations from big language models and theoretically proves it helps transferring pre-training performance to downstream task performances.

Abstract: Recently, researchers have found that representations learned by large-scale pre-trained language models are useful in various downstream tasks. However, there is little theoretical understanding of how pre-training performance is related to downstream task performance. In this paper, we analyze how this performance transfer depends on the properties of the downstream task and the structure of the representations. We consider a log-linear model where a word can be predicted from its context through a network having softmax as its last layer. We show that even if the downstream task is highly structured and depends on a simple function of the hidden representation, there are still cases when a low pre-training loss cannot guarantee good performance on the downstream task. On the other hand, we propose and empirically validate the existence of an ``anchor vector'' in the representation space, and show that this assumption, together with properties of the downstream task,  guarantees performance transfer.

Connecting Pre-trained Language Models and Downstream Tasks via Properties of Representations
Chenwei Wu Duke University cwwuêcs. duke. edu
Holden Lee Johns Hopkins University hlee2838jmu. edu
Rong Ge Duke University
ronggeEcs duke. edu
Abstract
Recently, researchers have found that representations learned by large-scale pre- trained language models useful in various downstream tasks. However, there is little theoretical understanding of how pre-training performance is related to downstream task performance. In this paper, analyze how this performance transfer depends on the properties of the downstream task and the structure of the representations. We consider log-linear model where word from its context through a network having softmax as its last layer. We show that even if the downstream task highly structured and depends on simple function of the hidden representation, there are still cases when low pre-training loss cannot guarantee good performance on the downstream task. On the other hand, we propose and empirically validate the existence of an "anchor vector" in the representation space, and show that this assumption, together with properties of the downstream task, guarantees performance transfer.
be predicted
Introduction
Large-scale pre-trained language models have achieved strong performance a wide range of down- stream tasks, including natural language inference (Devlin al. 2018) and reading comprehension [Brown et al. 2020). For many of these tasks, training linear classifier on top of the hidden-layer representations generated by the pre-trained models can already provide near state-of-the- results Bellinkvv et al. 2017]. Despite many empirical investigations about the zero-shot applications of these pre-trained models, there little theoretical understanding of their empirical success. this paper, we aim to theoretically investigate this core question:
When can the representations from pre-trained models transfer to downstream tasks that are very different from the pre-training task?
This a fundamental question in understanding why good performance in pre-training leads to good performance downstream tasks. Unlike the notion of generalization in traditional learning theory where the models are evaluated the same task and the test data are sampled from same distribution the training data, here the downstream tasks usually very different from pre-training. For instance, people can pre-train large language models using cross-entropy loss on language modeling task with webpage data, and evaluate the models using classification accuracy text classification news articles. The differences between pre-training and downstream tasks make it challenging to explain the success of these language models.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To overcome this challenge, we need a way to model the relationship between the pre-training and downstream tasks. Previous research has taken several approaches this direction: WWii et al. [202T] assumes a latent-variable generative model for the data and a downstream task depending variables; Saunshi et al. 202T] formulates the downstream classification task as a language modeling task which is similar to the pre-training task. These works either rely on strong explicit assumptions
the latent
about the structure of the data (i.e., assuming the data is generated from a simple generative model)
or treat the entire pre-trained model black box.
1.1 Our contributions
In this paper, we consider a very general model for the data and open the black box of the pre-trained model at the last layer. Specifically, for input sequence 1.1,--..: (L) where the entries comes from a dictionary {1....,n), n.,, we assume the observation probability of z;; satisfies a log-linear model
p= = j|æ=-)) x exp((v",(xx))) uz)),
where x=; is the sequence z without Tg, Vz is a vector only depending on word j. and v-, can be an arbitrary function. This aligns with commonly used networks whose last layer is usually softmax layer. Moreover, since our model does not put any constraint on the function v=' it can be arbitrarily complicated, such huge transformer model such as BERT [Devlin et al., 2018 GPT-3 [Brown) ct al. 2020). We also allow the distribution of the input to be different in pre-training and downstream tasks. This makes our setting more general than previous latent models Wei et al. [021] Arora et al. 2016)
We assume the pre-training task is to predict a word from its context. During pre-training, for every input sequence z, we want our model to predict the "label" x; from Z-i-- The model we use in training (which we call the "student model") has the same log-linear structure: p(z; = j|æ_,) x exp((r_((x_)) vz)).
For the downstream task, for simplicity we focus on binary sequence classification, e.g., sentiment classification. To define downstream tasks, let the "logits" of the ground -ruth model be defined as 2" = ((v", "= (these are just the outputs before the softmax computation), and assume that the downstream task is specified by function of the logits, f²")))
In reality, do not have access to the ground-trutt truth model v-, and v.. Instead, only have access to the student model v_g and UU that achieves low pre-training loss. We can define the student logits as = = ((v_>(x==)) (z) "z)) In some sense, is the representation learned by the pre-training step. natural idea for solving the downstream task would be to learn function f=)). More details about this model will be provided in Section 2
Our goal is to understand the properties of the learned representation. In order to simplify the problem, we assume that the student model can be optimized to achieve a small KL divergence with the true word probabilities during training. Under this setting, the question ask above becomes:
If the downstream task depends on a simple function of the logits f²(x") and we have access to a student model p such that Ez [z(Drr (p" (xx( (p) is small, under what conditions is there a function f such that f(z) R (**))²
priori, one might imagine if the function f* very simple (e.g., a linear function), then should be easy to find a function f. However, we give two counter-examples that show there are additional properties that !* needs to satisfy: (i) I" should not distinguish between words with small probabilities and words with super-small probabilities, and (īi) the hidden representations must have some structure that deals with the shift- inft-invvriaanee of the softmax function (that is, the result of softmax does not change if all the logits are shifted by the same constant). We present the counterexamples in Section
To further investigate the structure of the hidden representations and see how we can deal with the inftt-invariance property of softmax, in Section we propose and empirically verified the "anchor vector hypothesis": there exists an "anchor vector" in the representation space that can be used to estimate the bulk partition function, which we define to the sum of the exponential of all logits
except the largest few, i.c., Σjn; not large 3 We show how the anchor vector can be used to address the shift- inft-innvaiaacee of softmax.

---

Based on the observation that anchor vectors exist, in Section S we give sufficient conditions that shows when a sparse one-hideen-layer ReLU network f can be learned by our student model f. Specifically, assuming that f* is one-hidden-laye ReI.U network depending on a small set of words and the downstream task is binary classification depending on f*(="), ), the existence of the anchor vector enables us to upper bound the loss on the downstream task attained by the student model f(z). in terms of its KL divergence in pre-training. In other words, a small pre-training loss is guaranteed to transfer to small downstream classification error.
1.2 Related works
Theoretical understanding why pre-training helps downstream tasks: Most of the relevant existing works rely on latent variable models and show that pre-training could recover some form of the latent variables. [Arora et al. [2116] proposed the RAND-WALL WALK latent model and explains the empirical success of word embedding methods such as word2vec [Mikolov et al. 2013 and
GloVe [Pennington et al., 2014). Arora et al. [2017] extended the previous model to justity sentence embeddings, and Arora et al. 2018] explained sentence embedding via compressed sensing. Other models are also used in this line of work, e.g., hidden Markov models [Wei et al., (021] and graphical
models (Zhang and Hashimoto| (02T]. Lee al. [202T] Tosh et al. [2021] assume conditional
independence or multi-view structure in the pre-training data and prove that training an additional linear layer on top of learned representations can perform well in downstream tasks.
The problem setting in our paper is similar to that of Saunshi et al. 2021] which also analyzes the performance transfer of pre-trained language models binary classification downstream tasks. They treat the pre-trained model as black box and assume that the downstream task can be formulated as a sentence completion task, while we open the black box at the last layer and connect pre-training with downstream tasks by the "anchor vector" and function f". Moreover, they focus on the prediction probabilities of the pre-trained model while we instead focus on the representations.
Applications and analysis of hidden representations from large-scale language models: The hidden representations produced by large language models such as BERT (Devlin et al. 2018 or ELMo [Peters et al. 2018 have been very useful in various NLP tasks. A standard method to train a linear classifier on these representations. though there are other methods such as using the normalized mean of concatenated word embeddings (Tanaka al., [020]. . To understand why these word embeddings are useful, people have empirically showed that BERT word embeddings contain information about sentence-level context [Miaschi and Orletta, 2020), word sense [Wiedemann] et al. 2019]. J and syntactic phenomena [Tlennyy et al. 2019] including parse trees (Hewitt and Manning 2019 Kim et al.) 2020]. Other empirical explanations include the flatness of local minima achieved by pre-training [Hao et al. 2019). connection to deep metric learning (Tschannen et al.
2019]. and the attention patterns across different heads Kovaleva et al. 2019
Language modeling can also be considered as way of using the hidden representations because the next word probability is usually the softmax of the product of the representation and the dictionary matrix. Therefore. any zero-shot application of pre-trained auto-regressive language models, e.g., GPT-3 [Brown et al. 2020 and T5 RRafel et al. 2020 is specific method of using the hidden representations.
Some previous works have found that the word embeddings learned by language models can lie in narrow cone: Gao et al. 2019] empirically found this phenomena for the learned word embeddings in LSTM and vanilla transformers. [Ethayarajh] [2019), Cai al. 22000) found similar phenomena for contextualized embeddings in BERT-like models.
2 Problem setup
Notations. We use [n]] to denote the set (1, ........., n.. For an input sequence x = (11,...,3I1, we use x-; to denote the input sequence without the i-th entry where i E [L]. i.e., x= == (xi,..., ,......,,,,11. 1 1e.1,...,
We let (xl(p) be the KL--divergence between distributions p and g and define (p) to be the entropy of distribution p.
3

---

Ground-truth model. We consider the following model: There is set of words [rz]. each with a fixed corresponding vector vj E Td (j E [n]). We refer to each oj as an atom and the matrix [uf...., , R the dictionary. At each position i, let æ; be the value of the word at that position; then 24 E [n]. Assume that the probability of z; given Z-₁ follows a log-linear model, ie.
p² = j(x-)) x
oj)),
where (-) is function that encodes the remaining sequence æ-; into vector in R2
We also use zj (zz, i) := (u-,(x=))) uz) to denote the j-th logit and (xxx,)) i) == E exp(= (2]](; (2; ()) to denote the partition function, i.e., the normalization factor of equation ① In other words,
exp(2)(x,1)) exp((v" (F=) Vj E [n], pi = s|x--)= Z* (x,
(2)
(x,,t)
Student model. We use black-box neural network model whose penultimate layer outputs a d'-dimensional vector v=)(x=s) E IRd and the last layer is fully-connected layer with weight matrix [v,, U2, ... ,2,] E Rd x followed by the softmax function. In other words, the model output is
p(a; = 5| &
0;)).
Similar the ground-truth model, we use 2, i) (v_1(x=+)) Uz} denote the j-th logit and ( i) = ン (x, to denote the partition function of equation (3) so
05)))
Vj [n]. p(a, j|x=1)
(4)
Z (xx
Pre-training, For self-supe pre-training, are given (data, "label") pairs (x_1, and want our model to predict the "label" x; given x-1. The pre-training loss we use here cross-entropy loss:
((v_;) = Bx(--"((aaaar-))) pog = E, ( (p" (p"(m=(==:) + (x/ (x)(=-:)))
Note that E,, (p "p" (x=(==;))) is a constant, and we assume that our student model achieves a small loss value that the KI KI-ddivereence term EE,, (D.(( (p"(s; (px;)) < 6KI. for some eKL.
Downstream task. The downstream task we are considering is binary sequence classification, e.g., sentiment classification. For instance, given sentence, we (probably after adding some prompts) use our pre-trained model to predict the missing word æ; from the given input x We assume that there is perfect classifier (xx i) only depending on vi, that
distinguish between positive and negative samples. In other words, for all x E PO,, (x, >> 0 and for all æ E NEG, (" i) < O. Here POS and NEG the set of positive and negative input sequences, respectively. We choose to focus on binary classification for theoretical analysis for simplicity, but most of the ideas can be extended to a multi-class classification setting.
simple downstream task is one whose classifier is linear in oi,,, that is, f"(x, i) == (u=,(x=;), u²) for some u" E Rd One might also expect more structures in the vectors u" and v- For u", the downstream task usually depends on only small of the words which are related this task. For example, for sentiment classification, the sentiment of a sentence depends mostly on words similar to "positive" or "negative"" This can be formalized as the following assumption:
Assumption (u" is k-sparse combination of (u: )=1)) Assume u" is a sparse combination of at most vectors in {oj}í=1 and without loss of generality assume these k vectors are [V1,..., V.
ie., there exist coefficients (c5))=1 E & such that u² = Σ C..
Note that the fixed representations fei...., }h correspond the weights of the last layer in large language models instead of the token representations (x....... In other words, the token representations can still be different depending on their context.
There are usually differences between input distributions of pre-training and downstream tasks. The difference may be due to different data sources. For example, the samples in the downstream task

---

may only contain movie reviews while the pre- pre-training dataset can include all kinds of texts on the Internet. It can also result from prompting. which has become the dominant way of using large language models for downstream tasks [Brown et al., 2020 Radford et al. 2019]. For instance, for movie review classification, appending "This movie was" to the original input could improve the classification accuracy. Similar to the assumption made in Saunshi et al. (021), I, we use (0, 1] to capture this difference. A smaller μ indicates a larger difterence between the two distributions, and if and only if these two distributions are the same. In most tasks, the two distributions are not too different and we would expect a reasonable value of μ.
A Assumption 2 (Difference between pre-training and downstream distribution). Let Pere and PDs be the probability density functions of the pre-training and downstream task, respectively. We assume that there exists E (0. 1] such that
Vi, Vx E POS UEG,, Ppre (x) 2 J. PDs (r-))
If the ground truth classifier f can be too close to O, it would not be robust to small perturbations. We use the standard margin assumption to avoid such cases:
Assumption 3 (Margin for downstream task). There exists margin y E B.+ such that at any position i, x x E POS, then (*, i) 2 7, and i E NEG, then (*),, S -7, where POS and NEG are the sets of positive and negulive samples, rexpectively.
Ideally, want to show that such simple downstream tasks also be solved well with the representations learned by our student model, i.e., V_; and {u, 1/ However, as we will see in Section 3 our current model with this margin assumption still doesn't guarantee good downstream task performances. More structures in the model and the downstream task is necessary to make sure that the pre-trained representations are useful for the downstream task.
3 Cases when learned representations are insufficient for downstream tasks
The problem setting in Section 2 seems reasonable at first sight, but the following subsections, we will show that this model is not enough to guarantee good pre-training performance to generalize to downstream tasks. In other words, there are ways for the student model to approximate the ground- truth probabilities very well in terms of KL divergence but perform very badly at the downstream task. Therefore, we need to put further constraints on the ground- -ruth model and the downstream task.
3.1 Downstream tasks sensitive to words with super-small probability
Intuitively, KL divergence is a weighted log probability difference between two distributions where the weight is the ground-truth probability. Therefore, for the entries with small ground-truth probabilities, a large log probability difference will not result in a large KL divergence. However, the log probability difference proportional to the difference in the value of f* (z, i). This makes it possible for the student model to flip the sign of f (z, i) without incurring a large KL divergence, as presented in Theorem I whose proof is given in Appendix A
Theorem 1. Suppose the downstream task performance depends only on a function I (x, = (x_((=-)), )²)
-k oi (i(2,((==)) vt). For t- [b]], define p= := p'(x; t (i)). and assume p- S 2. Then for all 8 IRt,, there exist functions V- and } such that DKL(D" (xx (xnla=-)ll((xxp)) S 2sp and [xx ==
S (x, s.G.
Theorem shows that if there is some word of interest t- that has small probability p", then is possible to have model with small KL divergence in pre-training but bad downstream performance. This is because changing the KL divergence by only 2p- (z,1) is enough change the label of the downstream prediction. In other words, long the KL divergence is higher than the threshold
2p (z.i) we cannot distinguish between the case where the student model makes an already
small probability smaller (which can hurt the downstream task performance) and the case where random approximation errors are spread across the entries. In this case, a small KL divergence does not necessarily imply good downstream performance.
Note that this sensitivity of the downstream task to very small logits not natural. For the downstream tasks in practice, after conditioning on the context, whether word has probability of 10-5 or
5

---

10 should not influence the label of the sequence. Thus, we need to impose additional structure on our model. We make the downstream task ignore super-small entries by setting a threshold for the logits and ignoring the logits smaller than that threshold. In this case, making the logits smaller when they are already small will have no influence on the downstream task performance. Concretely, the enhanced model will be (a(x) == max(2; 0} is the ReLU function):
(x, Σ Eajo(z3(2,)) b3) - Σ
-b))
(5)
3.2 Representations are not shift-invariant
The softmax function is invariant under shift, i... the oulput stays the same if we add the same value to every coordinate of the input. In the current model, we have no control over the shift of student model logits on unseen data. Consequently, even if we get a student model that performs well on the training data for the downstream task, we cannot guarantee the performance of this model on new data. This can be formalized in the following theorem.
Theorem 2. Assume 2" (x, i) is bounded. For any function (x, E a
by). there exist functions (ž)( }= such that for all x and i, we have p(x;(o-i) pp (xilor-)) and (zx = E}=1 (2)(( - b7) is always equal to In other words, the pre-training loss of the model (z, the same as {zg (x, i)}r=1 but its logits are useless for the downstream task.
Proof. We choose T E R such that Vz, i, T < minseln b3 maxjelal 2z i), and Vz, i, Vj E [m].
we set (x( i) = 27 (z. i) + T, then
Vj [m] (2, 1) bg < (z, + min bj max 2j (I, - b3 S 0,
(6)
jE(m]
jE[n]
which implies that i) - bb) = 0. Therefore, Va. i, we have f(x, =0 0.
Theorem 2 iindicates that without any structure in the representations, the student model able to shift the logits for any sample and keep the pre-training loss unchanged. In the worst case, it can shift the logits for unseen data drastically, resulting in a bad downstream performance. Therefore, a theoretical guarantee for downstream performance requires structure in the representations learned by the pre-trained model.
" "Anchor vector" hypothesis and empirical verifications
we showed that the shift-invariance of the softmax function can potentially make the student logits useless for the downstream task. Therefore, to understand why the downstream tasks benefit from representations from the pre- pre-trained model, need to understand the structure of these representations, and this structure must be able to handle the shift-invariance problem.
In Section 3.2,
4.1 "A "Anchor vector" hypothesis
There are different ways prevent the ihift-invvriance of softmax from influencing the performance of the downstream tasks. One way of doing this to keep the partition function stable. Recall that in ( the probability of word is the exponential of the corresponding logit divided by the partition function. If the partition function is constant for different samples, the logits can be uniquely determined by the probabilities, which solves the shift-invariance problem. Arora et al. [0166] showed that when both the word embeddings and the latent representation are uniformly distributed on sphere, the partition function is close to a constant with high probability. They also empirically verified this uniformity of word embeddings trained using GloVe [Pennington et al.) 2014] and
word2vec [Mikolov al. [013] English Wikipedia. However, as we will see in later experiments in Sections 4.2 F this is not true for recent large-scale pre-trained language models.
Instead of uniformity of word embeddings. in large pre-trained models such GPT-2 Radford et al. 2019], we observe that if we remove several most frequent words from the computation of the log partition function, the remaining part can be well approximated by the inner product between the

---

hidden representation (=)(x) and a fixed vector. This motivates us to have the following "anchor vector" hypothesis:
Definition 1. For a sample a and position could select a set of bulk words B(x, i) C [n]. and we define the bulk partition functions as (2, == ΣJEB(E,)) exp((v2,(x-i))) 0z)) and (z = EjeB(u.s) exp( (xp((=((==s) 0z)).
The selection of bulk words Bax, i) can usually be selected manually or by a simple algorithm. For instance, we can construct B(x, i) by taking out the words corresponding to the largest entries in p(xx{/r_1). We can also manually select all the words that are irrelevant to the downstream task.
Hypothesis ("Anchor vector" hypothesis). There exists to E TRd such that
(e-1(x-s): vo) = log (2, (2,
If our hypothesis holds, we can use Up as anchor for the logits because it can be used estimate the partition fuunction, and this does not change the form of the downstream task by much. In the next subsection, we will show that Hypothesis holdd for most popular language models and provide some discussions. More details about these models are provided in Appendix
4.2 Empirical verification of "anchor vector" hypothesis
Figure la plots the mean squared approximation error of the log bulk partition function. We use different versions of pre-trained GPT-2 [Rafford et al., 2019] and OPT [Zhang et al., (2022), and use the first 1/4 of WikiText-2 |Merity et 2016 as the input text. The hidden representations we use in this experiment are the last hidden states of these models, i.e., the output of the penultimate layer. The dimension of the hidden representations ranges from 768 to 2048, and the number of tokens about 70k. We choose the bulk words to be all the words except those having top-k probabilities and compute the optimal anchor vector using the closed- form least-squares solution. In our experiments, use the mean squared error (MSE) to measure the approximation quality. Formally, the MSE is defined as
EMSE
min
IE,,,, (((,((((((xx<))) vo) log Zoulk 1)) #0
The values of the bulk partition functions are around 10 (with comparable standard deviation), and from Figure that the MSE is usually several orders of magnitude smaller. Therefore, the inner product between the hidden representation and the optimal anchor vector can usually approximate the log bulk partition funetion well, and the approximation improves as & increases, i.e. when we ignore more top words. This validates our "anchor vector" hypothesis.
10P
meen Square approxmmation entor
10
GPT-2 Large
GPT-2 XL
10
OPT-350M
OPT-1.38
C20
(a) Mean squared error of log bulk partition function (b) Singular values of dic-(e) Histogram of log origi- with different & in different models. Bulk partition tionary matrices (up: GPT-nal/bul partition functions function includes logits except top-k ones. 2 XL, down: OPT-1.3B) (GPT-2 XL/OPT-1.3B)
Figure 1: Empirical verifications of "anchor vector" hypothesis
The existence of anchor vector is quite surprising. Although there are some settings where the partition function can become easy to predict, egg., when the word embeddings are uniformly distributed on sphere, we have empirically shown that the large-scale language models considered in this paper
7

---

do not fall under these settings. Figure shows that the word embeddings are far from uniformly distributed on sphere, and Figure Indiiccates that the values of log partition functions and log bulk partition functions vary greatly for different samples. More experiments and discussions are provided Section
Anchor vector guarantees performance transfer from pre-training to downstream tasks
Based on the counterexamples in Section 3 and the observations in Section we now give sufficient conditions on the downstream tasks so that the downstream classification accuracy provably benefits from pre-trained representations.
5.1 Model and assumptions
For huge language models, the anchor vector hypothesis states that there exists a vector eg such that its inner product with the hidden state (a=( well approximates the logarithm of the bulk partition function. Therefore, we can use v0 as an anchor to handle the shift invariance problem of the softmax function, i.e., we want subtract (x=,(x-s), vg) from (v (x-;(--), uf). As result, we modify our model for the downstream task to be
& f'(x,) : [ajo((( 26) -b)).
(7)
j=l
The student model is also modified accordingly:
k
f)x = Σ ( - vo) - bb))
(8)
1=1
Note that in the above model we have already adapted Assumption [ which was originally stated for the linear model. Therefore, we also update the assumption and restate it below:
Assumption 4 (At most words of interest). Assume there are at most k vectors in {uz 1i=1 whose logits relevant to the downstream task and WLOG assume these k vectors are (ui...., v.] other words, we assume there exist coefficients {aş))= E IR such that (( vg) b3).
We are still assuming the teacher- stuuent setting and the log-linear word production model for the word probabilities, as restated below:
p² (a. = j|x_,) exp( (v" (z),
p(z; = j|m_,)= exp((r_ v,))
Z
Z
Under the modified model defined above, we will make some additional assumptions. As noticed in the experiments, the log bulk partition function (as defined in Definition L can be linearly approximated by the hidden state. Normally, the bulk only contains words that are not related the downstream task, and every single word usually has small probability, but the total probability of the bulk words not negligible, as reflected in the following two assumptions. Note that the set of bulk words can be strictly contained in the complement of the set of words of interest. This is useful especially when we are not certain about which words are important for the downstream task.
Assumption (Bulk contains no words of interest). For all x and B(x, i) (1,...,A]) k)
Assumption (Lower bound bulk probability). For all a and i,
Z (z. i) 7" (xx i) 2 Pb-
The "anchor" vector (i) used to handle the instability of the partition function, and we formalize anchor vector hypothesis the following assumption

---

Assumption (Linear approximation of log bulk partition function), There exist vo. vo E RR R Rt, S.L.,
Vx, max v6) log Tout (x, a)1b (v_<(x), vo) log Zoulk (I, i)13 S Eb.
Furthermore, we assume Eb S maxje(M] [e5]
For notational simplicity, we will use all the notations (including f", vō. and op) without i when the selection of is clear from the context.
5.2 Main theorem and interpretations
In our model, the ground-truth function f" contains & terms with coefficients {aj ) and we define the margin y the margin for f* If we scale {a increase k by adding duplicated terms to f", we can scale y arbitrarily without changing the pre-training performance or the downstream prediction of the student model. To construct quantity that better indicates the difficulty of the downstream task, we introduce the following definition the normalized margin that is invariant to the scaling and {az
Definition 2. The normalized margin is defined as D 1 ma 1951"
Then we are ready to state our main result. The proof and further discussions for this theorem will be provided in S
Theorem 3. Let EKL == BewDpr. [DKL (p)))() (p(x))] be the pre-training loss. and ECLS
PrawDos (z) f=(x < 0j be the downstream classification error rate, where Dpre and DDss are the input distributions of pre-training and downstream data. Under Assumptions 277 fur- ther assuming minzelk] by 2 EG log(4k) and Se, < r < 6, there exists a set of parameters (az)}=1, (b;)}=1 such that
288 1r Pb 1
ECLS S EKL
(10)
This theorem shows that when an "anchor vector" exists, we can upper bound the downstream classification error by the KL divergence of the student model during pre-training. This upper bound becomes smaller when the distribution of downstream input close to that of pre-training, the bulk probability is non-negligible, and the normalized margin of the ground-truth classifier is large. Here we discuss these ways to decrease the upper bound and their corresponding intuitions.
Large μ.. μ is larger when the data distributions of pre-training and the downstream task closer, which helps with the performance transfer.
Zinia is usually least constant in practice. When the bulk Large ps. The bulk probability 7
probability becomes larger, the anchor vector plays a more important role the partition function and the downstream task.
Large T. A larger normalized margin makes it harder for the student model to make mistakes in downstream prediction.
Remark 1. For ease of presentation, minjelk bj 2 E - (4k and Se, < < 3. More general cases are covered in Lemma 7 We also discuss potential ways improve the bound in Section B.l
making additional assumptions in Theorem e.g..
Conclusions, limitations and future work
In this paper, analyzed when and how the representations generated by pre-trained large-scale language models can be used in downstream classification tasks. We found two necessary conditions for guaranteeing the usefulness of these representations: the insensitivity super-small probability words, and the underlying structure in the representations to handle the shift- invariance of softmax. We also provide sufficient condition, i.e., the existence of an anchor vector, that can guarantee the representations from pre-trained language model to help with downstream tasks. We verify this existence empirically various large language models and believe that this an important reason why recent large-scale language models can adapt to different downstream tasks.
the downstream task to

---

We made some assumptions to provide theoretical understanding on how representations can be helpful for downstream applications, and these assumptions may not always hold in practice. For instance, some downstream tasks require multiple rounds of reasoning and do not rely small set of words, and the "anchor vector" hypothesis can be weaker for some models such as OPT-350M and GPT-2 Medium. Relaxing these assumptions is meaningful future direction.
While our work showed the existence of the anchor vector, remains unclear why this vector exists in most large language models. It might be related to the initialization, optimization, and structure of the neural networks, especially transformers, and it could also related to the underlying structure of the training data. Digging deeper into this may reveal more fundamental properties of these models. Our analysis of downstream tasks focuses on classification tasks. Since these large networks perform well in various types of downstream tasks, another future direction would be to analyze other kinds of downstream tasks.
We only consider using the last layer representation without fine-tuning the whole model, which is usually weaker performance. Furthermore, model the network before the softmax function as black box, ignoring its inner structure. Further opening up this black box and consider fine-tuning the whole model are important for deeper understanding of the structures in the learned representations from pre-training.
Acknowledgements
This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER) and a Sloan Research Fellowship.
References
S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399 2016.
S. Arora, Y. Liang, and T. Ma. simple but tough-to-beat baseline for sentence embeddings. In International conference learning representations, 2017.
Arora, M. Khodak, N. Saunshi, and Vodrahalli. A compressed sensing view of unsupervised text embeddings, bag-of-n-grams, and Istms. In International Conference on Learning Representations, 2018.
Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? arXiv preprint arXiv: 1704.03471, 2017.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
X. Cai, Huang, Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations, 2020.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
K. Ethayarajh. How contextual are contextualized word representationss? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1999.00511 2019.
J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009 2019.
Y. Hao, L. Dong, Wei, K. Xu. Visualizing and understanding the effectiveness of bert, arXiv preprint ar 1rXiv:1908.00500 2019.
10

---

J. Hewitt and C. D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- lational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1129-4138, 2019.
T. Kim, J. Choi, D. Edmiston, and S.-g. Lee. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. arXiv preprint arXiv:2002.00737 2020.
O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the dark secrets of bert, arXiv preprint 1998
2019.
J. D. Lee, Q. Lei, N. Saunshi, and J. Zhuo. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems, 34:309-323, 2021.
S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models, 2016.
A. Miaschi and Oretta. Contextual and non-contextual word embeddings: an in-depth linguistic investigation. In Proceedings of the 5th Workshop Representation Learning for NLP, pages 110-119, Online, July 2020, Association for Computational Linguistics. doi: 10.18653/v1/ 2020.repl4nlp- 1.15. htt.itttss//aclaathollooeo
2020 wwwp411pp11pppp 1.
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations vector space. arXiv preprint 1rXXiv:301.378/1 2013.
J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
M. E. Peters, M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computa- tional Linguistics. doi: 10.18653/w//N18- 1202. URL /tRLtttt/////aaattttolo
org/N18 NN/ 1202
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1)=5485-5511, 2020.
N. Saunshi, S. Malladi, and S. Arora. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations, 2021.
H. Tanaka, H. Shinnou, R. Cao, J. Bai, and W. Ma. Document classification by word embeddings of bert. In Computational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam. October 11-13, 2019, Revised Selected Papers 16, pages 145-154. Springer, 2020.
I. Tenney, Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. Van Durme, S. R. Bowman, D. Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv: 19051099566666 16, 2019.
C. Tosh, A. Krishnamurthy, and D. Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMIR, 2021.
M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907, 13625, 2019.
C. Wei, S. M. Xie, and T. Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34: 16170, 2021.
11

---

G. Wiedemann, S. Remus, A. Chawla, and C. Biemann. Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint 1rX4v:1999.10430 2019.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01065 2022.
Zhang and T. Hashimoto. On the inductive bias of masked language modeling: From statistical to syntactic dependencies. arXiv preprint 2Xxv:2104.05699 2021.
12

---

A Proof of Theorem 1
Theorem Suppose the downstream task performance depends only on a function f (x. i) (1=,()-)), w²) = Di= & (v=,(x==;),
For t- E [b], define p = p²(x; = 1 [i),
and assume p- S 2' Then for all s E Rt, there exist functions v-i and {uz)em such that PRI(p" (x:/-==))ll(x((x=-))
28p and (xx = Ei= G (v-i(( m) S (x, Gj-
Proof. We choose functions v-; and {ok, li=q such that Var, i, V E [m] (1)) (v- = (x=((x=)), ()). Besides, Vx, i, (v=s(x-i)) 1/2)) (x- (
From the definition pf p- and t we know that
exp((" )) 2 (x, i)
p -- -x;
(1I)
Following our construction of the student model, its partition function satisfies
((x, expl (w)1 >
Σ exp((e_s 07))
(12)
j=1
Z (x, exp((1 11 (1-p))11))))
(13)
Besides,
7(z, Σ exp((u (v-i:
expl (op((c=10)))) uf1) (x,
(14)
Therefore,
exp(("",.))))
Z(x,1)
tt,
(15)
plz; = j(z=
expl (v2
(x(2,))
2(0,6)
exp(("_,."?))
pd (2: t xx-
(x,)))
2(z,i)
e.
(16)
p(#; 1- (1-0)
exp( (VZ,,)))
(=(x;
2(0,1)
Thus,
p²(x; j(x
(x.l==/a)) ()) = Σ
p" (z; = j) x1) log
(17)
p(z,- = j(x_1)
5=1
Z(x, i) (x, s sog
Z(z,i) i)
Σ (x; = 5| log
8
(18)
(x,
< sp log
2sp
(19)
1 1 p
and
f(r,1) = Σ (o)))(())))))))
= Σ a (x__,(z=),, w/) - -. =f((x - 8. c.
(20)
B Proof and discussions for main theorem
Here we provide proof sketch of Theorem
Theorem 3. Let CKT. Ey-Dpra [DKI. (p' (p)))) be the pre-training loss. and coLS
Prew (f()) (x) 0] be the downstream classification error rate, where D Prre and Dps
13

---

are the input distributions of pre-training and downstream data. Under Assumptions 2 fur- ther assuming minzels] b3 2 E, log(4k) and SE, T there exists a set of parameters (1)))=1,b))) such that
288
ECLS S EKT. 1r P5 1
(10)
Proof. We have the following lemma providing lower bound of the TV distance between p" (xa( and p(rs( which will be proved Section C
Lemma 1. There exists a choice of (a, (a;);= (b,) such that if Assumptions 2 hold and there exists & E POS NEG and such that (xx (a, i) < O, then we must have
(x=("(=;(<=>)) p(rel 2 pb
min Ep
(1 (1-e) e1- =)) + h min c -
(eo (200-2)0- - 1)
JE(M
(21)
From Lemma and Pinsker's ' inequality we can lower bound the KL divergence at any incorrectly classified sample: For all POS NEG,
KL(p" (= 2(TV (p [
(22)
(23)
57 20 -2 200-)20000) -1) ?
2 2ph min (x) + k min 2,
(24)
jE(k]
Since the pre-training loss is the expected KI. divergence for pre-training data, it is lower bounded by the KL divergence the incorrectly classified samples, which is related to the downstream classification error rate:
<KI_ RaD pre [KT (xelar (x/==-)))(((s===-)))) min
(25)
Pr [f(z,)) f" (z, i) < 0]
KL(P" (a.(=-:))lp( (m-,))
(26)
z 2-1 D
(:/(a))) (2,
p. Pr [5 1" (ar, <0] - =(((a.))
min
KT.(p" (=sl=_))ll(<<((==)) (27)
2 ~DDs
2
2 ECLS 2p Ph 2 min (1 (1-e =r) + k. min , e (c0(f-20-0)) AE)
(28)
20
Here Drre and DDS are the data distributions of pre-training and downstream task, and the second inequality comes from Assumption 2
Now we have established the relationship between CKL and ECLS. and the only step left is bound the minimum over Ep. For notation simplicity, we define A == 2eg.
If Ep $. we have (1 - Ew) (1 - c1- $).
If Ep
A know that
mare seac))) 18g1 220 -1
bj - (e3 -1) (1)-1) bb c sinh
A
Thus,
(5 220 - -1)
min (1 - (6) + k- min
(29)
jEA(1)
A
min
1 - e
2k
min jE((
0 sinh
(30)
3
Plugging this into 28 gives us
2 A
6KL 2 ECLS 24 pd min (1- E $32 1,2,415
462
min JE(M)
263-22.
25. sinh?
(31)
14

---

Since r > Sca. know that 1 < 3, - (g)
r }
576
Besides, min bj 2 co -log(4k) implies 4k² minj ت26ع
We also have sinh (g))
(3)
r
ITI
Thus, min {11 (1-e- )2, 42,2 minje[M] 262--22 sinh? sinh?( ($)
r
Plugging this into 31
fin-
576
ishes the proof of Theorem 3
B.1 Further discussions of Theorem 3
Potentially improving the upper bound by better thresholding. We taking the minimum over two terms in the proof of Theorem These two terms come from two parts of the difference between the ground-truth probability and our learned probability: the bulk part and the top part. Both parts are functions of Ep. which measures the approximation error for the bulk probability. A large Ep will incur a large eтror the bulk part while a smaller 6, will make the top part larger. We are setting A as the threshold for the current bound Theorem 3 The bound could be improved by choosing other thresholds if we know more about the values (bg) and the normalized margin r.
More general cases for thresholds {b; } J==1 and normalized margin r. The thresholds (b3
can considered as the model's sensitivity to the logits. A smaller threshold indicates a higher sensitivity. In the proof of Theorem 3 we are assuming that the thresholds are not very small. In general cases where the thresholds can be small, an important observation is that: For the same set of samples, decreasing the thresholds by some number s will increase the normalized margin by as well. Since both the thresholds and the normalized margin are in the exponent in our bound (sinh function is close to exponential when I' is large), these two effects can cancel out to a large degree and don't 't influence the bound by much.
Prompt engineering can help with downstream performance. Prompt engincering can help with the downstream task performance in multiple ways. One direct way is to make the data distribution closer to that of the pre-training stage increase μ. Furthermore, can also indirectly help improve the bound by decreasing the number of activated neurons, increasing the margin, etc.
C Proof of Lemma I
Here provide proof of Lemmal with the proofs of the necessary technical lemmas postponed to Appendix D
Lemma 1. There exists choice of (ag -1 (by) such that if Assumptions 2 exists E POS NEG and such that f(x, i) .f" (x, i) < 0, then must have
hold and there
TV(p" (x:l==))) p(x((==-)))) ps min 50
(1 -1- (1-e-5r) + h min
-- c -e- (B-22en-5p) -1)
jE(M]
(21)
Proof. For all j [n], we set a, = az and by = b3. Since is fixed in the proof, we will omit when the selection of is clear from context.
WLOG, assume E POS because otherwise we can flip the sign of all az's. In this case, from Assumption 3 we know that (x, i) 2 7. Therefore, f(r f (z. i) < 0 implies f(x i) < 0, so fx f(, >.
There are two possible ways to make f(x i) smaller than (xx i): Making the neurons with positive coefficients smaller or making those with negative coefficients larger. The total difference must be at least 7, so there must exist one sign whose change in value is at least 3. Formally, we decompose the
15

---

difference (z, i) f(z, i) into two terms:
(zx, f(x = E aj
(0 (2), -6)) bb) o((u // - vo) bb)) (32)
J:az
+
Σ
(a((v oj - vo) b1) o (10 (x), oz (6) - -b)) (33)
zaj <0
Note that at least one of the two terms on the right-hand side must be at least 3 because their sum is at least 7. We first consider the first case, i.e., -6)
Ejazzo aj (z), - 0 - b3) - (vv_(<)), By - uo) - b7)) 2 2 The analysis of the second term is almost the same the first one.
For notational simplicity, we define
A = ((=2 oj r6) [0)b-b)))-) -(((v=(x)) uo) - bbj),
(34)
(n.b. a only appears in the second term) which implicitly depends on x. and and define the set
S(z) = [j : aj > 0,4, > 0}.
(35)
If A > O, we must have (x",(z), 0j v by 0 because ReL.U is non-negative. This means that the neurons corresponding to the words in S(x) must be activated.
Intuitively, S(x) the set of words whose corresponding neuron in our student model has smaller value than the ground-truth model, and A, for word j E S(x) is the difference between these two neurons. In other words, if we want make a mistake in classifying sample x, A, is the obstacle for the neuron to overcome, and the sum of all these obstacles must be at least the margin Formally, we have the following lemma for (a) and lower bound for Dy:
Lemma 2. S(x) #
Lemma 3. EjES(a) Aj Z max "5"
Considering the TV distance, we get
pp (I+ j(x=s) - p(z; = ا(جー1|ز
(36)
(x; = j( p(a; = + Σ
(2; = 5| p(2; = x|x=
(37)
jEB(z,))
jES(z)
Σ
p¹ pi j(xx Σ
pla, = j(x=
Σ
p' f|x= - pla; = j x:
jEB(z,i)
jEB(z.))
SES(z)
(38)
7oun (x,
Zhuulle (xx ((x,
Σ
(x; p(#: j|mm1)l
(39)
JES(Z)
where the first inequality comes from Assumption 5 i.e., S(x) C
[ C [m] B(r,t).
We define Ep := log Zian (a.1) logg Zoniz(z.)) (-,a))
Z(a,) which is the approximation error of the log bulk probability and an important quantity trade off the two terms the TV distance. There are two terms in (39). The first term corresponds to the bulk probability and the second term corresponds to the activated neurons. When Ep large, the bulk probability has large approximation error, resulting in large TV distance. We lower bound this term in the following lemma:
Lemma 4.
Ziun (I, (x,
Zuulk (z, i)
2 pb61 - -.v)
(40)
2(I, i)
16

---

The eight-hand side of the above lemma is an increasing function of Ep, and this lower bound becomes large when the approximation error of the bulk probability large.
In the other regime when Ep is small, the bulk partition function must be approximated accurately, Thus,
i... Lina (2,3)
(a.i)
(=,2,) ((a,))
(xp((vi((x)))))
exp((v-((z).0g))
1p" (2+ j(x=1) p(a; -)(=,)|
(41)
Zz
((x,
bulle (x. i) exp( (up(((2 ((),
exp((u_:(x)..))
(42)
Z (2r,
(r, (x,
Zbulk( (2,
From Assumption 7 we know that the bulk partition functions can be accurately approximated using the vectors 2i and vo. Thus, exp((v ((x),;))) & (x)) 2g)). Therefore, this difference is (213)) exp((x 1j -o-v)) ~ exp((r-(x))) 12 - -o>) -1).
approximately (x)((r"((x))..))) v5 -))) As Ep becomes smaller, the aforementioned approximations become more accurate, making the TV distance suffer from a term that is roughly proportional to (e - 1).. This is formalized in the following lemma:
Lemma 5. For all EE S(x),
-= - p(æ, = j(ز=1)l e - pb (0(A)-26b-s))-1)
(43)
Plugging Lemma 4 and 5 into 39
gives us
(44)
Thiik (2,1)
Zonlk (x)
(xp((12((x))
exp((u-((z), u3))
2
(45)
Z (z,
((z,i)
(x,
Z (I,
JES(a)
2 (1 - -x) + Σ
bj co 5 ph
=x) -1)
(46)
(E)(w)
2 po(1 Fx)
pb min
Σ
(cols) (coLA)-20-e0)- -1)
(47)
(
JES(x)
2 (1 pb min
d c Ep.
|S(x)
(eEscose alA, -60 2665-266655))15500 -1). (48)
jES(#)
where the last inequality comes from Jensen's 's inequality.
Let g(u, v) = u.u.e (e - 1) for v > 0, then dg Tu we know that
= = = S 0, so from |S(x) s
[a())
o(A, ((
-1) 2 k. (eIrEsts) o 200 200- - 1)
Using the fact that E. = E, max 03 2 max EE, Mis 0f=0== =0 W:).
Σ σ (A, 22a Ep) o
Σ jES(a)
(Ay-2es 22s -8p)
Ep)
20 2 22max aj
2k -8 Eb kεp
JES(z)
(49)
where the last inequality follows from Lemma
Therefore,
TV(p" (x0g((x))_))) ((1/1=-)),((xxx11)))
(50)
(f
( pb l min
5 -eb-p JES(z)
220 -20
-1) -1).
(51)
Since we do not have any assumption for Ep, we take the minimum over all possible values of Ep and get the following bound:
17

---

(52)
( (E of 2:56 -1 -
PPb min 6p20
(1 (1-0 Fv) + k min e -
(53)
JES(a)
Above we have derived a lower bound for the TV distance between p' and when the first term of (33) is at least 3. When the second term of (33) is least 3. the proof is symmetric to that of the first term and we only need to exchange the role of parameters from the teacher model and student model and the related definitions. Therefore, with the second term being least 3. we have
TV [V(p"(x)l<=)))
(54)
TVmos 1927 -200-pp) -1) )
Pps min 5,
(1 (1-6-
+ k- min (E)
(55)
where S'(x) {j: aj 2 0} and A
25 26) 63).
Since C [ merging 53 and
(5) finishes the proof of Lemma |
D Detailed proofs of auxilliary lemmas
Lemma 2. S(x) 0.
Proof. Assume by way of contradiction that S(z) = 0. Then by the definition of S(z), we know that for all j such that a; > 0,
al a((e1 13 v6) b3) ((v vo) - bÿ)
(56)
(((vI,()), 07 - v5) bb)) ((v By vo) by) -, S
(57)
Therefore,
Σ ag (o((u2, 46) - a ((( no) bb)) s < 7-
2
3,330
which is a contradiction. Thus, S(x) + 0.
Lemma 3. ESE(S)) Aj Z mmxxjeste) o5
Proof. Note that for all j E S(x),
- -()) v6) b > o((v-(()),) - vo) - 63) 2 0, so
d((o2(x)), Uz v²) - bj) = (0",(x), - -b.. Therefore,
Σ a Σ af ((1o2(()), v6) -6) bb) o(((v_(z)) Uy vo) bU)))
(58)
JES(z)
jES(x)
= Σ
(o)(0_1))))) - 1 - o((v_,()), - vo) - bj))
(59)
JES(#)
Y 2
(60)
which implies
Ejes(e) max A, Ejes(H) ajAj
Σ 4
(61)
maX az
maxxe5(z) a;
maxxes(2) at
(ES(z)
18

---

Lemma 4.
Zonlle (z,
Zuuik i)
2 Pb(1 -.
(40)
(x,
((x,
Proof. Ep = 0, both sides of the inequality equal O, and the inequality holds. When Ep > 0,
Bull (I,,
(62)
2 (x,,)
((x,i)
Tull (I,
Tuul (xnlk((r,
Zt (x),
(63)
Z(z,i) i)
Zouuk (2)
Zonk
Zuulk (z. i)
log Ziuik (2,,1) )
(x,
exp
log
(xx, i)
2x 1)
(64)
(uux(1,))
exp (-1 log
Zoul (I, 1)
log Zbulk (2,,)) z= 1)
(65)
(x,
((xr
2 po(1 - 55).
(66)
Lemma 5. For all j E (z))
[ = a: - -p(:; = j| 2 2 - 50- pb 200 -1)
(43)
Proof. We know that the word probabilities come from a log-linear model, so
10" (2= = j|xm-)) p(a; j|x= (xpp(vu,((x)))
exp((u-i(x), 7(x,
Z0 i)
If the probability p(r; = j| is reasonably large, i.e., its corresponding neuron is activated, then we can use that with A bound this difference. In other words, when (v-r(x), U/ - vo) 2 bj. we know that
((11)()) 23) log 2²) ((v_,(1), 02) log 2)
(67)
Zuulk z
Zulk
= =(v2,()), log 4bulk) - ((v_((z), 12)) - log Zbulk) + log
log
(68)
Z
2 ((v2,()), ug) log Zhulk) - ((v_((z), 02)) - -log Zbulk) - Ep
(69)
=u]2,(x), og - ui) (v_:( U1 - vo) + ((v2, 28) -log Zbulk)
(70)
((v= vo) log Zbulk) Ep
(71)
24j 26p- Ep-
(72)
Note that the last inequality 72 comes from Assumption and the following property Ay:
(oi,(x), - v (v) - vo) = (vi,(z), 03 - vt) bj ((v_i(z), Uj vo) b))
(73)
2 (vi,(x), - vi) - b3 < ((00)())) - vo) b9)
(74)
=,,-
(75)
Then we bound the difference in probabilities: for all j E S(x).
exp((v2 (x)),
exp((v_((x))) r,))
(76)
7-
7
exp((v- ((x), vz))
Z
exp((u=i(1)) 0z)) -1
(77)
2
Z
exp((u_<(x). vz)))
exp ((vi,(z), 0z) - log z²) - ((v 23)) - log 2)) - -1.. (78)
Z
19

---

Note that
exp( (v- (2)=(2),00))
log
(u_1(z), 22)) log Z
(79)
Zullk
[V->(1z), 1/j vo) + (u-;(2), - log Zmilk + log Z
(80)
bull +
Zbulk Z
bullk
2 bj Eb + loog
log
-log
(81)
Z
Z
2 bj - Eb + log Pb - Ep.
(82)
Moreover, when A 26b Ep > 0. we have by
72) that
exp (((=2 ,3) - log 2n) -(0--(()))))))1113)))) -1
- log 2))
(83)
2 exp ر - 28h - Ep) - 1.
(84)
When 4; 256 Ep S 0, we have
exp (((o", - log Z*) - -(vo-(x), - log Z)) -1 1/ 2 0 -1
(85)
Thus,
exp (((o2 u5) log Z*) ((0_ v²) log Z)) 1 2 -0( -1. 1.
(86)
Therefore, when (w-) Dj vo) 2 bj. we must have
j - p(=; = -لحا 2 c0j c0 po
1).
(87)
In the second case where (v_:(x), Uj - vo) < bj. we have (v=i(z), Uz) < (v-:(x); vo) + bb The proof for the second case is very similar that of the first case, with the main difference being we replace (z- 2y) in the first case by (v-((z), vo) + b3 in the second case.
In the second case, by the definition of D we know that A = ((v2,()), - v6) -b)).
Similar to the first case, we can get
((v2,(z), of) - log 2²) - ((v_:()), 0o) b3 - log 2)
(88)
= ((02,(2), of) log 4bulk) - -( ((v_(x)), vo) + bū - log Zoulk) + log Ziulk
Z
log
z
Z
(89)
2 (102,(1), 07)) - log Zbuk) - ((v_(()), uo) + トb - log Zrollk EEp
(90)
2 (vi,(4), 1nj - -v)) bi + (1v2, v6) log Ziulk) - ((v=1xz), vo) - logg Zbmlkk - Ep (91)
2 A- - 20b- Ep-
(92)
Therefore, for the difference in probabilities, if A, 22a Ep ((v2(()) of) - log 2²) > ((v=;(z),v)) vo) + bī log 2). so 10" (z. = j(x_;) - p(a; = أل exp (xp((vu,(x)) (x),((
0, we have
(93)
exp( (p ( mz)))
(94)
7.
7
exp((e2,(2) uz),
exp((e- (x), vo) + b7)
(95)
Z exp((e- (x), vo) + bj) (exp ((122 0z) log 2*) - ((v_: 00) + b2 - log Z)) - 1)
Z+
2
(96)
exp((v=>(2)) vo) + +b])
-205-pp- -1) 1).
(97)
Z
20

---

If A 25b - Ep S O, the absolute value of probability difference can be trivially bounded below by Thus,
(0, = 5|x_1) - pl = 5|(=-)1 exp((v-<(x))) vo) bj) (2(l))-20--0)-1 -1)
(98)
Z
Since
exp((u-<(a)) vo) + +b])
Zbulk
log
= bj (v- vo) - log Zbulk + log
(99)
Z
Z
Zoulk + z
Zuulk
bulk
bj Eb + log
logg
-log
(100)
Z
2 bj - Eb + log - Ep:
(101)
we finally get the same bound as the first case:
(x; j p(a, = e -o s...p.. ps (E((A)-200)) 2200-00) -1)
(102)
Merging the two cases, we know that for all j E S(x),
Ip (z) j x p(=; = )- e -6--5 ps (c((2)-200-5)) -200-6p) -1). This finishes the proof of Lemma 5
(103)
E Experiment details
Language models. We use various versions of GPT-2 Radford et al. [019] and OPTTZaan et al. G
[2022) with number of parameters ranging from 125M T.5B. We the base, medium, large, xi version of GPT-2 which have #parameters 124M / 355M / 774M 1.5B and hidden dimension 768 / 1024 / 1280 1600. For OPT, we use three versions, with #parameters 125M 350M 1.3B and hidden dimension 768 / 1024 2048. For all these models, the word probability is the softmax of the product of the penultimate layer representation and the dictionary. This is consistent with our theoretical model introduced in Section 2 The parameter settings and performances of these models shown Table
Table 1: Parameters and performances of languge models
Model
#Param
Hidden dim
Perp
GPT-2
124M
768
25.92
GPT-2 Medium
355M
1024
19.19
GPT-2 Large
774M
1280
17.13
GPT-2 XL
1.5B
1600
15.34
OPT-
125M
768
25.01
OPT-350M
350M
1024
19.69
0.T-
1.3B
2048
13.16
Dataset. We use WikiText-2 Merity et al. (0166] as the text corpus. WikiText-2 has about 280k tokens, and only the first 1/4 of it for computational efficiency. The perplexities of the language models on this corpus are shown in Table
Existence of anchor vector not trivial
In this section, will empirically verify that the existence of the anchor vector in large-scale language models does not arise from the trivial reasons that we could think of.
Linear approximation is not accurate for original partition function. Table 2 shows the mean squared approximation error of the log bulk partition function for different models and different k. For the models whose log bulk partition functions can be well approximated linearly, their log original partition function cannot. This confirms the necessity of removing the words of interest in the partition function.
21

---

Table Mean squared approximation error of log bulk partition function for different models and different k.
Model A:
10
100
1000
10000
GPT-2
625.8
651.4
665.2
685.5
727.4
GPT-2 M
0.7647
0.3444
0.3005
0.2683
0.2396
GPT-2 L
0.7296
0.1288
0.0740
0.0317
0.0022
GPT-2 XL.
0.7716
0.1219
0.0648
0.0279
0.0019
OPT-125M
0.6688
0.1010
0.0443
0.0117
0.0006
OPT-350M
2.2449
0.9393
0.7587
0.6182
0,4275
OPT-1.3B
0.5234
0.0984
0.0402
0.0117
0.0006
Word embeddings are not uniformly distributed on sphere. Figure shows the singular values of dictionary matrices from different models. For the four selected models, their word embeddings close to being low-rank. Therefore, the word embeddings in these models are far from uniformly distributed on a sphere.
(a) GPT-2 Large
(b) GPT-2 XL
(c) OPT-125M
(d) OPT-1.3B
Figure Singular values of dictionary matrices.
Dictionary atoms not uniformly distributed on sphere. Figure shows the histogram of the f2-norm of the dictionary atoms from different language models, and Figure 4 shows the distribution somewhat
of the Cosine similarity between two random atoms The norms of these atoms bounded, but their cosine similarity is strongly biased towards the positive part, and the dictionary matrices close low rank, indicating that these vectors are far from uniformly distributed sphere. Instead, they may concentrate around a cone-shaped region.
750
D5C
(a) GPT-2
(b) GPT-2 Medium
(c) GPT-2 Large
(d) GPT-2 XI
(e) OPT-1P1-2MM
OPT-350M
(g) OPT-1.3B
Figure 3: ---orms of atoms
(Bulk) partition functions have large variations. Figure shows the histogram of log partition function and log bulk partition function for the language models, where the bulk words are defined as all the words except the top 100, in terms of logit values. We see from the figures that both the partition function and the bulk partition function vary a lot depending on the samples. Therefore, in recent large-scale language models, the partition functions are not stable across samples.
22

---

(a) GPT-2
(b) GPT-2 Medium
(c) GPT-2 Large
(d) GPT-2 XI.
(e) OPT 125M
OPT 350M
(g) OPT 1.3B
Figure 4: Cosine similarity between two random atoms
(a) GPT-2 Large
(b) GPT-2 XI.
(c) OPT-125M
(d) OPT-1.3B
Figure 5: Histogram of log original and bulk partition functions
23


Reviews:
[{"review_id": "YHxbFrzVHV", "reviewer": "Reviewer_8QyB", "summary": "This paper explores how to connect pretraining performance with downstream task performance (i.e., binary classification). The theoretical analysis is based on token representations. The authors find the ``anchor vector'' in the representation space and bridge pretraining and downstream tasks performance based on it.\n\n---\n\nRebuttal response:\n\nThanks for the clarification! It would be better if these parts can be described clearly in the paper.\n\nAs for the responses to the two questions, I don't think they are convincing. As I mentioned, previous work used ``surprisal'' to measure the information amount brought by that word. “low probability words” could be vital and carry essential information in that sentence. But this is a theoretical analysis paper. So, I'm fine with this assumption. Therefore, I would only slightly raise my score.", "questions": "Q1 (Line 204-206): Why does this claim hold? First of all, the vocabulary size of LM could be very large (e.g., for BLOOM, its vocabulary size is 250,000). A word with probability 1e-5 is not super-small. Second, words with small probability could be vital to the downstream tasks. Some people use ``surprisal'' to measure the information gained from that word. It means words with small probability could convey information.\n\nQ2 (Line 246-248): Does it mean that the frequent words are not that useful? Is the claim contradictory to the aforementioned one?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper discusses a vital problem in language model understanding from a theoretical view. The idea of the proofs is interesting.", "weaknesses": "1. There are many assumptions and some of them may not hold in practice. A1 could fail for contextualized language models (e.g., BERT, GPT-2) since one token could have very different representations with different contexts. A4 may fail since the prediction head of downstream tasks could be nonlinear. Besides, there are some claims that could be wrong (see Q1, Q2).\n\n\n2. The ``anchor vector'' hypothesis could fail. Even if the authors show that the MSE for the approximation is less than 1. However, it is not sufficient to prove the hypothesis empirically. Language models could have quite similar representations for all tokens, and they may be far from the zero point. More baselines should be included to support the hypothesis. For example, calculating the average/minimum MSE distance between two random words.\n\n3. The scope of this paper is not a little limited. Even if in the introduction and empirical section (4.2) popular language models (e.g., GPT-2) are included, the theoretical discussion only considers binary classification as the downstream task. Besides, not sure if I understand it correctly, the analysis of language models assumes that they are uncontextualized (e.g., GloVe and ELMo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. However, there is likely some serious misunderstanding which we try to clarify below:\n\nQ: Many assumptions; some of them can fail.\nA: In general we agree that we make many assumptions and they may not always hold in practice. The goal of our paper is to give some theoretical understanding on how representations can be helpful for downstream applications, and unfortunately this is extremely difficult without assumptions. We tried to give explanations/motivations for every assumption we make, but it is of course still subjective on whether those are sufficient or not.\n\nAs for the concrete assumptions, we want to emphasize that A1 *does not* need to fail for contextualized language models. It is indeed true that for these models, the representation of a token can be different depending on their context. However, the “vectors” in A1 refer to the weights of the last layer for these large language models, which are fixed. More precisely, referring to Equation (1) in the paper, the representation of a token is the v*_{-i}(x_{-i}) which is indeed allowed to depend on the context (x_{-i}), while the vectors v*_j’s are the weights of the last softmax layer and are therefore fixed. We believe this is a major confusion that the reviewer had and we will clarify this in the final version.\n\nQ: Anchor vector assumption might fail.\nA: Although we provided empirical evidence that anchor vector assumption holds for the language models we considered, it is indeed possible that for different language models it can fail. However, we would like to point out that the possibility mentioned in the review (that anchor vector assumption might be true because the language model has similar representations for all tokens) is ruled out by our experiments: as we can see from Figure 1, when we choose k to be small (i.e., we do not exclude the frequent words) the approximation guarantee is much worse. This is not possible if all the words have similar representations and is a stronger indicator than measuring the MSE between random words.\n\nQ: Limited scope: binary classification, simple language models.\nA: As the analysis is already quite complicated for binary classification, we choose to focus on that for simplicity. However, most of the ideas can be extended to a multi-class classification setting. The paper does not just apply to simple language models and we suspect the misunderstanding is similar to the one we explained in the first question - we will make sure to clarify this point carefully.\n\nQ1: Why does 204-206 hold?\nA: We think there is again a misunderstanding here. When we talk about “low probability words”, we are not talking about the probability of the word in the dataset/without context. Of course, as the reviewer pointed out, a word with probability 10^-4 is still fairly frequent when we don’t condition on context, and it’s unreasonable to leave those words out. However, in this discussion by “low probability words” we mean the word has low probability after *conditioning” on the context. For example, when we use a prompt like “this movie is ***” for a sentiment classification task, we expect the blank to be some of the words that are extremely relevant to the task (such as good, bad, exciting, boring). For a word that is meaningless in this context (say “for”), we shouldn’t care whether the model predicts it with low or extremely low probability. We believe this is justified in practice because the distribution of words changes significantly after conditioning, and in applying language models people frequently use top-k words after conditioning for k that is not large. We will make this more clear.\n\nQ2: Line 246-248, does that mean that the high probability words are not useful?\nA: No, in fact it means quite the opposite - the high probability words have such a strong influence on the partition function, so that if we don’t remove them, we cannot well-approximate the log-partition function."}, {"review_id": "XKnh633rD5", "reviewer": "Reviewer_joK7", "summary": "\nThis paper presents a sequence of assumptions along with their respective conclusions, which advance the objective of comprehending the intricate connection between the performance of pre-training and downstream tasks in language models.\nBy framing the prevailing language models as log-linear models, this paper initially presents a mathematical demonstration showcasing instances where pre-trained models encounter limitations in effectively transferring their knowledge to address downstream tasks.\nAdditionally, the paper enumerates several imperative prerequisites for successful knowledge transfer, such as the requirement for shift invariance. \nFurthermore, the authors introduce the \"anchor vector hypothesis,\" which serves as a crucial framework for elucidating the remarkable adaptability of language models to various tasks.", "questions": "If we examine the standard (masked) language modeling during the pre-training phase, as far as my understanding goes, $p^*(x_i|x_{-i})$ should correspond to an instance of the Dirac delta distribution. If this understanding is correct, I'm wondering the subsequent discussions in the paper, such as the one in Section 3.1 where scenarios are considered where $p^*$ is relatively small but possibly not zero, are still valid.\n", "limitations": "While the mathematical framework proposed by the authors is compelling, there is a valid concern regarding the explanatory power of the paper's contents in relation to the inner workings of current language models. This concern arises due to the series of (unrealistic) assumptions upon which the framework is built.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Attempted to rigorously prove the effectiveness of pre-training language models with a mathematical framework.\n- The claims posited in the paper appear to be reasonable, albeit necessitating a potential round of further verification, provided that readers accept the underlying assumptions upon which the proofs are based.\n- The storyline of the paper is intuitive and persuasive.\n\n\n\n", "weaknesses": "- Given the non-negligible series of assumptions outlined in the paper, a legitimate concern arises regarding the practical implications associated with the claims put forth. \nFor instance, the authors commence the proof procedures by assuming that \"a downstream task depends on only a small set of words related to this task,\" which may not hold true for tasks that involve complex rounds of reasoning.\n- In addition to Section 4.2, it would be advantageous for the paper to include additional empirical experiments that substantiate the claims presented, thereby strengthening the overall argument.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: Many assumptions\nA: In general we agree that we make many assumptions and they may not always hold in practice. The goal of our paper is to give some theoretical understanding on how representations can be helpful for downstream applications, and unfortunately this is extremely difficult without assumptions. We tried to give explanations/motivations for every assumption we make, but it is of course still subjective on whether those are sufficient or not. We will add more discussions on the limitations of these assumptions.\n\nQ: p*(x_i|x_{-i}) being a dirac delta\nA: Note that x_i and x_{-i} are discrete (they are words) instead of vector representations. If they are vector representations (which we denote as v*_i(x_{-i}) as in Equation (1)) then that is indeed often deterministic/dirac delta distributions. The distribution p*(x_i|x_{-i}) is the distribution of a word given its context. It usually is not a dirac delta function as that would mean we are absolutely certain which word needs to be at position i given its context, which is rarely true given the inherent amount of ambiguity in language. None of the language models we used in our experiments give dirac delta distributions for p*(x_i|x_{-i}). As for the specific instance of Section 3.1, Theorem 1 is talking about a word that has low probability, which will exist even if p*(x_i|x_{-i}) is almost concentrated on a single word.\n"}, {"review_id": "wIrjiY2QIL", "reviewer": "Reviewer_1JJM", "summary": "This paper investigates the relationship between language model pretraining and downstream classification tasks. Under certain assumptions, the author theoretically demonstrated that pre-trained models can guarantee performance on downstream tasks with the existence of proposed “anchor vectors”.", "questions": "See weaknesses.", "limitations": "The downstream tasks discussed in this paper are somewhat limited, e.g. the sentiment classification. Some of the assumptions are also based on the importance of words that are related to the task itself, which makes the overall framework not general enough.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research question of “why pre-trained models can help with downstream tasks” is quite important.\n2. The paper is generally quite clearly written, the question definition is clear and the mathematical settings are reasonable.\n3. For me, the method to connect minimizing the KL divergence during pre-training and the performance of downstream tasks is interesting and valuable.\n", "weaknesses": "1. I think there is a difference between “pretrained embedding” (e.g., word2vec) and “large language models” (e.g., BERT/GPT), also the large language model is not equal to “the last layer embeddings”. In this paper, the author takes the model fixed and takes the extracted representations for analysis, however, fine-tuning the whole model is actually a more common way for models such as BERT/GPT on downstream classification tasks.\n2. Some of the claims and assumptions may be incorrect. For example, “We can also manually select all the words that are irrelevant to the downstream task.” in Line 255 and “Assume there are at most $k$ vectors are relevant to the downstream task” in assumption 4. These only hold true for certain tasks like SST-2, which is not generalizable.\n3. There exists more works need to be discussed and compared. e.g.,\n[1] Visualizing and Understanding the Effectiveness of BERT (EMNLP2019)\n[2] Revealing the Dark Secrets of BERT (EMNLP2019)\n[3] On Mutual Information Maximization for Representation Learning (ICLR 2020)\n4. I see some findings from Appendix F (Existence of anchor vector is not trivial) in the supplementary material, however, most of the conclusions are already found from previous research. e.g.,\n[4] How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP 2019)\n[5] Representation Degeneration Problem in Training Natural Language Generation Models (ICLR 2019)\n[6] Isotropy in the Contextual Embedding Space: Clusters and Manifolds (ICLR 2021)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Large language model is not just last layer embeddings.\nA: Indeed, this paper only considers the setting where one takes the last layer representation of a language model and uses that in downstream tasks without fine tuning the whole model. While this gives reasonable performance for many tasks, it is indeed often weaker than fine tuning the whole model (although it’s significantly cheaper). We will highlight this limitation in the final version.\n\nQ2: Some assumptions may only hold for certain tasks and not generalize.\nA: In general, we agree that we make many assumptions and they may not always hold in practice. The goal of our paper is to give some theoretical understanding on how representations can be helpful for downstream applications, and unfortunately this is extremely difficult without assumptions. We tried to give explanations/motivations for every assumption we make, but we agree that they certainly don’t cover all the settings. We will add more discussions on the limitations of these assumptions.\n\nWe also thank the reviewer for the pointers to more references in weaknesses 3 and 4, we will add discussions to these very relevant works.\n"}, {"review_id": "4iyo7zo6Uq", "reviewer": "Reviewer_TyuF", "summary": "The paper mainly addresses two conditions that enable the representation of pre-trained LLMs to be transferred effectively to the downstream tasks which are usually different from pre-trained objectives. \n\n1. The insensitivity of the downstream task of \"super-small\" probability words must be guaranteed for good downstream performance. Thus, a function like ReLU is needed as an additional structure of the student model to ignore “small” logits which are considered to have no influence on the downstream task. \n\n2. Using a shift-invariant softmax function can render the logit values meaningless in unseen data. Here, the need for structure in the representation space of the pre-trained model arises and leads to stabilization of the partition function with the “Anchor vector” hypothesis. \n\nThe anchor vector hypothesis has backed up with empirical verification. The authors demonstrate that as more words with top probabilities are excluded to compute the optimal anchor vector, the error of approximated bulk partition function decreases. This finding supports that the anchor vector actually exists and can be utilized to address the shift-invariance of the softmax function. Furthermore, with the anchor vector, the upper bound of the downstream task’s error rate can be made.\n", "questions": "(1) Based on Figure 1 (a), it is evident that OPT-350M and GPT-2 Medium show minimal decrease in mean squared approximation error, while other three models exhibit a significant decrease in mean squared approximation error as k increases. What could be the underlying reasons for the different tendency? It cannot be attributed solely to the difference in model scale (number of parameters), since OPT-125M displays a considerable decrease in error. Without further analysis, the findings from two of five models (OPT-350M and GPT-2 Medium) present weak evidence to support the “anchor vector” hypothesis. This raises some concerns regarding the generalizability of the proposed hypothesis to pretrained LLMs as a whole. \n\n(2) Do you consider including a broader range of activation functions like swish [1] and gelu [2]? Does the proposed hypothesis still hold for these activation functions, other than ReLU, that do not completely ignore (treat as 0) small logit values?\n\n[1] https://arxiv.org/abs/1710.05941 \n[2] https://arxiv.org/abs/1606.08415 \n\n[1] Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941.\n[2] Hendrycks, D., & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415.\n", "limitations": "Limitations are listed in section 6: Conclusions and future work. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written with a clear flow. Starting from the problem setting, the properties of the learned representation that affect the performance of the downstream task are well stated and addressed with the anchor vector hypothesis.\n- The empirical verifications provide substantial evidence in support of the anchor vector hypothesis, solidifying its credibility and reinforcing its validity.\n", "weaknesses": "(1) The empirical verification of the “anchor vector” hypothesis lacks clear explanations and detailed information regarding the experimental setup, leading to confusion and a sense of insufficiency. One particular point that requires clarification is the choice of using auto-regressive models, specifically GPT-2 and OPT, for the verification.\n\nAs mentioned earlier in Line 138, the pre-training task of the language model is described as predicting a “label” given a sequence without “label”, which aligns with the mask infilling task associated with masked language models. However, in section 4.2, the verification process utilizes the auto-regressive models, GPT-2 and OPT, which have a pre-training task of predicting the next token of the given sequence. \n\nBased on the provided context, it appears that a single token which can be an intermediate token of the sequence is removed from the sequence (-i) rather than considering the last token of the sequence. This discrepancy raises questions about the alignment between the pretraining task and the verification task depicted in the provided context in section 4.2. \nAddressing these issues and providing a clear explanation of the rationale behind using auto-regressive models instead of masked language models for the verification process would enhance the overall understanding of the presented claim. \n\n(2) In Figure 1, the labels for the x and y axes should be provided. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: Verification of anchor vector using autoregressive models.\nA: In our definition we allow v*_i to depend on the entire x_{-i} to capture both autoregressive models and masked language models for generality. An autoregressive model can still be a special case of the theory as their v*_i can just depend on the prefix. We chose OPT and GPT-2 mostly because those are the largest models we have access to.\n\nQ: x/y axis of Figure 1\nA: The x axis is the number of highest frequency words that are removed (k in the text), and the y axis is the mean-squared error for the prediction. We will add these labels in the final version.\n\nQ: Figure 1 OPT-350M and GPT-2 Medium.\nA: We honestly don’t know why those models behave that way and we agree with this weakness.\n\nQ: Considering other nonlinearities.\nA: That is a very interesting direction. We suspect it is not difficult to extend our result to swish and GeLU as they are still very flat in the large negative range and that is mostly what we need. However there will certainly be more difficulty in the proof due to the nonlinearity of these functions."}]

Author rebuttal:

Meta-review: The paper explores the connection between downstream task performance and pretraining performance. The authors show the existence of an anchor vector that serves as a critical element to ensure the performance on downstream tasks.
Decision: Accept (poster)