Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning

1 Introduction . Natural language processing ( NLP ) has been revolutionized by large-scale pretrained language models such as BERT [ 4 ] and GPT [ 25 ] , which are adapted to a variety of downstream NLP tasks . Although a large body of empirical work seeks to understand the effectiveness of pretrained models [ 7 , 5 , 12 , 35 , 34 , 11 , 27 , 15 ] , theoretical understanding is scarce . Theoretically analyzing the relationship between the pretraining and downstream tasks is challenging because pretraining and downstream settings can greatly differ . The key starting point for our analysis is to link the pretraining and downstream settings through an underlying generative model of the data . We model the data distribution as a latent variable model and the downstream task as a function of the latent variables . Assuming that pretraining on a large corpus allows us to learn the generative model , the conditional token probabilities predicted by the pretrained model carry information about the hidden variables . In downstream adaptation , we aim to recover this information to solve the downstream task . Though full finetuning is the de facto empirical standard , analyzing it is challenging because it requires characterizing the weights of the pretrained model . In this paper , we focus on head tuning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and prompt tuning , which both freeze all pretrained parameters and allow us to treat the pretrained model as a black box . Head tuning [ 22 ] trains task-specific heads on top of the pretrained model outputs . Prompt tuning [ 31 , 19 , 9 , 21 ] optimizes a task-specific “ prompt ” that is concatenated to the model input . Studying prompt tuning is particularly interesting since it can match the performance of full finetuning with less computation time [ 19 , 9 , 21 ] . Our work contrasts with prior theoretical work [ 28 ] , which assumes that downstream labels are recoverable via a linear head applied to the conditional token probabilities , and analyze how errors in pretraining or model misspecification propagate downstream . We consider specific generative distributions for which we can prove these assumptions , showing that head and prompt tuning can recover the downstream labels . Our analysis considers two data-generating distributions with increasing realism . First , we consider data generated from a Hidden Markov Model ( HMM ) , where the downstream task is to learn a linear classifier on the posterior distribution over the hidden states ( Section 3 ) . We prove that , under strong non-degeneracy conditions on token emission probabilities , a linear head applied to a pretrained model G which outputs exact conditional token probabilities ( Gipxq “ P rXi |x´is ) can recover the downstream label ( Theorem 3.3 ) . Furthermore , we can prove better recovery guarantees with relaxed non-degeneracy assumptions ( Assumption 3.1 ) by using continuous prompt tuning ( Theorem 3.6 ) , reflecting the strong empirical performance of prompt tuning [ 19 , 9 , 21 ] . Intuitively , prompt tuning conditions the latent variables so that nonessential information for the downstream task can be ignored during the tuning phase , making task-essential information easier to recover . Second , we also strengthen our analysis by leveraging additional structure in the data . Motivated by long-range dependences in natural language , we analyze HMM variants with additional latent “ memory ” variables that can store long-term information more easily than vanilla HMMs ( Section 4 ) . Here , the downstream task is to learn a linear classifier on the posterior distribution of the memory variables . We show that , under weaker non-degeneracy conditions than the first setting , an attentionbased classification head can recover ground-truth downstream labels from pretrained model outputs ( Theorem 4.3 ) . Intuitively , our recovery guarantees improve because the classification head can focus on the persistent , task-essential information in the memory while ignoring other transient and nonessential aspects of the latent variables . As with the vanilla HMM , we analyze prompt tuning for relaxing the non-degeneracy conditions even further ( Theorem 4.6 ) . In summary , we relate the pretraining and downstream tasks by assuming that the downstream task is to learn a classifier on the posterior distributions of the latent variables defined by an underlying generative model of text . Our theoretical contributions are : 1 ) in this setting we analyze an HMM generative model show that simple classification heads can recover the true downstream labels under certain non-degeneracy assumptions , 2 ) we prove that soft prompt tuning can relax the non-degeneracy assumptions needed for downstream recovery making it easier to extract task-specific information , and 3 ) our recovery guarantees are stronger for memory-augmented HMMs in comparison to the vanilla HMM when tuning an attention-based classfication head . We empirically evaluate our theoretical results with language models pretrained on synthetically generated data from HMMs . We find that prompt tuning obtains good downstream performance when our non-degeneracy conditions are relaxed , whereas head tuning performs poorly . Furthermore , we show that head tuning obtains better downstream performance when data is generated from a memory-augmented HMM , compared to a vanilla HMM , as is predicted by our theory . 1.1 Related works . The black box nature of BERT and related models has inspired a variety of empirical works which seek to understand them . Probing papers study whether a pretrained model computes various types of structured information ( e.g. , syntactic [ 35 , 11 ] ) by evaluating the performance of simple classifiers , or probes , on the representations [ 7 , 12 , 34 , 27 , 15 ] . Other papers ablate various aspects of pretraining , such as changing the masking scheme [ 14 , 20 , 40 ] or permuting the word order [ 32 ] . In comparison , theoretical analysis of pretrained language models is limited . Besides [ 28 ] , which we discussed in Section 1 , Zhang and Hashimoto [ 40 ] analyze using a linear classifier to approximately recover the latent variable in a Gaussian graphical model with sparse dependencies between observed variables . However , their analysis and setting are focused towards understanding syntactic dependencies between tokens , whereas we directly model and analyze downstream performance . Prompt-based tuning [ 31 , 19 , 9 , 21 , 13 , 6 , 41 , 2 , 23 ] , which has improved empirical downstream performance for lightweight adaptation methods beyond head tuning to approach full finetuning , is an important focus of our theoretical analysis . Shin et al . [ 31 ] employ task-specific prompts that are optimized over the discrete token space . Schick and Schütze [ 29 , 30 ] reformulate natural language tasks as cloze-style phrases to enable few-shot learning . Subsequent methods [ 19 , 9 , 21 ] optimize “ soft ” prompts , or continuous embedding vectors . Lester et al . [ 19 ] employ soft prompts on pretrained large-scale T5 [ 26 ] models and show that as the model size increases , prompt tuning performance can eventually match finetuning . Hambardzumyan et al . [ 9 ] applies a variant of soft prompt tuning to MLM models . Li and Liang [ 21 ] propose prefix tuning , which prepends a trainable prefix embedding sequence to all layers of the transformer . More broadly , Lee et al . [ 18 ] analyze reconstruction-based self-supervised learning methods in a general setting and show that under certain conditional independence assumptions , predicting one observed variable from another allows recovery of the latent with a linear head . Other theoretical works analyzing self-supervised or constrastive learning include [ 1 , 10 , 36 , 38 , 37 ] , but they do not directly relate to our particular setting . 2 Formulations and notations . We analyze models pretrained on masked language modeling ( MLM ) objectives . Let X denote a finite vocabulary of input tokens , X ˚ the set of variable-length sequences of tokens , and X “ pX1 , . . . , XT q P X ˚ a random sequence of T tokens . Let ∆|X | denote the space of probability distributions over tokens . Pretraining and downstream task . Let Gpxq “ pG1pxq , G2pxq , . . .q denote the masked language model which predicts a probability vector for each timestep in the input x . Our theoretical abstraction is that Gi perfectly computes the distribution of Xi , the i-th token , conditioned on all other tokens : Gipxq “ P rXi|X´i “ x´is . Here P rXi |X´i “ x´is P ∆|X | is a probability vector . In particular , Gipxq does not depend on xi . The downstream task involves labeled examples px , F ‹pxqq P X ˚ˆY , where F ‹ : X ˚ Ñ Y provides ground-truth downstream labels and Y is a discrete set of labels for classification . Head and prompt tuning . Head tuning trains a classification head f on top of fixed model outputs , resulting in the classifier F pxq “ 1pfpGpxqq ě 0q . We expect f to be a simple function such as a linear or one layer attention model . We also analyze variants where f also takes the tokens x or embeddings of x as input , which provides additional information . Soft prompt tuning requires viewing the pretrained model G as a function of the token embeddings ; we refer to this model by G. Letting epxq “ epx1q , . . . , epxtq denote the token embeddings , we have Gpepxqq “ Gpxq . Soft prompt tuning concatenates a trainable prompt u so that the model output isGppu , epxqq . We consider simultaneously training the prompt parameter u and a classification head to fit the downstream task . Notations . Let ∆d denote the space of d-dimensional probability vectors . We work with discrete random variables V taking values in a finite set V . We use P rV s P ∆|V| to denote the distribution of V and P rU |V “ vs P R|U | the conditional distribution of U given V “ v. PrpV “ vq P r0 , 1s will denote the probability that V takes values v. We also let P rU “ u |V s P R|V| denote the vector with entries PrpU “ u |V “ vq . P rU |V s P R|U |ˆ|V| will describe the matrix with entries P rU |V su , v “ PrpU “ u |V “ vq . For a sequence v “ pv1 , . . . , vtq , we use the notation vi : j for i ď j to denote pvi , . . . , vjq , and v´i to denote pv1 : i´1 , vi ` 1 : tq . We let 1 denote the indicator function . For set V , we let V˚ “ V1YV2Y¨ ¨ ¨ denote variable-length sequences of elements of V . Let d denote elementwise product . Let 1d,0d denote the d-dimensional all-1 ’ s and all-0 ’ s vector . We omit the subscript if the dimension is clear from context . For two vectors a , b P Rd , we let a { b denote their element-wise division . We use supppaq to denote the set of indices where vector a is non-zero . 3 Analysis for Hidden Markov Models . Defining a relation between pretraining and downstream tasks is the foremost challenge for analysis . We propose to link the two via latent variable generative assumptions on the input distribution . We model the downstream task as a function of the posterior distribution of the latent variables . Towards a first result , this section studies the case where inputs are generated by HMMs ( see Figure 1 ( left ) ) , which have been well-studied in the context of language and speech processing ( see e.g . [ 24 , 17 , 3 ] ) . Data distribution . Let H denote the hidden state space of the HMM . We use H “ pH0 , H1 , . . . , HT q P H˚ to denote the sequence of hidden states . For all timesteps i ą 0 , the transition probabilities are time-invariant , i.e . P rHi |Hi´1s “ A for A P R|H|ˆ|H| . For each timestep i ě 1 , tokens Xi are emitted following some time-invariant probability : P rXi |His “ W for W P R|X |ˆ|H| . The joint probability of X , H is PrpX , H “ x , h |T “ tq “ PrpH0 “ h0q t ź i “ 1 PrpHi “ hi |Hi´1 “ hi´1qPrpXi “ xi |Hi “ hiq . Downstream tasks . We assume that H0 has the meaningful information for the downstream task , which is a binary classification task where the ground-truth labeling F ‹ is assumed to be a linear classifier on the posterior P rH0 |X1 : T “ xs : F ‹pxq “ 1pµJP rH0 |X1 : T “ xs ě 0q ( 3.1 ) for µ P R|H| . Our results are easily extended to the multiclass setting . We consider tuning a linear head for the downstream classifier , which formally computes 1pbJG1pxq ě 0q for b P R|X | . The following non-degeneracy condition is crucial for our recovery result in this setting . Assumption 3.1 ( Non-degeneracy , vanilla HMM ) . The token emission probability matrix W has linearly independent columns . We also require the following regularity conditions on H0 and the state transitions . Assumption 3.2 ( Regularity ) . The Markov chain H0 , H1 , . . . is ergodic , and P rH0s has full support . We show that if W has linearly independent columns , a linear head fits downstream labels . Theorem 3.3 . Assume that non-degeneracy ( Assumption 3.1 ) and regularity ( Assumption 3.2 ) hold . Then any downstream task F ‹pxq of the form ( 3.1 ) can be computed by a linear head onG applied to a shifted sequence . That is , there exists linear head weights b P R|X | such that for all x P supppP rXsq , F ‹pxq “ 1pbJG1px1q ě 0q where x1 “ p∅ , x1 : tq is the concatenation of a special token ∅ with x.1 The key for the proof is to leverage the following general statement about random variables U , V , Z such that U K V |Z , which decomposes the expression for P rU |V s. 1We note that G1px1q does not depend on x11 and therefore x11 can be any token . Proposition 3.4 . Let U , V , Z be random variables such that U K V |Z . Then for any v , P rU |V “ vs “ P rU |Zs ¨ P rZ |V “ vs . Thus , if P rU |Zs has a left inverse pP rU |Zsq : , then P rZ |V “ vs “ pP rU |Zsq : P rU |V “ vs. By the conditional independence structure of the HMM , Proposition 3.4 immediately implies G1px1q “ WP rH1|X2 : T ` 1 “ xs ùñ P rH1|X2 : T ` 1 “ xs “ W : G1px1q where W : is the left inverse for W , guaranteed to exist by Assumption 3.1 . This lets us recover P rH1|X2 : T ` 1 “ xs by applying a linear function to G1px1q . Additional linear functions will be sufficient to obtain µJP rH0|X1 : T “ xs from P rH1|X2 : T ` 1 “ xs . We provide the full proof in Section B . Proposition 3.4 is reminiscent of the arguments of [ 18 ] , which leverages the independence structure in the same way . Subsequent sections will require more complicated analyses and recovery procedures . A drawback of Theorem 3.3 is that it relies heavily on assuming W has full column rank , which implies the necessary condition that |H| ď |X | . Without this assumption , it is unclear how to recover P rH0 |X1 : T “ xs from Gpxq alone . However , in realistic settings we would expect |H| ą |X | , as increasing the size of the hidden state space improves language modeling capabilities of HMMs [ 3 ] .

Summary: [UPDATE: Thanks to the authors for engaging with the comments in the review.  I hope to see them fully addressed in the revision and look forward to seeing the next version of the paper!]  The paper investigates how pretrained representations might be expected to help prediction on a downstream task.  In some sense, they should "obviously" help: the downstream predictor can learn to exploit any mutual information between its target output and the pretrained representations.  That's the standard motivation for multitask learning.  However, the paper exhibits specific settings in which the target output can be *exactly* predicted from the pretrained representations, using only a *simple* probe such as a linear classifier.    The paper is primarily theoretical, although a few experiments are included by way of illustration.  It is not clear what the experiments are intended to verify.  To make the analysis possible, the pretrained representations in the paper are not underlying representations such as word embeddings.  Rather, they are identifiable given the pretraining data, which makes it clear what "successful pretraining" means.  For example, the representation of word x_i in a sentence is the conditional distribution of x_i given all other words in the sentence, as predicted by a pretrained cloze language model.  The analysis assumes (though the experiments do not check) that the predicted conditional distribution actually matches the true conditional distribution.  This should be the case in the limit of infinite training data, provided that the cloze language model has adequate capacity and can be trained to a global optimum of log-loss.  It is shown that if the vocabulary is large enough, or under other assumptions, then this representation of x_1 is enough to recover the posterior distributions over h_1 and h_0, which are assumed to be sufficient for the downstream task.  Caveat: In the analysis, the observed sentences are assumed to be generated by a latent-variable model.  One would then expect the downstream task target output to be predictable from the actual latent variables.  However, the paper assumes more strongly that the target is a function of the *posterior distribution* of the latent variables given the observed sentences.  In other words, the goal of the downstream task is not to recover what the speaker *actually* meant, but only to recover what a hearer who knows the true generating distribution *thinks* it meant (given only the sentence and no other context).  Two structures are considered for the latent-variable model and its relationship to the downstream task.  For the more complex version, the predictor is also more complex, requiring argmax attention over all positions i.  In both cases, the paper considers allowing the predictor to prepend an arbitrary word embedding x_1 to the input.  This soft prompt can serve to shift the posterior predictions of x_i so as to be more informative for the downstream task.