Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning

1 Introduction . Natural language processing ( NLP ) has been revolutionized by large-scale pretrained language models such as BERT [ 4 ] and GPT [ 25 ] , which are adapted to a variety of downstream NLP tasks . Although a large body of empirical work seeks to understand the effectiveness of pretrained models [ 7 , 5 , 12 , 35 , 34 , 11 , 27 , 15 ] , theoretical understanding is scarce . Theoretically analyzing the relationship between the pretraining and downstream tasks is challenging because pretraining and downstream settings can greatly differ . The key starting point for our analysis is to link the pretraining and downstream settings through an underlying generative model of the data . We model the data distribution as a latent variable model and the downstream task as a function of the latent variables . Assuming that pretraining on a large corpus allows us to learn the generative model , the conditional token probabilities predicted by the pretrained model carry information about the hidden variables . In downstream adaptation , we aim to recover this information to solve the downstream task . Though full finetuning is the de facto empirical standard , analyzing it is challenging because it requires characterizing the weights of the pretrained model . In this paper , we focus on head tuning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and prompt tuning , which both freeze all pretrained parameters and allow us to treat the pretrained model as a black box . Head tuning [ 22 ] trains task-specific heads on top of the pretrained model outputs . Prompt tuning [ 31 , 19 , 9 , 21 ] optimizes a task-specific “ prompt ” that is concatenated to the model input . Studying prompt tuning is particularly interesting since it can match the performance of full finetuning with less computation time [ 19 , 9 , 21 ] . Our work contrasts with prior theoretical work [ 28 ] , which assumes that downstream labels are recoverable via a linear head applied to the conditional token probabilities , and analyze how errors in pretraining or model misspecification propagate downstream . We consider specific generative distributions for which we can prove these assumptions , showing that head and prompt tuning can recover the downstream labels . Our analysis considers two data-generating distributions with increasing realism . First , we consider data generated from a Hidden Markov Model ( HMM ) , where the downstream task is to learn a linear classifier on the posterior distribution over the hidden states ( Section 3 ) . We prove that , under strong non-degeneracy conditions on token emission probabilities , a linear head applied to a pretrained model G which outputs exact conditional token probabilities ( Gipxq “ P rXi |x´is ) can recover the downstream label ( Theorem 3.3 ) . Furthermore , we can prove better recovery guarantees with relaxed non-degeneracy assumptions ( Assumption 3.1 ) by using continuous prompt tuning ( Theorem 3.6 ) , reflecting the strong empirical performance of prompt tuning [ 19 , 9 , 21 ] . Intuitively , prompt tuning conditions the latent variables so that nonessential information for the downstream task can be ignored during the tuning phase , making task-essential information easier to recover . Second , we also strengthen our analysis by leveraging additional structure in the data . Motivated by long-range dependences in natural language , we analyze HMM variants with additional latent “ memory ” variables that can store long-term information more easily than vanilla HMMs ( Section 4 ) . Here , the downstream task is to learn a linear classifier on the posterior distribution of the memory variables . We show that , under weaker non-degeneracy conditions than the first setting , an attentionbased classification head can recover ground-truth downstream labels from pretrained model outputs ( Theorem 4.3 ) . Intuitively , our recovery guarantees improve because the classification head can focus on the persistent , task-essential information in the memory while ignoring other transient and nonessential aspects of the latent variables . As with the vanilla HMM , we analyze prompt tuning for relaxing the non-degeneracy conditions even further ( Theorem 4.6 ) . In summary , we relate the pretraining and downstream tasks by assuming that the downstream task is to learn a classifier on the posterior distributions of the latent variables defined by an underlying generative model of text . Our theoretical contributions are : 1 ) in this setting we analyze an HMM generative model show that simple classification heads can recover the true downstream labels under certain non-degeneracy assumptions , 2 ) we prove that soft prompt tuning can relax the non-degeneracy assumptions needed for downstream recovery making it easier to extract task-specific information , and 3 ) our recovery guarantees are stronger for memory-augmented HMMs in comparison to the vanilla HMM when tuning an attention-based classfication head . We empirically evaluate our theoretical results with language models pretrained on synthetically generated data from HMMs . We find that prompt tuning obtains good downstream performance when our non-degeneracy conditions are relaxed , whereas head tuning performs poorly . Furthermore , we show that head tuning obtains better downstream performance when data is generated from a memory-augmented HMM , compared to a vanilla HMM , as is predicted by our theory . 1.1 Related works . The black box nature of BERT and related models has inspired a variety of empirical works which seek to understand them . Probing papers study whether a pretrained model computes various types of structured information ( e.g. , syntactic [ 35 , 11 ] ) by evaluating the performance of simple classifiers , or probes , on the representations [ 7 , 12 , 34 , 27 , 15 ] . Other papers ablate various aspects of pretraining , such as changing the masking scheme [ 14 , 20 , 40 ] or permuting the word order [ 32 ] . In comparison , theoretical analysis of pretrained language models is limited . Besides [ 28 ] , which we discussed in Section 1 , Zhang and Hashimoto [ 40 ] analyze using a linear classifier to approximately recover the latent variable in a Gaussian graphical model with sparse dependencies between observed variables . However , their analysis and setting are focused towards understanding syntactic dependencies between tokens , whereas we directly model and analyze downstream performance . Prompt-based tuning [ 31 , 19 , 9 , 21 , 13 , 6 , 41 , 2 , 23 ] , which has improved empirical downstream performance for lightweight adaptation methods beyond head tuning to approach full finetuning , is an important focus of our theoretical analysis . Shin et al . [ 31 ] employ task-specific prompts that are optimized over the discrete token space . Schick and Schütze [ 29 , 30 ] reformulate natural language tasks as cloze-style phrases to enable few-shot learning . Subsequent methods [ 19 , 9 , 21 ] optimize “ soft ” prompts , or continuous embedding vectors . Lester et al . [ 19 ] employ soft prompts on pretrained large-scale T5 [ 26 ] models and show that as the model size increases , prompt tuning performance can eventually match finetuning . Hambardzumyan et al . [ 9 ] applies a variant of soft prompt tuning to MLM models . Li and Liang [ 21 ] propose prefix tuning , which prepends a trainable prefix embedding sequence to all layers of the transformer . More broadly , Lee et al . [ 18 ] analyze reconstruction-based self-supervised learning methods in a general setting and show that under certain conditional independence assumptions , predicting one observed variable from another allows recovery of the latent with a linear head . Other theoretical works analyzing self-supervised or constrastive learning include [ 1 , 10 , 36 , 38 , 37 ] , but they do not directly relate to our particular setting . 2 Formulations and notations . We analyze models pretrained on masked language modeling ( MLM ) objectives . Let X denote a finite vocabulary of input tokens , X ˚ the set of variable-length sequences of tokens , and X “ pX1 , . . . , XT q P X ˚ a random sequence of T tokens . Let ∆|X | denote the space of probability distributions over tokens . Pretraining and downstream task . Let Gpxq “ pG1pxq , G2pxq , . . .q denote the masked language model which predicts a probability vector for each timestep in the input x . Our theoretical abstraction is that Gi perfectly computes the distribution of Xi , the i-th token , conditioned on all other tokens : Gipxq “ P rXi|X´i “ x´is . Here P rXi |X´i “ x´is P ∆|X | is a probability vector . In particular , Gipxq does not depend on xi . The downstream task involves labeled examples px , F ‹pxqq P X ˚ˆY , where F ‹ : X ˚ Ñ Y provides ground-truth downstream labels and Y is a discrete set of labels for classification . Head and prompt tuning . Head tuning trains a classification head f on top of fixed model outputs , resulting in the classifier F pxq “ 1pfpGpxqq ě 0q . We expect f to be a simple function such as a linear or one layer attention model . We also analyze variants where f also takes the tokens x or embeddings of x as input , which provides additional information . Soft prompt tuning requires viewing the pretrained model G as a function of the token embeddings ; we refer to this model by G. Letting epxq “ epx1q , . . . , epxtq denote the token embeddings , we have Gpepxqq “ Gpxq . Soft prompt tuning concatenates a trainable prompt u so that the model output isGppu , epxqq . We consider simultaneously training the prompt parameter u and a classification head to fit the downstream task . Notations . Let ∆d denote the space of d-dimensional probability vectors . We work with discrete random variables V taking values in a finite set V . We use P rV s P ∆|V| to denote the distribution of V and P rU |V “ vs P R|U | the conditional distribution of U given V “ v. PrpV “ vq P r0 , 1s will denote the probability that V takes values v. We also let P rU “ u |V s P R|V| denote the vector with entries PrpU “ u |V “ vq . P rU |V s P R|U |ˆ|V| will describe the matrix with entries P rU |V su , v “ PrpU “ u |V “ vq . For a sequence v “ pv1 , . . . , vtq , we use the notation vi : j for i ď j to denote pvi , . . . , vjq , and v´i to denote pv1 : i´1 , vi ` 1 : tq . We let 1 denote the indicator function . For set V , we let V˚ “ V1YV2Y¨ ¨ ¨ denote variable-length sequences of elements of V . Let d denote elementwise product . Let 1d,0d denote the d-dimensional all-1 ’ s and all-0 ’ s vector . We omit the subscript if the dimension is clear from context . For two vectors a , b P Rd , we let a { b denote their element-wise division . We use supppaq to denote the set of indices where vector a is non-zero . 3 Analysis for Hidden Markov Models . Defining a relation between pretraining and downstream tasks is the foremost challenge for analysis . We propose to link the two via latent variable generative assumptions on the input distribution . We model the downstream task as a function of the posterior distribution of the latent variables . Towards a first result , this section studies the case where inputs are generated by HMMs ( see Figure 1 ( left ) ) , which have been well-studied in the context of language and speech processing ( see e.g . [ 24 , 17 , 3 ] ) . Data distribution . Let H denote the hidden state space of the HMM . We use H “ pH0 , H1 , . . . , HT q P H˚ to denote the sequence of hidden states . For all timesteps i ą 0 , the transition probabilities are time-invariant , i.e . P rHi |Hi´1s “ A for A P R|H|ˆ|H| . For each timestep i ě 1 , tokens Xi are emitted following some time-invariant probability : P rXi |His “ W for W P R|X |ˆ|H| . The joint probability of X , H is PrpX , H “ x , h |T “ tq “ PrpH0 “ h0q t ź i “ 1 PrpHi “ hi |Hi´1 “ hi´1qPrpXi “ xi |Hi “ hiq . Downstream tasks . We assume that H0 has the meaningful information for the downstream task , which is a binary classification task where the ground-truth labeling F ‹ is assumed to be a linear classifier on the posterior P rH0 |X1 : T “ xs : F ‹pxq “ 1pµJP rH0 |X1 : T “ xs ě 0q ( 3.1 ) for µ P R|H| . Our results are easily extended to the multiclass setting . We consider tuning a linear head for the downstream classifier , which formally computes 1pbJG1pxq ě 0q for b P R|X | . The following non-degeneracy condition is crucial for our recovery result in this setting . Assumption 3.1 ( Non-degeneracy , vanilla HMM ) . The token emission probability matrix W has linearly independent columns . We also require the following regularity conditions on H0 and the state transitions . Assumption 3.2 ( Regularity ) . The Markov chain H0 , H1 , . . . is ergodic , and P rH0s has full support . We show that if W has linearly independent columns , a linear head fits downstream labels . Theorem 3.3 . Assume that non-degeneracy ( Assumption 3.1 ) and regularity ( Assumption 3.2 ) hold . Then any downstream task F ‹pxq of the form ( 3.1 ) can be computed by a linear head onG applied to a shifted sequence . That is , there exists linear head weights b P R|X | such that for all x P supppP rXsq , F ‹pxq “ 1pbJG1px1q ě 0q where x1 “ p∅ , x1 : tq is the concatenation of a special token ∅ with x.1 The key for the proof is to leverage the following general statement about random variables U , V , Z such that U K V |Z , which decomposes the expression for P rU |V s. 1We note that G1px1q does not depend on x11 and therefore x11 can be any token . Proposition 3.4 . Let U , V , Z be random variables such that U K V |Z . Then for any v , P rU |V “ vs “ P rU |Zs ¨ P rZ |V “ vs . Thus , if P rU |Zs has a left inverse pP rU |Zsq : , then P rZ |V “ vs “ pP rU |Zsq : P rU |V “ vs. By the conditional independence structure of the HMM , Proposition 3.4 immediately implies G1px1q “ WP rH1|X2 : T ` 1 “ xs ùñ P rH1|X2 : T ` 1 “ xs “ W : G1px1q where W : is the left inverse for W , guaranteed to exist by Assumption 3.1 . This lets us recover P rH1|X2 : T ` 1 “ xs by applying a linear function to G1px1q . Additional linear functions will be sufficient to obtain µJP rH0|X1 : T “ xs from P rH1|X2 : T ` 1 “ xs . We provide the full proof in Section B . Proposition 3.4 is reminiscent of the arguments of [ 18 ] , which leverages the independence structure in the same way . Subsequent sections will require more complicated analyses and recovery procedures . A drawback of Theorem 3.3 is that it relies heavily on assuming W has full column rank , which implies the necessary condition that |H| ď |X | . Without this assumption , it is unclear how to recover P rH0 |X1 : T “ xs from Gpxq alone . However , in realistic settings we would expect |H| ą |X | , as increasing the size of the hidden state space improves language modeling capabilities of HMMs [ 3 ] .

Summary: The paper studies the question why head and prompt tuning work for pre-trained LM. Analyzing complicated neural networks is a daunting task. The paper offers an analysis framework based on the Hidden Markov Model (HMM) and memory augmented HMM. Under this framework, the paper makes a connection with masked LM (BERT) that at time step i, the output of BERT(`[mask]_i`) represents the marginal P(X_i) of the HMM. The framework further allows for analysis of a binary classification task using head tuning or prompt tuning via the posterior distribution of a HMM. To this end, the paper runs a simulation study with data generated from HMM and memory augmented HMM.