When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
Authors: Aleksandar Petrov
Philip Torr
Adel Bibi
Venue: ICLR 2024 poster (2024)
Keywords: prompt
prefix
LLM
fine-tuning
theory
Primary area: general machine learning (i.e., none of the above)
TL;DR: We show that prompting and prefix tuning have limitations that make them less powerful than full fine-tuning and explain how despite these limitations they can work well in practice for some tasks.

Abstract: Context-based fine-tuning methods, including prompting, in-context learning, soft prompting (also known as prompt tuning), and prefix-tuning, have gained popularity due to their ability to often match the performance of full fine-tuning with a fraction of the parameters. Despite their empirical successes, there is little theoretical understanding of how these techniques influence the internal computation of the model and their expressiveness limitations. We show that despite the continuous embedding space being more expressive than the discrete token space, soft-prompting and prefix-tuning are potentially less expressive than full fine-tuning, even with the same number of learnable parameters. Concretely, context-based fine-tuning cannot change the relative attention pattern over the content and can only bias the outputs of an attention layer in a fixed direction. This suggests that while techniques like prompting, in-context learning, soft prompting, and prefix-tuning can effectively elicit skills present in the pretrained model, they may not be able to learn novel tasks that require new attention patterns.

Published as a conference paper at ICLR 2024
WHEN Do PROMPTING AND PREFIX-TUNING WORK? A THEORY OF CAPABILITIES AND LIMITATIONS
Aleksandar Petrov, Philip H.S. Torr & Adel Bibi
Department of Engineering Science University of Oxford
Oxford, United Kingdom {aleksandar, petrov, pheetrov,phiippe torr, adel adel.bibijjeng ox ack .uk
ABSTRACT
Context-based fine-tuning methods, including prompting, in-context learning, soft prompting (also known as prompt tuning), and prefix-tuning, have gained popu- larity due to their ability to often match the performance of full fine-tuning with a fraction of the parameters. Despite their empirical successes, there little theoret- ical understanding of how these techniques influence the internal computation of the model and their expressiveness limitations. We show that despite the contin- uous embedding space being more expressive than the discrete token space, soft- prompting and prefix-tuning are potentially less expressive than full fine-tuning, even with the same number of learnable parameters. Concretely, context- context-based fine-tuning cannot change the relative attention pattern over the content and can only bias the outputs of an attention layer in a fixed direction. This suggests that while techniques like prompting, in-context learning, soft prompting, and prefix- tuning can effectively elicit skills present in the pretrained model, they may not be able to learn novel tasks that require new attention patterns.
INTRODUCTION
Language model advances are largely driven by larger models and more training data (Kaplan al., 2020; Rae al., 2021). Training cutting-edge models is out of reach for most academic researchers, small enterprises, and individuals, and has become common to instead fine-tune open-source pre- trained models (Devlin et al., 2019; Min et al., 2021). Yet, due to escalating computational demands, fine-tuning of the larger models has become prohibitively expensive (Lialin et al., 2023).
As a result, there is an acute need for more efficient fine-tuning methods, either by sparsely modi- fying the parameters of the model or modifying its input context. Examples of the first type include adapter modules which introduce a few trainable layers to modify the behaviour of the frozen pre- trained network (Rebuffi al., 2017; Houlsby al., 2019; Hu al., 2023). One can also low-rank updates, which also results in a reduced number of trainable parameters (Hu et al., 2021).
Context-based fine-tuning has been motivated by the success of few-shot and zero-shot learning (Wei et al., 2021; Kojima et al., 2022). The most popular context-based approach is prompting, where generation is conditioned on either human-crafted or automatically optimized tokens (Shin et al., 2020; Liu et al., 2023). In-context learning prompting via providing input-label examples another widely used technique (Brown al., 2020). Given the challenges of discrete optimization over tokens, there is a growing interest in methods that optimize real-valued embeddings (Lester al., 2021). It is widely believed that these soft prompts offer greater expressiveness due to the expansive nature of continuous space. Furthermore, beyond only optimizing input embeddings, one can optimize the inputs of every attention layer (Li and Liang, 2021). This technique, prefix-tuning, has proven to be very successful and competitive full fine-tuning (Liu et al., 2022).
While context-based fine-tuning approaches have witnessed impressive empirical successes and widespread adoption, we have little theoretical understanding of how they work. In this work, analyse the influence of prompts and prefixes on the internal computations of pretrained model and delineate their limitations. Specifically, we address the following questions:

---

Published as a conference paper at ICLR 2024
1. Soft prompting and prefix-tuning motivated by the embedding space being larger than the token space. However, can a transformer utilize the additional capacity? We show that with a careful choice of transformer weights, controlling single embedding can generate any of the VN completions of N tokens, while controlling a token can produce only V completions, with V being the vocabulary size. Thus, transformer can indeed exploit the embedding space.
2. Since prefix-tuning is more expressive than prompting, is it as expressive as full fine-tuning? Despite the expressiveness of continuous space, prefix-tuning has structural limitations. A prefix cannot change the relative attention over the content tokens and can only bias the output of the attention block in a constant direction. In contrast, full fine-tuning can learn new attention patterns and arbitrarily modify attention block outputs, making it strictly more powerful.
3. If context- fine-tuning methods suffer from such structural limitations, how come they have high empirical performance? We show that the prefix induced bias can steer the model towards a pretraining task. Prefix-tuning can also combine skills picked up during pretraining to solve some new tasks similar to pretraining tasks. However, it may not learn a completely new task. This is not simply because of the small number of learnable parameters: fine-tuning the same number of parameters can be sufficient to learn the novel task. Hence, context-based fine- tuning can elicit combine pretrained model skills but cannot learn completely new behaviors.
2 BACKGROUND
2.1 THE TRANSFORMER ARCHITECTURE
We outline a simplified decoder-only transformer architecture (Vaswani et al., 2017). Assume that the model has vocabulary size V (also referred to number of tokens). The input is sequence (x1...., Xpp), (,, V}, Vi. Each token is mapped to a de-dimensional vector that is the x,- th column of an embedding matrix EERd,VV XV The attention mechanism is position-invariant, so typically position encodings are added. For model with maximum input length N (context size), we use a one-hot position encoding e (i) concatenated with the embedding. Therefore, the embedding for the i-th position provided to the first attention block would be x; e
A transformer consists of alternating attention blocks which operate across the whole sequence and Multi-Layer Perceptrons (MLPs) that operate each individual element. Each attention block consists of H heads. Each head h is parameterized by query, key, and value matrices W W e W/ IR xda The attention matrix A" A "ERRPP for head then has elements
exp (1/M((W)))) ((okW/)))((()))))))
4
ΣP= exp (Z))E(W (
where p S. N is the current length of the input and T 0 is inverse temperature parameter. Equation (1) is the softmax function, hence with high enough T, will result in approximately one-hot encoding of the maximum j. The output of the attention block A with H heads is then (f1,..., (p)). where each position is the sum of the attention-weightee values across all heads:
A((W)..... W (W ...... W (Wi,
W [[ xp) = (l,,...,4p)
t, t D - A Wo
(2)
A transformer then applies an MLP to each output of an attention block before passing them to next attention block. We will consider linear layers L,M, b)x= and ReLU-activated linear layers L M, Re)):RRLU(MM++++
When we compose attention blocks and linear or softmax layers, we will implicitly assume that the linear layer is applied to all positions of the sequence. Furthermore, we will use the then operator ; for left-to-right function composition. Therefore, a transformer model predicting confidences over the vocabulary can, for example, be represented as:
((EV)))
E
E.,, LeN (p)
) (3)
( Up) (1 C1,1 L1,2 A2 ; L2,1 ; L2,2 ; softmax)
(N)
For the first block, di must he de + N but may he different for the deeper blocks. A causal model has Au 0 for j > i. This does affect our results we will skip the masking step.
2

---

Published as a conference paper at ICLR 2024
where the output dimension of the last layer has to be V. The next token for deterministic trans- former is selected to be the last element's 's largest logit: Xp+1 = arg maX yp.u. Given an input (x)...., xp), the model then autoregressively extends this sequence one token at time, fol- lowing Equation (3) either until the sequence reaches a length N or until a special termination token.
A transformer has separation between the system prompt S, user provided input X and the autoregressively response Y. Thus, a sequence conditional on user input is denoted as (S1, Sngr X1, my Xnx X1,,, Yny) and one without user input as (S, Sng Yay).
2.2 CONTEXT-BASED FINE-TUNING OF A PRETRAINED MODEL
We now define prompting, soft prompting and prefix-tuning with the previously introduced notation.
Prompting. The most frequently used content-basee based fine-tuning approach is prompting: prefix- ing the input (X,, Xnx) with a token sequence E {1,... V) to guide the model response: (Si...., Sng XI,.,, Xn This is how most people interact with language models such ChatGPT.
Soft prompting. Soft prompting replaces the embeddings of the system input E,s, with learned vectors 8, E Rd called virtual tokens (Hambardzumyan et al., 2021; Lester et al., 2021; Qin and Eisner, 2021). Hence, the input in Equation (3) is modified to be:
$1 (lexin))
..... [
E (ns))) LEv(ns + n)
8ns
]
Exxax 1. LeN(ns + )
(4)
with S chosen to maximize the likelihood of target response Y=( Kny i.e., arg max tng ERd. ا log yostnx where ynstaxts are autoregressively generated.
Prefix-tuning. Prefix-tuning applies soft prompting across the depth of the model (Li and Liang, 2021; Liu et al., 2022). The first ns positions for all attention blocks are learnable parameters, replacing the input (ai...., nx) for layer with (si...., 8 -I...., Tn where all 8f constitute the prefix. Hence, prefix- tuning Prefix-tuning has been successful at fine-tuning models (Vu et al., 2022; Wu and Shi, 2022; Choi and Lee, 2023; Ouyang et al., 2023; Bai et al., 2023), leading to calls for language models provided as service (La Malfa et al., 2023) to allow providing prefixes instead of prompts (Sun al., 2022).
be formulated arg max ....... ( 11 Dj=1 log Ung+nx
Any token-based prompt (S,,..., has a corresponding soft prompt (s;= E.s,)) but the reverse does hold. Similarly, every soft prompt ($1, 8ns) ) can be represented the deeper prefixes to be the values that the model would compute at these positions (s/-
prefix by setting
... ( (1)] ..... [s] (T) ))) The reverse also does not hold: there are prefixes that cannot be represented as a soft prompt. A hierarchy emerges: prompting < soft prompting < prefix-tuming, with prefix-tuning the most powerful of the three. Hence, we focus on examining its performance relative to full fine-tuning but our findings also apply to prompting and soft prompting.
SOFT PROMPTING HAS MORE CAPACITY THAN PROMPTING
The success of soft prompting (and prefix-tuning is commonly attributed to the larger capacity of the continuous embeddings compared the finite tokens. Yet, increased capacity beneficial only if the model can utilize it. We show this is indeed the case by constructing transformer generating exponentially more completions by varying a single virtual token than by varying a hard token.
Consider unconditional generation (representing a function with no inputs) with single system token: (Y,, (f)==((()==ff For deterministic autoregressive function, there are a total V functions in this family, hence the upper bound on the number of outputs of length N that one can generate by varying the first token S1 is V: the first token fully determines the rest of the sequence. Generally, if one varies the first Ns tokens, there are at most VoN unique outputs. What if instead of the token S we vary a single virtual token $1 (X,,.., Y ) This family of functions indexed by real vector and hence infinite: in principle, one could generate all V possible
3

---

Published as a conference paper at ICLR 2024
output sequences by only controlling $1.² Still, a transformer may not be able to represent function that achieves that in practice, i.c., it is not obvious if there is a surjective map from (fe, $1 E R*) to {1,..., VJA. We show that, in fact, there is transformer f for which such surjective map exists:
Theorem 1 (Exponential unconditional generation capacity of a single virtual token). For any V, N>0, there exists a transformer with vocabulary size V, context size N, embedding size de=N, one attention layer with two heads and three-layer MLP such that it generates any token sequence [ V when conditioned on the single virtual token $1 ((=1-1//V,... (Fx (xx-))/v).
However, conditional generation is more interesting: given a user input (X1, Xnx ). we want to generate a target response provides one token and the model generates one token in response (Y==f($ X1 f= (X))), we cannot control response of the model to any user input with the system token. There are VV maps from X to Yı, but S can take on only V values: |[s, Si E 1, ...,, VH = < VV. Hence, tokens cannot be used to specify single virtual token can specify any of the VV maps, i.e., there exists a transformer fa, (X2) for which there is a surjective map from {fe, 81 E (Z.) to {1, (1,, Vlll....)
Even in the simple case of one system token, the user
arbitrary map from user input to model output. However,
Theorem (Conditional generation capacity for a single virtual token (nx=ny-1)), For any V>0, there exists transformer with vocabulary size V, context size N -2, embedding size d, -V. V one at- tention layer with two heads and a three-layer MLP that reproduces any map ...; [1-+[1,.... V] may
from user input token to a model response token when conditioned on single virtual token ,..,
81 =mm(1)/v ",, mV(V//v). That is, by selecting 81 we control the model response to any user input.
Theorem builds on Theorem I by showing that soft prompting is also more expressive for govern- ing the conditional behavior of a transformer model. This also holds for longer responses ny > by increasing the length of the soft prompt, or longer user imputs n 1, by increasing the depth of the model. We provide proofs in Appendix A, as well as working Python implementations.
This section showed that soft prompting, and by implication, prefix-tuning, possess greater expres- siveness than prompting. As we can fully determine the map from user input to model response using virtual tokens, our findings may appear to suggest that soft prompting is as powerful as full fine-tuning. However, this is not at all the case. There are structural constraints on the capabilities of soft prompting and prefix-tuning; they cannot facilitate the learning of an entirely new task. The following section elucidates this discrepancy and reconciles these seemingly contradictory results.
4 PREFIX-TUNING CAN ONLY BIAS THE OUTPUT OF AN ATTENTION HEAD
We just saw that soft prompting and prefix-tuning can fully control the conditional behavior of a transformer. However, that assumed specific design for the network weights. Given a fixed pretrained model, as opposed to manually crafted one, can prefix-tuning be considered equally powerful to full fine-tuning? In this section, we show that, for an arbitrary pretrained model, a prefix S cannot change the relative attention over the content X, Y and can only bias the attention block outputs in a subspace of rank n, the prefix length, making it less powerful than full fine-tuning.
While full fine-tuning can alter the attention pattern of an attention head, prefix-tuning cannot. Recall the attention Aij position i gives to position j for a trained transformer (Equation (1)):
exp (T/VE x. W WK.3,)
exp (T/VT a.. Hx,)
Aij
(5)
D exp
(T/VR x W WK
E exp (T/Vi x. Ha,)
where W Wx-H. H. Full fine-tuning can enact arbitrary changes to We and WK and hence, assum- ing the input does not change (e.g., at the first attention layer), we get the following attention:
cxp (T/VE x Ha, + T/VE a. AHæ,
n
D exp (T/VER x. HE, T/VT a AHæ.)
FFor example, LLaMA-7B 7B (Touvron et al., 2023) has 24 426 unique completions when prompted with each of its 32 000 tokens and we found a non-exhaustive set of 46 812 unique 10-token-long sequences by controlling the first virtual token. Hence, in practice, one can generate more outputs by soft prompting than by prompting.
4

---

Published as a conference paper at ICLR 2024
where the changes to Wq and WK are folded into AH. It is clear that by varying AH full fine- tuning can change the attention patterns arbitrarily. However, let us see how is attention affected by the presence of prefix. For now, assume we have prefix of length one (81) at position O.
exp(X/VE 2. Hai)
exp exp(2/VE a Ha,)
A
A
for j21. exp Je H -
VT Hai)
Σ -
exp
Hz.)
Hoy
E
V
The numerator of P) is the same as in Equation (5), i.e., the prefix does not affect it. It only adds the term exp(T/vī 2 H to the denominator. Therefore, the attention position gives to the content positions j21 is simply scaled down by the attention now gives to the prefix. If tomato attends the most to salad in a particular context, no prefix can change that. This becomes evident by rewriting A as the attention of the pretrained model scaled by the attention "stolen" by the prefix:
A ,,
A:j(1 A
Hence, prefix-tuning cannot affect the relative attention patterns across the content, it will only scale them down. In other words, one cannot modify what an attention head attends to via prefix-tuning."
Prefix-tuning only adds a bias to the attention block output. Let us see how this attention scaling down affects the output of the attention block. Following Equation (2), the output at position i for the pretrained (t,), the fully fine-tuned (t!!) and the prefix-tuned (tf)) models are as follows:
4; = == Wvzzj:
t = Σ EE=
AB (Wv AW,
P WvsI+ Ai1(1 W Wvst -1- (7)
t.t = 4% Wvs Σ Wra, @ A
W
j=1
Hence, prefix-tuning only biases the attention block value at each position i towards the constant vector Wys1, 81, which is independent of the content (Z1, Tp). Lee., the prefix-tuned activation is a linear combination of the pretrained activation and the constant vector SVs1. The content only affects the scale A% of the bias via the amount of attention on the prefix. In contrast, in full fine- tuning AWQ. AWK K and AWr allow for content-dependent change of the attention and value computation. These results hold for suffix-tuning (placing the prefix after the input) but not for suffix soft-prompting. We validate that this indeed is the case when prefix-tuning real-world transformers. In Figures 5 and 6, we show that prefix applied to LLaMA's first layer does not change the relative attention distribution over the content positions and results in a bias with a constant direction.
Longer prefixes define larger subspaces for the bias but are not fully utilized in practice. In the case of a longer prefix (8)...., Sng), the bias vector is in a subspace of dimensionality ngt t.' = ンド= A.s,, Wys, + (1 > A,,, )d, where i goes over the content and j over the prefix posi- tions. Larger prefixes thus have larger subspace to modify the attention block output. The specific direction is determined by the relative distribution of attention across the prefix positions. However, when examine the distribution of attention across the prefix positions for various inputs in Appendix B, it appears that the prefixes do not span this subspace. Regardless of the input, the attention A.,., over the prefix positions remains nearly constant. Thus, prefix-tuning does not seem to make full use of the space that the vectors Wvs8 span. We hypothesise that this is due to the two competing optimization goals for the vectors 8y: the same time they need to "grab attention" when interacting with WK and determine the bias direction when multiplied with Wv.
So, is prefix-funing equivalent to full fine-tuning or is it less powerful than full fine-tuning? In Section 3, we showed that prefix- tuning, in principle, has a large capacity to influence the behavior of the model. But then, in this section, we showed that has some severe limitations, including not being able to affect the attention pattern and only biasing the attention layer activations. These two results seem to be contradicting one another, so how do we reconeile them?
The constructions for the results in Section 3 (described in Appendix A) are simply an algorithm that extracts the completion from a lookup table encoded in the virtual tokens. The attention patterns
"iikhosherstov et al. (2021) show that fixed attention head can approximate any sparse attention pattern. However, they require control over all the input embeddings while we can only control the prefix ones. He et al. (2021) show a similar analysis but do not study the expressiveness of prefix-tuning.
5

---

Published as a conference paper at ICLR 2024
Pretrained model on ascending
Full fine-tuning on descending
Prefix-tuning on descending
pramitdd rodi bus
Selttie cas diaage lepand wellemeeent
Pudli-ezingg the hand, carnee charg lorkeyy and oderee mooes theateskin whin petien cos only
waing rosooee regerie Hewin
Usey
teragge otter firs
muker
krouditatteann tovd bod as cellegrrrnnen
Tama, amtratee Thara the
port sould.
Model to peppeos
Sanaldaareing
Nacuunal
360
3333333
Figure I: Attention patterns a small transformer pretrained on sorting in ascending order. The model given the prefix S and user input X and generates Y autoregressively. We have highlighted the attention when the first response Y, is being generated. Full fine-tuning sorts in descending order but prefix-tuning cannot as it cannot update the leamned attention. Note how the relative attention of X to X in the left and right plots exactly the same: the prefix cannot change attention pattern the same inputs. The relative attention of X to X in the center plot is very different because full fine-tuning can arbitrarily change Wo and Wx.
Head 1
Head 2
Head
Head 4
the
Loge systh
epos norlo
shull irppn
Figure 2: Model pretrained on the four tasks. The four attention heads specialize in the skills necessary to solve these tasks: look at the elements in order, look first at the smallest elements or first at the largest elements.
are simply extracting the current position embedding and the virtual token and hence the attention does not depend the actual content in the tokens. There is need to learn new attention pattern to learn a different map from input to output. 6 Furthermore, the virtual token designates the map precisely by acting bias. Therefore, the observations in these two sections do not contradict one another. Soft prompting and prefix-tuning can be on par with full fine-tuning but only in very limited circumstances: when all the knowledge is represented in the virtual token lookup table and the model simply extracts the relevant entry. Transformers do not behave like this in practice. Models are typically trained with token inputs rather than virtual tokens. Moreover, if we had a lookup table of the responses to each input we would not need learning algorithm in the first place.
Therefore, the limitations from this section hold for real-world pretrained transformers. Then how come prefix-tuning has been reported to achieve high accuracy and often to be competitive to full fine-tuning? The next section aims to explain when and why prefix-tuning can work practice.
5 THE BIAS CAN ELICIT SKILLS FROM THE PRETRAINED MODEL
Pretraining potentially exposes a model to different types of completions for the same token quence. For a string like I didn't enjoy the movie, the model may have seen completions such as I found the acting to be sub par, This is negative sentiment or Je n ai pas aimé le film. Hence, pretrained model could do text completion, sentiment analysis, or translation. Still, the input does not fully determine the desired completion type and the model generate any one of them. Hence, following our results from Section 4, we hypothesise that prefix-tuning cannot gain new knowledge but can bring the surface latent knowledge present the pretrained model.? We test this hypoth-
'In follow-up work (Petrov al., 2024), utilize this observation to show that, in fact, there exist pretrained weights for which a transformer can be a universal approximator for sequence-to tooeenee----eueeeee functions when prefixed. This is not in contradiction with the present results as these transformers can approximate any function without having to modify their attention mcchanism.
A similar hypothesis has also been proposed by Reynolds and McDonell (2021) for fine-tuning in general.

---

Published as a conference paper at ICLR 2024
esis by constructing small transformers trained model (Karpathy, 2020) to show that prefix- ttnniig struggles to learn a new task that full fine-tuning can. Then, that prefix-tuning can easily elicit latent skill from pretraining. Finally, show how it can even learn some new tasks, provided they can be solved by combining pretraining skills.
or few tasks. We minimal transformer
Prefix-tuning may not learn a new task requiring a different attention pattern. To check if prefix-tuning can learn a new task, we train a I-layer, 1-head trans- former to sort numbers into ascending order and then fine-tune it sort in descending order. During training, the model sees random sequences of 10 digits from O to followed by their ascending sorted order. The pre- trained accuracy (fully matching the sorted sequence) is 91%. Full fine-tuning on the descending task leads to 85% test accuracy, hence full fine-tuning successfully learns the new task. However, prefix-tuning with a prefix size ng=1 results in 0% accuracy, hence prefix-tuning fails to learn the new task at all.
Table I: A transformer pretrained on sorting in ascending order cannot be prefix-tuned to sort in descending order. 10 random seeds.
Ascending Descending
Pretrain on asc.
91+55%
0+-0%
Full fine-tune on desc.
0+-0%
85+55%
Prefix-tune on desc.
0.10%
0100%
The attention patterss in Figure show why this is the case: the pretrained model learns to attend first to the smallest numbers and then to the larger ones. When fully fine-tuned, the attention patterns reversed: they now first attend to the largest values. However, following Section 4, prefix-tuning cannot change the attention pattern over the input sequence and will still attend to the smallest values. Hence, prefix-tuning may indeed struggle to learn a new task requiring new attention patterns.
Prefix-tuning can elicit skill from the pre- trained model. The second part of our hy- pothesis was that prefix-tuning can elicit latent skills in the pretrained model. To test that, we pretrain a 1-layer, 4-head model with solutions sorted in ascending (/²) or descending ( (%) or- der, or adding one (+1) or two (+2) to each el- ement of the input sequence. Each solution is Prefix-tune on +1 0 0%
Table 2: A transformer pretrained on several tasks can be prefix-tumed for one of them. 10 random seeds.
Accuracy on:
+1
+2
Pretrained
25=13% 12%
24
11%
22:77%
Prefix-tune tune-ttueon on x 95 2%
0 0%
0 0%
0+:0%
Prefix-tune tune on
0 0%
90 3%
1 1%
1+1
1 3%
95 6%
0+11%
shown with 25% probability. The model has no indication of what the task is, hence, it as- signs equal probability all tasks, as shown in the first row in Table 2. Full fine-tuning for each task
Prefix-tune on +2 0 0%
0+ 0%
1 2% 98%
naturally results in high accuracy. However, prefix-tuning (n1-1) can also reach accuracy above 90% for all tasks. Compared the previous case, prefix-tuning is more successful here because the pretrained model contains the attention mechanisms for solving the four tasks, as shown in Figure 2.
If all prefix does is bias the attention layer activations, how can steer the model to collapse its distribution onto one task? This is likely due to the attention block solving all tasks in parallel and placing their solutions in different subspaces of the residual stream (intermediate representation, Elhage et al., 2021). As the MLP needs to select one solution to generate, a further indicator on the selected task (or lack of selection thereof) should also be represented. The bias induced by the prefix then acts on this "selection subspace" to nudge the MLP to select the desired solution.
This can be clearly seen from the activations of the attention layer at the last input position (Xnx)) ): the position where the task selection happens as the first output element fully describes the task. Figure 4 shows plots of randomly selected dimensions of the residual stream with and without a prefix. The attention block activations of the pretrained model (without prefix) show no correlation with the output it is about to generate, demonstrating that the choice of completion is indeed not determined by the attention block. However, the prefix-tuned activations for the same inputs clustered as a result of the prefix-induced bias. This indicates that the bias induced by the prefix may act as "task selector"' of the subspace of the residual stream specializing in the desired task.
Prefix-tuning can combine knowledge from pretraining tasks to solve new tasks. Prefix-tuning cliciting one type of completion learned in pretraining starts to explain its practical utility. Still, prefix-tuning seems to be successful also at tasks that the pretrained model has not seen. As we showed above, a model trained to sort in one order cannot prefix- tuned to sort in the other. Then how is it possible for prefix-tuning to learn a new task? We posit that this can happen, as long as the "skill" required to solve the new task is a combination of "skills" the pretrained model has seen.
7

---

Published as a conference paper at ICLR 2024
We test this by pretraining a 40-layer 4-head model with the same four tasks. We prefix- tune (m)=12) for two new tasks: increment- ing the ascending sorted sequence (/²+)) and double histogram (mapping each element to the number of elements with the same value, e.g., I,0.0,1-+1,,2,1,, HD). The pretrained model has not seen either task. Prefix-tuning results 93% accuracy for /+1 which is a combination of the / and +1 pretraining tasks and just I 1% for the task which re- quires different skills: finding other instances of the same token and counting. H is not a hard task: it requires 2 layers and 2 heads to be solved exactly (Weiss et al., 2021). Therefore, prefix-tuning is can indeed combine skills that the model has learned in order to solve a novel task but may not learn a completely new task requiring new skills.
Table 3: Prefix tuning can learn a new task requiring only pretraining skills (/+1) but cannot learn a completely new task (H), Average accuracy over 3 seeds.
Accuracy on:
+1
+2
/
Pretrained Prefix-tune on
17%
23%
34%
25%
0% 0%
100%
0%
0%
0%
0% 0%
Prefix-tune on Prefix-tune +1 0%
0%
100%
0%
0%
0% 0%
0%
100%
0%
0% 0%
Prefix-tune +2 0%
0%
0%
100%
0% 0%
Prefix-tune on 7+1
0%
0%
0%
93% 0%
Prefix-tune on H
0% 0%
0%
0%
0% 1%
EFFECTS OF PREFI--TNNING BEYOND THE SINGLE ATTENTION LAYER
Section 4 focused exclusively on its output, this bias can exhibit complex behaviors via the subsequent MLPs and attention layers. This section shows how prefix can change the attention pattern of the following attention layer but only in a linear fashion while full fine-tuning also has bilinear effects. Appendix C further argues that the representational capacity of prefix-tuning may be limited. Therefore, prefix-tuning appears to be less expressive than full fine-tuning, even with the same number of learnable parameters.
single attention layer. Still, even if prefix only induces bias
Prefix-tuning can change the attention, albeit the one of the next layer Let examine how the prefix of one attention layer affects the following one. Assume no MI.ss, residual connections or layer norms: the output g of the first is the input x. of the second. The pretrained outputs are ( Ei A w/ ", ",)", resulting in the second layer attention A//-ryy/ (D) H Here A,j is the pre-softmax attention, i.c., =ex= Au/ exp A... For prefix-tuning we then have:
(2())
APII Wys Σ A wilh
pl() Wys +(1 1
1
( g1
vi
I (asa; jz H p (1)))))) H p (1:(1-a)) ) p H +(1 (1-os)(1 (1)))) (D)
H
( (.)
constant
depends only
depends only on (1))
preemined oftectioo ((2)
The presence of ja shows that the prefix of layer can change the attention pattern of the following layer. This change is content-specifer the second and the third terms depend on the inputs, hence simple bias can affect the attention when passed through MLPs and further attention blocks. Com- pare with Equation (6), which showed prefix cannot change the attention of the same layer. Still, even considering this cross-layer effect, prefix-tuning is more limited in its expressiveness than full fine-tuning. While the second and the third terms are input- deputddependent, each depends position only. The prefix does not change the bilinear dependency on both the query and key. This something that the full fine-tuning can achieve: ( T/VE 6
input
((T) (2 + (H())) (()))
Even if prefix-tuning could be universal approximator, would not be parfmeter-efffiient. Prefix-tuning appears to be less parameter-efficient than other comparable approaches. We designed an experiment to this end. Our pretrained model in Section 5 failed to learn the double histogram task (HH). A rank-1 Low Rank Adaptation (LoRA, Hu et al., 2021) applied only to the MLPs in a 4-layer 4-head model pretrained in the exact same way results in 92% accuracy on the H task. The number of parameters for the LoRA fine-tuning is exactly the same as for prefix of size 12. However, as can be expected from the results in Section 5, training this prefix results 0% accuracy. Hence, prefix-tuning fails task that LoRA with the same number of parameters can learn.
8

---

Published as a conference paper at ICLR 2024
7 DISCUSSION AND RELATED WORKS
Understanding fine-tuning and prefix-tuning. Prior works show that prefixes have low intrinsic dimension allowing transfer to similar tasks and initialization of prefixes for new tasks (Qin et al., 2021; Su et al., 2022: Zhong al., 2022; Wang et al., 2022b; Zheng et al., 2023). In this work, offered theoretical insights into their results: this subspace is the span of the prefix- reefi--indueed bias. Another line of work shows that skills can be localized in the parameter space of pretrained models (Wang et al., 2022a; Panigrahi et al., 2023). Here, we showed that it is also possible to identify subspaces of the residual stream corresponding to individual tasks and select them via prefix-tuning.
special case of prefix-tuning. Therefore, the limitations and mechanisms discussed in this work apply to prompt- ing as well: prompts cannot change the distribution of attention of the first attention layer over the content following it and can only induce a bias on the output of this layer (Section 4), Even consid- ering the cross-layer effects, a prompt is strictly less expressive than full fine ttenin (Section 6) and prompting is unlikely to enable the model to solve a completely new task. Our theory thus explains why Kossen et al. (2023) observed that in-context examples cannot overcome pre-training skills.
Prompting and in-context learning. Prompting and in-context learning
While context-based fine-tuning approaches may not learn arbitrary new tasks, as shown in Sec- tion 5, they can leverage pre-trained skills. Wies et al. (2023) have PAC-learnability results that also show that when pretraining is on mixture tasks, they can be efficiently learned via in-context learning, Moreover, transformers can learn linear models in-context by mimicking gradient descent (Von Oswald et al., 2023) or approximating matrix inversion (Akyürek et al., 2022). This is consis- tent with our theory: the prediction updates are enacted as biases in the attention block activations. Hence, despite the limitations discussed in this work, bontext-based methods can result in powerful fine-tuning if the pretrained model has "transferable skills" such as algorithmic fundamentals. Still, in-context learning will likely fail for non-algorithmic tasks, e.g., translating to language that the model has never seen before, even if large number of translation pairs are provided in-context.
Implications for model interpretability. An open question for language model interpretability is whether attention is sufficient for explainability (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Section 5 points toward the negative: by interfering in the output of the attention layer with the bias induced by prefix, we can change the behavior of the model, without changing its attention. On the flip side, prefix-tuning can be used to understand what "skills" a model has: if prefix-tuming for a task fails, then the model likely lacks one of the key "skills" for that task.
Limitations. The present analysis is largely limited prefixing with prompts, soft prompts and for prefix-turr -urfing While our theoretical results hold for suffix-tuning they do not necessarily apply to suffixing with prompts or soft prompts. That is because the deeper representations for prompt and soft prompt suffixes would depend on the previous positions. This does not apply to suffix-tuning as it fixes all intermediate representations. Therefore, whether suffixing is more expressive than prefixing remains open question. Separately, while we provided evidence towards context-based fine-tuning methods being parameter inefficient learners, the formal analysis of the conditions under which they may be universal approximators remain an open question. Finally, we mostly considered simple toy problems. In practice. however, language models are pretrained with very large datasets and can pick up very complex behaviors. Hence, the extent to which the limitations demonstrated apply to large-scale pretrained transformers also remains for future work.
expressie
8 CONCLUSION
This paper formally showed that fine-tuning techniques working in embedding space, such as soft prompting and prefix-tuning, are strictly more expressive than prompting which operates in the discrete token space. However, we then demonstrated that despite this larger expressivity, prefix- tuning suffers from structural limitations that prevent it from learning new attention patterns. As a result, it can only bias the output of the attention layer in direction from a subspace of rank equal to the size of the prefix. We showed that this results in practical limitations by constructing minimal transformers where prefix tuning fails to solve a simple task. This result seems to be at odds with the empirical success of prefix-tuning. We provided explanations towards that. First, we showed that prefix-tuning can easily elicit a skill the pretrained model already has and can even learn a new task, if it has picked up the skills to solve it during pretraining. Second, we showed that the effect of the prefix- prefix-inducee induced bias is more complicated and powerful when combined with downstream non-liner operations. However, it appears to be still less expressive than full fine-tuning.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
In order to facilitate the reproduction of our empirical results, validating our theoretical results, and further studying the properties of context-based fine-tuning, release all our code and resources used this work. Furthermore, in Appendix A we offer explicit constructions transformers with the properties discussed in Section 3. We also provide Python implementations of these construc- tions that validate their correctness.
ACKNOWLEDGEMENT
This work is supported by a UKRI grant Turing AI Fellowship (EP/W002981/1) and the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1), AB has received funding from the Amazon Research Awards. We also thank the Royal Academy of Engineering and FiveAI.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learn- ing algorithm is in-context learning? Investigations with linear models. In International Confer- ence on Learning Representations.
Jiaqi Bai, Zhao Yan, Jian Yang, Xinnian Liang, Hongcheng Guo, and Zhoujun Li. 2023. Knowprefix-tuning: A two-stage prefix-tuning framework for knowledge-groundee dialogue gen- eration. arXiv preprint arXiv:2306.15542
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems.
YunSeok Choi and Jee-Hyong Lee. 2023. CodePrompt: Task-agnostic prefix tuning for program and language generation. In Findings of the Association for Computational Linguistics: ACI. 2023.
Ana Santos Costa, Montserrat Comesaña, and Ana Paula Soares. 2022. PHOR-in-One: A mul- tilingual lexical database with PHonological, ORthographic and PHonographic word similarity estimates in four languages. Behavior Research Methods.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers).
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-oodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adver- sarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Compu- fational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers).
Mohamad Hassoun. 1995. Fundamentals of artificial neural networks. MIT press.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpaiika,,i,ck, and Graham Neubig. 2021. To- wards a unified view of parameter-efffcient transfer learning. In International Conference on Learning Representations.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NL.P. In International Conference Machine Learning.
10

---

Published as a conference paper at ICLR 2024
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations.
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ke-Wei Lee, Lidong Bing, and Soujanya Poria. 2023. LLM-Adapters: An adapter family for parameter-efficier fine-tuning of large language models. arXiv preprint vrXiv:2304.01933.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers).
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Andrej Karpathy, 2020. minGPT GitHub Repository.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems.
Jannik Kossen, Tom Rainforth, and Yarin Gal. 2023. In-context learning in large language models learns label relationships but is not conventional learning. arXiv preprint arXiv:2307.12375
Emanuele La Malfa, Aleksandar Petrov, Christoph Weinhuber, Simon Frieder, Ryan Burnell, An- thony G. Cohn, Nigel Shadbolt, and Michael Wooldridge. 2023. The ARRT of Language-Models as-a-Service: Overview of a new paradigm and its challenges. arXiv preprint arXiv:2309.16573.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing.
Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning; Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky, 2023. Scaling down to scale up: A guide to parameter-efficien fine-tuning, arXiv preprint 2333
Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. 2021. On the expressive power of self-attention matrices. arXiv preprint orXiv:2106.03764.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P- Tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In P Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Pupers).
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey, ACM Computing Surveys
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma. Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
11

---

Published as a conference paper at ICLR 2024
Yawen Ouyang, Yongchang Cao, Yuan Gao, Zhen Wu, Jianbing Zhang, and Xinyu Dai. 2023. On prefix-tuning for lightweight out- out-of- -f-ttiibiiiittoi detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers).
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. 2024. Prompting a pretrained transformer can be universal approximator. arXiv preprint arXIv:2402.14753
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. 2021. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:21 10.077677
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Lan- guage models are unsupervised multitask learners.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song,
John Aslanides, Sarah Henderson, Roman Ring, et al. 2021. Scaling language models: Methods, analysis & insights from training Gopher.
Sylvestre-Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems.
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems.
Elvis Saravia, Hsien Chii Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Auto- Prompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Mirko Solazzi and Aurelio Uncini 2004. Regularising neural networks using flexible multivariate activation function. Neural Networks, 17(2):247-260
Yusheng Su, Xiaozii Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for mawagg--mdolll-----eee
In International Conference on Machine Learning.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint a3311.2302.13971
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural
Information Processing Systems.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvint- sev, Andrey Zhmoginov, and Max Vladymyrov. 2023. Transformers learn in-context by gradient descent. In International Conference on Machine Learning.
12

---

Published as a conference paper at ICLR 2024
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022a. Find- ing skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2022b. Multitask prompt tuning enables parameter-efficient transfer learning. In International Confer- ence on Learning Representations.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. 2021. Finetumed language models zero-shot learners. In International Conference on Learning Representations.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021. Thinking like transformers. In International Conference on Machine Learning.
Sarah Wiegreffe and Yuval Pinter: 2019. Attention is not not explanation. Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
Noam Wies, Yoav Levine, and Amnon Shashua. 2023. The learnability of in-context learning. arXiv preprint arXiv:2303.0789)
Hui Wu and Xiaodong Shi. 2022. Adversarial soft prompt tuning for cross-domain sentiment analy- sis. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers).
Yuanhang Zheng, Zhixing Tan, Peng Li, and Yang Liu, 2023. Black-box prompt tuning with sub- space learning. arXiv preprint arXiv:2305.03518.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao. 2022. PANDA: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint arXiv:2208..00660
13

---

Published as a conference paper at ICLR 2024
A
CONSTRUCTING TRANSFORMERS THAT UTILIZE THE CAPACITY OF THE EMBEDDING SPACE
A.1 UNCONDITIONAL GENERATION FOR SINGLE VIRTUAL TOKEN
This section provides an explicit construction of a transformer with the properties described in The- orem The goal is construct a transformer that, by varying the choice of the virtual token, can generate any sequence of N tokens.
First, we need specify how we encode the target sequence (Yi,..., Yx) into the virtual token 81- We chose the size of the embedding (and hence of s1) to be N. This way, each element of $1 can represent one position of the target sequence. We then represent the token value by discretizing each element of into V levels:
81 = Note that this means that each element of 81 is in 1).
(2w-1l/v).
When predicting the token for the i position, the transformer needs to pick the i-th element of 81, and then decode the corresponding value one-hot encoding representing the Y,-th token.
We extract the i-th element of 81 using one attention block of two heads. The fst head always looks at the first position which is our virtual token 81. For that purpose create an attention head that always has fst 1 if = and A = 0 otherwise together with a value matrix W that extracts the embedding. This is achieved with
W = [0 1yl.
w K
[O. 1, 1,1,,
W V INN.ONNI,
(8)
and sufficiently high inverse temperature parameter T.
The pos head instead extracts the one-ho hot encoding of the current position. This can be done with an attention head that always attends only to the current position and value matrix WD0s that extracts the position embedding as a one-hot vector:
m 1ONNN IN];
K ONNN, IN],
W = OONN IN].
When the outputs these two attention heads are summed, then only the element of $1 that corre- sponds to the current position will be larger than From Equation (2) the output at the i-th position of the attention block is:
A
+
A eN (i) =s te)(
where æ1 and 2; E.,x,-; for 1.
We can extract the value of 81 corresponding to the current position by substracting 1 from the hidden state and apply ReLU: Les = I - -1] Now, we are left with only one non-zero entry and that's the one corresponding the next token. We can retain only the non-zero entry if we just sum all the entries of the hidden state with C.um _[1 0].
The final step is to map this scalar to a -ddimensional vector which has its maximum value at index designing V linear functions, each attaining its maximum at one of
Y, This task is equivalent 0, l/v..... (V-l//v. To construct this, we use the property of convex functions that their tangent is always under the plot of the function. Therefore, given convex function 7 (xx) we construct the i-th linear function to be simply the tangent of y at 5-1/v. If we take ((x)) = (x 1/2)2, this results in following linear layer:
2(( 21 2(1-1) [[ V
2(V
1)
T
[
(1-1)
1)2
1
(V 1)2
Ppos L
1
(10)
V
V2
4
1/2
Figure shows the predictors for each individual token id.
With just two attention heads A((W Q w (Whi,
and three linear layers, the transformer K W (W-b, wF" fex ; Samm ; Cproj 3 softmax achieves the up-
per bound of V unique outputs by controlling single virtual token at its input. Note that for this
14

---

Published as a conference paper at ICLR 2024
Logit for token (argest token when the isput (s
Logit for token (brggst token impert 7/N)
ondhno loudry
0/V
2/V
4/V
6/V
8/V
V/V
prorj input
Figure 3: Illustration of the predictors for each token in the Lyei linear layer for V 10. The layer is constructed in such way that the i-th token has the highest confidence when the input i-v/v.
construction, the choice of embedding matrix E E NXV does not matter. The same transformer architecture can generate only V unique outputs we only control the first token instead. Therefore, it is indeed the case that the embedding space has exponentially more capacity for control than the token space. You can see this transformer implemented and running in practice in Section of this notebook.
A.2 CONDITIONAL GENERATION FOR SINGLE VIRTUAL TOKEN (nx ny my =)
This section provides an explicit construction of a transformer with the properties described in The- orem 2. The goal is construct a transformer that, by varying the choice of the virtual token, can cause the model to act as any map m : [1..... V] - [1...., VJ. In other words, by selecting the virtual token, we can fully control how the model will respond to any token the user may provide.
First, we need to specify how the map will be encoded in the virtual token the embedding size de to be V. Now, we can use the same encoding scheme as before, but now each element in s1 corresponds to different user token, rather than to position in the generated sequence:
We choose
(mo())/v.... m(V)vv)
Therefore, the first element s designates the response if the user provides token the second element is the response to the token 2, and so on.
Extracting the Y--th value from 81 and decoding it be done in very similar way as for the unconditional case. The only difference is that instead of looking at the user input position, we look its value. Take Iy and N 2.
Hence we have the following val head (only differing in the Wy matrix from Equation (9)):
W [22v 12),
W K 02x 12),
W [Iv Ov
We also need embedding of the first token, so we have a modified version of Equation (8):
wha 00v 1,
W [0v 1,0),
W [Iv Oy
And hence the output of this attention block at the second position would be:
2
t2 Σ
Σ
A A
W
= eY,))
J=1
Similarly to the unconditional case, only the entry of t2 corresponding to the user token will have a value above and that value would be m(x))/V.
15

---

Published as a conference paper at ICLR 2024
We now extract the one-hot representation of the target token using the same approach before, just adjusting for the different hidden state size: fex -[ lv]. Lum f 0j, and the same projection had as before (Equation (10)). The final transformer is then: A Q We (W Wk",, (WV) W fex Sum Prooj softmax. You can see this trans- former implemented and running in practice Section here.
A.3 CONDITIONAL GENERATION FOR LONGER RESPONSES (nx = ny > 1)
We can obtain longer responses via simple extension. the response length is No. then we can encode the map m [l.... V] + ,I,..., VI in No virtual tokens, each corresponding to one of the target positions:
8; (m(1)./v,.... m(Vvv/v for = = 1,..,,
For this model we would then have N = 2No and de V.
First, we need head that always looks at the token provided by the user, which will be at position No + 1:
W = 1x].
user = [0 N.). 1, -ino
w = [Iv, Ov
In order to consume the map at the right location, we need to also look at the embedding of the token trying to generate:
No positions before the
Whick
0 No (No+ O Nox xoox(++V (No+V)
IN O NN
W [ONXV IN]; W
[Tv, Ov.N]=
From here on, the decoding is exactly the same in the nxx ny = 1 case. The final transformer then: A(W Q Whick), K" Whick)), (WV Whiak)] 3 Lex 3 Ssum Lpau softmax. You can see this transformer implemented and running in practice in Section 4 here.
A.4 CONDITIONAL GENERATION FOR LONGER USER INPUTS (nx > ny = 1)
Finally, we consider the case when the user input X is longer. This is a bit more complicated because we need to search through domain of size VV. We will only consider the case with nx = 2 where we would need two attention layers. A similar approach can be used to construct deeper models for nx > Finally, combining the strategy in the previous section for longer responses with the strategy in this section for longer user inputs allows to construct transformers that map from arbitrary length user strings to arbitrary length responses.
In order to encode map [l...., vJ2 .I...., Vj into single virtual token would need more involved construction than before. Similarly to how we discretized each element of the virtual token $1 in V levels before, levels would be one the VV possible maps from the second user token to the response. The first user token would be used to select the corresponding element of 81. Then this scalar will be "unpacked"" into a new vector of V elements using the first attention block. Then, the second user token will select element from this unpacked vector, which will correspond to the target token.
going to now discretize it into VV levels. Each one of these
We construct the virtual token as follows:
Pi-1
V Σ
Vi-1 VV
81
(i)
VV
mvli)
where mx(x)) = m(ffar) .r) a map from the second user token the response when the first token fixed to be f.
An additional change from the previous constructions is that we are going to divide the residual stream into two sections. This is in line with the theory that different parts of the residual stream specialize for different communications needs by different attention heads (Elhage et al., 2021). We will use the first half of the residual stream to extract and "unpack" the correct mapping from second token to target token, while the second half of the residual stream will be used to copy the second token value that the second attention layer
it to extract the target. As usual, the embedding
16

---

Published as a conference paper at ICLR 2024
matrix will be the identity matrix: E Iv. Finally, for convenience, will also use dummy zero virtual token that we will attend to when we want to not attend to anything. This results in context size N with the imput being
(l (en((D)
81 [ex(2)
Exx eN(3
(ewvs) -
Or ((x())))
[er(xi) ex((3
(er(xa)] exx(4)
(ex(2)
We want the output the last position to be the target m (X1, X2), that is:
arg max weI..... V y4.u m(x1, X2) for any m. X1, X2-
The first attention block will have three attention heads.
As before, we want to extract the value of 81 that corresponds to the first token the user provided (x1) and place it in the first half of the residual stream. We want only the third position do that, while the rest of the positions keep the first half of their residual stream with zeros. Hence we have the following fst head:
who
02xV
1
w
02xV
0
W
Iv
Ovx
0 0
0
1
0
0
Ovvv
OVxN
The userl head extracts the value of the first user-provided token (X1) and also places it in the first half of the residual stream:
Winserl
O2XV
1 1101
Wuseer
02xV
0 0
W Wuserl
Iv
OVxN OVxN
0 0 1
0
0 1
0
Ovvv
And the user2 head does the same for the value of the second user-provided token (X2), placing in the second half of the residual stream:
2
02xV
0
WK
O2xV
0
0
W
xv 2Iv
Ovx Ovxn
where the factor 2 is there because, as usual, the first linear layer will subtract from everything in order to extract the value selected by the first token.
This linear layer looks as usual: fox = -. 122 The result is that the first V elements will be except one which designates which map from second user token to output we should use, and the second V elements have one hot-encoding of the second user token. Constructing an MLP that unpacks the mapping can become quite involved so we do not provide an explicit form for it. But from the universal approximation theorems and the finiteness of the domain and range, know that such an MIP should exist. We thus designate by unpack the MI.P that decodes the first half of the residual stream to:
mx, (1) V
Myx (V) V
and keeps the second half unchanged.
And now, by using two attention heads, the second attention block extracts the value of the above vector the position designated by the second token, in fashion not dissimilar to all the previous cases:
Wemb [0%, [.
Wemb [1V, 07),
W = [Iv Ovxv),
WH [0,, 1
Waser2' 10v, 17..
W =Ovxv Ivl.
And finally, with S.r tion had as before (Equation (10),, we get the target token. The final transformer is K: W W (W W w Loo 3 unpack
[[Iv,-Iv], Sum
L 0). and the same projec-
then: Al(Wi Q Wiwser, W (Wihi A[(W Wy ), (Weuh, W ), (Wenb, W )] Lox ; Esum ; Eproj softmax. You can see this transformer implemented and running in practice in Section here.
17

---

Published as a conference paper at ICLR 2024
Dincnuion 148 of the resdull sceam
Resporse socted ascending
Ad42 prehs
Reeporse sorted descending
Figure 4: Attention block activations for len sequences the last input position (10) when pretrained the four tasks. The left plot shows pretrained activations 4ı0 are not predictive of the completion. The right
Response incressed by
Ascending pretx
Response incressed 2
Wishout prefix (points the left ploc)
plot shows prefixes cluster the activations too Connecting the pretrained and prefixed activations highlights the bias. dimen-
Without prefixes
Descending pretix
(pretrained model)
With prefixes
sionality reduction is used; the clustering solely due to the prefixes.
0. Dimension of residual stream
0.
20
Dimansson 94 of the resdual stream
Attention for layer 1, head 12 of LLaMA with a prefix
natr
Attention for layer 1. head of LLaMA with a prefix normalized over the content
Attention for layer 1, head 12 LLaMA without prefix
Figure The attention the twelfth head the layer LLaMA (Touvron et al., 2023). The left plot shows the attention with a prefix of length one. The second plot shows the same attention but normalized such that the attenion over the non-prefix positions sums to 1. The right plot shows the at- tention of the pre pre-traine trained model (without prefix). The center and the right plots the same, illustrat- ing that the presence of the prefix indeed only scales down the attention positions) but does not change relative distribution, providing empirical validation The test sequence is TABLE; Fourth Round Qual i fying NEW ENTR ES THIS S ROUND 24 TEXT: Fourth round qualifying had 24 new entries. from the DART table-to-test dataset (Nan et al., 2021).
the content (non-prefix
Equation (6).
Actual prefixed activations
Predicted activations A Wvs1 + (1 A,h));
pt
Input position
Residual stream
Residual stream
Figure 6: The activations of the twelfth head of the first layer of LLaMA (Touvron et al., 2023). The left plot shows activations
right plot shows activations of the pretrained model, scaled by one minus the attention that the prefix would take and then biased the direction Wysj. The two plots are the same, illustrating that our thcory, Equation (7) in particular, also holds for real-world large transformer models. The test sequence the same in Figure
presence
prefix.
18

---

Published as a conference paper at ICLR 2024
Prefx
Prefx
Prcix 3
P
Prefx 5
Prdft
Prefx 8
Prefix 9
Profix 10
50
02
04
0.
6a
1
60
0.2
066
06
1.
0.0
02
94
06
10
60
62
84
05
ca
10
laayerr
Amention a
Astention Layer
Attention Layer
Prefs
Prcfe
3
Prefx
5
Prdix
Prefxx 8
Profix 9
Padix
100
1.0
Aneetionn of Tayer
Attention of layer
Attention of byyrr
Attention of layer 8
Padix
Prefec
Prefxx 5
6
Proftc
Profee
Prefxy y
Prefis 10
1/0
000
01.2
1.0
D.6
10
02
0.6
100
Attenttion of layer
Attention
layer
Astention layer
Ameention layer
Figure 7: The range of attention (Ist to 99th percentile) for a single GPT-2 (Radford et al., 2019) prefix trained the Emotion dataset (Saravia et al., 2018). The prefix is of size 10 (n.s 10). This is the attention of the last user input token (nx) because this the position which the class prediction is done. For illustration purposes, we have normalized the attention so that the attention over 10 prefix positions sums to The range of attention over the 10 positions for each layer are shown.
B ATTENTION DISTRIBUTION OVER THE PREFIX
As discussed in Section 4, longer prefixes define a subspace from which the bias for the attention block selected. For a prefix of size n, that means that this subspace is ny--dimensional. Each of prefix position j defines basis vector Wys, , for this subspace, while the attention A S.S, on this position determines how much of this basis component contributes to the bias.
In ordered to span the whole subspace and make full use of the capacity the prefix, s, should vary between and for different inputs. However, we observe that this does not happen prac- tice. Figure 7 shows the ranges of attention the different prefix positions take for the GPT-2 model (Radford et al., 2019). For layer for example, the attention each prefix positions gets is almost constant hence, the effective subspace is collapsed and there is single bias vector that's applied to the attention layer output, regardless the user input X.
Some other layers show slightly higher variation. For example, layer has three prefix positions with large variations. Therefore, the effective bias subspace is 3-dimensional and the user input X governs which bias vector from this subspace will be selected.
19

---

Published as a conference paper at ICLR 2024
linear
exp
linear
fixed MLP
linear
exp
linear
fixed
fixed feature extraction
fixed feature extraction
MLP
fixed feature extraction
fixed
fixed feature extraction
fixed
linear
exp
linear
ML.P
linear
exp
linear
soaarr veltor
cunilioca
Figure 8: Prefix-tuning as a neural network architecture. While linearities and non-linearities are present, the have limited interaction with the inputs æ1 and æ2- The interaction of the prefix parameters with each input only via the scalar attention, shown here with a light connection. The mixing of information between the inputs happens via residual connections with the pretrained fixed feature extraction and hence is not learnable. The MLP is also fixed and hence only acts as multivariate activation function. This limited interaction explains why prefix-tuning struggles learn tasks in deeper models.
only learnable parameters ( s(?)
C EXPRESSIVITY OF PREFIX- TUUNING ACROSS DEEPER MODELS
In Section 6, considered the effect of the presence of the prefix in the first attention layer the attention of the second. However, the effects become more complex as one adds more attention attention layers. We also ignored the MLPs between, but in practice they can play an important role. Here, instead, we analyse prefix-tuning as learning a neural network. We argue that while the resulting architecture includes both linear operation and non-linear activations, the structure is unlikely to learn efficiently.
For simplicity, we will consider two inputs 2₁ and 2g and a single prefix The output of the attention head, parameterized by s is then:
A, (21, = (y), y2)
y1 exp(x1 H (Hs)Wys exp(2
Hæ Wy + (x) Hæ Wyaz
(11)
C₁
exp(2x2 T Hs) exp(222 Ha
W v +
exp((22
Ha W
V2
C2
where we have omitted the T/Vk factors and have folded the softmax normalization into C₁ and C2. The layer inputs, pretrained parameters and the learnable parameters are correspondingly high- lighted. The attention head is clearly a non-linear operation. However, the learnable parameter s participates only in the left term. It interacts with only one of the inputs time and only by computing a single scalar value x; Hs. As we discussed above, the interaction between x and æ2 is non-trainable and be thought a hard-coded feature extraction. Each of the outputs is then passed to a pretrained MLP which can be thought of as an activation function. This would be multivariate activation function, which while unusual in the contemporary practice has been studied before (Solazzi and Uncini, 2004). Figure 8 illustrates the computation graph of the resulting neural network and shows that the only learnable interaction between the inputs is indirect. Therefore, prefix-tuning can be considered learning neural network where the only interaction between the inputs happens via non-learnable residual connections. Nevertheless, the alternating linear and nonlinear operations are reminiscent of the standard neural network architecture and their universal approximation properties (Hassoun, 1995). That begs the question if the prefix-tuning architecture can be universal approximator and whether it would be parameter-efficieet one.
An example of prefix-tuning failing to be a universal approximator. While we leave the formal analysis of the representational capacity of prefix-tuning future work, we provide example of pretrained parameters for which the architecture is not universal approximator. As can be seen in Figure 8, all information must pass through the leamnaaeeble MLPs. Thus, the MLPs destroy all input information, there is value for the prefixes (1), ((2) that can change that fact. The ML.Ps
20

---

Published as a conference paper at ICLR 2024
can destroy all information, if, e.g., one of their linear layers has zero weight matrix. While this unlikely to be learned with typical pretraining, this demonstrates that if prefix-tumin could a universal approximator, that would pose specific requirements the pretrained MLPs and it is not clear whether real-world pretraining would satisfy these requirements.
D EXTENDED RESULTS
In this appendix we present further experiments in the context of Section 5. We consider different prefix lengths (ng (10, 50, 100}) and different model sizes (4, 16 and 32 layers). We consider two extensions, the first one maintains the pretraining step as in Section 5, the other one extends the set of pretraining tasks with 4 additional tasks. Both pretraining and prefix-tuning are done for 100 000 iterations with the prefix-tuning accuracy reported every 10 000 iterations.
D.1 PRETRAINING AS IN SECTION 5
The first setting has the same pretraining tasks as in Section 5, namely sorting in ascending (/*) or descending (x) order, and adding one (+1) or two (+2) each element of the input sequence. We evaluate by prefix-tuning on the same four tasks plus incrementing the ascending sorted sequence (/+1), double histogram (HH), element-wise modulo operation (with respect the first element of the sequence), and Filter AttterAAlleee eemeee which puts zeros at the positions of elements whose value appears at less than N times in the sequence with N being the first element of the sequence:
Sort ascending order (?)
Sort in descending order (
Pretraining tasks:
Add 1(+1)
Add (+2)
Sort in ascending order (/)
Sort descending order (N)
Add I(+1)
Add 2 (+2)
Prefix-tuning tasks:
Sort ascending and add (n*+)
Modulo the first element (Modulo)
Double Histogram (TH)
Filter the elements that are at least as large as the first element (FilterAtl .atierAmtl.asNNTimess
The results are plotted in Figure 9. As expected, the prefix- prefix-tuned accuracy on the pretraining tasks close to 100%. Interestingly, prefix length 50 for the largest (32-layer) architecture appears to be an exception and does not learn the +1, and Y
As also observed in Section 5, regardless of the prefix length and the model size, prefix-tuning a model pretrained with these four tasks cannot learn the double histogram task (lHl). FilterAtLeast- NTimes also appears to be challenging but the 16-layer and 32-layer experiments reach about 50% accuracy for the longest prefix size n 100. This is curious as FilterAtLeastNTimme is related to the double histogram task: can be considered as thresholding the double histogram output. Fil- terAtLeastNTimes similarly to double histogram, is therefore not compositional in the pretraining task. is surprising then that FilterAtl eastNTimes would achieve higher accuracy than double his- togram. This hints that, perhaps, compositionality does not fully explain why prefix-tuning works for some and not other downstream tasks.
The results in Figure 9 also hint that bigger is always better. For example, the 4-layer model prefix-tuned for the Modulo task performs better with prefix size 50 than the larger prefix size 100. similar effect can be observed with the 16-layer model prefix-tuned for the 7+1 task. Larger models are also not necessarily more conducive to successful prefix-tuning for many of the cases the 32- laylr models perform worse when prefix-tuned than the 16-layer models.
D.2 EXTENDED PRETRAINING
The second setting extends the of pretraining tasks. On top of the original three pretraining tasks 7, +1 (without +2) we also pretrain on element-wise modulo operation (with respect to the first element of the sequence), element-wise less than (less than the first element in the sequence),
21

---

Published as a conference paper at ICLR 2024
Models pretrained with the original 4 tasks (+1, +2, 7.7) )
Prefix length:
Models with attention layers
Models with 16 atlention layers
Models with 32 attention layers
vole
Prefix uining
wws.graabeeeeee beraboon
Figure 9: Extended result with pretraining in Section S. The pretraining and the prefix-tuning tasks are described in Appendix D.I. Each prefix trained for 100000 iterations and report accuracy at 10 iterations. Each experiment is performed with random seeds, except the 32 layer case which, due to the computational costs involved, was performed only with one seed.
every
element-wise divisible (by the first element of the sequence), and inverse binary (element-wise nega- tion). For the inverse binary task, the input restricted be binary. We evaluate by prefix-tuning on the same seven tasks, as well as 11 additional ones as listed in the following table:
Sort ascending order Sort descending order (N)
Add (+1)
Pretraining tasks:
Modulo the first element (Modulo)
Filter the elements that are less than the first element (LessThan) Filter elements that are divisible by first element (Divisible)
Element- wise negation (InverseBinary)
Sort ascending order
Sort descending order (N)
Add 1 (+1)
Add 2 (+2)
Add (+3)
Modulo the first element (Modulo)
Filter the elements that are less than the first element (LessThan)
Filter the elements that are not less than the first element (MoreThanEEqull)
Filter the elements that are divisible first element (Divisible)
Trefix-tuning tasks:
Filter elements that are not divisible by first element (NotDivisible) Element- wise negation (InverseBinary)
Double Histogram (I=I
Filter the elements that are least large as the first element (FilterAtl NastNTimess Sort ascending, followed by add (/7+1)
Add 1, followed by LessThan (+1 LessTThan)
Less Than, followed by Add (LessThan +1)
LessThan, Than, followed by sort ascending (LessThan + 7
Divisible, followed by Add (Divisible +1)
22

---

Published as a conference paper at ICLR 2024
Models pretrained with an extended set tasks
Prefix length:
Models with 4 attention layers
Models with t6 attention layers
Models with 32 attention layers
M00
1881
Prefts -rning
Setieerstrr snieatte
Figure 10: Extended result with pretraining with additional tasks. The pretraining and the veffi-ttnning tasks described in Appendix D.2. Each prefix trained for 000 000 iterations and report accuracy every 10 10000 iterations. experiment is performed with random seeds, except the 32 layer case which, due to the computational costs involved, was performed only with one seed.
23

---

Published as a conference paper at ICLR 2024
The results are shown in Figure 10. Even though the model did not see the +2 and +3 tasks, it appears that prefix-tuning can generalize from the +1 task. The +1 + LessThan task is another example of successful prefix-tuning for task that is compositional in pretraining tasks. Similarly, for Divisible +1, TessThan +1, 11 Less' Than + /. MoreThanFFqua,, NotDivisible,
Similarly the case in Appendix D.I, observe several instances in which the largest, 32-layer, model performs worse, when prefix-tuned, than the smaller models, as well as cases where shorter prefixes perform better than longer ones.
E FURTHER EXPERIMENTS WITH MEMORIZATION
Our experiments in Section 5 focused on algorithmic tasks such sorting, incrementing and count- ing. However, learning natural language also includes a substantial memorization component. For example, learning new language requires learning its vocabulary. Hence, if novel task requires memorization of novel concepts. the fine-tuning method should be able to memorize them. To this end, we evaluate and compare the abilities of prefix-tuning and LoRA to learn to memorize large number of new words and show that, for the same amount of learnable parameters, LoRA can learn to translate to new language while prefi-ttuning cannot. These findings further strengthen our results from Section 5.
We use the PHOR-in-One dataset (Costa et al., 2022) that contains 4921 unique translations of English words into German, Spanish and Portuguese. The dataset is preprocessed to remove all accents in order to ensure that fin-ttunig does not require characters that the model has not seen during pre-training (c.g., éçï would become ecu). Character-level tokenization is used, with the additional <TR> token that separates the source and target words and a <PAD> token that we use to ensure all training sequences of the same length.
We then pre-train a 4-layer 4-head transformer to translate the English words to German. As seen in Table 4, this model successfully memorizes 99.3% of the English-German pairs. We then want to fine-tune this model to instead translate the English words to Spanish. Spanish is linguistically closer to English than German, so, in a way, the fine-tuning task simpler than the pre-training task.
We do prefix-tuning on English-Spanish pairs with a prefix of size ns = 66 but it achieves only 0.18% accuracy (about 10 words which are the same in English and Spanish, e.g., instrumental, orbital, solar). However, rank-4 LoRA is able to achieve 94.8% accuracy with half the training iterations. Both fine-tuning methods have similar numbers of learnable parameters (see Table 4). Therefore, this is further evidence that prefix- tunfin fails to learn new task, while LoRA with the same number of learnable parameters can.
Note that there is no train-test split for the word pairs. We are evaluating the accuracy on the training set as this experiment is measuring memorization rather than generalization.
Table 4: Further experiments on memorization word pairs form (Costa al., 2022) different languages. The model pre-trained on the English-to-German word pairs to almost perfect accuracy. Prefix-tuning fails to modify it to memorize English-Spanish word pairs while LoRA A with the same number of parameters and half the training iterations reaches 94% percent accuracy. 5 random seeds.
Pre-training on English to German
Prefix-tume tune (n==66) on English to Spanish
Rank-4 LoRA on English to Spanish
Accuracy
99.3 + 0.1 %
0.18 = 0.01%
94.8 : 1.1%
Learnable parameters
3.19M
67 584
66 780
Training iterations
50000
100 000
50000
24


Reviews:
[{"review_id": "05T2SEtSgm", "reviewer": "Reviewer_RrcA", "summary": "The paper presents an insightful theoretical analysis of the limitations of context-based fine-tuning methods like prompting and prefix-tuning. While these methods have been empirically successful, the paper argues they are structurally less expressive than full fine-tuning and cannot learn tasks requiring new attention patterns. The paper tests these theoretical claims with minimal transformers and discusses the practical implications of these limitations.", "questions": "The theoretical framework presented is compelling, but how does it hold up against the more recent transformer models that might use different mechanisms or have additional layers/heads?\n\nHow can the results inform the development of more efficient training procedures that could circumvent the limitations of prefix-tuning?\n\nCould the authors provide a more detailed discussion on the practical implications of these findings for various NLP applications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper provides a valuable theoretical perspective on the limitations of prompting and prefix-tuning. It contributes to the understanding of how these methods compare to full fine-tuning in terms of their ability to learn new attention patterns, which is a significant contribution to the field.\n\n+ The authors support their theoretical framework with empirical evidence. The use of minimal transformer models for testing provides clear illustrations of the theoretical limitations in a controlled experimental setting.\n\n+ The paper's findings have practical relevance for the design of fine-tuning strategies in NLP applications. It helps practitioners understand when to employ prompting and prefix-tuning and when to opt for more expressive fine-tuning methods.", "weaknesses": "It would be beneficial to validate the theoretical findings with a broader set of experiments, including a variety of tasks, models, and datasets to confirm the universality of the proposed limitations.\n\nA comparison with other fine-tuning methods such as transfer learning or domain-adaptive pretraining could provide a more comprehensive view of where prefix-tuning stands in the spectrum of fine-tuning techniques.\n\nThe paper identifies important limitations but does not provide detailed potential solutions or alternative methods that could overcome these limitations. Expanding on this could make the paper more impactful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuiLykKpcA", "reviewer": "Reviewer_R11r", "summary": "This paper discusses the roles and limitations of context-based fine-tuning approaches (e.g. prefix fine-tuning) from a theoretical perspective. By analyzing the effects of attention mechanisms and computation within the model, the authors illustrate that there are structural limitations of prefix fine-tuning, while prefix fine-tuning is expressive due to continuous space. These limitations preclude prefix fine-tuning from learning new attention patterns, which makes it less expressive than full fine-tuning. The paper then reveals that the success of the context-based fine-tuning approach depends on the eliciting of skills in the pre-trained model, rather than learning new skills.", "questions": "-\tIs the cross-layer effect enough to make the prefix fine-tuning learn new tasks when the transformer consists of more attention layers like many current language models?\n-\tIs there any potential improvements for tuning methods based on the analysis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies a valuable problem, which focuses on the capabilities and limitations of context-based fine-tuning, making a valuable contribution to the research community. \n- The theoretical discussions effectively highlight the problems posed by context-based fine-tuning.", "weaknesses": "- It would be better to further test models larger than LLaMA-7B as models with more parameters may exhibit different properties.\n\n- An additional ablation experiment may be required in subsection 'Prefix-tuning can combine knowledge from pretraining tasks to solve new tasks'. The authors utilize a 4-layer 4-head model to validate the point that prefix-tuning can learn a new task as long as the “skill” required to solve the new task is a combination of “skills” the pretrained model has seen. However, in a 4-layer 4-head model, a prefix-induced bias can have non-linear behavior when passed through non-linear MLPs and attention blocks as mentioned In Section 5.2. It would be better to further test a 1-layer 4-head model for further clarification.\n\n- Minor grammar issues\nThere are also several minor grammar issues, just a few:\n'However, generation generation is more interesting'\n'This can be be clearly from the activations'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jxaCrTpnJ", "reviewer": "Reviewer_7Rcz", "summary": "Context-based fine-tuning techniques like prompting, in-context learning, soft prompting, and prefix-tuning have gained popularity due to their ability to achieve good results with fewer parameters compared to full fine-tuning. However, we lack a theoretical understanding of how these methods affect the model's internal operations and their limitations. This paper reveals that while continuous embedding space is more flexible than discrete token space, soft prompting and prefix-tuning are less expressive than full fine-tuning. This means that techniques like prompting and in-context learning can leverage existing skills in a model but can't learn entirely new tasks that require different attention patterns. This understanding provides insights into the capabilities and limitations of these fine-tuning methods, helping researchers and practitioners make informed choices when applying them to various tasks.", "questions": "see W1", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. theoretical support to prompt learning is a pressing need and this paper provided a comprehensive view from theoretical analysis.\n\nS2. I like their presentation, which is clear.\n\nS3. their theoretical analysis is interesting.", "weaknesses": "Overall, this paper tried to solve a very interesting problem. I would be very happy to raise my score if the following concerns are addressed:\n\n\nW1: prompting is not only used in linear data like text but also applied to non-linear data recently like graphs. It would be more solid to discuss them in the related work section (e.g. X Sun, et al. \"All in One: Multi-task Prompting for Graph Neural Networks\". KDD2023). It would be even better if the author could further confirm whether their theoretical analysis applies to the graph prompting area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2h4qKE0FFf", "reviewer": "Reviewer_rXHB", "summary": "The paper presents an analysis of how prompting, soft prompting, and pre-fix tuning work for transformer models and why these methods are not as capable as full-finetuning for transformer performance. The paper presents theorems on why soft prompting is able to elicit a wider range of behaviors than standard prompting, and prefix tuning than soft prompting. The paper then presents an explanation of why prefix-tuning cannot change the behavior of a transformer model as much as full fine-tuning. The paper then investigates why, if prefix-tuning is less powerful than fine-tuning, then why does it produce good results in practice? The paper posits that this result is due to prefix-tuning being very good for biasing a transformer toward performing pre-trained tasks and that the results in practice are a result of the model already understanding the task from pretraining.", "questions": "In addition to the previously mentioned questions, in equation 7, what is the equivalence between the pre-trained ($t_i^{ft}$) and prefix-tuned ($t_i^{pt}$) model outputs, or is there one? While I generally by the argument that the change to $W_v$ means more changes to the outputs than adding the $W_V s_1$ term to the equation for the outputs, I am not convinced that that is always true or under what conditions it might achieve equivalence. Thus, to really cement the claim that pre-training holds more potential for outputs than prefix-tuning, it would be interesting to see where adding the bias term from prefix-tuning can be (and can’t be) equivalent to adding an update to $W_V$.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The paper presents some very significant theoretical results and their implications for using transformer-based models. In particular, the idea that biasing the outputs through prefix (or even prompting or soft prompting) elicits pre-trained skills from the transformer model and that these pre-trained skills can be combined through this means, is important to really understanding both why techniques like prefix and prompting work, but also provide insight into the emergent behavior of models like LLMs. It also leads to the intriguing question of whether there is some basic set of tasks that something like an LLM needs to be pretrained on, in order to be able to practically accomplish just about any task in natural language. \n\nThe paper is also thorough in its investigation of the phenomenon of prefix-tuning and prompting by including both mathematical arguments for the claims made as well as simplified examples with actual code.", "weaknesses": "The paper has some clarity and soundness issues, from my reading. For clarity, I having trouble interpreting the attention figures in Figures 1, 3, etc. to see the patterns that the authors are trying to call out. Perhaps the image captions could include some interpretation guidance for the readers (e.g, the figures are meant to be read left-to-right, where …). \n\nFor soundness, there were a couple of areas, where I was not fully convinced of claims by the provided proofs. For Theorems 1 &2, I think can see why those are true, but having more of a sketch as to why they are true would improve both the clarity and firmly establish why soft prompting and prefix-tuning are more expressive in output generation than prompting. And in section 5.2, how does prefix-tuning change the attention of the next layer? Earlier on in the article, it is argued that changes to the attention layer have the form of $ W_{v} + \\Delta W_{V}$ (i.e., equation 7), and yet the equations of section 5.2 do not have any alterations to $W_{V}$ or $H$. Rather it looks like the prefix-tuning changes the inputs to the next layer of attention rather than the attention block itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)