Title: Universality and Limitations of Prompt Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: prompt-tuning; language model; expressive power
Abstract: Despite the demonstrated empirical efficacy of prompt tuning to adapt a pretrained language model for a new task, the theoretical underpinnings of the difference between "tuning parameters before the input" against "the tuning of model weights" are limited. We thus take one of the first steps to understand the role of soft-prompt tuning for transformer-based architectures. By considering a general purpose architecture, we analyze prompt tuning from the lens of both: universal approximation and limitations with finite-depth fixed-weight pretrained transformers for continuous-valued functions. Our universality result guarantees the existence of a strong transformer with a prompt to approximate any sequence-to-sequence function in the set of Lipschitz functions. The limitations of prompt tuning for limited-depth transformers are first proved by constructing a set of datasets, that cannot be memorized by a prompt of any length for a given single encoder layer. We also provide a lower bound on the required number of tunable prompt parameters and compare the result with the number of parameters required for a low-rank update (based on LoRA) for a single-layer setting. We finally extend our analysis to multi-layer settings by providing sufficient conditions under which the transformer can at best learn datasets from invertible functions only. Our theoretical claims are also corroborated by empirical results.

Universality and Limitations of Prompt Tuning

Abstract

Despite the demonstrated empirical efficacy of prompt tuning to adapt a pretrained language model for a new task, the theoretical underpinnings of the difference between "tuning parameters before the input" against "the tuning of model weights" are limited. We thus take one of the first steps to understand the role of soft-prompt tuning for transformer-based architectures. By considering a general purpose architecture, we analyze prompt tuning from the lens of both: universal approximation and limitations with finite-depth fixed-weight pretrained transformers for continuous-valued functions. Our universality result guarantees the existence of a strong transformer with a prompt to approximate any sequence-to-sequence function in the set of Lipschitz functions. The limitations of prompt tuning for limited-depth transformers are first proved by constructing a set of datasets, that cannot be memorized by a prompt of any length for a given single encoder layer. We also provide a lower bound on the required number of tunable prompt parameters and compare the result with the number of parameters required for a low-rank update (based on LoRA) for a single-layer setting. We finally extend our analysis to multi-layer settings by providing sufficient conditions under which the transformer can at best learn datasets from invertible functions only. Our theoretical claims are also corroborated by empirical results.

1 Introduction

The surge in the empirical research of large-scale models has led to the emergence of a new paradigm of prompt tuning. Current large models consist of billions of parameters [Brown et al., 2020, Chowdhery et al., 2022], which greatly exacerbate the cost of tuning the entire model weights via gradient-based optimization. On the other hand, the power of scale in both model size and pretraining dataset size has demonstrated strong capabilities by achieving reasonable performance through a learnable prompt appended before the input [Li and Liang, 2021, Lester et al., 2021]. Despite this, several questions emanate around the abilities and limitations of prompt tuning.

In this work, we aim to characterize some natural yet essential questions about prompt tuning with transformer architectures. Firstly, are prompts universal approximators, i.e. with a fixed pretrained transformer network, can we find a prompt to approximate any sequence-to-sequence function in a given space? If yes, can we construct the transformer for this universality result? Second, can we identify failure modes of prompt tuning when applied on potentially non-optimal but non-trivial transformers? Moreover, since prompt tuning is usually compared against LoRA[Hu et al., 2021] in consideration to parameter-efficient tuning, is prompt tuning then more/less parameter-efficient than LoRA? Answering these questions can lead to important insights on when and how to perform prompt tuning to adapt a pretrained transformer network to a given downstream task of interest.

In this work, we seek to answer these questions with appropriate theoretical analysis and further validate our claims with empirical results. We first characterize the universal nature of prompt tuning by constructing a specific transformer network. We show that for a given approximation error and the space of sequence-to-sequence Lipschitz functions, we can construct a transformer network, with a suitable number of layers, that can leverage prompt tuning to approximate any function in this space. Despite this universality of prompt tuning with a carefully constructed pretrained transformer, we then identify some limitations of prompt tuning with weaker but non-trivial transformers. We prove this by constructing sequence-to-sequence datasets with shared input tokens, which are surprisingly simple but cannot be memorized by prompt tuning for a given transformer. We also extend our analysis to more general settings where the shared token is not required. In this setting, we first prove that prompt tuning on a single-layer transformer requires Ω( n ) trainable parameters to memorize n training examples, wherein for LoRA, it suffices with O ( n ) trainable parameters. We finally extend our analysis to the multi-layer setting and provide sufficient conditions under which prompt tuning exhibits extremely limited capacity to at best memorizing datasets from invertible functions.

Our contributions can be summarized as below:

Wecharacterize the universal nature of prompt tuning by explicitly constructing a transformer network (Theorem 1).

We provide a construction-based argument for sequence-to-sequence datasets that cannot be learned by prompt tuning with a given single-layer transformer (Theorem 2).

We provide the lower bound on the required number of parameters for prompt tuning to memorize any sequence-to-sequence functions (Theorem 3).

We provide a sufficient condition for multi-layer transformers, under which datasets with shared output tokens cannot be learned with prompt tuning (Theorem 4).

We conduct empirical studies, including real-world datasets, to verify our theoretical claims.

2 Related Work

Theoretical Analysis of Transformers Various works have characterized the theoretical properties of transformers and its primary self-attention component. Yun et al. [2020] studies the universal approximation ability of transformers for continuous permutation equivariant sequence-to-sequence functions with compact support and further examined the role of using positional encodings to circumvent permutation equivariant condition. Pérez et al. [2021] show that transformer with a hardattention is Turing complete based on their capacity to perform computations and access the internal dense representations of the data. Wei et al. [2021] further shows that transformers can approximate Turing machines with bounded computation time with a new notion of approximation. [Dong et al., 2021] provides a negative yet interesting result signifying the limitations of pure self-attention in terms of rank diminishing of the input. Other works including Kim et al. [2021], Dasoulas et al. [2021] derive upper bounds on the Lipschitz constant of respective modifications of the attention mechanism. The works by Li et al. [2022], Zhang et al. [2020] documented optimization perspective on transformer training via SGD.

Fine-tuning and Prompt Tuning Fine-tuning is the standard way to adapt a pretrained model to downstream tasks. The most standard and popular paradigm is tuning the model weights via a suitable optimization procedure along with a linear head on the output representations [Radford et al., 2018, Devlin et al., 2019]. Subsequent works studied more parameter-efficient ways of fine-tuning by updating either a subset of model parameters [Ben Zaken et al., 2022] or restricting the parameterupdates to a low-dimensional subspace [Aghajanyan et al., 2021, Hu et al., 2021, Mahabadi et al., 2021]. The work by Hu et al. [2021] (their framework referred to as LoRA) has garnered particular interest in the community and Malladi et al. [2023] has provided an interpretation of LoRA via the kernel mechanism. In the particular context of LLMs, prompt tuning has emerged as the de facto approach where only the prompt is updated while keeping the rest of the transformer weights and architecture fixed [Shin et al., 2020, Lester et al., 2021, Li and Liang, 2021].

Analysis of Prompt Tuning Wei et al. [2022] studies the link between prompt tuning and downstream tasks with an underlying latent variable generative model of text, which is confined to a Hidden Markov Model. However, they focused on the discrete vocabulary setting contrary to our

results for continuous sequence-to-sequence functions. Some more recent works [Akyürek et al., 2023, Von Oswald et al., 2023] characterize an intriguing property of a specific form of prompting, referred to as in-context learning, where they proved by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. This work however pursued a different and specific direction from the prompting results we aim to provide for generic settings.

Memorization Capacity of Neural Networks A series of works have sought to provide finite sample universal memorization capacity results of neural networks and the understanding of expressive power of neural networks. Huang and Huang [1990], Huang and Babri [1998], Huang [2003], Yamasaki [1993] analyze the memorization capacity of FNNs with sigmoid and other bounded activation functions. Hardt and Ma [2016], Zhang et al. [2021], Nguyen and Hein [2018] provide results for modern ReLU networks including FNNs and CNNs. For transformer architectures, Kim et al. prove that transformers can memorize a dataset with finite parameters. To the best of our knowledge, similar results for prompt tuning have not been studied in continuous settings for transformer architectures.

3 Transformers and Parameter Efficient Training

3.1 Preliminaries

We use the following notations throughout the paper. A bold lower case character, e.g. x , denotes a vector. A bold upper case character, e.g. W , denotes a matrix while W i,j , W i, : and W : ,j is the ( i, j ) -th element, i -th row, j -th column, respectively. We use a single superscript or subscript to denote the index of a matrix, e.g. X i , X i denote the i -th matrix in a matrices sequence. We use σ and ¯ σ for softmax and hardmax operators, respectively. We use ReLU ( v ) = max( v , 0 ) to denote the ReLU activation function where max( · ) function is applied entry-wise to a vector. We use Cone ( a 1 , a 2 , ..., a m ) to denote a cone without its origin point where Cone ( a 1 , a 2 , ..., a m ) = { x : x = ∑ m i =1 a i a i , a i > 0 } . We also define the minus operation between a set S and a vector v as S -v = { x -v : x ∈ S } . In Section 4, we use [ a : b : c ] to denote a grid { a, a + b, a +2 b, ..., c -b } from a to c , with an interval b .

Transformer networks [Vaswani et al., 2017] are a stack of multiple transformer layers, composed subsequently. A transformer layer has two key components: an attention layer and a token-wise MLP layer, with residual connections around both blocks. We consider the input and output to be sequences of tokens X ∈ R d × m and Y ∈ R d × m , where m is the number of tokens in the sequence and d is the token dimension.

Definition 1 (Attention Layer) . We define an h -head attention layer parameterized with W q , W k , W v , W o between a single token x and a token sequence X as

The normalizing factor of 1 √ d kq is subsumed in the weight matrices W i k for notational simplicity.

We can then define the cross attention between two sequences X 1 ∈ R d × m 1 and X 2 ∈ R d × m 2 (We use x k = ( X 1 ) : ,k for simplicity):

Definition 2 (Standard Transformer Layer) . With definition 1, we define a standard transformer layer τ as

The definition here omits the layer normalization block for simplicity (following [Kim et al., 2021]).

We denote the set of transformer networks with h heads of size s and r MLP hidden neurons with T h,s,r . In Section 4, we utilize a modified transformer network with hardmax operation ¯ σ instead of softmax σ . We denote this modified version of transformer networks as ¯ T h,s,r .

During fine-tuning, we optimize the matrices W i q , W i k , W i v in the attention layer and W 1 , W 2 , b 1 , b 2 in the MLP layer pertaining to a loss function L . However in prompt tuning, the pretrained model weight matrices are fixed and we optimize a tunable sequence prepended to the input.

Prompt Tuning Given a pretrained transformer network g ∈ T and a downstream training dataset S = { ( X 1 , Y 1 ) , ..., ( X n , Y n ) } , prompt tuning seeks to find a prompt P ∗ ∈ R d × m p with m p tunable tokens under the loss function L :

The tunable prompt P is shared amongst all the inputs in a task. Note that P in prompt tuning is a continuously trainable parameter, alternately referred to as soft prompt, which is different from hard prompt in that the latter operates on a discrete space of predefined vocabulary. Since the representation power of soft prompts is strictly more than the hard prompts, the limitations studied in this paper also extend to hard prompts.

In the subsequent sections, we analyze the universality and limitations of prompt tuning while comparing the latter against fine-tuning and LoRA[Hu et al., 2021], which is a low-rank version of model fine-tuning. In Section 4, we prove that prompt tuning can be universal approximators for sequence-to-sequence functions, while providing the construction for the same. In Sections 5 and 6, we identify the failure modes where prompt tuning cannot learn with a possibly non-optimal but non-trivial pretrained transformer network.

4 Universality of Prompt Tuning

Without loss of generality, we assume that the support and range set of all considered sequence-tosequence functions f is [0 , 1] d × m in this section. We define F L as the collection of all continuous sequence-to-sequence L -lipschitz functions under norm p and sequence length m . For f ∈ F L and any two inputs X , X ′ ∈ [0 , 1] d × m , we have ∥ f ( X ) -f ( X ′ ) ∥ p ≤ L ∥ X -X ′ ∥ p . Furthermore, given functions f 1 , f 2 , the approximation error under a p -norm (which is entry-wise) is measured as:

Primarily, we show that there exists a Transformer network g ∈ T 2 , 1 , 4 such that for any f ∈ F L , prompt tuning on g can approximate this function upto some error budget ϵ > 0 .

Theorem 1. Let 1 ≤ p < ∞ and ϵ > 0 , there exist a transformer network g ∈ T 2 , 1 , 4 and prompt length m p , such that for any f ∈ F L we can find a prompt P ∈ R d × m p with d p ( g ([ P , · ]) : ,m p : , f ) ≤ ϵ .

Here we use the transformer in a encoder mode which generates the m outputs in one step. In Appendix C.4, a similar result can be obtained for next-token prediction, which is widely used in many recent language models.

The proof is inspired from [Yun et al., 2019a], which follows the typical construction based proof mechanism to show universality. Thereby, we can construct a 'meta-transformer' for prompt tuning to approximate any sequence-to-sequence function with prompt tuning. Next we briefly describe the two steps for the construction of this meta-transformer. We start by building a meta-function for F L .

Building the Meta-Function We denote the length of all inputs as m and the prompt length as m p . Then we can build a sequence-to-sequence meta-function that accepts inputs with length m + m p .

Lemma 1. For the sequence-to-sequence function space F L with functions f : [0 , 1] d × m → [0 , 1] d × m , we can build a sequence-to-sequence function ¯ g : [0 , 1] d × ( m p + m ) → [0 , 1] d × ( m p + m ) such that for any f ∈ F L , we can find P ∈ R d × m p , d p (¯ g ([ P , · ]) : ,m p : , f ) ≤ ϵ/ 2 .

The complete proof is given in Appendix C.1. Succinctly, we first quantize the input and output sequence space of [0 , 1] d × m into a grid G δ,m = { 0 , δ, 2 δ, ..., 1 -δ } d × m , thus leading to C = ( 1 δ d × m ) 1 δ d × m possible functions mappings from the input to the output, in this discrete space. By

this quantized function space as ¯ F L = { ¯ f 1 , ¯ f 2 , ..., ¯ f C } , we can select δ such that the approximation error for any function is less than ϵ/ 2 . Then we construct a set of quantized prompts in G δ,m p = { 0 , δ, 2 δ, ..., 1 -δ } d × m p to index these C functions and construct a quantized function ¯ g where ¯ g ([ P i , X ]) : ,m p : = ¯ f i ( X ) , i = 1 , 2 , ..., C , for all X ∈ G δ,m , thereby concluding the lemma.

Next we can utilize some conclusions in [Yun et al., 2019a] to construct a transformer for ¯ g .

Constructing the Meta-Transformer We first introduce a useful lemma which enables the construction of a transformer for any quantized sequence-to-sequence function.

The proof mainly follows the discussions in Section C of [Yun et al., 2019a]. To prove this lemma, the network ¯ h can be constructed in the following three steps. We first use a series of MLP layers to quantize the input to grid [0 : δ : 1 -δ ] d × m and then a series of attention layers to obtain a unique contextual mapping for each quantized input. Finally we can use a series of MLP layers to map the unique contextual mapping to the desired outputs. While a transformer network usually stacks self-attention and MLP layers alternately within a single layer, the aforementioned construction can be trivially attained via the use of skip connections. The complete proof of Lemma 2 is deferred to Appendix C.2.

Since ¯ g is a quantized function in grid G δ,m + m p , following Lemma 2 we can find a modified version of transformer ¯ h ∈ ¯ T 2 , 1 , 1 such that ¯ g ([ P , X ]) = ¯ h ([ P , X ])) . The modified version of transformer ¯ g with hardmax operators can then be approximated with a standard transformer g with softmax operators by Lemma 3.

Lemma 3 (Lemma 9 in [Yun et al., 2019a]) . For each ¯ h ∈ ¯ T 2 , 1 , 1 , ϵ > 0 and 1 ≤ p < ∞ , ∃ g ∈ T 2 , 1 , 4 such that d p ( ¯ h, g ) ≤ ϵ/ 2 .

Since the approximation error can be treated uniformly amongst the P i , we have that d p ( ¯ h ([ P i , · ]) : ,m p : , g ([ P i , · ]) : ,m p : ) ≤ d p ( ¯ h ([ P i , · ]) , g ([ P i , · ]) ≤ ϵ/ 2 . Therefore, we can build a transformer g ∈ T 2 , 1 , 4 , such that for any sequence-to-sequence f ∈ F L , we can find a quantized version ¯ f i ∈ ¯ F L and the corresponding prompt P i ∈ G δ,m p such that

Theorem 1 provides the construction for a large transformer (discussed more in appendix) that is sufficient for prompt tuning to exhibit universal approximation over a Lipschitz function space. However, even this strong transformer also has limitations with prompt tuning when the target function f / ∈ F L . Is this an essential limitation for prompt tuning on any transformer? In the next section, we will theoretically analyze the limitations of prompt tuning with transformers and target functions under more general conditions.

5 Limitations of Prompt-Tuning: Single Layer Transformer

To analyse the failure modes and therefore the limitations under the setting where a transformer has fixed pretrained weights, we follow the lens of exact memorization in the subsequent sections.

Definition 3 (Memorization of a Sequence-to-Sequence Dataset) . Given a sequence-to-sequence dataset S = { ( X 1 , Y 1 ) , ..., ( X n , Y n ) } where X i , Y i ∈ R d × m are the input/output sequences, we consider a function f exactly memorizing dataset S if f ( X i ) = Y i . In the following proofs of this section, we explicitly focus on the last output token, ie: f ( X i ) : , -1 = ( Y i ) : , -1 .

We start from the analysis on a single layer transformer and extend to multi-layer settings in Section 6.

5.1 Failure modes of Prompt Tuning

It is straightforward to note that prompt tuning has limited expressive power when the number of trainable parameters is limited. A natural question to then ask is: Does increasing the number of trainable prompt tokens suffice? While it is known that for MLPs, even with a single hidden layer, increasing the number of hidden neurons can memorize any training data [Yun et al., 2019b]. However, as we will prove next, this is not the case for prompt tuning. This result highlights an essential limitation of prompt tuning compared to model fine-tuning.

Before providing the theorem statement, we first outline some straightforward assumptions on the pretrained transformer and datasets, without which prompt tuning trivial loses expressive power.

We consider sequence-to-sequence datasets of the form S = { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , ..., ( X n , Y n ) } with n distinct examples and a single-layer single-head standard transformer defined in Definition 2. The results can be directly extended to the single-layer multi-head scenario, which we skip here to avoid notational clutter.

Assumption 1 (Non-trivial conditions) . We assume that all output tokens ( Y i ) : ,k are in the range set of MLP , otherwise the expressivity becomes trivially weak. We assume that W q , W k , W v are full rank matrices and that Att ( X i , X i ) + X i are distinct for i = 1 , 2 , ..., n .

Assumption 2 (Assumption for the MLP layer) . We assume that d ≥ 2 + dim (( MLP -1 ( y 10 ) -x 0 ) ∪ ( MLP -1 ( y 20 ) -x 0 )) for the dataset constructed in Theorem 2 and token dimension d . dim ( S ) measures the dimension of subspace spanned by vectors in a set S and MLP -1 ( y ) = { x : MLP ( x ) = y } .

We provide an example for this assumption in Example 1 and a sufficient condition in the following Lemma 4.

Lemma 4. If ∥ W 1 ∥ 2 ×∥ W 2 ∥ 2 < 1 , where ∥ · ∥ 2 is the matrix spectral norm, then the MLP block in Definition 2 is invertible, ie, MLP -1 is a singleton set.

Therefore, if Lemma 4 holds and d ≥ 4 , Assumption 2 also holds.

Proof of Lemma 4 can be found in Appendix C.5. The experimental evidence in [Dong et al., 2021] shows that for most architectures, the norm of the weight matrices indeed admits small values and thus the requirement that ∥ W 1 ∥ 2 ×∥ W 2 ∥ 2 < 1 is a mild condition.

With these assumptions, here we introduce our first theorem on the unlearnability of prompt tuning.

Theorem 2. For a single layer transformer τ defined above with Assumptions 1 and 2, we can build a sequence-to-sequence dataset S = { ( X 1 = [ x 1 , x 0 ] , Y 1 = [ y 11 , y 10 ]) , ( X 2 = [ x 2 , x 0 ] , Y 2 = [ y 21 , y 20 ])) } , and we cannot find a prompt P ∈ R d × m p with any m p > 0 such that τ ([ P , X i ]) = Y i holds for any i = 1 , 2 . The vectors x 0 , x 1 , x 2 are denoted post positional encodings.

An important feature of this dataset is that the same token x 0 is shared between the two examples, and the expressive capability of prompt tuning is limited by the correlation of outputs corresponding to this token in different examples. We show a concrete example here to illustrate this theorem (note that Lemma 4 is in fact not required in the following construction) and defer the formal proof to Appendix C.6.

Example 1. We consider a single-head transformer layer τ , where b 1 = b 2 = 0 , W 1 = 1 r × d , W 2 = 1 d × r . Then the token-wise MLP layer is a concatenation of two linear functions:

Here ( W 1 x ) 0 denotes the first element of vector W 1 x .

W 2 W 1 + I is a non-singular matrix. Therefore, for any y in MLP ( X ) 's output set, MLP -1 ( y ) contains at most two points { y , ( W 2 W 1 + I ) -1 y } . We arbitrarily choose x 0 , y 10 and y 20 .

As long as d ≥ 6 (from Assumption 2), we can find c 1 , c 2 such that c 1 , c 2 ⊥ y 10 -x 0 , y 20 -x 0 , ( W 2 W 1 + I ) -1 y 10 -x 0 , ( W 2 W 1 + I ) -1 y 20 -x 0 , c 1 ⊥ c 2 . Then we choose x 1 and x 2 such that Att ( x 0 , X 1 ) ∥ c 1 and Att ( x 0 , X 2 ) ∥ c 2 (Lemma 7 in Appendix). Then Cone ( -Att ( x 0 , X 1 ) , a -x 0 ) ∩ Cone ( -Att ( x 0 , X 2 ) , b -x 0 ) = ∅ , for any a ∈ { y 10 , ( W 2 W 1 + I ) -1 y 10 } and b ∈ { y 20 , ( W 2 W 1 + I ) -1 y 20 } . Here Cone stands for a convex cone as defined in Section 3.1.

If a P exists such that τ ([ P , X i ]) = Y i holds for both i = 1 , 2 , then we have

where λ ( · , · , · ) is a positive scalar. We also have

Therefore, Att ( x 0 , P ) must be in both Cone ( a -x 0 , -Att ( x 0 , X 1 )) and Cone ( b -x 0 , -Att ( x 0 , X 2 )) , where a ∈ { y 10 , ( W 2 W 1 + I ) -1 y 10 } and b ∈ { y 20 , ( W 2 W 1 + I ) -1 y 20 } , which contradicts the existence of P as Cone ( -Att ( x 0 , X 1 ) , a -x 0 ) ∩ Cone ( -Att ( x 0 , X 2 ) , b -x 0 ) = ∅ . Therefore, in this example, even though we allow an arbitrary number of trainable parameters in prompt P , we cannot find one to exactly memorize the training set with only two training examples.

This theorem reveals an important difference between prompt tuning and adjusting the model weights directly. For any training dataset S with two training examples { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) } , so long as Att ( X 1 , X 1 ) + X 1 and Att ( X 2 , X 2 ) + X 2 are distinct, MLP can easily map the post-attention features to expected output tokens with finite number of hidden neurons. As a result, tuning the MLP parameters for this pretrained transformers can memorize any dataset in the form of Assumption 1. However, prompt tuning cannot achieve this even if the number of tunable tokens → infinity, thereby limiting the expressiveness of prompt tuning when compared to model fine-tuning.

5.2 Comparison with a More General Dataset

In Section 5.1, we constructed sequence-to-sequence datasets that cannot be learned by a given transformer layer with prompt tuning, by utilizing the shared token between different training examples. In this section, we compare the expressive power of prompt tuning and fine-tuning under a more general dataset construction where the former requirement can be relaxed.

Since the primary essence of prompt tuning is to perform parameter-efficient tuning, wherein we seek to adapt a pretrained large model to a new task with fewer tunable parameters, we compare prompt tuning with another parameter-efficient version of model-tuning: LoRA [Hu et al., 2021]. Succinctly, we compare the required number of parameters to memorize a given dataset. Again, consider a sequence-to-sequence dataset S = { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , ..., ( X n , Y n ) } , where X i = [ x i 1 , x i 2 , ..., x im ] and Y i = [ y i 1 , y i 2 , ..., y im ] . We again discuss the memorization of the last output token for simplicity and results can be directly extended.

We first give the required number of parameters of LoRA to memorize dataset S .

Lemma 5 (LoRA) . For a standard single-layer transformer τ defined in Definition 2 with r ≥ n MLP hidden neurons, for any sequence-to-sequence dataset S satisfying Assumptions 1, we can apply a low-rank update to MLP weights with O ( nd ) parameters to memorize τ ( X i ) : ,m = y im .

This lemma is derived based on the memorization capabilities of 1-hidden layer MLPs [Yun et al., 2019b]. As the post-attention values for different training inputs are different from Assumption 1, we can construct a low rank update with O ( nd ) parameters on the MLP layer to memorize S . We defer the complete proof to Appendix C.7.

For prompt tuning, we derive a result in the next theorem which shows that it requires Ω( nd ) tunable parameters to memorize some constructed dataset S with n examples.

Theorem 3 (Lower bound on Tunable Prompt Parameters) . For any single layer transformer τ defined in Definition 2, there exists a sequence-to-sequence dataset { ( X 1 = [ x 10 , x 1 ] , [ y 10 , y 11 ]) , ( X 2 = [ x 20 , x 2 ] , [ y 20 , y 21 ]) , ..., ( X n = [ x n 0 , x n ] , [ y n 0 , y n 1 ]) } that satisfies Assumption 1 with n < d training examples such that we need at least n prompt tokens in P to memorize the training set, ie, for τ ([ P , X i ]) : , -1 = y i 1 to hold for all i = 1 , 2 , ..., n .

This dataset can be constructed by including n examples that require n linearly independent prompts tokens. The complete proof is deferred to Appendix C.8.

Note that in Theorem 3, we provide a key lower bound on the required number of prompt tokens for exact memorization and this can very well more than nd . This partially (but not necessarily) explains the worse empirical performance of prompt tuning against LoRA under a comparable number of trainable parameters.

6 Extension to Multi-Layer Setting

In this section, we extend our analysis to multi-layer setting and provide a sufficient condition under which the expressiveness of prompt tuning is restricted. An immediate consequence of our result is an interesting connection to the spectral norm of soft prompts surfaces. This result provides us a partial understanding of the phenomenon that soft prompt P vectors typically exhibit larger norms compared to the actual input X , after the tuning.

With some further notation adjustments, we denote an H layer pretrained transformer network as g ( ∈ T ) = τ 1 ◦ τ 2 ◦ ... ◦ τ H , the input set as X 1 , and the set of possible prompts as P 1 . We assume that the following compactness condition is satisfied:

Here [ P 1 , X 1 ] is the input to the first layer τ 1 with P 1 ∈ P 1 , X 1 ∈ X 1 and ∥ · ∥ 2 is the spectral norm. Similarly, [ P H +1 , X H +1 ] denotes the output set.

We start by providing an upper bound to the Lipschitz constant of attention, pertaining to eq 9. This derivation is different from the works of [Dasoulas et al., 2021, Vuckovic et al., 2020] and thus can be of independent interest.

Lemma 6. Under the compactness condition, the Lipschitz constant of the i -th attention head in the l -th transformer layer, denoted for simplicity as Att i,l , admits the following bound w.r.t the entire input sequence of length m :

and the Lipschitz constant of the entire attention block in layer l , denoted as Att l , admits the bound:

It is noteworthy that this upper bound is dependent on D l , the spectral norm of the input prepended with the prompt. In conjunction with the following theorem, we obtain a result on limited expressivity of prompt tuning by showing that the transformer becomes invertible, in consideration to functions from P 1 ×X 1 →P H +1 ×X H +1 (an extension to functions of the from X 1 →X H +1 is provided in Appendix Section C.11).

Theorem 4. A transformer g ∈ T is invertible, ie , g -1 ( Y ) = { X : g ( X ) = Y } is a singleton set ∀ Y in range of g , if:

The Lipschitz constant of the attention block in each layer τ l is strictly less than 1

The Lipschitz constant of the 2-layer ReLU block in each layer τ l , which is bounded by ∥ W l 2 ∥ 2 ×∥ W l 1 ∥ 2 , is strictly less than 1.

Proof of Theorem 4 can be found in Appendix C.9. Combining Lemma 6 and Theorem 4, we observe that the invertibility is guaranteed if the upper bound for the Lipschitz constant of the attention, eq 11, and the MLP layer, is strictly less than 1. In this case, we can then construct arbitrarily many datasets where two different inputs share the same output, and prompt tuning cannot learn (more subtly: memorize) these datasets with a restricted prompt norm.

7 Experiments

7.1 Experimental Settings

In Section 7.2, we use a standard single-layer single-head transformer from Definition 2, to justify the infinite prompt-length limitation. In Section 7.3, we justify the increasing prompt norm on the pretrained LLaMA 7B model [Touvron et al., 2023]. For prompt tuning and LoRA, we use the Huggingface Peft library [Mangrulkar et al., 2022]. On the dataset front, we utilize the RTE subtask of SuperGlue dataset [Wang et al., 2019] and WMT14 En-Fr translation [Bojar et al., 2014]. More details and hyperparameter settings can be found in Appendix A.

7.2 Limited Expressivity of Infinite Length Prompt

We first construct the dataset following the proof of Theorem 2 and then show that prompt tuning cannot memorize this simple dataset { ( X 1 = [ x 1 , x 0 ] , Y 1 = [ y 11 , y 10 ]) , ( X 2 = [ x 2 , x 0 ] , Y 2 = [ y 21 , y 20 ]) } even with very large prompt lengths.

We set the token dimension d = 10 . We follow the default pytorch weight initialization and then normalize W 1 , W 2 such that ∥ W 2 ∥ 2 ×∥ W 1 ∥ 2 < 1 , following Assumption 2. We randomly sample x 0 , y 10 , y 20 in a uniform distribution in [0 , 1) d and construct the corresponding vectors: x 1 and x 2 following Theorem 2. To compute MLP -1 ( y ) , we follow [Kim et al., 2021] Section 4.1 with 5000 iterations at convergence. We solve Att ( x 0 , [ x 0 , x 1 ]) ∥ c in Lemma 7 with gradient descent terminating at ∠ ( Att ( x 0 , [ x 0 , x 1 ]) , c ) < 0 . 0001 . We repeat this setup to obtain 3 different datasets for distinct x 0 , y 10 , y 20 and denote these with S i , i = 1 , 2 , 3 .

We perform prompt tuning, MLP fine-tuning and MLP LoRA training on the constructed datasets for 5 runs and report the mean and standard deviation of per-element Mean Squared Error (MSE) loss y 10 , y 20 at convergence. We show the comparison between prompt-tuning and MLP fine-tuning in Figure 1. As we can observe from the figure, increasing the number of soft prompt tokens post a certain threshold that does not exhibit any reduction in MSE. On the contrary, fine-tuning on the MLP layer tend to easily memorize the training set by reducing the training loss to almost zero (all the three curves for fine-tuning overlap and thus not differentiated). Note that we plot the standard deviation, however it is negligible in the range. Similar to fine-tuning on the MLP layer, LoRA with width 2 on the MLP layer also achieves near-zero training loss which is less than 10 -10 on the constructed dataset. We don't plot the comparison on Figure 1 as all the six curves are overlapped). This result validates our Theorem 3 that LoRA can memorize a dataset with n examples with trainable parameters O ( n ) while prompt-tuning may require more.

7.3 Increasing Prompt Spectral Norm during Tuning

As discussed in Section 6, a major constraint on the expressive power of prompt tuning is the spectral norm of soft prompts. In Figure 2, we plot the curve for spectral norm of soft prompt as training progresses and the loss reduces on RTE dataset. The curve for WMT14 En-Fr dataset can be found in Appendix B. This trend clearly highlights that in order to counter the limit on the capacity, the spectral norm consistently increases till the training loss saturates.

8 Conclusions

In this work, we embark on exploring the capabilities of prompt tuning in the continuous regime, contrasting it with fine-tuning, as an initial endeavor towards a theoretical comprehension. We prove by construction that prompt tuning admits universal approximation within the space of Lipschitz functions. Additionally, we identified inherent limitations of prompt tuning on single-layer transformers by constructing theoretically difficult datasets for prompt tuning. These limitations are then extended to multi-layer setting under a specific prompt-norm restriction.

From the analysis in Theorem 2 and 3, we note that the limitation of prompt-tuning primarily arises from the correlation across different inputs. Broadly describing, prompt-tuning implements transformation on different inputs via 'additional attention values', which is more restrictive as compared to the transformations from MLP layers on input tokens. An interesting potential direction to improve prompt-tuning is: 'designing a mechanism to leverage prompting in order to generate

10

100

1000

5000

Length of Soft Prompt

0.00

0.02

0.04

0.06

0.08

0.10

Training MSE loss

Prompt Tuning on

S

1

Fine-tuning on

S

1

Prompt Tuning on

S

2

Fine-tuning on

S

2

Prompt Tuning on

S

3

Fine-tuning on

S

3

20

40

60

80

100

120

140

160

180

200

Training Steps

2.7

2.8

2.9

3.0

3.1

3.2

3.3

3.4

3.5

Prompt Spectral Norm

Prompt Spectral Norm

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Validation Loss

Validation Loss

prompt-dependent adapter/LoRA updates'. We expect to have some future work focusing on designing novel prompt-tuning strategies along this direction.

Limitations While our results provide valuable insights, extending the construction in Theorem 2 to multiple layers and deriving tighter bounds for Lemma 6 are critical steps for a deeper understanding of the limitations of prompt tuning.

Acknowledgments and Disclosure of Funding

We thank the reviewers for their invaluable feedbacks. The work is supported in part by NSF 2008173, 2048280, 2325121, 2331966, ONR N00014-23-1-2300:P00001.

References

Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long.568 .

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= 0g0X4H8yN4I .

Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 573-582. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/behrmann19a.html .

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-short.1. URL https://aclanthology.org/2022.acl-short.1 .

Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation , pages 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http://www. aclweb.org/anthology/W/W14/W14-3302 .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.

George Dasoulas, Kevin Scaman, and Aladin Virmaux. Lipschitz normalization for self-attention layers with application to graph neural networks, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https: //aclanthology.org/N19-1423 .

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2793-2803. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/dong21a.html .

Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.04231 , 2016.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Guang-Bin Huang. Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE transactions on neural networks , 14(2):274-281, 2003.

Guang-Bin Huang and Haroon A Babri. Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE transactions on neural networks , 9(1):224-229, 1998.

S-C Huang and Y-F Huang. Bounds on number of hidden neurons of multilayer perceptrons in classification and recognition. In 1990 IEEE International Symposium on Circuits and Systems (ISCAS) , pages 2500-2503. IEEE, 1990.

Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention, 2021.

Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations .

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.243. URL https: //aclanthology.org/2021.emnlp-main.243 .

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.353. URL https://aclanthology.org/2021.acl-long.353 .

Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12656-12684. PMLR, 17-23 Jul 2022. URL https://proceedings. mlr.press/v162/li22b.html .

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning, 2023.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Belkada Younes, and Paul Sayak. Peft: Stateof-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft , 2022.

Quynh Nguyen and Matthias Hein. Optimization landscape and expressivity of deep cnns. In International conference on machine learning , pages 3730-3739. PMLR, 2018.

Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing-complete. Journal of Machine Learning Research , 22(75):1-35, 2021. URL http://jmlr.org/papers/v22/20-302.html .

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023.

James Vuckovic, Aristide Baratin, and Remi Tachet des Combes. A mathematical theory of attention, 2020.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems , 32, 2019.

Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. CoRR , abs/2107.13163, 2021. URL https: //arxiv.org/abs/2107.13163 .

Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning, 2022.

Masami Yamasaki. The lower bound of the capacity for a neural network with multiple hidden layers. In ICANN'93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3 , pages 546-549. Springer, 1993.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 , 2019a.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memorizers: a tight analysis of memorization capacity. Advances in Neural Information Processing Systems , 32, 2019b.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=ByxRM0Ntvr .

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?, 2020.

A Experimental Details

All the experiments are run on a NVIDIA RTX A6000 GPU. For experiments with Llama 7B model, we use batch size 32 and learning rate 0.001. For experiment on WMT14 En-Fr translation, we only compute the loss on the first 100 examples for computational efficiency.

We use Adam optimizer and optimal learning rate from grid search at 0.1 for prompt-tuning and at 0.001 for fine-tuning in Section 7.2.

In Section 7.3, we use the default loss function in Huggingface implementation for causal language models. We use prompt length m = 10 and the prompt tokens are initialized as the first m tokens in the model vocabulary.

B Additional Experiments

As mentioned in Section 7.3, the second real world dataset used in our experiment is WMT14 En-Fr translation in order to illustrate that the spectral norm of soft prompts increases during training. We show the curve in Figure B.

5

10

15

20

25

30

35

40

45

50

Training Steps

2.350

2.375

2.400

2.425

2.450

2.475

2.500

2.525

Prompt Spectral Norm

Prompt Spectral Norm

0.65

0.70

0.75

0.80

0.85

0.90

Validation Loss

Validation Loss

C Proof of Lemmas and Theorems

C.1 Proof of Lemma 1

For the sequence-to-sequence function space F L with functions f : [0 , 1] d × m → [0 , 1] d × m , we can build a sequence-to-sequence function ¯ g : [0 , 1] d × ( m p + m ) → [0 , 1] d × ( m p + m ) such that for any f ∈ F L , we can find P ∈ R d × m p , d p (¯ g ([ P , · ]) : ,m p : , f ) ≤ ϵ/ 2 .

Proof. we first quantize the input and output sequence space of [0 , 1] d × m into a grid space G δ,m = { 0 , δ, 2 δ, ..., 1 -δ } d × m , which leads to C = ( 1 δ d × m ) 1 δ d × m functions considering all input and output mappings in this grid. We index these C functions as ¯ F L = { ¯ f 1 , ¯ f 2 , ..., ¯ f C } . For X / ∈ G δ,m , we let ¯ f i ( X ) = ¯ f i ( X ∗ ) if k i,j δ < X i,j , X ∗ i,j ≤ ( k i,j +1) δ and X ∗ ∈ G δ,m .

Then for any f ∈ F L , we can find a function ¯ f ∈ ¯ F L such that d p ( ¯ f, f ) = ( ∫ ∥ ¯ f ( X ) -f ( X ) ∥ p p d X ) 1 /p ≤ ( ∫ L p mdδ p d X ) 1 /p = L ( md ) 1 p δ . We choose δ = δ 1 such that L ( md ) 1 p δ ≤ ϵ/ 2 . For the prompt part, we choose m p such that 1 δ d × mp ≥ C . Then we can build a set of quantized prompts in G δ,m p = { 0 , δ, 2 δ, ..., 1 -δ } d × m p to index these C functions. We denote this set of prompts as { P 1 , P 2 , ..., P C } . Finally we can create the quantized function ¯ g and let

¯ g ([ P i , X ]) : ,m p : = ¯ f i ( X ) and ¯ g ([ P i , X ]) : , : m p = 0 , ∀ X ∈ [0 , 1] d × m , P ∈ G δ,m p . For P / ∈ G δ,m p , we set ¯ g ([ P , X ]) = ¯ g ([ P ∗ , X ]) if k i,j δ < P i,j , P ∗ i,j ≤ ( k i,j +1) δ and P ∗ ∈ G δ,m p .

Therefore, with a properly chosen δ = δ 1 , for any f ∈ F L , we can find P ∈ R d × m p such that d P ( f, ¯ g ([ P , · ]) : ,m p : ) = d p ( ¯ f, f ) ≤ ϵ/ 2 .

C.2 Proof of Lemma 2

For any given quantized function ¯ f : [0 , 1] d × m → [0 , 1] d × m with quantization at interval δ , ∃ ¯ h ∈

¯ T 2 , 1 , 1 such that ¯ f = ¯ h with positional embedding E =

Proof. The proof is given following Section C in Yun et al. [2019a] appendix. With Section C.1 in Yun et al. [2019a], there exists a function g q composed of dm δ token-wise feed-forward layers with hidden layer size r = 1 and ReLU activation to implement this scalar quantization on each input element:

Then with Section C.2 in Yun et al. [2019a], we can stack m (1 /δ ) d +1 attention layers to map all possible input sequences in grid [0 : δ : 1 -δ ] d × [1 : δ : 2 -δ ] d × ... × [ m -1 : δ : m -δ ] d to distinct numbers which are at least δ from each other.

Finally we only require O ( m (1 /δ ) dm ) layers to map these distinct numbers to expected outputs.

C.3 Proof of Lemma 3

Lemma 3 is alsmost the same as [Yun et al., 2019a] except that we use ϵ/ 2 instead of ϵ/ 3 .

C.4 Extension of Theorem 1 to Next-token Predictors

As an extension of Theorem 1, we consider approximating a set of sequence-to-sequence functions when we use a transformer layer as a next-token predictor. We consider a set of sequence-to-sequence functions F L with Lipschitz constant L under norm p . f ∈ F L : [0 , 1] d × m 1 → [0 , 1] d × m 2 accepts an input of length m 1 and outputs a sequence of length m 2 . For any X , X ′ ∈ [0 , 1] d × m 1 , we have ∥ f ( X ) -f ( X ′ ) ∥ p ≤ L ∥ X -X ′ ∥ p .

Next we show that we can construct a transformer τ which can approximate any f ∈ F L with prompt-tuning when we use it as a next-token predictor.

Theorem 5. For any f ∈ F L , we can construct a transformer τ such that for any f ∈ F L , 1 ≤ p < ∞ and ϵ > 0 , we can find a prompt P ∈ [0 , 1] d × m p , such that d p ( f, h ( P )) ≤ ϵ , where

τ i is the sequence-to-sequence function implemented with the transformer τ when accepting sequences with length m p + m 1 + i .

Proof. Similar to Theorem 1, we quantize the inputs to grid of [0 : δ : 1 -δ ] with interval δ and set m p = ( δ d × m 2 ) δ d × m 1 . δ is chosen such that L ( m 1 d ) 1 /p δ ≤ m -1 /p 2 ϵ . We index the C = ( δ d × m 2 ) δ d × m 1 different f s as f 1 , f 2 , ..., f C and its sub-function to generate the i -th output token as f j i . The C sequence-to-sequence functions can then be indexed by C distinct prompts. Similar to Lemma 2, we can construct a transformer which can map all possible input sequences in grids [0 : δ : 1 -δ ] × ... × [ m -1 : δ : m -δ ] d , 0 < m ≤ m 1 + m 2 + m p -1 to distinct numbers. A final series of MLP layers then map these distinct numbers to desired output vectors where inputs

in the same grid are mapped to the same output token at each step. Then for any input x ∈ [0 , 1] d × m 1 and any f j , we can find a prompt P j such that

...

Then we have d p ( h ( P j ) , f j ) ≤ ϵ .

C.5 Proof of Lemma 4

If ∥ W 1 ∥ 2 ×∥ W 2 ∥ 2 < 1 , where ∥ · ∥ 2 is the matrix spectral norm, then the MLP block in Definition 2 is invertible, ie, MLP -1 is a singleton set.

Proof. Based on the sufficient conditions for invertibility of a residual block Behrmann et al. [2019], we have that if the feedforward part of a residual block W 2 ReLU ( W 1 X : , 1 + b 1 )+ b 2 is a contraction with respect to some metric, i.e. its Lipschitz constant < 1 , and the metric space on which is defined is complete, then MLP in eq 2 is invertible. Since we are dealing with the euclidean space, any metric induced by the ∥ · ∥ p norm for p ∈ [1 , ∞ ] ensures the space is complete.

The Lipschitz constant of W 2 ReLU ( W 1 x + b 1 )+ b 2 is simply ∥ W 1 ∥ 2 ×∥ W 2 ∥ 2 . Thus the statement of the lemma follows.

C.6 Proof of Theorem 2

For a single layer transformer τ defined above with Assumptions 1 and 2, we can build a seq-to-seq dataset { ( X 1 = [ x 1 , x 0 ] , Y 1 = [ y 11 , y 10 ]) , ( X 2 = [ x 2 , x 0 ] , Y 2 = [ y 21 , y 20 ])) } , and we cannot find a prompt P ∈ R d × m p with any m p > 0 such that τ ([ P , X i ]) = Y i holds for any i = 1 , 2 . The vectors x 0 , x 1 , x 2 are denoted post positional encodings.

Proof. Before proving Theorem 2, we first provide a lemma that will be used in proof and also Theorem 3.

Lemma 7. Given any c ∈ R d × m , there are x 0 almost anywhere for which we can find another vector x 1 ∈ R d × m such that Att ( x 0 , [ x 0 , x 1 ]) ∥ c with full rank attention weights W q , W k , W v .

Proof. If W v x 0 ∥ c , we can just set x 1 = x 0 , which makes Att ( x 0 , [ x 0 , x 1 ]) ∥ c hold.

If W v x 0 ∦ c , let v = α c -W v x 0 where α ∈ R . As W v is full-rank, we can find x such that x = W -1 v v = α W -1 v c -x 0 . Then we will have

Therefore, as long as W q x 0 ̸⊥ W k ( W -1 v c ) , we can change α such that Att ( x 0 , [ x 0 , x 1 ]) = β W v x 0 +(1 -β )( α c -W v x 0 ) where β = exp (( W q x 0 ) ⊤ ( W k x 0 )) exp (( W q x 0 ) ⊤ ( W k x 0 ))+exp (( W q x 0 ) ⊤ ( W k ( α W -1 v c -x 0 ))) . When α = 0 , Att ( x 0 , [ x 0 , x 1 ]) = W v x 0 , when α → -∞ or α → ∞ , Att ( x 0 , [ x 0 , x 1 ]) → α c -W v x 0 . As Att ( x 0 , [ x 0 , x 1 ]) is continuous w.r.t changing α , there must exist an α such that Att ( x 0 , [ x 0 , x 1 ]) ∥ c .

Pass the two input sequences X 1 , X 2 through the attention layer Att with any prompt P , we can get the last output token as:

Here λ ( X 1 , x 2 , X 3 = [ X 1 , X 2 ]) ∈ (0 , 1) is a positive scalar, defined as

x ij is the j th token in X i for notation simplicity.

Then from equation 12, Att ( x 0 , P ) must be on Cone ( -Att ( x 0 , X 1 ) , Att ( x 0 , [ P , X 1 ])) and Cone ( -Att ( x 0 , X 2 ) , Att ( x 0 , [ P , X 2 ])) .

On the otherhand, as we want to memorize the two examples, we must have Att ( x 0 , [ P , X 1 ]) + x 0 ∈ MLP -1 ( y 10 ) and Att ( x 0 , [ P , X 2 ]) + x 0 ∈ MLP -1 ( y 20 ) .

We construct the dataset S with arbitrary x 0 , y 10 and y 20 . Then if dim (( MLP -1 ( y 10 ) -x 0 ) ∪ ( MLP -1 ( y 20 ) -x 0 )) + 2 ≤ d (Assumption 2), we can find two vectors c 1 , c 2 such that c 1 , c 2 ⊥ v : v + x 0 ∈ MLP -1 ( y 10 ) or v + x 0 ∈ MLP -1 ( y 20 ) and c 1 ⊥ c 2 . Then we can choose x 1 , x 2 such that Att ( x 0 , X 1 ) ∥ c 1 and Att ( x 0 , X 2 ) ∥ c 2 (Lemma 7). Combine this construction with assumption 1, we have that Cone ( -Att ( x 0 , X 1 ) , Att ( x 0 , [ P , X 1 ])) and Cone ( -Att ( x 0 , X 2 ) , Att ( x 0 , [ P , X 2 ])) has no intersection, which means that we cannot find a P to memorize this constructed dataset.

C.7 Proof of Lemma 5

For a standard single-layer transformer τ defined in Definition 2 with r ≥ n MLP hidden neurons, for any sequence-to-sequence dataset S satisfying Assumptions 1, we can apply a low-rank update to MLP weights with O ( nd ) parameters to memorize τ ( X i ) : ,m = y im .

Proof. We use MLP j ( x ) to denote the jth output of the MLP layer for an input token x , which is

According to our assumption, Att ( x im , X i ) are unique vectors for i = 1 , 2 , ..., n . Then we only need to use the MLP layer to map each x i = Att ( x im , X i ) + x im to y im , where we get a new token-wise dataset { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) }

Then we need to find w k , a k and b k such that

, which is equivalent to constructing a standard MLP to memorize a dataset:

Follow Thoerem 1 in Yun et al. [2019b], we can construct a , b 1 , ..., b n such that for x 1 , x 2 , ..., x n , we have z i = ⟨ a , x i ⟩ , b 1 < z 1 < b 2 < ... < b n < z n . Then we can find w 1 , ..., w n which solves equation 15. For d -dimension output, we need to find W ∈ R n × d and a ∈ R d and b ∈ R n . With LoRA, we need a low-rank update of size m × n + n × d for W 2 , a low-rank update of size d × n + n × m for W 1 and an update of size n for b 1 , which is O ( n × d ) in total. Normally we have m ≃ d , then we need an update with parameter size around (4 n +1) d to memorize the last token of n training examples.

C.8 Proof of Theorem 3

For any single layer transformer τ defined in Definition 2, there exists a seq-to-seq dataset { ( X 1 = [ x 10 , x 1 ] , [ y 10 , y 11 ]) , ( X 2 = [ x 20 , x 2 ] , [ y 20 , y 21 ]) , ..., ( X n = [ x n 0 , x n ] , [ y n 0 , y n 1 ]) } that satisfies Assumption 1 with n < d training examples such that we need at least n prompt tokens in P to memorize the training set, ie, for τ ([ P , X i ]) : , -1 = y i 1 to hold for all i = 1 , 2 , ..., n .

Proof. Without loss of generality, we assume W 2 has no zero elements, otherwise we can just ignore this hidden neuron in MLP layer.

R d has d bases { t j : j = 1 , 2 , ..., d } , then MLP -1 ( y i 1 ) must be bounded on either positive or negative part of these d directions, which means there exists B ≥ 0 such that

Otherwise ∀ B > 0 , ∃ t j , we can find a v ∈ MLP -1 ( y i ) that v ⊤ t j ∥ t j ∥ ≥ B . Meanwhile we have MLP ( v ) = v + b 2 + W 2 ReLU ( W 1 v + b 1 ) . As ∥ v ∥ can be arbitrarily large, if W 1 v = 0 , ∥ MLP ( v ) ∥ → ∞ if ∥ v ∥ → ∞ . if W 1 v = 0 , ∥ MLP ( v ) ∥ can also be arbitrarily large when increasing the norm of v due to the non-linearity of ReLU ( W 1 v + b 1 ) .

̸

Then we can find a set of n linearly independent vectors { c 1 , c 2 , ..., c n } such that { a i : a i -c i ⊥ c i , a i ∈ MLP -1 ( y i 1 ) } = ∅ by enlarging the norm of c i . With the n c i vectors, we can begin to construct our dataset:

We set x i = c i , i = 1 , 2 , ..., n and find x i 0 such that c i ⊥ Att ( x i , X i ) (Lemma 7) and Att ( x i , X i ) are distinct for i = 1 , 2 , ..., n (Assumption 1), which makes { a 1 -x 1 -λ ( X 1 , x 1 , [ P , X 1 ]) Att ( x 1 , X 1 ) , ..., a n -x n -λ ( X n , x n , [ P , X n ]) Att ( x n , X n ) } linearly independent for any a i ∈ MLP -1 ( y i 1 ) . Here λ ( · , · , · ) is the same as defined in Section C.6.

Moreover, we have

Then Att ( x i , P ) , i = 1 , 2 , ..., n must be n linearly independent vectors, which requires

where A ∈ R m p × n is the attention score matrix between x i and P . P ∈ R d × m p is the prompt token sequence and W v is the attention value weight. Therefore, we must have m p ≥ n .

C.9 Proof of Theorem 4

A transformer T is invertible if:

The Lipschitz constant of the attention block in each layer τ l is strictly less than 1

The Lipschitz constant of the 2-layer ReLU block in each layer τ l , which is bounded by ∥ W l 2 ∥ 2 ×∥ W l 1 ∥ 2 , is strictly less than 1

Proof. This proof is based on the proof provided for lemma 4, thus we restrict ourselves to the sketch: Based on the sufficient condition for invertibility in Behrmann et al. [2019], condition (1) implies that the attention block (eq 1) with the residual connection, ie X + Att ( X , X ) , is an invertible function. Similarly, condition (2) implies that the MLP block which constitutes of the 2-layer ReLU block with the residual connection (eq 2) also exhibit invertibility.

Thus each transformer layer τ l (eq 3) is invertible by noting that its a composition of two invertible functions. The same property ensures that the entire transformer architecture T is also invertible.

C.10 Proof of Lemma 6

Under the compactness condition, the Lipschitz constant of the i -th attention head in the l -th transformer layer, denoted for simplicity as Att i,l , admits the following bound w.r.t the entire input sequence of length m :

and the Lipschitz constant of the entire attention block in layer l , denoted as Att l , admits the bound:

Proof. We drop the superscripts i, l in the proof to avoid notation clutter. Similarly, we denote the concatenation of the prompt matrix P and the original input matrix X , simply with X .

Derivation for single head eq 18:

Consider two matrices X 1 , X 2 ∈ X = { X ∈ R d × m ; ∥ X ∥ 2 ≤ D } . Denote with A 1 , A 2 the corresponding attention matrices respectively, which can be defined as:

The output of the attention head, denoted with Att ( · ) admits the following:

where ( a ) holds from the spectral norm properties and in ( b ) we use the bounded input spectral norm assumptions.

We now focus on the second term ∥ A 1 -A 2 ∥ 2 in eq 25 . From the bound in lemma 9, we have:

where G is the diagonal matrix with entires described in lemma 8

We can now invoke lemma 10 to obtain the following :

Combining the previous inequality with eq 25, we have the following bound:

Derivation for the entire block eq 11:

The proof follows simply by leveraging the following property:

Property: for a matrix C = [ A , B ] , the spectral norm of C admits the bound:

We then simply combine the definition of the attention block and the lipschitz constant bound in eq 18 with the above property in order to obtain the desired bound.

Lemma 8 (Dong et al. [2021] Lemma A.1) . For the column stochastic matrix A 1 obtained by performing column-wise softmax of some matrix Z 1 (where in our setting Z 1 = ( W k X 1 ) ⊤ W q X 1 , and another row stochastic matrix A 2 obtained by performing column-wise softmax of some matrix Z 2 , where Z 2 = Z 1 -E (for some E , which need not belong to X ), we have the following bound:

where the inequality is elementwise and G is a diagonal matrix with entries as G ii = max j,j ′ | δ T i E ( δ T j -δ T j ′ ) | . Here δ k is a one-hot vector with the entry 1 in the k th dimension.

Lemma 9. Following the notations of lemma 8, we have the following spectral norm bound:

Proof. We begin by noting the following entry-wise inequality from eq 29:

which ensures that ∥ A 1 -A 2 ∥ F ≤ 2 ∥ A 2 G ∥ F .

We also have the following using matrix norm equivalence:

Invoking the matrix norm equivalence again, we have that

where rank ( · ) is the matrix rank.

Combining the inequalities, we attain the bound :

since A 2 is column-stochastic , ∥ A 2 ∥ 2 = 1

Lemma 10. The term G in lemma 9 admits the following spectral norm bound:

here D is the previously stated spectral norm bound of the inputs X l ∈ X l .

Proof. We begin by noting that since G is a square diagonal matrix with non-negative real values, the singular values of G are the corresponding diagonal elements.

We thus have that ∥ G ∥ max = ∥ G ∥ 2 , where ∥ · ∥ max is the max norm.

Since G admits the form described in lemma 8, it is trivial to note that:

where the second inequality follows from the matrix norm equivalence.

Now, we can bound the last term ∥ E ∥ 2 by noting that the inputs X belong to a bounded set. This allows us to provide the following bounds:

C.11 Extension of Lemma 6

Lemma 6 and theorem 4 operate over functions from P 1 ×X 1 → P L +1 ×X L +1 . We can relax the requirement of the prompt and provide the Lipschitz constant upper bound in consideration to functions of the form X 1 →X L +1 by using the following assumption:

Assumption 3. Assume for simplicity that ∥ P l 1 -P l 2 ∥ 2 ≤ α l ∥ X l 1 -X l 2 ∥ 2 ; ∀ l ≥ 1 . α l = 0 when l = 1 .

Note: A recursive expression for α l in the above assumption can be provided, but the expression does not admit a simplified form and we thus omit it here.

We will use D l X , akin to eq 9, to denote the compactness corresponding to the input matrix across the layers.

Based on this assumption, we have the following Lipschitz constant upper bound:

Lemma 11. The Lipschitz constant of the single head Att i,l admits the following bound w.r.t the input part, X 1 of length m X , of the input sequence:

For l = 1 , α l = 0 in the above bound.

The Lipschitz constant of the entire attention block in layer l follows similarly.

Proof. For some first layer input X 1 1 and prompt P , let us denote the direct output of the attention head in the l -th layer with - → X l 1 . We have the following update rule for - → X l 1 :

Here, P l 1 is the updated prompt matrix w.r.t the input. For two different inputs X 1 1 and X 1 2 at the first layer, P 1 1 = P 1 2 = P , since the prompt is same across all inputs. A l 1 is then simply the corresponding column-stochastic matrix.

With the context clear, we now drop the superscripts i, l , as done previously. For ∥ - → X 1 - - → X 2 ∥ 2 , we have:

where the second inequality is attained using the property of spectral norm of concatenated matrices.

We now consider the term ∥ A 1 -A 2 ∥ 2 . By invoking lemmas 9 and 10, we have that:

Invoking assumption 3 , ∥ E ∥ 2 can further be bounded as:

Finally, by combining the bound for ∥ E ∥ 2 and assumption 3 with eq 44, we obtain:

which provides us the desired bound.

By setting α l = 0 , the case when there is no prompt, we obtain a similar bound as lemma 6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the theoretical capacity and limitations of prompt tuning. The authors have demonstrated the possibility of constructing a large transformer model that is sufficient for prompt-tuning to exhibit universal approximation over a Lipschitz function space. However, they have also shown the limitations of prompt tuning compared to model fine-tuning by constructing sequence-to-sequence datasets that cannot be learned by prompt-tuning with a given single-layer transformer, even if the soft-prompt embedding size approaches infinity. Additionally, compared with LoRA, prompt-tuning may require more prompt tokens for exact memorization.\n\nSTRENGTHS:\n- Originality\n    - This paper has pretty good originality. It first theoretically studies the theoretical capacity and limitations of prompt tuning.\n\n- Clarity\n    - The structures are clear and conclusions are emphasized.\n\n- Significance\n    - The problem studied in this paper is important\n\nWEAKNESSES:\n- The authors have conducted experiments to support the proposed limitation. However, additional experimental verifications may be necessary to reinforce the theoretical claims and demonstrate the impact on a wider range of tasks and datasets.\n\n- The authors have created sequence-to-sequence datasets that are simple in structure but cannot be memorized by prompt-tuning for a given transformer model. Therefore, they argue that prompt-tuning has limited expressiveness compared to model fine-tuning. However, is memorization typically necessary in real-world applications? Does memorization lead to overfitting? As I understand it, such a phenomenon could also be viewed as a potential advantage in preventing overfitting and achieving generalization ability in the process of downstream adaptation.\n\n- The authors have provided quite a few mathematical proofs and concluded several insights. I wonder whether there are any guidance for practical prompt-tuning applications.  For example, they provide a lower bound on the required number of prompt tokens for exact memorization and point out that such lower bound can be higher than LoRA’s. I would like to know that on which scenarios this could happen and it is recommended to user LoRA rather than prompt-tuning.\n\n- The authors need to correct some minor typos.\n    - line 30, typo:  the or a\n    - line 503, line 508, are they the same reference?\n\nQUESTIONS:\nPlease respond to questions in the \"Weaknesses\" section.\n\nLIMITATIONS:\nsome assumptions are given and not sure whether they are applied in real-world applications", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper wants to provide the insights on \"when and how to perform prompt tuning to adapt a pretrained transformer to downstream tasks\", by theoretically quantifying the universality (i.e., universal approximators) and limitations (i.e., representation capacity) of prompt tuning, in complementing a lot of work focusing on empirically improve soft prompt tuning, such as better initialization or hyper-parameter tuning. Crucially, they show prompt tuning with theoretically very strong transformer models can approximate any seq2seq functions in the set of Lipschitz functions, and they theoretically and empirically show some limitations on weaker transformers (constructed seq2seq datasets as low bounds) by their relatively lower memorization ability.\n\nSTRENGTHS:\n1. This work shows the universality of prompts to the Lipschitz function space, and the limitations of prompt tuning at their memorization ability. The proofs are interesting and well-guaranteed. \n2. The mathematical derivations do show your deliberate thinking and rigorous reasoning. \n3. The paper writing is almost well-structured.\n\nWEAKNESSES:\n1. Just for curious, you define F_L by directly posting the L-lipschitz functions. However, indeed, this causes some unavoidable confusions to readers, as what are \"L-lipschitz functions\"? Can you provide any context? Especially for some highly theoretical papers?\n2. Some writings make me confusing, even though the general structure looks good. \n3. Just one question: Is the first section of universality looks very similar to your cited ICLR work on Transformer's universal approximators?  \n\nI am happy to raise scores if you can make this more clear to me, such that this highly theoretical paper can obtain a more rigorous consideration for its ratings.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on understanding the role and theoretical underpinnings of soft-prompt tuning in transformer-based architectures, which is a technique used to adapt pretrained language models for new tasks. Despite the empirical effectiveness of this method, there's a lack of theoretical understanding of how it differs from tuning the model weights.\n\nThe paper takes significant steps in several areas:\n\n1. **Universal Approximation Analysis:** The researchers prove a universality result that guarantees the existence of a transformer with a prompt that can approximate any sequence-to-sequence function within the set of Lipschitz functions, which are a class of functions that have a particular property of bounded rate of change.\n\n2. **Limitations of Limited-Depth Transformers:** The researchers construct datasets that cannot be memorized by a prompt of any length for a single-layer transformer, demonstrating the inherent limitations of prompt-tuning in this setting. The paper extends the analysis to multi-layer transformer settings, providing conditions under which transformers can only learn from invertible functions. This shows additional limitations of the transformer model.\n\n3. **Lower Bound on Tunable Prompt Parameters:** They establish a lower bound on the required number of tunable prompt parameters and compare this with the parameters needed for a low-rank update (LoRA). This helps to illuminate the resource trade-offs between these two methods.\n\n4. **Empirical Results:** The researchers also provide empirical results that back up their theoretical findings, adding credibility to their conclusions.\n\nSTRENGTHS:\nOne of the strengths of the paper is its theoretical grounding. It provides a theoretical understanding of the difference between 'tuning parameters before the input' and 'the tuning of model weights'. The paper presents a strong universality result, stating the existence of a strong transformer with a prompt that can approximate any sequence-to-sequence function in the set of Lipschitz functions. \n\nIn addition, the paper presents a detailed analysis of the limitations of prompt-tuning, especially for limited-depth transformers. The authors create specific datasets that cannot be memorized by any prompt of any length for a single encoder layer. This analysis provides useful insights into the constraints of prompt-tuning.\n\nThe paper is generally easy-to-follow in idea. Although I do not check the theoretical correctness of these proofs.\n\nWEAKNESSES:\nWhile this paper takes a commendable effort in delving into the theoretical underpinnings of prompt-tuning in transformer-based architectures, it has several significant shortcomings.\n\nFirstly, the focus on sequence-to-sequence encoder models poses a substantial gap in the relevance of these findings to practical, real-world application. The universality of sequence-to-sequence transformers cannot be immediately generalized to decoder models, which are commonly used in practice and can handle an arbitrary number of output tokens. Therefore, the study needs to address how its theoretical results can be extended to these types of models. Without this, the scope of the paper's relevance becomes significantly narrow and its conclusions may not be directly applicable to most use-cases.\n\nSecondly, the reliance on the universality results from Yun et al.'s work (\"Are transformers universal approximators of sequence-to-sequence functions?\") reduces the novelty of the paper. While it is valid to build upon previous work, the authors have not demonstrated a significant extension or new application of these universality findings. It would be beneficial for the paper to provide more novel insights or extend the known results to a more general setting beyond what is already known from the referenced work.\n\nLastly, the exploration of limitations in prompt-tuning is an interesting aspect of the paper. However, the limited breadth of the experiments conducted undermines the impact of these findings. The limitation proofs revolve around specific, constructed single-layer transformers, which do not cover the wide array of transformer architectures used in practice. The implications of these limitations would be far more convincing if the authors could generalize the findings to multi-layer transformers or a wider variety of architectures.\n\nTo improve upon these shortcomings, the authors might consider broadening their analysis to include decoder models, providing more novel extensions to the universality results, and expanding the scope of their limitation analysis to apply to a broader range of transformer architectures.\n\nQUESTIONS:\n1. How to extend the conclusion to decoder models like GPTs that can generate new content?\n\n2. Can you provide some specific examples (with detailed values) for the limitation analysis? E.g. assume the dimension $d = 1$, then x and y are all just numbers. Then I can have a better understanding of why some functions  are not learnable via prompt tuning.\n\nI'm not familiar with theoretical analysis, and skipped the proofs. I will leave the judgement of proofs to my peer reviewers and area chairs. \n\nIn addition, several typos should be fixed:\n\n\"We use the following notations throughout the paper. A bold lower case character, eg x,\": eg --> e.g.\n\n\"If yes, can we construct the a transformer for this universality result?\": the a --> a\n\nLIMITATIONS:\nNot fully addressed. The authors do not mention the limitation in encoder-only models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper presents theoretical analysis of prompt tuning by 1) formally define the attention and prompting operations, 2) deriving the universality of prompt tuning with a strong transformer model, 3) further discussing the limitation of prompt tuning under the case of single and multi-layered transformer models. Explorative experiments on standard models and benchmark datasets demonstrate the correctness of the derived theorems.

STRENGTHS:
- The paper is well written with clearly defined definitions and theorem presentations.
- Exploring prompt tuning from a theoretical perspective is very interesting and important, since better understanding the limitations of prompts is the key to improve prompting techniques.
- The theorems seems correct and solid.

WEAKNESSES:
- The experimental part is a bit weak compared with the theoretical part. 
- It will be nice to have some further discussions on how to leverage the current limitations of prompt tuning to design better prompting-based models.

QUESTIONS:
Please see weaknesses part.

LIMITATIONS:
No limitations or potential negative societal impacts discussed in the main context.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

comments: SUMMARY:
In this work, the authors embark on exploring the capabilities of prompt tuning in the continuous regime,
contrasting it with fine-tuning, as an initial endeavor toward theoretical comprehension. The authors prove
by construction that prompt tuning admits universal approximation within the space of Lipschitz
functions. Additionally, the authors identified inherent limitations of prompt tuning on single-layer transformers by constructing theoretically difficult datasets for prompt tuning. These limitations are then
extended to multi-layer settings under a specific prompt-norm restriction.

STRENGTHS:
1. The authors characterize the universal nature of prompt tuning by explicitly constructing a transformer
network.

2. The authors provide a construction-based argument for sequence-to-sequence datasets that cannot be
learned by prompt tuning with a given single-layer transformer.

3. The authors provide the lower bound on the required number of parameters for prompt tuning to
memorize any sequence-to-sequence functions.

4. The authors provide a sufficient condition for multi-layer transformers, under which datasets with
shared output tokens cannot be learned with prompt tuning.

Overall, this paper is well written. The hypothesis of these theories is acceptable and the theoretical analysis is solid.

WEAKNESSES:
Please add more experiments to support the theoretical result.

QUESTIONS:
N/A

LIMITATIONS:
the authors adequately addressed the limitations in last page.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26