SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1 (Non-trivial conditions). We assume that all output tokens  $(\mathbf{Y}_i)_{:k}$  are in the range set of MLP, otherwise the expressivity becomes trivially weak. We assume that  $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v$  are full rank matrices and that  $A t t(\mathbf{X}_i, \mathbf{X}_i) + \mathbf{X}_i$  are distinct for  $i = 1, 2, \dots, n$ .
Assumption 2 (Assumption for the MLP layer). We assume that  $d \geq 2 + \dim((MLP^{-1}(\mathbf{y}_{10}) - \mathbf{x}_0) \cup (MLP^{-1}(\mathbf{y}_{20}) - \mathbf{x}_0))$  for the dataset constructed in Theorem 2 and token dimension  $d$ .  $\dim(S)$  measures the dimension of subspace spanned by vectors in a set  $S$  and  $MLP^{-1}(\mathbf{y}) = \{\mathbf{x} : MLP(\mathbf{x}) = \mathbf{y}\}$ .
We provide an example for this assumption in Example 1 and a sufficient condition in the following Lemma 4.
Lemma 4. If  $\| \mathbf{W}_1\| _2\times \| \mathbf{W}_2\| _2 < 1$  , where  $\| \cdot \| _2$  is the matrix spectral norm, then the MLP block in Definition 2 is invertible, ie,  $MLP^{-1}$  is a singleton set.
Therefore, if Lemma 4 holds and  $d \geq 4$ , Assumption 2 also holds.
Proof of Lemma 4 can be found in Appendix C.5. The experimental evidence in [Dong et al., 2021] shows that for most architectures, the norm of the weight matrices indeed admits small values and thus the requirement that  $\| \mathbf{W}_1\| _2\times \| \mathbf{W}_2\| _2 < 1$  is a mild condition.
With these assumptions, here we introduce our first theorem on the unlearnability of prompt tuning.
Theorem 2. For a single layer transformer  $\tau$  defined above with Assumptions 1 and 2, we can build a sequence-to-sequence dataset  $S = \{(\mathbf{X}_1 = [\mathbf{x}_1,\mathbf{x}_0],\mathbf{Y}_1 = [\mathbf{y}_{11},\mathbf{y}_{10}])$ $(\mathbf{X}_2 = [\mathbf{x}_2,\mathbf{x}_0],\mathbf{Y}_2 = [y_{21},y_{20}])\}$ , and we cannot find a prompt  $\mathbf{P}\in \mathbb{R}^{d\times m_p}$  with any  $m_p > 0$  such that  $\tau ([\mathbf{P},\mathbf{X}_i]) = \mathbf{Y}_i$  holds for any  $i = 1,2$ . The vectors  $\mathbf{x}_0,\mathbf{x}_1,\mathbf{x}_2$  are denoted post positional encodings.
An important feature of this dataset is that the same token  $\mathbf{x}_0$  is shared between the two examples, and the expressive capability of prompt tuning is limited by the correlation of outputs corresponding to this token in different examples. We show a concrete example here to illustrate this theorem (note that Lemma 4 is in fact not required in the following construction) and defer the formal proof to Appendix C.6.
Example 1. We consider a single-head transformer layer  $\tau$ , where  $\mathbf{b}_1 = \mathbf{b}_2 = \mathbf{0}$ ,  $\mathbf{W}_1 = 1^{r\times d}$ ,  $\mathbf{W}_2 = 1^{d\times r}$ . Then the token-wise MLP layer is a concatenation of two linear functions:
$$
M L P (\mathbf {x}) = \left\{ \begin{array}{l} \left(\mathbf {W} _ {2} \mathbf {W} _ {1} + \mathbf {I}\right) \mathbf {x}, \left(\mathbf {W} _ {1} \mathbf {x}\right) _ {0} > 0 \\ \mathbf {x}, \left(\mathbf {W} _ {1} \mathbf {x}\right) _ {0} \leq 0 \end{array} \right. \tag {7}
$$
Here  $(\mathbf{W}_1\mathbf{x})_0$  denotes the first element of vector  $\mathbf{W}_1\mathbf{x}$ .
$\mathbf{W}_2\mathbf{W}_1 + \mathbf{I}$  is a non-singular matrix. Therefore, for any  $\mathbf{y}$  in  $\mathrm{MLP}(\mathbf{X})$  's output set,  $\mathrm{MLP}^{-1}(\mathbf{y})$  contains at most two points  $\{\mathbf{y},(\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}\}$ . We arbitrarily choose  $\mathbf{x}_0,\mathbf{y}_{10}$  and  $\mathbf{y}_{20}$ .
As long as  $d \geq 6$  (from Assumption 2), we can find  $\mathbf{c}_1, \mathbf{c}_2$  such that  $\mathbf{c}_1, \mathbf{c}_2 \perp \mathbf{y}_{10} - \mathbf{x}_0, \mathbf{y}_{20} - \mathbf{x}_0, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{10} - \mathbf{x}_0, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{20} - \mathbf{x}_0, \mathbf{c}_1 \perp \mathbf{c}_2$ . Then we choose  $\mathbf{x}_1$  and  $\mathbf{x}_2$  such that  $\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_1) \| \mathbf{c}_1$  and  $\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_2) \| \mathbf{c}_2$  (Lemma 7 in Appendix). Then  $\mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_1), \mathbf{a} - \mathbf{x}_0) \cap \mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_2), \mathbf{b} - \mathbf{x}_0) = \emptyset$ , for any  $\mathbf{a} \in \{\mathbf{y}_{10}, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{10}\}$  and  $\mathbf{b} \in \{\mathbf{y}_{20}, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{20}\}$ . Here  $\mathrm{Cone}$  stands for a convex cone as defined in Section 3.1.
If a  $\mathbf{P}$  exists such that  $\tau ([\mathbf{P},\mathbf{X}_i]) = \mathbf{Y}_i$  holds for both  $i = 1,2$ , then we have
$$
\operatorname {A t t} \left(\mathbf {x} _ {0}, [ \mathbf {P}, \mathbf {X} _ {1} ]\right) = \lambda \left(\mathbf {X} _ {1}, \mathbf {x} _ {0}, [ \mathbf {P}, \mathbf {X} _ {1} ]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {X} _ {1}\right) + \lambda \left(\mathbf {P}, \mathbf {x} _ {0}, [ \mathbf {P}, \mathbf {X} _ {1} ]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {P}\right) \tag {8}
$$
$$
\operatorname {A t t} \left(\mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) = \lambda \left(\mathbf {X} _ {2}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {X} _ {2}\right) + \lambda \left(\mathbf {P}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {P}\right)
$$
where  $\lambda (\cdot ,\cdot ,\cdot)$  is a positive scalar. We also have
$$
\operatorname {A t t} \left(\mathbf {x} _ {0}, [ \mathbf {P}, \mathbf {X} _ {1} ]\right) + \mathbf {x} _ {0} \in \mathrm {M L P} ^ {- 1} (\mathbf {y} _ {1 0})
$$
$$
\mathbf {A t t} (\mathbf {x} _ {0}, [ \mathbf {P}, \mathbf {X} _ {2} ]) + \mathbf {x} _ {0} \in \mathbf {M L P} ^ {- 1} (\mathbf {y} _ {2 0})
$$
as MLP(Att(x0, [P, Xi]) + x0) = yio, i = 1, 2.
Therefore,  $\mathsf{Att}(\mathbf{x}_0,\mathbf{P})$  must be in both  $\mathsf{Cone}(\mathbf{a} - \mathbf{x}_0, -\mathsf{Att}(\mathbf{x}_0,\mathbf{X}_1))$  and  $\mathsf{Cone}(\mathbf{b} - \mathbf{x}_0, -\mathsf{Att}(\mathbf{x}_0,\mathbf{X}_2))$ , where  $\mathbf{a} \in \{\mathbf{y}_{10}, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{10}\}$  and  $\mathbf{b} \in \{\mathbf{y}_{20}, (\mathbf{W}_2\mathbf{W}_1 + \mathbf{I})^{-1}\mathbf{y}_{20}\}$ , which contradicts the existence of  $\mathbf{P}$  as  $\mathsf{Cone}(-\mathsf{Att}(\mathbf{x}_0,\mathbf{X}_1), \mathbf{a} - \mathbf{x}_0) \cap \mathsf{Cone}(-\mathsf{Att}(\mathbf{x}_0,\mathbf{X}_2), \mathbf{b} - \mathbf{x}_0) = \emptyset$ . Therefore, in this example, even though we allow an arbitrary number of trainable parameters in prompt  $\mathbf{P}$ , we cannot find one to exactly memorize the training set with only two training examples.
This theorem reveals an important difference between prompt tuning and adjusting the model weights directly. For any training dataset  $S$  with two training examples  $\{(\mathbf{X}_1, \mathbf{Y}_1), (\mathbf{X}_2, \mathbf{Y}_2)\}$ , so long as  $\mathrm{Att}(\mathbf{X}_1, \mathbf{X}_1) + \mathbf{X}_1$  and  $\mathrm{Att}(\mathbf{X}_2, \mathbf{X}_2) + \mathbf{X}_2$  are distinct, MLP can easily map the post-attention features to expected output tokens with finite number of hidden neurons. As a result, tuning the MLP parameters for this pretrained transformers can memorize any dataset in the form of Assumption 1. However, prompt tuning cannot achieve this even if the number of tunable tokens  $\rightarrow \infty$  infinity, thereby limiting the expressiveness of prompt tuning when compared to model fine-tuning.
5.2 Comparison with a More General Dataset
In Section 5.1, we constructed sequence-to-sequence datasets that cannot be learned by a given transformer layer with prompt tuning, by utilizing the shared token between different training examples. In this section, we compare the expressive power of prompt tuning and fine-tuning under a more general dataset construction where the former requirement can be relaxed.
Since the primary essence of prompt tuning is to perform parameter-efficient tuning, wherein we seek to adapt a pretrained large model to a new task with fewer tunable parameters, we compare prompt tuning with another parameter-efficient version of model-tuning: LoRA [Hu et al., 2021]. Succinctly, we compare the required number of parameters to memorize a given dataset. Again, consider a sequence-to-sequence dataset  $S = \{(\mathbf{X}_1,\mathbf{Y}_1),(\mathbf{X}_2,\mathbf{Y}_2),\dots,(\mathbf{X}_n,\mathbf{Y}_n)\}$ , where  $\mathbf{X}_i = [\mathbf{x}_{i1},\mathbf{x}_{i2},\dots,\mathbf{x}_{im}]$  and  $\mathbf{Y}_i = [\mathbf{y}_{i1},\mathbf{y}_{i2},\dots,\mathbf{y}_{im}]$ . We again discuss the memorization of the last output token for simplicity and results can be directly extended.
We first give the required number of parameters of LoRA to memorize dataset  $S$ .
Lemma 5 (LoRA). For a standard single-layer transformer  $\tau$  defined in Definition 2 with  $r\geq n$  MLP hidden neurons, for any sequence-to-sequence dataset  $S$  satisfying Assumptions 1, we can apply a low-rank update to MLP weights with  $O(nd)$  parameters to memorize  $\tau (\mathbf{X}_i)_{:m} = \mathbf{y}_{im}$
This lemma is derived based on the memorization capabilities of 1-hidden layer MLPs [Yun et al., 2019b]. As the post-attention values for different training inputs are different from Assumption 1, we can construct a low rank update with  $O(nd)$  parameters on the MLP layer to memorize  $S$ . We defer the complete proof to Appendix C.7.
For prompt tuning, we derive a result in the next theorem which shows that it requires  $\Omega(nd)$  tunable parameters to memorize some constructed dataset  $S$  with  $n$  examples.
Theorem 3 (Lower bound on Tunable Prompt Parameters). For any single layer transformer  $\tau$  defined in Definition 2, there exists a sequence-to-sequence dataset  $\{(\mathbf{X}_1 = [\mathbf{x}_{10},\mathbf{x}_1],[\mathbf{y}_{10},\mathbf{y}_{11}]),(\mathbf{X}_2 = [\mathbf{x}_{20},\mathbf{x}_2],[\mathbf{y}_{20},\mathbf{y}_{21}]),\dots,(\mathbf{X}_n = [\mathbf{x}_{n0},\mathbf{x}_n],[\mathbf{y}_{n0},\mathbf{y}_{n1}])\}$  that satisfies Assumption 1 with  $n < d$  training examples such that we need at least  $n$  prompt tokens in  $\mathbf{P}$  to memorize the training set, ie, for  $\tau ([\mathbf{P},\mathbf{X}_i])_{:, - 1} = \mathbf{y}_{i1}$  to hold for all  $i = 1,2,\ldots ,n$
This dataset can be constructed by including  $n$  examples that require  $n$  linearly independent prompts tokens. The complete proof is deferred to Appendix C.8.
Note that in Theorem 3, we provide a key lower bound on the required number of prompt tokens for exact memorization and this can very well more than  $nd$ . This partially (but not necessarily) explains the worse empirical performance of prompt tuning against LoRA under a comparable number of trainable parameters.
6 Extension to Multi-Layer Setting
In this section, we extend our analysis to multi-layer setting and provide a sufficient condition under which the expressiveness of prompt tuning is restricted. An immediate consequence of our result is an interesting connection to the spectral norm of soft prompts surfaces. This result provides us a partial understanding of the phenomenon that soft prompt  $\mathbf{P}$  vectors typically exhibit larger norms compared to the actual input  $\mathbf{X}$ , after the tuning.
With some further notation adjustments, we denote an  $H$  layer pretrained transformer network as  $g(\in \mathcal{T}) = \tau^1\circ \tau^2\circ \ldots \circ \tau^H$ , the input set as  $\mathcal{X}^1$ , and the set of possible prompts as  $\mathcal{P}^1$ . We assume that the following compactness condition is satisfied:
$$
\| [ \mathbf {P} ^ {l}, \mathbf {X} ^ {l} ] \| _ {2} \leq D ^ {l} \tag {9}
$$
$$
\mathrm {s . t .} [ \mathbf {P} ^ {l + 1}, \mathbf {X} ^ {l + 1} ] = \tau^ {l} ([ \mathbf {P} ^ {l}, \mathbf {X} ^ {l} ]), \forall l = 1, \dots , H.
$$
Here  $[\mathbf{P}^1, \mathbf{X}^1]$  is the input to the first layer  $\tau^1$  with  $\mathbf{P}^1 \in \mathcal{P}^1$ ,  $\mathbf{X}^1 \in \mathcal{X}^1$  and  $\|\cdot\|_2$  is the spectral norm. Similarly,  $[\mathcal{P}^{H+1}, \mathcal{X}^{H+1}]$  denotes the output set.
We start by providing an upper bound to the Lipschitz constant of attention, pertaining to eq 9. This derivation is different from the works of [Dasoulas et al., 2021, Vuckovic et al., 2020] and thus can be of independent interest.
Lemma 6. Under the compactness condition, the Lipschitz constant of the  $i$ -th attention head in the  $l$ -th transformer layer, denoted for simplicity as  $A t t^{i,l}$ , admits the following bound w.r.t the entire input sequence of length  $m$ :
$$
L i p \left(\boldsymbol {A} t \boldsymbol {t} ^ {i, l} (\cdot , \cdot)\right) \leq (1 + 8 \sqrt {m} (D ^ {l}) ^ {2} \| \left(\mathbf {W} _ {k} ^ {i, l}\right) ^ {T} \mathbf {W} _ {q} ^ {i, l} \| _ {2}) \| \mathbf {W} _ {v} ^ {i, l} \| _ {2}, \tag {10}
$$
and the Lipschitz constant of the entire attention block in layer  $l$ , denoted as  $A \oplus B^l$ , admits the bound:
$$
L i p \left(\boldsymbol {A} t \boldsymbol {t} ^ {l} (\cdot , \cdot)\right) \leq \sqrt {\sum_ {i = 1} ^ {h} \left(\left\| \mathbf {W} _ {o} ^ {i , l} \right\| _ {2} \times L i p \left(\boldsymbol {A} t \boldsymbol {t} ^ {i , l}\right)\right) ^ {2}}. \tag {11}
$$
It is noteworthy that this upper bound is dependent on  $D^{l}$ , the spectral norm of the input prepended with the prompt. In conjunction with the following theorem, we obtain a result on limited expressivity of prompt tuning by showing that the transformer becomes invertible, in consideration to functions from  $\mathcal{P}^1 \times \mathcal{X}^1 \to \mathcal{P}^{H + 1} \times \mathcal{X}^{H + 1}$  (an extension to functions of the from  $\mathcal{X}^1 \to \mathcal{X}^{H + 1}$  is provided in Appendix Section C.11).
Theorem 4. A transformer  $g \in \mathcal{T}$  is invertible, ie,  $g^{-1}(\mathbf{Y}) = \{\mathbf{X} : g(\mathbf{X}) = \mathbf{Y}\}$  is a singleton set  $\forall \mathbf{Y}$  in range of  $g$ , if:
1. The Lipschitz constant of the attention block in each layer  $\tau^l$  is strictly less than 1
2. The Lipschitz constant of the 2-layer ReLU block in each layer  $\tau^l$ , which is bounded by  $\| \mathbf{W}_2^l \|_2 \times \| \mathbf{W}_1^l \|_2$ , is strictly less than  $1$ .
Proof of Theorem 4 can be found in Appendix C.9. Combining Lemma 6 and Theorem 4, we observe that the invertibility is guaranteed if the upper bound for the Lipschitz constant of the attention, eq 11, and the MLP layer, is strictly less than 1. In this case, we can then construct arbitrarily many datasets where two different inputs share the same output, and prompt tuning cannot learn (more subtly: memorize) these datasets with a restricted prompt norm.
7.1 Experimental Settings
In Section 7.2, we use a standard single-layer single-head transformer from Definition 2, to justify the infinite prompt-length limitation. In Section 7.3, we justify the increasing prompt norm on the pretrained LLaMA 7B model [Touvron et al., 2023]. For prompt tuning and LoRA, we use the Huggingface Peft library [Mangrulkar et al., 2022]. On the dataset front, we utilize the RTE subtask of SuperGlue dataset [Wang et al., 2019] and WMT14 En-Fr translation [Bojar et al., 2014]. More details and hyperparameter settings can be found in Appendix A.
7.2 Limited Expressivity of Infinite Length Prompt
We first construct the dataset following the proof of Theorem 2 and then show that prompt tuning cannot memorize this simple dataset  $\{(\mathbf{X}_1 = [\mathbf{x}_1,\mathbf{x}_0],\mathbf{Y}_1 = [\mathbf{y}_{11},\mathbf{y}_{10}]),(\mathbf{X}_2 = [\mathbf{x}_2,\mathbf{x}_0],\mathbf{Y}_2 = [\mathbf{y}_{21},\mathbf{y}_{20}])\}$  even with very large prompt lengths.
We set the token dimension  $d = 10$ . We follow the default pytorch weight initialization and then normalize  $\mathbf{W}_1, \mathbf{W}_2$  such that  $\| \mathbf{W}_2 \|_2 \times \| \mathbf{W}_1 \|_2 < 1$ , following Assumption 2. We randomly sample  $\mathbf{x}_0, \mathbf{y}_{10}, \mathbf{y}_{20}$  in a uniform distribution in  $[0, 1)^d$  and construct the corresponding vectors:  $\mathbf{x}_1$  and  $\mathbf{x}_2$  following Theorem 2. To compute  $\mathrm{MLP}^{-1}(\mathbf{y})$ , we follow [Kim et al., 2021] Section 4.1 with 5000 iterations at convergence. We solve  $\mathrm{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1])$ $\| \mathbf{c}$  in Lemma 7 with gradient descent terminating at  $\angle (\mathrm{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]), \mathbf{c}) < 0.0001$ . We repeat this setup to obtain 3 different datasets for distinct  $\mathbf{x}_0, \mathbf{y}_{10}, \mathbf{y}_{20}$  and denote these with  $S_i$ ,  $i = 1, 2, 3$ .
We perform prompt tuning, MLP fine-tuning and MLP LoRA training on the constructed datasets for 5 runs and report the mean and standard deviation of per-element Mean Squared Error (MSE) loss  $\mathbf{y}_{10}, \mathbf{y}_{20}$  at convergence. We show the comparison between prompt-tuning and MLP fine-tuning in Figure 1. As we can observe from the figure, increasing the number of soft prompt tokens post a certain threshold that does not exhibit any reduction in MSE. On the contrary, fine-tuning on the MLP layer tends to easily memorize the training set by reducing the training loss to almost zero (all the three curves for fine-tuning overlap and thus not differentiated). Note that we plot the standard deviation, however it is negligible in the range. Similar to fine-tuning on the MLP layer, LoRA with width 2 on the MLP layer also achieves near-zero training loss which is less than  $10^{-10}$  on the constructed dataset. We don't plot the comparison on Figure 1 as all the six curves are overlapped. This result validates our Theorem 3 that LoRA can memorize a dataset with  $n$  examples with trainable parameters  $O(n)$  while prompt-tuning may require more.
7.3 Increasing Prompt Spectral Norm during Tuning
As discussed in Section 6, a major constraint on the expressive power of prompt tuning is the spectral norm of soft prompts. In Figure 2, we plot the curve for spectral norm of soft prompt as training progresses and the loss reduces on RTE dataset. The curve for WMT14 En-Fr dataset can be found in Appendix B. This trend clearly highlights that in order to counter the limit on the capacity, the spectral norm consistently increases till the training loss saturates.
In this work, we embark on exploring the capabilities of prompt tuning in the continuous regime, contrasting it with fine-tuning, as an initial endeavor towards a theoretical comprehension. We prove by construction that prompt tuning admits universal approximation within the space of Lipschitz functions. Additionally, we identified inherent limitations of prompt tuning on single-layer transformers by constructing theoretically difficult datasets for prompt tuning. These limitations are then extended to multi-layer setting under a specific prompt-norm restriction.
From the analysis in Theorem 2 and 3, we note that the limitation of prompt-tuning primarily arises from the correlation across different inputs. Broadly describing, prompt-tuning implements transformation on different inputs via "additional attention values", which is more restrictive as compared to the transformations from MLP layers on input tokens. An interesting potential direction to improve prompt-tuning is: "designing a mechanism to leverage prompting in order to generate
Figure 1: MSE losses at convergence for the 3 constructed datasets (following Theorem 2). We plot the bold curves with increasing prompt length in prompt tuning and dashed fixed lines in fine-tuning (all three datasets overlapping).
Figure 2: Increasing prompt spectral norm during tuning on SuperGlue RTE dataset.
prompt-dependent adapter/LoRA updates". We expect to have some future work focusing on designing novel prompt-tuning strategies along this direction.
Limitations While our results provide valuable insights, extending the construction in Theorem 2 to multiple layers and deriving tighter bounds for Lemma 6 are critical steps for a deeper understanding of the limitations of prompt tuning.
We thank the reviewers for their invaluable feedbacks. The work is supported in part by NSF 2008173, 2048280, 2325121, 2331966, ONR N00014-23-1-2300:P00001.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long.568.
Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 573-582. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/behrmann19a.html.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-short.1. URL https://aclanthology.org/2022.acl-short.1.
Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating Turing machines with transformers. CoRR, abs/2107.13163, 2021. URL https://arxiv.org/abs/2107.13163.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning, 2022.
Masami Yamasaki. The lower bound of the capacity for a neural network with multiple hidden layers. In ICANN'93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3, pages 546-549. Springer, 1993.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077, 2019a.
Chulhee Yun, Suvrit Sra, and Ali Jabbabaie. Small relu networks are powerful memorizers: a tight analysis of memorization capacity. Advances in Neural Information Processing Systems, 32, 2019b.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ByxRMONtvr.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?, 2020.
All the experiments are run on a NVIDIA RTX A6000 GPU. For experiments with Llama 7B model, we use batch size 32 and learning rate 0.001. For experiment on WMT14 En-Fr translation, we only compute the loss on the first 100 examples for computational efficiency.
We use Adam optimizer and optimal learning rate from grid search at 0.1 for prompt-tuning and at 0.001 for fine-tuning in Section 7.2.