SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 7.3, we use the default loss function in Huggingface implementation for causal language models. We use prompt length  $m = 10$  and the prompt tokens are initialized as the first  $m$  tokens in the model vocabulary.
As mentioned in Section 7.3, the second real world dataset used in our experiment is WMT14 En-Fr translation in order to illustrate that the spectral norm of soft prompts increases during training. We show the curve in Figure B.
Figure 3: Increasing prompt spectral norm during tuning on WMT14 En-Fr translation dataset.
C Proof of Lemmas and Theorems
For the sequence-to-sequence function space  $\mathcal{F}_L$  with functions  $f:[0,1]^{d\times m}\to [0,1]^{d\times m}$ , we can build a sequence-to-sequence function  $\bar{g}:[0,1]^{d\times (m_p + m)}\to [0,1]^{d\times (m_p + m)}$  such that for any  $f\in \mathcal{F}_L$ , we can find  $\mathbf{P}\in \mathbb{R}^{d\times m_p}$ ,  $d_{p}(\bar{g} ([\mathbf{P},\cdot ])_{:,m_{p}:},f)\leq \epsilon /2$ .
Proof. we first quantize the input and output sequence space of  $[0,1]^{d\times m}$  into a grid space  $G_{\delta,m} = \{0,\delta,2\delta,\ldots,1-\delta\}^{d\times m}$ , which leads to  $C = \left(\frac{1}{\delta^d\times m}\right)^{\frac{1}{\delta^d\times m}}$  functions considering all input and output mappings in this grid. We index these  $C$  functions as  $\bar{\mathcal{F}}_L = \{\bar{f}_1,\bar{f}_2,\dots,\bar{f}_C\}$ . For  $\mathbf{X}\notin G_{\delta,m}$ , we let  $\bar{f}_i(\mathbf{X}) = \bar{f}_i(\mathbf{X}^*)$  if  $k_{i,j}\delta < \mathbf{X}_{i,j},\mathbf{X}_{i,j}^* \leq (k_{i,j} + 1)\delta$  and  $\mathbf{X}^* \in G_{\delta,m}$ .
Then for any  $f \in \mathcal{F}_L$ , we can find a function  $\bar{f} \in \bar{F}_L$  such that  $d_p(\bar{f}, f) = (\int \| \bar{f}(\mathbf{X}) - f(\mathbf{X}) \|_p^p d\mathbf{X})^{1/p} \leq (\int L^p md\delta^p d\mathbf{X})^{1/p} = L(md)^{\frac{1}{p}}\delta$ . We choose  $\delta = \delta_1$  such that  $L(md)^{\frac{1}{p}}\delta \leq \epsilon/2$ . For the prompt part, we choose  $m_p$  such that  $\frac{1}{\delta^{d \times m_p}} \geq C$ . Then we can build a set of quantized prompts in  $G_{\delta, m_p} = \{0, \delta, 2\delta, ..., 1 - \delta\}^{d \times m_p}$  to index these  $C$  functions. We denote this set of prompts as  $\{\mathbf{P}_1, \mathbf{P}_2, \dots, \mathbf{P}_C\}$ . Finally we can create the quantized function  $\bar{g}$  and let
$\bar{g}([\mathbf{P}_i, \mathbf{X}])_{:,m_p} = \bar{f}_i(\mathbf{X})$  and  $\bar{g}([\mathbf{P}_i, \mathbf{X}])_{:,m_p} = 0$ ,  $\forall \mathbf{X} \in [0,1]^{d \times m}$ ,  $\mathbf{P} \in G_{\delta,m_p}$ . For  $\mathbf{P} \notin G_{\delta,m_p}$ , we set  $\bar{g}([\mathbf{P}, \mathbf{X}]) = \bar{g}([\mathbf{P}^*, \mathbf{X}])$  if  $k_{i,j} \delta < \mathbf{P}_{i,j}$ ,  $\mathbf{P}_{i,j}^* \leq (k_{i,j} + 1) \delta$  and  $\mathbf{P}^* \in G_{\delta,m_p}$ .
Therefore, with a properly chosen  $\delta = \delta_{1}$ , for any  $f\in \mathcal{F}_L$ , we can find  $\mathbf{P}\in \mathbb{R}^{d\times m_p}$  such that  $d_P(f,\bar{g} ([\mathbf{P},\cdot ])_{:,m_p:}) = d_p(\bar{f},f)\leq \epsilon /2$
For any given quantized function  $\bar{f}:[0,1]^{d\times m}\to [0,1]^{d\times m}$  with quantization at interval  $\delta$ ,  $\exists \bar{h}\in$ $\bar{T}^{2,1,1}$  such that  $\bar{f} = \bar{h}$  with positional embedding  $\mathbf{E} = \begin{bmatrix} 0 & 1 & 2 & \ldots & m - 1\\ 0 & 1 & 2 & \ldots & m - 1\\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 1 & 2 & \ldots & m - 1 \end{bmatrix}$ .
Proof. The proof is given following Section C in Yun et al. [2019a] appendix. With Section C.1 in Yun et al. [2019a], there exists a function  $g_{q}$  composed of  $\frac{dm}{\delta}$  token-wise feed-forward layers with hidden layer size  $r = 1$  and ReLU activation to implement this scalar quantization on each input element:
$$
g _ {q} ^ {e n t} (t) = \left\{ \begin{array}{l l} k \delta & \text {i f} k \delta \leq t <   (k + 1) \delta , k = 0, 1,..., m / \delta - 1 \\ - \delta^ {- m d} & \text {o t h e r w i s e} \end{array} \right.
$$
Then with Section C.2 in Yun et al. [2019a], we can stack  $m(1 / \delta)^{d} + 1$  attention layers to map all possible input sequences in grid  $[0:\delta :1 - \delta ]^d\times [1:\delta :2 - \delta ]^d\times \ldots \times [m - 1:\delta :m - \delta ]^d$  to distinct numbers which are at least  $\delta$  from each other.
Finally we only require  $O(m(1 / \delta)^{dm})$  layers to map these distinct numbers to expected outputs.
Lemma 3 is almost the same as [Yun et al., 2019a] except that we use  $\epsilon /2$  instead of  $\epsilon /3$
C.4 Extension of Theorem 1 to Next-token Predictors
As an extension of Theorem 1, we consider approximating a set of sequence-to-sequence functions when we use a transformer layer as a next-token predictor. We consider a set of sequence-to-sequence functions  $F_{L}$  with Lipschitz constant  $L$  under norm  $p$ .  $f \in F_{L} : [0,1]^{d \times m_{1}} \to [0,1]^{d \times m_{2}}$  accepts an input of length  $m_{1}$  and outputs a sequence of length  $m_{2}$ . For any  $\mathbf{X}, \mathbf{X}' \in [0,1]^{d \times m_{1}}$ , we have  $\| f(\mathbf{X}) - f(\mathbf{X}') \|_{p} \leq L \| \mathbf{X} - \mathbf{X}' \|_{p}$ .
Next we show that we can construct a transformer  $\tau$  which can approximate any  $f\in F_L$  with prompt-tuning when we use it as a next-token predictor.
Theorem 5. For any  $f \in F_L$ , we can construct a transformer  $\tau$  such that for any  $f \in F_L$ ,  $1 \leq p < \infty$  and  $\epsilon > 0$ , we can find a prompt  $\mathbf{P} \in [0,1]^{d \times m_p}$ , such that  $d_p(f, h(\mathbf{P})) \leq \epsilon$ , where
$$
h (\mathbf {P}) = \tau_ {1} ([ \mathbf {P}, \cdot ]) _ {:, - 1} \times \tau_ {2} ([ \mathbf {P}, \cdot ]) _ {:, - 1} \times \dots \times \tau_ {m _ {2}} ([ \mathbf {P}, \cdot ]) _ {:, - 1}.
$$
$\tau_{i}$  is the sequence-to-sequence function implemented with the transformer  $\tau$  when accepting sequences with length  $m_p + m_1 + i$ .
Proof. Similar to Theorem 1, we quantize the inputs to grid of  $[0: \delta: 1 - \delta]$  with interval  $\delta$  and set  $m_p = (\delta^{d \times m_2})^{\delta^{d \times m_1}}$ .  $\delta$  is chosen such that  $L(m_1 d)^{1/p} \delta \leq m_2^{-1/p} \epsilon$ . We index the  $C = (\delta^{d \times m_2})^{\delta^{d \times m_1}}$  different  $f_s$  as  $f^1, f^2, \ldots, f^C$  and its sub-function to generate the  $i$ -th output token as  $f_i^j$ . The  $C$  sequence-to-sequence functions can then be indexed by  $C$  distinct prompts. Similar to Lemma 2, we can construct a transformer which can map all possible input sequences in grids  $[0: \delta: 1 - \delta] \times \ldots \times [m-1: \delta: m - \delta]^d, 0 < m \leq m_1 + m_2 + m_p - 1$  to distinct numbers. A final series of MLP layers then map these distinct numbers to desired output vectors where inputs
in the same grid are mapped to the same output token at each step. Then for any input  $\mathbf{x} \in [0,1]^{d \times m_1}$  and any  $f^j$ , we can find a prompt  $\mathbf{P}_j$  such that
$$
\left\| \tau \left(\left[ \mathbf {P} _ {j}, \mathbf {x} \right]\right) _ {: -, 1} - f _ {0} ^ {j} (\mathbf {x}) \right\| _ {p} \leq m _ {2} ^ {- 1 / p} \epsilon
$$
$$
\left\| \tau ([ \mathbf {P} _ {j}, \mathbf {x}, \tau ([ \mathbf {P}, \mathbf {x} ]) _ {:, - 1} ], _ {:, - 1}, f _ {1} ^ {j} ([ \mathbf {x}, \tau ([ [ \mathbf {P}, \mathbf {x} ]) _ {:, - 1} ]) \right\| _ {p} \leq m _ {2} ^ {- 1 / p} \epsilon
$$
Then we have  $d_p(h(\mathbf{P}_j),f^j)\leq \epsilon$
C.5 Proof of Lemma 4
If  $\| \mathbf{W}_1\| _2\times \| \mathbf{W}_2\| _2 < 1$  , where  $\| \cdot \| _2$  is the matrix spectral norm, then the MLP block in Definition 2 is invertible, ie,  $\mathrm{MLP}^{-1}$  is a singleton set.
Proof. Based on the sufficient conditions for invertibility of a residual block Behrmann et al. [2019], we have that if the feedforward part of a residual block  $\mathbf{W}_2\mathrm{ReLU}(\mathbf{W}_1\mathbf{X}_{:,1} + \mathbf{b}_1) + \mathbf{b}_2$  is a contraction with respect to some metric, i.e. its Lipschitz constant  $< 1$ , and the metric space on which is defined is complete, then MLP in eq 2 is invertible. Since we are dealing with the euclidean space, any metric induced by the  $\| \cdot \|_p$  norm for  $p \in [1,\infty]$  ensures the space is complete.
The Lipschitz constant of  $\mathbf{W}_2\mathrm{ReLU}(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2$  is simply  $\| \mathbf{W}_1\| _2\times \| \mathbf{W}_2\| _2$ . Thus the statement of the lemma follows.
For a single layer transformer  $\tau$  defined above with Assumptions 1 and 2, we can build a seq-to-seq dataset  $\{(\mathbf{X}_1 = [\mathbf{x}_1,\mathbf{x}_0],\mathbf{Y}_1 = [\mathbf{y}_{11},\mathbf{y}_{10}]),(\mathbf{X}_2 = [\mathbf{x}_2,\mathbf{x}_0],\mathbf{Y}_2 = [\mathbf{y}_{21},\mathbf{y}_{20}])\}$ , and we cannot find a prompt  $\mathbf{P}\in \mathbb{R}^{d\times m_p}$  with any  $m_p > 0$  such that  $\tau ([\mathbf{P},\mathbf{X}_i]) = \mathbf{Y}_i$  holds for any  $i = 1,2$ . The vectors  $\mathbf{x}_0,\mathbf{x}_1,\mathbf{x}_2$  are denoted post positional encodings.
Proof. Before proving Theorem 2, we first provide a lemma that will be used in proof and also Theorem 3.
Lemma 7. Given any  $\mathbf{c} \in \mathbb{R}^{d \times m}$ , there are  $\mathbf{x}_0$  almost anywhere for which we can find another vector  $\mathbf{x}_1 \in \mathbb{R}^{d \times m}$  such that  $A t t(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]) \parallel \mathbf{c}$  with full rank attention weights  $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v$ .
Proof. If  $\mathbf{W}_v\mathbf{x}_0\parallel \mathbf{c}$ , we can just set  $\mathbf{x}_1 = \mathbf{x}_0$ , which makes  $\mathrm{Att}(\mathbf{x}_0,[\mathbf{x}_0,\mathbf{x}_1])\parallel \mathbf{c}$  hold.
If  $\mathbf{W}_v\mathbf{x}_0\nVdash \mathbf{c}$ , let  $\mathbf{v} = \alpha \mathbf{c} - \mathbf{W}_v\mathbf{x}_0$  where  $\alpha \in \mathbb{R}$ . As  $\mathbf{W}_v$  is full-rank, we can find  $\mathbf{x}$  such that  $\mathbf{x} = \mathbf{W}_v^{-1}\mathbf{v} = \alpha \mathbf{W}_v^{-1}\mathbf{c} - \mathbf{x}_0$ . Then we will have
$$
\begin{array}{l} \operatorname {A t t} \left(\mathbf {x} _ {0}, \left[ \mathbf {x} _ {0}, \mathbf {x} _ {1} \right]\right) \\ = \frac {\exp \left(\left(\mathbf {W} _ {q} \mathbf {x} _ {0}\right) ^ {\top} \left(\mathbf {W} _ {k} \mathbf {x} _ {0}\right)\right) \mathbf {W} _ {v} \mathbf {x} _ {0} + \exp \left(\left(\mathbf {W} _ {q} \mathbf {x} _ {0}\right) ^ {\top} \left(\mathbf {W} _ {k} \left(\alpha \mathbf {W} _ {v} ^ {- 1} \mathbf {c} - \mathbf {x} _ {0}\right)\right)\right) \left(\alpha \mathbf {c} - \mathbf {W} _ {v} \mathbf {x} _ {0}\right)}{\exp \left(\left(\mathbf {W} _ {q} \mathbf {x} _ {0}\right) ^ {\top} \left(\mathbf {W} _ {k} \mathbf {x} _ {0}\right)\right) + \exp \left(\left(\mathbf {W} _ {q} \mathbf {x} _ {0}\right) ^ {\top} \left(\mathbf {W} _ {k} \left(\alpha \mathbf {W} _ {v} ^ {- 1} \mathbf {c} - \mathbf {x} _ {0}\right)\right)\right)} \\ \end{array}
$$
Therefore, as long as  $\mathbf{W}_q\mathbf{x}_0 \not\subset \mathbf{W}_k(\mathbf{W}_v^{-1}\mathbf{c})$ , we can change  $\alpha$  such that  $\mathrm{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]) = \beta \mathbf{W}_v\mathbf{x}_0 + (1 - \beta)(\alpha \mathbf{c} - \mathbf{W}_v\mathbf{x}_0)$  where  $\beta = \frac{\exp((\mathbf{W}_q\mathbf{x}_0)^\top(\mathbf{W}_k\mathbf{x}_0))}{\exp((\mathbf{W}_q\mathbf{x}_0)^\top(\mathbf{W}_k\mathbf{x}_0)) + \exp((\mathbf{W}_q\mathbf{x}_0)^\top(\mathbf{W}_k(\alpha\mathbf{W}_v^{-1}\mathbf{c} - \mathbf{x}_0)))}$ .
When  $\alpha = 0$ ,  $\mathsf{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]) = \mathbf{W}_v \mathbf{x}_0$ , when  $\alpha \to -\infty$  or  $\alpha \to \infty$ ,  $\mathsf{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]) \to \alpha \mathbf{c} - \mathbf{W}_v \mathbf{x}_0$ . As  $\mathsf{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1])$  is continuous w.r.t changing  $\alpha$ , there must exist an  $\alpha$  such that  $\mathsf{Att}(\mathbf{x}_0, [\mathbf{x}_0, \mathbf{x}_1]) \| \mathbf{c}$ .
Pass the two input sequences  $\mathbf{X}_1, \mathbf{X}_2$  through the attention layer Att with any prompt  $\mathbf{P}$ , we can get the last output token as:
$$
\operatorname {A t t} \left(\mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {1} \right]\right) = \lambda \left(\mathbf {X} _ {1}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {1} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {X} _ {1}\right) + \lambda \left(\mathbf {P}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {1} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {P}\right) \tag {12}
$$
$$
\operatorname {A t t} \left(\mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) = \lambda \left(\mathbf {X} _ {2}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {X} _ {2}\right) + \lambda \left(\mathbf {P}, \mathbf {x} _ {0}, \left[ \mathbf {P}, \mathbf {X} _ {2} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {0}, \mathbf {P}\right) \tag {13}
$$
Here  $\lambda (\mathbf{X}_1,\mathbf{x}_2,\mathbf{X}_3 = [\mathbf{X}_1,\mathbf{X}_2])\in (0,1)$  is a positive scalar, defined as
$$
\lambda (\mathbf {X} _ {1}, \mathbf {x} _ {2}, \mathbf {X} _ {3}) = \frac {\sum_ {j} \exp ((\mathbf {W} _ {k} \mathbf {x} _ {1 j}) ^ {\top} (\mathbf {W} _ {q} \mathbf {x} _ {2}))}{\sum_ {j} \exp ((\mathbf {W} _ {k} \mathbf {x} _ {3 j}) ^ {\top} (\mathbf {W} _ {q} \mathbf {x} _ {2}))}.
$$
$\mathbf{x}_{ij}$  is the  $j$ th token in  $\mathbf{X}_i$  for notation simplicity.
1. Then from equation 12,  $\mathrm{Att}(\mathbf{x}_0,\mathbf{P})$  must be on  $\mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0,\mathbf{X}_1),\mathrm{Att}(\mathbf{x}_0,[\mathbf{P},\mathbf{X}_1]))$  and  $\mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0,\mathbf{X}_2),\mathrm{Att}(\mathbf{x}_0,[\mathbf{P},\mathbf{X}_2]))$
2. On the other hand, as we want to memorize the two examples, we must have  $\mathsf{Att}(\mathbf{x}_0,[\mathbf{P},\mathbf{X}_1]) + \mathbf{x}_0\in \mathsf{MLP}^{-1}(\mathbf{y}_{10})$  and  $\mathsf{Att}(\mathbf{x}_0,[\mathbf{P},\mathbf{X}_2]) + \mathbf{x}_0\in \mathsf{MLP}^{-1}(\mathbf{y}_{20})$
We construct the dataset  $S$  with arbitrary  $\mathbf{x}_0, \mathbf{y}_{10}$  and  $\mathbf{y}_{20}$ . Then if  $\dim((\mathsf{MLP}^{-1}(\mathbf{y}_{10}) - \mathbf{x}_0) \cup (\mathsf{MLP}^{-1}(\mathbf{y}_{20}) - \mathbf{x}_0)) + 2 \leq d$  (Assumption 2), we can find two vectors  $\mathbf{c}_1, \mathbf{c}_2$  such that  $\mathbf{c}_1, \mathbf{c}_2 \perp \mathbf{v}: \mathbf{v} + \mathbf{x}_0 \in \mathsf{MLP}^{-1}(\mathbf{y}_{10})$  or  $\mathbf{v} + \mathbf{x}_0 \in \mathsf{MLP}^{-1}(\mathbf{y}_{20})$  and  $\mathbf{c}_1 \perp \mathbf{c}_2$ . Then we can choose  $\mathbf{x}_1, \mathbf{x}_2$  such that  $\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_1) \| \mathbf{c}_1$  and  $\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_2) \| \mathbf{c}_2$  (Lemma 7). Combine this construction with assumption 1, we have that  $\mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_1), \mathrm{Att}(\mathbf{x}_0, [\mathbf{P}, \mathbf{X}_1]))$  and  $\mathrm{Cone}(-\mathrm{Att}(\mathbf{x}_0, \mathbf{X}_2), \mathrm{Att}(\mathbf{x}_0, [\mathbf{P}, \mathbf{X}_2]))$  has no intersection, which means that we cannot find a  $\mathbf{P}$  to memorize this constructed dataset.
C.7 Proof of Lemma 5
For a standard single-layer transformer  $\tau$  defined in Definition 2 with  $r\geq n$  MLP hidden neurons, for any sequence-to-sequence dataset  $S$  satisfying Assumptions 1, we can apply a low-rank update to MLP weights with  $O(nd)$  parameters to memorize  $\tau (\mathbf{X}_i)_{:,m} = \mathbf{y}_{im}$ .
Proof. We use  $\mathsf{MLP}_j(\mathbf{x})$  to denote the  $j$ th output of the MLP layer for an input token  $\mathbf{x}$ , which is
$$
\mathrm {M L P} _ {j} (\mathbf {x}) = x _ {j} + b _ {2, j} + \sum_ {k = 1} ^ {m} w _ {k, j} \max  (\langle \mathbf {a} _ {k}, \mathbf {x} \rangle + b _ {1, k}, 0)
$$
According to our assumption,  $\mathsf{Att}(\mathbf{x}_{im},\mathbf{X}_i)$  are unique vectors for  $i = 1,2,\ldots ,n$  . Then we only need to use the MLP layer to map each  $\mathbf{x}_i = \mathsf{Att}(\mathbf{x}_{im},\mathbf{X}_i) + \mathbf{x}_{im}$  to  $\mathbf{y}_{im}$  , where we get a new token-wise dataset  $\{\left(\mathbf{x}_1,\mathbf{y}_1\right),\left(\mathbf{x}_2,\mathbf{y}_2\right),\dots ,\left(\mathbf{x}_n,\mathbf{y}_n\right)\}$
Then we need to find  $w_{k},\mathbf{a}_{k}$  and  $b_{k}$  such that
$$
\mathrm {M L P} _ {j} \left(\mathbf {x} _ {i}\right) = x _ {i, j} + b _ {2, j} + \sum_ {k = 1} ^ {m} w _ {k, j} \max  \left(\left\langle \mathbf {a} _ {k}, \mathbf {x} _ {i} \right\rangle + b _ {1, k}, 0\right) = y _ {i, j}, i = 1, 2, \dots , n, j = 1, 2, \dots , d \tag {14}
$$
, which is equivalent to constructing a standard MLP to memorize a dataset:
$$
\sum_ {k = 1} ^ {n} w _ {k, j} \max  \left(\left\langle \mathbf {a} _ {k}, \mathbf {x} _ {i} \right\rangle + b _ {1, k}, 0\right) = y _ {i, j} - x _ {i, j} - \sum_ {k = n + 1} ^ {m} w _ {k, j} \max  \left(\left\langle \mathbf {a} _ {k}, \mathbf {x} _ {i} \right\rangle + b _ {1, k}, 0\right) - b _ {2, j} \tag {15}
$$
Follow Theorem 1 in Yun et al. [2019b], we can construct  $\mathbf{a}, b_1, \ldots, b_n$  such that for  $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n$ , we have  $z_i = \langle \mathbf{a}, \mathbf{x}_i \rangle$ ,  $b_1 < z_1 < b_2 < \ldots < b_n < z_n$ . Then we can find  $w_1, \ldots, w_n$  which solves equation 15. For  $d$ -dimension output, we need to find  $\mathbf{W} \in \mathbb{R}^{n \times d}$  and  $\mathbf{a} \in \mathbb{R}^d$  and  $\mathbf{b} \in \mathbb{R}^n$ . With LoRA, we need a low-rank update of size  $m \times n + n \times d$  for  $\mathbf{W}_2$ , a low-rank update of size  $d \times n + n \times m$  for  $\mathbf{W}_1$  and an update of size  $n$  for  $\mathbf{b}_1$ , which is  $O(n \times d)$  in total. Normally we have  $m \simeq d$ , then we need an update with parameter size around  $(4n + 1)d$  to memorize the last token of  $n$  training examples.
C.8 Proof of Theorem 3
For any single layer transformer  $\tau$  defined in Definition 2, there exists a seq-to-seq dataset  $\{(\mathbf{X}_1 = [\mathbf{x}_{10},\mathbf{x}_1],[\mathbf{y}_{10},\mathbf{y}_{11}]),(\mathbf{X}_2 = [\mathbf{x}_{20},\mathbf{x}_2],[\mathbf{y}_{20},\mathbf{y}_{21}]),\ldots ,(\mathbf{X}_n = [\mathbf{x}_{n0},\mathbf{x}_n],[\mathbf{y}_{n0},\mathbf{y}_{n1}])\}$  that satisfies Assumption 1 with  $n < d$  training examples such that we need at least  $n$  prompt tokens in  $\mathbf{P}$  to memorize the training set, ie, for  $\tau ([\mathbf{P},\mathbf{X}_i])_{:, - 1} = \mathbf{y}_{i1}$  to hold for all  $i = 1,2,\dots,n$
Proof. Without loss of generality, we assume  $\mathbf{W}_2$  has no zero elements, otherwise we can just ignore this hidden neuron in MLP layer.
$\mathbb{R}^d$  has  $d$  bases  $\{\mathbf{t}_j:j = 1,2,\ldots ,d\}$ , then  $\mathrm{MLP}^{-1}(\mathbf{y}_{i1})$  must be bounded on either positive or negative part of these  $d$  directions, which means there exists  $B\geq 0$  such that
$$
\frac {\mathbf {v} ^ {\top} \mathbf {t} _ {j}}{\| \mathbf {t} _ {j} \|} \leq B, \forall \mathbf {v} \in \mathrm {M L P} ^ {- 1} (\mathbf {y} _ {i 1}), j = 1, 2, \dots , d
$$
Otherwise  $\forall B > 0$ ,  $\exists \mathbf{t}_j$ , we can find a  $\mathbf{v} \in \mathrm{MLP}^{-1}(\mathbf{y}_i)$  that  $\frac{\mathbf{v}^\top \mathbf{t}_j}{\|\mathbf{t}_j\|} \geq B$ . Meanwhile we have  $\mathrm{MLP}(\mathbf{v}) = \mathbf{v} + \mathbf{b}_2 + \mathbf{W}_2 \mathrm{ReLU}(\mathbf{W}_1 \mathbf{v} + \mathbf{b}_1)$ . As  $\|\mathbf{v}\|$  can be arbitrarily large, if  $\mathbf{W}_1 \mathbf{v} = \mathbf{0}$ ,  $\|\mathrm{MLP}(\mathbf{v})\| \to \infty$  if  $\|\mathbf{v}\| \to \infty$ . If  $\mathbf{W}_1 \mathbf{v} \neq \mathbf{0}$ ,  $\|\mathrm{MLP}(\mathbf{v})\|$  can also be arbitrarily large when increasing the norm of  $\mathbf{v}$  due to the non-linearity of  $\mathrm{ReLU}(\mathbf{W}_1 \mathbf{v} + \mathbf{b}_1)$ .
Then we can find a set of  $n$  linearly independent vectors  $\{\mathbf{c}_1,\mathbf{c}_2,\dots,\mathbf{c}_n\}$  such that  $\{\mathbf{a}_i:\mathbf{a}_i - \mathbf{c}_i\perp \mathbf{c}_i,\mathbf{a}_i\in \mathrm{MLP}^{-1}(\mathbf{y}_{i1})\} = \emptyset$  by enlarging the norm of  $\mathbf{c}_i$ . With the  $n$ $\mathbf{c}_i$  vectors, we can begin to construct our dataset:
We set  $\mathbf{x}_i = \mathbf{c}_i, i = 1,2,\dots,n$  and find  $\mathbf{x}_{i0}$  such that  $\mathbf{c}_i\perp \mathrm{Att}(\mathbf{x}_i,\mathbf{X}_i)$  (Lemma 7) and  $\mathrm{Att}(\mathbf{x}_i,\mathbf{X}_i)$  are distinct for  $i = 1,2,\dots,n$  (Assumption 1), which makes  $\{\mathbf{a}_1 - \mathbf{x}_1 - \lambda (\mathbf{X}_1,\mathbf{x}_1,[\mathbf{P},\mathbf{X}_1])\mathrm{Att}(\mathbf{x}_1,\mathbf{X}_1),\dots,\mathbf{a}_n - \mathbf{x}_n - \lambda (\mathbf{X}_n,\mathbf{x}_n,[\mathbf{P},\mathbf{X}_n])\mathrm{Att}(\mathbf{x}_n,\mathbf{X}_n)\}$  linearly independent for any  $\mathbf{a}_i\in \mathrm{MLP}^{-1}(\mathbf{y}_{i1})$ . Here  $\lambda (\cdot ,\cdot ,\cdot)$  is the same as defined in Section C.6.
$$
\begin{array}{l} \operatorname {A t t} \left(\mathbf {x} _ {i}, \left[ \mathbf {P}, \mathbf {X} _ {i} \right]\right) = \lambda \left(\mathbf {X} _ {i}, \mathbf {P}, \left[ \mathbf {P}, \mathbf {X} _ {i} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {i}, \mathbf {P}\right) + \lambda \left(\mathbf {X} _ {i}, \mathbf {x} _ {i}, \left[ \mathbf {P}, \mathbf {X} _ {i} \right]\right) \operatorname {A t t} \left(\mathbf {x} _ {i}, \mathbf {X} _ {i}\right) \tag {16} \\ \in \operatorname {M L P} ^ {- 1} \left(\mathbf {y} _ {i 1}\right) - \mathbf {x} _ {i} \\ \end{array}
$$
Then  $\mathsf{Att}(\mathbf{x}_i,\mathbf{P}),i = 1,2,\dots,n$  must be  $n$  linearly independent vectors, which requires
$$
\operatorname {r a n k} \left(\mathbf {W} _ {v} \mathbf {P A}\right) = n, \tag {17}
$$
where  $\mathbf{A} \in \mathbb{R}^{m_p \times n}$  is the attention score matrix between  $\mathbf{x}_i$  and  $\mathbf{P}$ .  $\mathbf{P} \in \mathbb{R}^{d \times m_p}$  is the prompt token sequence and  $\mathbf{W}_v$  is the attention value weight. Therefore, we must have  $m_p \geq n$ .
C.9 Proof of Theorem 4
A transformer  $\mathcal{T}$  is invertible if:
1. The Lipschitz constant of the attention block in each layer  $\tau^l$  is strictly less than 1
2. The Lipschitz constant of the 2-layer ReLU block in each layer  $\tau^l$ , which is bounded by  $\| \mathbf{W}_2^l \|_2 \times \| \mathbf{W}_1^l \|_2$ , is strictly less than 1
Proof. This proof is based on the proof provided for lemma 4, thus we restrict ourselves to the sketch: Based on the sufficient condition for invertibility in Behrmann et al. [2019], condition (1) implies that the attention block (eq 1) with the residual connection, ie  $\mathbf{X} + \mathsf{Att}(\mathbf{X},\mathbf{X})$  , is an invertible function. Similarly, condition (2) implies that the MLP block which constitutes of the 2-layer ReLU block with the residual connection (eq 2) also exhibit invertibility.
Thus each transformer layer  $\tau^l$  (eq 3) is invertible by noting that its a composition of two invertible functions. The same property ensures that the entire transformer architecture  $\mathcal{T}$  is also invertible.
C.10 Proof of Lemma 6
Under the compactness condition, the Lipschitz constant of the  $i$ -th attention head in the  $l$ -th transformer layer, denoted for simplicity as  $\mathsf{Att}^{i,l}$ , admits the following bound w.r.t the entire input sequence of length  $m$ :
$$
L i p \left(\operatorname {A t t} ^ {i, l} (\cdot , \cdot)\right) \leq (1 + 8 \sqrt {m} (D ^ {l}) ^ {2} \| \left(\mathbf {W} _ {k} ^ {i, l}\right) ^ {T} \mathbf {W} _ {q} ^ {i, l} \| _ {2}) \| \mathbf {W} _ {v} ^ {i, l} \| _ {2}, \tag {18}
$$
and the Lipschitz constant of the entire attention block in layer  $l$ , denoted as  $\mathsf{Att}^l$ , admits the bound:
$$
L i p \left(\mathbf {A t t} ^ {l} (\cdot , \cdot)\right) \leq \sqrt {\sum_ {i = 1} ^ {h} \left(\left\| \mathbf {W} _ {o} ^ {i , l} \right\| _ {2} \times L i p \left(\mathbf {A t t} ^ {i , l}\right)\right) ^ {2}}. \tag {19}
$$
Proof. We drop the superscripts  $i, l$  in the proof to avoid notation clutter. Similarly, we denote the concatenation of the prompt matrix  $\mathbf{P}$  and the original input matrix  $\mathbf{X}$ , simply with  $\mathbf{X}$ .