SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yihan Wang
wangyihan617@gmail.com
Jatin Chauhan
chauhanjatin100@gmail.com
weiwang@cs.ucla.edu
Despite the demonstrated empirical efficacy of prompt tuning to adapt a pretrained language model for a new task, the theoretical underpinnings of the difference between "tuning parameters before the input" against "the tuning of model weights" are limited. We thus take one of the first steps to understand the role of soft-prompt tuning for transformer-based architectures. By considering a general purpose architecture, we analyze prompt tuning from the lens of both: universal approximation and limitations with finite-depth fixed-weight pretrained transformers for continuous-valued functions. Our universality result guarantees the existence of a strong transformer with a prompt to approximate any sequence-to-sequence function in the set of Lipschitz functions. The limitations of prompt tuning for limited-depth transformers are first proved by constructing a set of datasets, that cannot be memorized by a prompt of any length for a given single encoder layer. We also provide a lower bound on the required number of tunable prompt parameters and compare the result with the number of parameters required for a low-rank update (based on LoRA) for a single-layer setting. We finally extend our analysis to multi-layer settings by providing sufficient conditions under which the transformer can at best learn datasets from invertible functions only. Our theoretical claims are also corroborated by empirical results.
The surge in the empirical research of large-scale models has led to the emergence of a new paradigm of prompt tuning. Current large models consist of billions of parameters [Brown et al., 2020, Chowdhery et al., 2022], which greatly exacerbate the cost of tuning the entire model weights via gradient-based optimization. On the other hand, the power of scale in both model size and pretraining dataset size has demonstrated strong capabilities by achieving reasonable performance through a learnable prompt appended before the input [Li and Liang, 2021, Lester et al., 2021]. Despite this, several questions emanate around the abilities and limitations of prompt tuning.
In this work, we aim to characterize some natural yet essential questions about prompt tuning with transformer architectures. Firstly, are prompts universal approximators, i.e. with a fixed pretrained transformer network, can we find a prompt to approximate any sequence-to-sequence function in a given space? If yes, can we construct the transformer for this universality result? Second, can we identify failure modes of prompt tuning when applied on potentially non-optimal but non-trivial transformers? Moreover, since prompt tuning is usually compared against LoRA[Hu et al., 2021] in consideration to parameter-efficient tuning, is prompt tuning then more/less parameter-efficient than LoRA? Answering these questions can lead to important insights on when and how to perform prompt tuning to adapt a pretrained transformer network to a given downstream task of interest.
In this work, we seek to answer these questions with appropriate theoretical analysis and further validate our claims with empirical results. We first characterize the universal nature of prompt tuning by constructing a specific transformer network. We show that for a given approximation error and the space of sequence-to-sequence Lipschitz functions, we can construct a transformer network, with a suitable number of layers, that can leverage prompt tuning to approximate any function in this space. Despite this universality of prompt tuning with a carefully constructed pretrained transformer, we then identify some limitations of prompt tuning with weaker but non-trivial transformers. We prove this by constructing sequence-to-sequence datasets with shared input tokens, which are surprisingly simple but cannot be memorized by prompt tuning for a given transformer. We also extend our analysis to more general settings where the shared token is not required. In this setting, we first prove that prompt tuning on a single-layer transformer requires  $\Omega(n)$  trainable parameters to memorize  $n$  training examples, wherein for LoRA, it suffices with  $O(n)$  trainable parameters. We finally extend our analysis to the multi-layer setting and provide sufficient conditions under which prompt tuning exhibits extremely limited capacity to at best memorizing datasets from invertible functions.
Our contributions can be summarized as below:
- We characterize the universal nature of prompt tuning by explicitly constructing a transformer network (Theorem 1).
- We provide a construction-based argument for sequence-to-sequence datasets that cannot be learned by prompt tuning with a given single-layer transformer (Theorem 2).
- We provide the lower bound on the required number of parameters for prompt tuning to memorize any sequence-to-sequence functions (Theorem 3).
- We provide a sufficient condition for multi-layer transformers, under which datasets with shared output tokens cannot be learned with prompt tuning (Theorem 4).
- We conduct empirical studies, including real-world datasets, to verify our theoretical claims.
Theoretical Analysis of Transformers Various works have characterized the theoretical properties of transformers and its primary self-attention component. Yun et al. [2020] studies the universal approximation ability of transformers for continuous permutation equivariant sequence-to-sequence functions with compact support and further examined the role of using positional encodings to circumvent permutation equivariant condition. Pérez et al. [2021] show that transformer with a hard-attention is Turing complete based on their capacity to perform computations and access the internal dense representations of the data. Wei et al. [2021] further shows that transformers can approximate Turing machines with bounded computation time with a new notion of approximation. [Dong et al., 2021] provides a negative yet interesting result signifying the limitations of pure self-attention in terms of rank diminishing of the input. Other works including Kim et al. [2021], Dasoulas et al. [2021] derive upper bounds on the Lipschitz constant of respective modifications of the attention mechanism. The works by Li et al. [2022], Zhang et al. [2020] documented optimization perspective on transformer training via SGD.
Fine-tuning and Prompt Tuning Fine-tuning is the standard way to adapt a pretrained model to downstream tasks. The most standard and popular paradigm is tuning the model weights via a suitable optimization procedure along with a linear head on the output representations [Radford et al., 2018, Devlin et al., 2019]. Subsequent works studied more parameter-efficient ways of fine-tuning by updating either a subset of model parameters [Ben Zaken et al., 2022] or restricting the parameter-updates to a low-dimensional subspace [Aghajanyan et al., 2021, Hu et al., 2021, Mahabadi et al., 2021]. The work by Hu et al. [2021] (their framework referred to as LoRA) has garnered particular interest in the community and Malladi et al. [2023] has provided an interpretation of LoRA via the kernel mechanism. In the particular context of LLMs, prompt tuning has emerged as the de facto approach where only the prompt is updated while keeping the rest of the transformer weights and architecture fixed [Shin et al., 2020, Lester et al., 2021, Li and Liang, 2021].
Analysis of Prompt Tuning Wei et al. [2022] studies the link between prompt tuning and downstream tasks with an underlying latent variable generative model of text, which is confined to a Hidden Markov Model. However, they focused on the discrete vocabulary setting contrary to our
results for continuous sequence-to-sequence functions. Some more recent works [Akyurek et al., 2023, Von Oswald et al., 2023] characterize an intriguing property of a specific form of prompting, referred to as in-context learning, where they proved by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. This work however pursued a different and specific direction from the prompting results we aim to provide for generic settings.
Memorization Capacity of Neural Networks A series of works have sought to provide finite sample universal memorization capacity results of neural networks and the understanding of expressive power of neural networks. Huang and Huang [1990], Huang and Babri [1998], Huang [2003], Yamasaki [1993] analyze the memorization capacity of FNNs with sigmoid and other bounded activation functions. Hardt and Ma [2016], Zhang et al. [2021], Nguyen and Hein [2018] provide results for modern ReLU networks including FNNs and CNNs. For transformer architectures, Kim et al. prove that transformers can memorize a dataset with finite parameters. To the best of our knowledge, similar results for prompt tuning have not been studied in continuous settings for transformer architectures.
3 Transformers and Parameter Efficient Training
We use the following notations throughout the paper. A bold lower case character, e.g.  $\mathbf{x}$ , denotes a vector. A bold upper case character, e.g.  $\mathbf{W}$ , denotes a matrix while  $\mathbf{W}_{i,j}$ ,  $\mathbf{W}_{i,\cdot}$  and  $\mathbf{W}_{\cdot ,j}$  is the  $(i,j)$ -th element,  $i$ -th row,  $j$ -th column, respectively. We use a single superscript or subscript to denote the index of a matrix, e.g.  $\mathbf{X}_i,\mathbf{X}^i$  denote the  $i$ -th matrix in a matrices sequence. We use  $\sigma$  and  $\bar{\sigma}$  for softmax and hardmax operators, respectively. We use  $\mathrm{ReLU}(\mathbf{v}) = \max (\mathbf{v},\mathbf{0})$  to denote the ReLU activation function where  $\max (\cdot)$  function is applied entry-wise to a vector. We use  $\mathrm{Cone}(\mathbf{a}_1,\mathbf{a}_2,\dots,\mathbf{a}_m)$  to denote a cone without its origin point where  $\mathrm{Cone}(\mathbf{a}_1,\mathbf{a}_2,\dots,\mathbf{a}_m) = \{\mathbf{x}:$ $\mathbf{x} = \sum_{i = 1}^{m}a_{i}\mathbf{a}_{i},a_{i} > 0\}$ . We also define the minus operation between a set  $S$  and a vector  $\mathbf{v}$  as  $S - \mathbf{v} = \{\mathbf{x} - \mathbf{v}:\mathbf{x}\in S\}$ . In Section 4, we use  $[a:b:c]$  to denote a grid  $\{a,a + b,a + 2b,\ldots ,c - b\}$  from  $a$  to  $c$ , with an interval  $b$ .
Transformer networks [Vaswani et al., 2017] are a stack of multiple transformer layers, composed subsequently. A transformer layer has two key components: an attention layer and a token-wise MLP layer, with residual connections around both blocks. We consider the input and output to be sequences of tokens  $\mathbf{X} \in \mathbb{R}^{d \times m}$  and  $\mathbf{Y} \in \mathbb{R}^{d \times m}$ , where  $m$  is the number of tokens in the sequence and  $d$  is the token dimension.
Definition 1 (Attention Layer). We define an  $h$ -head attention layer parameterized with  $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o$  between a single token  $\mathbf{x}$  and a token sequence  $\mathbf{X}$  as
$$
A t t (\mathbf {x}, \mathbf {X}) = \sum_ {i = 1} ^ {h} \mathbf {W} _ {o} ^ {i} \mathbf {W} _ {v} ^ {i} \mathbf {X} \cdot \sigma \left(\left(\mathbf {W} _ {k} ^ {i} \mathbf {X}\right) ^ {\top} \mathbf {W} _ {q} ^ {i} \mathbf {x}\right). \tag {1}
$$
The normalizing factor of  $\frac{1}{\sqrt{d_{kq}}}$  is subsumed in the weight matrices  $\mathbf{W}_k^i$  for notational simplicity.
We can then define the cross attention between two sequences  $\mathbf{X}_1\in \mathbb{R}^{d\times m_1}$  and  $\mathbf{X}_2\in \mathbb{R}^{d\times m_2}$  (We use  $\mathbf{x}_k = (\mathbf{X}_1)_{:,k}$  for simplicity):
$$
A t t \left(\mathbf {X} _ {1}, \mathbf {X} _ {2}\right) = \left[ A t t \left(\mathbf {x} _ {1}, \mathbf {X} _ {2}\right), A t t \left(\mathbf {x} _ {2}, \mathbf {X} _ {2}\right), \dots , A t t \left(\mathbf {x} _ {m _ {1}}, \mathbf {X} _ {2}\right) \right].
$$
Definition 2 (Standard Transformer Layer). With definition 1, we define a standard transformer layer  $\tau$  as
$$
M L P (\mathbf {X}) = \left[ \mathbf {W} _ {2} R e L U \left(\mathbf {W} _ {1} \mathbf {X} _ {:, 1} + \mathbf {b} _ {1}\right) + \mathbf {b} _ {2} + \mathbf {X} _ {:, 1}, \dots , \mathbf {W} _ {2} R e L U \left(\mathbf {W} _ {1} \mathbf {X} _ {:, n} + \mathbf {b} _ {1}\right) + \mathbf {b} _ {2} + \mathbf {X} _ {:, n} \right] \tag {2}
$$
$$
\tau (\mathbf {X}) = M L P (A t t (\mathbf {X}, \mathbf {X}) + \mathbf {X}). \tag {3}
$$
The definition here omits the layer normalization block for simplicity (following [Kim et al., 2021]).
We denote the set of transformer networks with  $h$  heads of size  $s$  and  $r$  MLP hidden neurons with  $\mathcal{T}^{h,s,r}$ . In Section 4, we utilize a modified transformer network with hardmax operation  $\bar{\sigma}$  instead of softmax  $\sigma$ . We denote this modified version of transformer networks as  $\bar{\mathcal{T}}^{h,s,r}$ .
During fine-tuning, we optimize the matrices  $\mathbf{W}_q^i,\mathbf{W}_k^i,\mathbf{W}_v^i$  in the attention layer and  $\mathbf{W}_1,\mathbf{W}_2,\mathbf{b}_1,\mathbf{b}_2$  in the MLP layer pertaining to a loss function  $\mathcal{L}$ . However in prompt tuning, the pretrained model weight matrices are fixed and we optimize a tunable sequence prepended to the input.
Prompt Tuning Given a pretrained transformer network  $g \in \mathcal{T}$  and a downstream training dataset  $S = \{(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)\}$ , prompt tuning seeks to find a prompt  $\mathbf{P}^* \in \mathbb{R}^{d \times m_p}$  with  $m_p$  tunable tokens under the loss function  $\mathcal{L}$ :
$$
\mathbf {P} ^ {*} = \arg \min  _ {\mathbf {P}} \sum_ {i = 1} ^ {n} \mathcal {L} \left(g \left(\left[ \mathbf {P}, \mathbf {X} _ {i} \right]\right) _ {:, m _ {p}:}, \mathbf {Y} _ {i}\right). \tag {4}
$$
The tunable prompt  $\mathbf{P}$  is shared amongst all the inputs in a task. Note that  $\mathbf{P}$  in prompt tuning is a continuously trainable parameter, alternately referred to as soft prompt, which is different from hard prompt in that the latter operates on a discrete space of predefined vocabulary. Since the representation power of soft prompts is strictly more than the hard prompts, the limitations studied in this paper also extend to hard prompts.
In the subsequent sections, we analyze the universality and limitations of prompt tuning while comparing the latter against fine-tuning and LoRA[Hu et al., 2021], which is a low-rank version of model fine-tuning. In Section 4, we prove that prompt tuning can be universal approximators for sequence-to-sequence functions, while providing the construction for the same. In Sections 5 and 6, we identify the failure modes where prompt tuning cannot learn with a possibly non-optimal but non-trivial pretrained transformer network.
4 Universality of Prompt Tuning
Without loss of generality, we assume that the support and range set of all considered sequence-to-sequence functions  $f$  is  $[0,1]^{d \times m}$  in this section. We define  $\mathcal{F}_L$  as the collection of all continuous sequence-to-sequence  $L$ -lipschitz functions under norm  $p$  and sequence length  $m$ . For  $f \in F_L$  and any two inputs  $\mathbf{X}, \mathbf{X}' \in [0,1]^{d \times m}$ , we have  $\| f(\mathbf{X}) - f(\mathbf{X}') \|_p \leq L \| \mathbf{X} - \mathbf{X}' \|_p$ . Furthermore, given functions  $f_1, f_2$ , the approximation error under a  $p$ -norm (which is entry-wise) is measured as:
$$
d _ {p} \left(f _ {1}, f _ {2}\right) = \left(\int \left\| f _ {1} (\mathbf {X}) - f _ {2} (\mathbf {X}) \right\| _ {p} ^ {p} d \mathbf {X}\right) ^ {\frac {1}{p}}. \tag {5}
$$
Primarily, we show that there exists a Transformer network  $g \in \mathcal{T}^{2,1,4}$  such that for any  $f \in \mathcal{F}_L$ , prompt tuning on  $g$  can approximate this function up to some error budget  $\epsilon > 0$ .
Theorem 1. Let  $1 \leq p < \infty$  and  $\epsilon > 0$ , there exist a transformer network  $g \in \mathcal{T}^{2,1,4}$  and prompt length  $m_p$ , such that for any  $f \in \mathcal{F}_L$  we can find a prompt  $\mathbf{P} \in \mathbb{R}^{d \times m_p}$  with  $d_p(g([\mathbf{P}, \cdot])_{:, m_p:}, f) \leq \epsilon$ .
Here we use the transformer in a encoder mode which generates the  $m$  outputs in one step. In Appendix C.4, a similar result can be obtained for next-token prediction, which is widely used in many recent language models.
The proof is inspired from [Yun et al., 2019a], which follows the typical construction based on the mechanism to show universality. Thereby, we can construct a "meta-transformer" for prompt tuning to approximate any sequence-to-sequence function with prompt tuning. Next we briefly describe the two steps for the construction of this meta-transformer. We start by building a meta-function for  $\mathcal{F}_L$ .
Building the Meta-Function We denote the length of all inputs as  $m$  and the prompt length as  $m_p$ . Then we can build a sequence-to-sequence meta-function that accepts inputs with length  $m + m_p$ .
Lemma 1. For the sequence-to-sequence function space  $\mathcal{F}_L$  with functions  $f:[0,1]^{d\times m}\to [0,1]^{d\times m}$ , we can build a sequence-to-sequence function  $\bar{g}:[0,1]^{d\times (m_p + m)}\to [0,1]^{d\times (m_p + m)}$  such that for any  $f\in \mathcal{F}_L$ , we can find  $\mathbf{P}\in \mathbb{R}^{d\times m_p}$ ,  $d_{p}(\bar{g} ([\mathbf{P},\cdot ])_{:,m_{p};},f)\leq \epsilon /2$ .
The complete proof is given in Appendix C.1. Succinctly, we first quantize the input and output sequence space of  $[0,1]^{d\times m}$  into a grid  $G_{\delta ,m} = \{0,\delta ,2\delta ,\ldots ,1 - \delta \}^{d\times m}$ , thus leading to  $C = \left(\frac{1}{\delta^{d\times m}}\right)^{\frac{1}{\delta^{d\times m}}}$  possible functions mappings from the input to the output, in this discrete space. By
this quantized function space as  $\bar{\mathcal{F}}_L = \{\bar{f}_1,\bar{f}_2,\dots,\bar{f}_C\}$ , we can select  $\delta$  such that the approximation error for any function is less than  $\epsilon /2$ . Then we construct a set of quantized prompts in  $G_{\delta ,m_p} = \{0,\delta ,2\delta ,\dots,1 - \delta \}^{d\times m_p}$  to index these  $C$  functions and construct a quantized function  $\bar{g}$  where  $\bar{g} ([\mathbf{P}_i,\mathbf{X}])_{:,m_p} = \bar{f}_i(\mathbf{X}),i = 1,2,\ldots,C$ , for all  $\mathbf{X}\in G_{\delta ,m}$ , thereby concluding the lemma.
Next we can utilize some conclusions in [Yun et al., 2019a] to construct a transformer for  $\bar{g}$ .
Constructing the Meta-Transformer We first introduce a useful lemma which enables the construction of a transformer for any quantized sequence-to-sequence function.
Lemma 2. For any given quantized function  $\bar{f}:[0,1]^{d\times m}\to [0,1]^{d\times m}$  with quantization at interval
$$
\delta , \exists \bar {h} \in \bar {\mathcal {T}} ^ {2, 1, 1} \text {s u c h t h a t} \bar {f} = \bar {h} \text {w i t h p o s i t i o n a l e m b e d d i n g} \mathbf {E} = \left[ \begin{array}{c c c c c} 0 & 1 & 2 & \ldots & m - 1 \\ 0 & 1 & 2 & \ldots & m - 1 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 1 & 2 & \ldots & m - 1 \end{array} \right].
$$
The proof mainly follows the discussions in Section C of [Yun et al., 2019a]. To prove this lemma, the network  $\bar{h}$  can be constructed in the following three steps. We first use a series of MLP layers to quantize the input to grid  $[0:\delta:1-\delta]^{d\times m}$  and then a series of attention layers to obtain a unique contextual mapping for each quantized input. Finally we can use a series of MLP layers to map the unique contextual mapping to the desired outputs. While a transformer network usually stacks self-attention and MLP layers alternately within a single layer, the aforementioned construction can be trivially attained via the use of skip connections. The complete proof of Lemma 2 is deferred to Appendix C.2.
Since  $\bar{g}$  is a quantized function in grid  $G_{\delta, m + m_p}$ , following Lemma 2 we can find a modified version of transformer  $\bar{h} \in \bar{\mathcal{T}}^{2,1,1}$  such that  $\bar{g}([\mathbf{P}, \mathbf{X}]) = \bar{h}([\mathbf{P}, \mathbf{X}])$ . The modified version of transformer  $\bar{g}$  with hardmax operators can then be approximated with a standard transformer  $g$  with softmax operators by Lemma 3.
Lemma 3 (Lemma 9 in [Yun et al., 2019a]). For each  $\bar{h} \in \bar{\mathcal{T}}^{2,1,1}$ ,  $\epsilon > 0$  and  $1 \leq p < \infty$ ,  $\exists g \in \mathcal{T}^{2,1,4}$  such that  $d_p(\bar{h}, g) \leq \epsilon / 2$ .
Since the approximation error can be treated uniformly amongst the  $\mathbf{P}_i$ , we have that  $d_p(\bar{h}([\mathbf{P}_i, \cdot])_{:,m_p:}, g([\mathbf{P}_i, \cdot])_{:,m_p:}) \leq d_p(\bar{h}([\mathbf{P}_i, \cdot]), g([\mathbf{P}_i, \cdot]) \leq \epsilon/2$ . Therefore, we can build a transformer  $g \in \mathcal{T}^{2,1,4}$ , such that for any sequence-to-sequence  $f \in \mathcal{F}_L$ , we can find a quantized version  $\bar{f}_i \in \bar{\mathcal{F}}_L$  and the corresponding prompt  $\mathbf{P}_i \in G_{\delta,m_p}$  such that
$$
d _ {p} \left(g \left(\left[ \mathbf {P} _ {i}, \cdot \right]\right) _ {:, m _ {p}:}, f\right) \leq d _ {p} \left(g \left(\left[ \mathbf {P} _ {i}, \cdot \right]\right) _ {:, m _ {p}:}, \bar {h} \left(\left[ \mathbf {P} _ {i}, \cdot \right]\right)\right) + d _ {p} \left(\bar {h} \left(\left[ \mathbf {P} _ {i}, \cdot \right]\right) _ {:, m _ {p}:}, \bar {f} _ {i}\right) + d _ {p} \left(\bar {f} _ {i}, f\right) \leq \epsilon . \tag {6}
$$
Theorem 1 provides the construction for a large transformer (discussed more in appendix) that is sufficient for prompt tuning to exhibit universal approximation over a Lipschitz function space. However, even this strong transformer also has limitations with prompt tuning when the target function  $f \notin \mathcal{F}_L$ . Is this an essential limitation for prompt tuning on any transformer? In the next section, we will theoretically analyze the limitations of prompt tuning with transformers and target functions under more general conditions.
5 Limitations of Prompt-Tuning: Single Layer Transformer
To analyse the failure modes and therefore the limitations under the setting where a transformer has fixed pretrained weights, we follow the lens of exact memorization in the subsequent sections.
Definition 3 (Memorization of a Sequence-to-Sequence Dataset). Given a sequence-to-sequence dataset  $S = \{(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)\}$  where  $\mathbf{X}_i, \mathbf{Y}_i \in \mathbb{R}^{d \times m}$  are the input/output sequences, we consider a function  $f$  exactly memorizing dataset  $S$  if  $f(\mathbf{X}_i) = \mathbf{Y}_i$ . In the following proofs of this section, we explicitly focus on the last output token, ie:  $f(\mathbf{X}_i)_{:, -1} = (\mathbf{Y}_i)_{:, -1}$ .
We start from the analysis on a single layer transformer and extend to multi-layer settings in Section 6.
5.1 Failure modes of Prompt Tuning
It is straightforward to note that prompt tuning has limited expressive power when the number of trainable parameters is limited. A natural question to then ask is: Does increasing the number of trainable prompt tokens suffice? While it is known that for MLPs, even with a single hidden layer, increasing the number of hidden neurons can memorize any training data [Yun et al., 2019b]. However, as we will prove next, this is not the case for prompt tuning. This result highlights an essential limitation of prompt tuning compared to model fine-tuning.
Before providing the theorem statement, we first outline some straightforward assumptions on the pretrained transformer and datasets, without which prompt tuning trivial loses expressive power.
We consider sequence-to-sequence datasets of the form  $S = \{(\mathbf{X}_1, \mathbf{Y}_1), (\mathbf{X}_2, \mathbf{Y}_2), \dots, (\mathbf{X}_n, \mathbf{Y}_n)\}$  with  $n$  distinct examples and a single-layer single-head standard transformer defined in Definition 2. The results can be directly extended to the single-layer multi-head scenario, which we skip here to avoid notational clutter.