SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junda Wu $^{1*}$  Tong Yu $^{2*}$  Rui Wang $^{3}$  Zhao Song $^{2}$  Ruiyi Zhang $^{2}$  Handong Zhao $^{2}$  Chaochao Lu $^{4\dagger}$  Shuai Li $^{5\dagger}$  Ricardo Henao $^{3,6}$ $^{1}$ University of California, San Diego  $^{2}$ Adobe Research  $^{3}$ Duke University  $^{4}$ University of Cambridge  $^{5}$ Shanghai Jiao Tong University  $^{6}$ KAUST juw069@ucsd.edu  $\{\mathrm{tu}, \mathrm{z} \mathrm{Song}, \mathrm{ru} \mathrm{i} \mathrm{z} \mathrm{h} \mathrm{a} \mathrm{g}, \mathrm{h} \mathrm{a} \mathrm{z} \mathrm{h} \mathrm{a} \} @$ adobe.com  $\{\mathrm{rw} 161, \mathrm{ricardo.henao}\} @$ duke.edu cl641@cam.ac.uk shuaili8@sjtu.edu.cn
Soft prompt tuning achieves superior performances across a wide range of few-shot tasks. However, the performances of prompt tuning can be highly sensitive to the initialization of the prompts. We have also empirically observed that conventional prompt tuning methods cannot encode and learn sufficient task-relevant information from prompt tokens. In this work, we develop an information-theoretic framework that formulates soft prompt tuning as maximizing the mutual information between prompts and other model parameters (or encoded representations). This novel view helps us to develop a more efficient, accurate and robust soft prompt tuning method, InfoPrompt. With this framework, we develop two novel mutual information based loss functions, to (i) explore proper prompt initialization for the downstream tasks and learn sufficient task-relevant information from prompt tokens and (ii) encourage the output representation from the pretrained language model to be more aware of the task-relevant information captured in the learnt prompts. Extensive experiments validate that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods. Finally, we provide a formal theoretical result to show that a gradient descent type algorithm can be used to train our mutual information loss.
Soft prompt tuning has shown great successes in a wide range of natural language processing tasks, especially in low-resource scenarios [60, 37, 66]. With a relatively small size of prompt parameters appended to the input of the context, the language model can be adapted to the downstream tasks with the large scale pretrained parameters frozen. Compared with conventional fine tuning methods, prompt tuning requires less memory and computational resources to update these significantly smaller sized prompt parameters. In addition, in low-shot learning scenarios, prompt tuning can prevent the language model from overfitting on the training data, thus maintaining the generalization ability of pretrained language models.
However, recent works reveal that it is non-trivial to find a proper initialization of the prompt tokens. Several works have investigated the effect of prompt initialization on the prompt tuning performances
†The work was done previously when this author was at the University of Cambridge. This author is now at Shanghai AI Laboratory.
(a) MRPC. Prompt handcraft 1 is 'it is equivalent to' and prompt handcraft 2 is 'has the same meaning'.
(b) SST-2. Prompt handcraft 1 is 'this is positive' and prompt handcraft 2 is 'this is negative'.
Figure 1: Distributions of the CMI metrics of the prompts learned or handcrafted from different methods on MRPC and SST-2 [95]. By our InfoPrompt, the relevance between the prompt tokens and downstream tasks is the highest among all methods.
[91, 101] and showed that the performances of prompt tuning are highly sensitive to the prompt initialization. However, since the proper prompt initialization can vary to different downstream tasks and pretrained language models, it is hard to find very accurate knowledge to guide us to obtain the proper initialization [13].
In addition to the above limitations, , we also empirically observe that conventional prompt tuning methods cannot effectively learn sufficient task-relevant information from prompt tokens. Specifically, the prompts may fail to learn sufficient information that is relevant to the downstream tasks. To understand the relevance between the prompt tokens and downstream tasks, we calculate the conditional mutual information (CMI) between the prompt tokens and the latent representation from the language model conditioned on the input context. We follow [40] in determining the positions of prompt tokens inserted between the input sentences. Figure 1 shows the distribution of CMI of the prompts resulting from different methods. The randomly sampled prompts have the lowest CMI. The prompts learned by a soft prompt tuning method, WARP [40], can have relatively higher CMI than the handcrafted ones. By directly maximizing the CMI, our InfoPrompt (detailed in Section 3) facilitates learning of more informative prompts. Without the guidance of task-relevant information, randomly exploring the optimal prompts within the large continuous embedding space of the prompt tokens can be inefficient, i.e. a similar challenge is also discussed in [75]. Some related results [75, 37, 78] show that prompt tuning takes much larger numbers of epochs to converge than fine tuning. Comparatively, thanks of the guidance of CMI, our propose InfoPrompt allows prompt tuning to converge much faster. We also provide theoretical guarantees of the convergence of the proposed losses when training with gradient descent based algorithms.
Overall, we develop an information-theoretic framework that formulates soft prompt tuning as maximizing mutual information between prompts and other model parameters (or encoded representations), conditioned on the input context. With this framework, we develop InfoPrompt with two novel mutual information based loss functions. (i) To explore proper prompt initialization for the downstream tasks and learn prompts with sufficient task-relevant information, we optimize the head loss which maximizes the mutual information between the prompt and the task-specific classification head. By optimizing this head loss, the prompt can effectively learn task-relevant information from the downstream tasks, since the task-specific classification head usually contains the information from the downstream tasks. Besides, the mutual information loss can help to guide the learning of the prompt tokens in the early steps to tackle the initialization challenge, since the classification head parameters can learn the downstream task information more quickly. (ii) To further encourage the output representation from the pretrained language model (i.e., encoded representation) to be more aware of the task-relevant information captured in the learnt prompt, we optimize the representation loss which maximizes the conditional mutual information between the prompt embeddings and the feature representations conditioned on the input context.
Our contributions are summarized as:
- We revisit the challenge of initialization with prompt tuning and show that existing prompt tuning methods fail to learn sufficient task-relevant information.
- We propose InfoPrompt, a framework to solve such challenges from a information-theoretic perspective. Specifically, we develop two novel loss functions that effectively find proper
prompt initialization and learn sufficient task-relevant information from down-stream tasks, without requiring any prior knowledge.
- Extensive experiments on multiple tasks and datasets validate that, InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform existing prompt tuning methods with higher accuracy.
- We provide a formal theoretical result to show that our proposed loss functions can be optimized using gradient descent based algorithm with convergence guarantees.
2.1 Prompt Tuning
Prompt tuning has shown great successes in a wide range tasks of NLP [60, 37, 66]. Let  $\Phi$  denote the encoder of a pretrained language model, e.g., the Roberta-Large [67]. Assume  $X = \{x_{1}, x_{2}, \dots, x_{n}\}$  is a length- $n$  text sequence of and  $Y$  is its classification label. In prompt tuning, we add extra information  $P$  for the encoder to condition on for its prediction of  $Y$ .  $P = \{p_{1}, \ldots, p_{n_{p}}\}$  is a sequence of prompt embeddings and  $n_{p}$  is the number of prompt tokens.  $p_{i} \in \mathbb{R}^{D}$ ,  $i = 1, \dots, n_{p}$ , is an embedding vector with dimension  $D$  and  $D$  is also the embedding dimension of the pretrained language model. We first embed each token of  $X$  into its corresponding token embedding from the pretrained language model.  $P$  is inserted into the resulting embedding sequence of  $X$ , and the resulting sequence is further encoded by the pretrained encoder  $\Phi$  into the representation space of the pretrained language model. The template for inserting of prompt tokens is detailed in Section 4. Formally, we denote such a process by  $Z = \Phi(P, X)$ , with  $Z$  being the output representation from the pretrained encoder. The model prediction for  $X$  is made on top of  $Z$  with a trainable classification head parameterized by  $\theta$ , denoted as  $h_{\theta}$ , whose output  $h_{\theta}(Z)$  is the probability distribution over all possible classification labels. For classification, the prompts are trained via minimizing the following loss function,
$$
\mathcal {L} _ {\text {p r e d}} = \text {c r o s s - e n t r o p y} (h _ {\theta} (Z), Y)
$$
Parameters of the pretrained encoder  $\Phi$  is frozen during training. Different from previous works (e.g., [60]) where the prompts are directly learnt, the prompts in our approach are encoded from the input  $X$ . In this way, the resulting prompts can better capture task-relevant information from the training text  $X$ . We will elaborate on how the prompts are encoded from input  $X$  in Section 4.
2.2 Mutual Information
Mutual information (MI) is a metric in information theory [79, 19], which quantifies the amount of information shared between two random variables. The mutual information between two random variables  $A$  and  $B$  is
$$
\mathcal {I} (A; B) = \mathbb {E} _ {p (a, b)} [ D _ {K L} [ p (a | b) \| p (b) ] ].
$$
Inspired by [89], we use MI as the criterion for comparing prompts and other model parameters. MI has also been applied to measure the similarity between the masked token and the corresponding context in the pretraining of Multilingual Masked Language Modeling [15], the relevance between documents and sentences in document summarization [72] and the source and target sentences in Neural Machine Translation (NMT) [107].
3 Our Method: InfoPrompt
As mentioned above, we want the learnt prompts to be task-relevant. To achieve this, we notice that the classification head is trained with both the data representation and the classification labels, thus should contain rich information of the learnt tasks. In encouraging the task-relevancy of the learnt prompts, we consider maximizing the mutual information between the prompt and the parameters of the classification head, denoted as  $\theta$ . By maximizing such mutual information, the learnt prompt will be more aligned with the training data with which the classification head is trained, thus captures more task-relevant information from training. Further, in order for the pretrained language model to
properly leverage the task-relevant information in the prompt, we additionally maximize the mutual information between the prompt and the representation from the pretrained language model, so that the encoded representation can be aware of the task-relevant information captured by the prompt. In addition, we also provide theoretical guarantees of the convergence of those losses when training with gradient descent, demonstrating that our method can converge more easily than existing prompt tuning methods. Below, we denote the negative mutual information between the prompt and parameters of the classification head as the head loss. The negative mutual information between the prompt and representations from the pretrained language model is denoted as the representation loss.
3.1 The Head Loss
The head loss is the negative mutual information between the prompt  $P$  and parameters  $\theta$ , i.e.,  $-I(P; \theta | X)$ . In maximizing  $I(P; \theta | X)$ , we follow [68] that approximates it with the following lower bound,
$$
\mathcal {I} (P; \theta | X) \geq C + \mathcal {L} _ {N C E} (P, \theta , X),
$$
where  $C$  is a constant,  $\mathcal{L}_{NCE}$  is a Noise Contrastive Estimation (NCE) of mutual information,
$$
\mathcal {L} _ {N C E} = \mathbb {E} \left[ \log \frac {\exp (l (P , \theta , X))}{\sum_ {k = 1} ^ {K} \exp (l (P _ {k} , \theta | X))} \right],
$$
and  $\{P_k\}_{k=1}^K$  are the negative prompt samples for contrastive learning. In practice, we randomly sample  $K-1$  tokens from the context as the negative samples, i.e.,  $\{P_k\}_{k=2}^K$ , and the positive sample is  $P_1 = P$ .
We model the score function  $l(P, \theta | X)$  as a standard bilinear function with the learnable matrix  $W_{1}$
$$
l (P, \theta | X) = P ^ {\top} W _ {1} \theta .
$$
where  $\theta$  and  $P$  are encoded from  $X$ , and  $W_{1}$  is a trainable matrix. Since the classification head is learnt on top of the output from the last layer of the pretrained language model, the learning of its parameters  $\theta$  is easier than the learning of the prompt  $P$  (the input layer of the pretrained language model). Therefore,  $\theta$  may capture more task-relevant information than  $P$  in the early stage of training. By maximizing the mutual information between  $\theta$  and  $P$ , the task-relevant information can be transferred to  $P$  in the initial training steps. In this way,  $P$  can be more task-relevant especially in the early training stage. Experiments in Section 6 also show that our head loss,  $\mathcal{I}(P; \theta | X)$ , can facilitate the training of the initial training steps.
3.2 The Representation Loss
The representation loss, denoted as  $-\mathcal{I}(P;Z|X)$ , is defined as the negative of mutual information between the prompt  $P$  and the encoded representation from the pretrained language model, i.e.,  $Z = \Phi(P,X)$ . Similar to the head loss, we approximate the representation loss with its lower bound,
$$
\mathcal {I} (P; Z | X) \geq \log (N) + \mathcal {L} _ {N C E} (P, Z | X),
$$
$$
\mathcal {L} _ {\mathrm {N C E}} = \mathbb {E} \left[ \log \frac {\exp (l (P , Z | X))}{\sum_ {k = 1} ^ {K} \exp (l (P , Z _ {k} | X))} \right],
$$
$\{Z_k\}_{k=1}^K$  are the negative samples. Here, we overload the notations of InfoNCE loss  $\mathcal{L}_{NCE}$  and score function  $l$  for conciseness. Let  $W_2$  be a trainable matrix, the function  $l$  for the representation loss is defined by,
$$
l (P, Z | X) = P ^ {\top} W _ {2} Z.
$$
We use variational inference methods [46] to recover the latent distribution of  $Z$ . Specifically, we assume that the latent distribution is  $N(\mu, \sigma)$ , where  $N(\mu, \sigma)$  is the normal distribution with mean  $\mu$  and diagonal covariance matrix  $\sigma$ . We model  $\mu$  and  $\sigma$  via,
$$
\mu = f _ {\mu} (Z), \sigma = f _ {\sigma} (Z).
$$
$f_{\mu}$  and  $f_{\mu}$  are trainable fully connected layers. Since the negative samples of  $Z$ , i.e.,  $\{Z_k\}_{k=1}^K$ , should not be paired with  $P$ , we assume the  $\{Z_k\}_{k=1}^K$  are drawn from  $N(\mu', \sigma')$ , s.t.,
$$
\mu^ {\prime} = f _ {\mu} \left(Z ^ {\prime}\right), \sigma^ {\prime} = f _ {\sigma} \left(Z ^ {\prime}\right).
$$
In contrast to  $Z = \Phi(P, X)$ , we have  $Z' = \Phi(X)$  where  $\{Z_k\}_{k=1}^K$  are not paired with  $P$ . By maximizing the representation loss  $I(P; Z|X)$ , we encourage the encoded representation  $Z$  to be more aware of the prompt  $P$ , so that the task-relevant information in  $P$  can be properly encoded by the pretrained language model in producing  $Z$ .
3.3 Overall Objective
We minimize the following objective in prompt tuning:
$$
\mathcal {L} = \mathcal {L} _ {\text {p r e d}} - \beta \cdot \mathcal {I} (P; Z | X) - \gamma \cdot \mathcal {I} (P; \theta | X). \tag {1}
$$
We denote  $\mathcal{L}_{\mathrm{pred}}$  as the task loss.  $\beta$  and  $\gamma$  are balancing parameters for the proposed representation loss and head loss, respectively. We denote our approach as InfoPrompt. More details about the implementation and configurations are provided in Section 4.2.
3.4 Theoretical Guarantees
We state our main theoretical result as follows. Due to the space limit, we delay the proof into Appendix.
Theorem 3.1. Given the Loss function  $\mathcal{L}$  (Eq. (1)) and conditions specified in Appendix C.1 and  $D$ , using gradient descent type of greedy algorithm, we can find the optimal solution of that loss function.
We provide theoretical guarantees of the convergence of those losses trained by conventional gradient descent type algorithms. In Section 4, we empirically observe that our method converges more easily than traditional soft prompt tuning methods and requires fewer training epochs.
We conduct experiments with datasets of sequence classification from the GLUE benchmark [95], along with those of relation extraction tasks and NER tasks. We choose four sequence classification tasks from the GLUE benchmark: RTE (Recognizing Textual Entailment, [7]), MRPC (Microsoft Research Paraphrase Corpus, [28]), CoLA (Corpus of Linguistic Acceptability, [98]) and SST-2 (Sentence Sentiment Treebank, [81]). We choose these tasks because their datasets are of smaller sizes and prompt tuning is comparably more effective in low-resource settings [40, 62]. For the task of relation extraction, we evaluate our method on the ACE2005 corpus and the Semeval-2010 datasets [44]. We also use the ACE2005 corpus for the task of NER. Note that the entity spans for NER have been given ACE2005. Unlike the standard NER model that learns to predict the entity span and entity type simultaneously from the raw text sequence, our model only predicts the entity type based on the given entity span. We follow the same data splitting strategy for ACE2005 corpus as the previous work [103, 71]. For the Semeval-2010 tasks, we follow the official data partition [44].
4.2 Experiment Settings
We follow the resource constrained scenario in [40] that trains each task with only 64 or 256 samples. We experiment with  $n_p = 1$  and  $n_p = 4$  prompt tokens for each task. The prompt tokens are inserted into the template for each task. Similar to [40], we adopt the RoBERTa-large model as our pretrained encoder. We freeze the pretrained parameters and only train the parameters of the prompt head and prompt tokens. During training, we empirically set  $\beta = 0.1$  and  $\gamma = 0.05$ . The number of negative samples is  $K = 32$ . The learning rate is  $1e - 3$  and the batch size is 8. For each task, we report the results after 30 epochs, averaged over 5 random seeds. To encode the prompt  $P = [p_1,\dots ,p_{n_p}]$  from  $X$ , we first encode  $X$  into  $P^{\prime}\in \mathbb{R}^{D}$  via  $P^{\prime} = \Phi (X)$ . We denote the up-sampling and down-sampling
Table 1: Results on Sequence Classification.
CoLA
np=1 np=4	RTE
np=1 np=4	MRPC
np=1 np=4	SST2
np=1 np=4	Average
Finetuning	0.6131	0.7798	0.8873	0.9427	0.8057
Adapter [45]	0.5552	0.5776	0.6814	0.9472	0.6904
WARP [40]	0.5282	0.5911	0.6282	0.6426	0.8039	0.8186	0.9507	0.9587	0.7403
IDPG [102]	0.5556	0.5646	0.6282	0.6534	0.7941	0.8039	0.9587	0.9587	0.7396
InfoPrompt	0.5631	0.6018	0.6751	0.6968	0.8039	0.8137	0.9576	0.9599	0.7590
γ=0	0.5699	0.5853	0.6751	0.6787	0.7941	0.8137	0.9495	0.9587	0.7531
β=0	0.5546	0.5579	0.6065	0.6318	0.7892	0.7966	0.9472	0.9610	0.7306
γ=0,β=0	0.5032	0.5732	0.6173	0.6029	0.7917	0.7672	0.9495	0.9564	0.7202
Table 2: Results on Relation Extraction and NER.
RE	NER	SemEval	Average
np=1	np=4	np=1	np=4	np=1	np=4
Finetuning	0.8119	0.9054	0.8506	0.8560
Adapter [45]	0.5073	0.8329	0.6570	0.6657
WARP [40]	0.6384	0.6596	0.8174	0.8607	0.6702	0.7284	0.7291
IDPG [102]	0.6079	0.6132	0.8360	0.8931	0.6408	0.6776	0.7114
InfoPrompt	0.6914	0.7616	0.8526	0.8962	0.7563	0.7917	0.7916
γ=0	0.6914	0.7285	0.8452	0.8635	0.7471	0.7865	0.7770
β=0	0.6967	0.7470	0.8351	0.8698	0.7449	0.7538	0.7746
γ=0,β=0	0.5364	0.7285	0.8512	0.8661	0.7490	0.7799	0.7519
projections similar in [31]. For each  $p_i \in \mathbb{R}^D$ , we have  $p_i = W_i^{\mathrm{up}}W_i^{\mathrm{down}}P'$ ,  $W_i^{\mathrm{up}} \in \mathbb{R}^{D \times 64}$ ,  $W_i^{\mathrm{down}} \in \mathbb{R}^{64 \times D}$ .
For the tasks of sequence classification and relation extraction, we follow the template of [40] that contains a [mask] token. The representation  $Z$  is obtained from the [mask] token from the last layer of the RoBERTa-Large encoder. For the task of NER, we have the [mask] token before the given entity span, with the rest being the same as for sequence classification.
4.3 Baselines and Ablations
As mentioned above, our method with Eq. (1) in denoted as InfoPrompt. In the experiments, we compare our method with the following baselines:
- Finetuning: We fine tune all the parameters from the pretrained encoder on each task. Finetuning is included as the upper bound for the model performance, since it is more computational expensive compared with only training the prompt parameters.
- Adapter [45]: Similar to prompt tuning, this is also a way of parameter-efficient training for pretrained language models. Specifically, instead of adding the prompt tokens in the input, we add adapters after the feed-forward module in each transformer layer.
- WARP [40]: Different from our approach, the prompt tokens of WARP are not generated from the input sequence. The prompt tokens are insert into the input sequence. During training, the pretrained encoder is frozen and only the prompt tokens are trainable.
- IDPG [102]: Similar to our approach, the prompt tokens are generated from the input sequence. The pretrained encoder is frozen and the prompt generator is trainable.
In evaluating the effectiveness of our proposed loss functions, we consider the following two ablations:
-  $\gamma = 0$ : We disable the head loss during training via  $\gamma = 0$ , while keeping  $\beta = 0.05$ .
-  $\beta = 0$ : We disable the representation loss during training via  $\beta = 0$ , while keeping  $\gamma = 0.1$ .
-  $\beta = \gamma = 0$ : We disable both the losses. The prompt parameters are trained with  $L_{\mathrm{pred}}$ .
Table 3: Few-shot results on Sequence Classification. We experiment with  $N = 64$  and  $N = 256$  samples for each task. The number of prompt is fixed to  $n_p = 4$  for all soft prompt tuning methods.
CoLA	RTE	MRPC	SST2	Average
N = 64	N = 256	N = 64	N = 256	N = 64	N = 256	N = 64	N = 256
Finetuning	0.1746	0.4086	0.4801	0.6787	0.7107	0.7819	0.8027	0.8853	0.6153
Adapter [45]	0.0627	0.2486	0.5487	0.5668	0.5931	0.6250	0.4908	0.664	0.4750
WARP [40]	0.0749	0.0785	0.5596	0.5812	0.7083	0.7083	0.5872	0.7638	0.5077
IDPG [102]	0.0902	0.1513	0.5018	0.5523	0.6593	0.7010	0.5424	0.8188	0.5021
InfoPrompt	0.1567	0.1750	0.6137	0.6580	0.7059	0.7377	0.6697	0.7305	0.5559
γ = 0	0.1479	0.1447	0.5776	0.6318	0.6936	0.7328	0.664	0.7294	0.5402
β = 0	0.1372	0.1433	0.5812	0.5957	0.6838	0.7132	0.5631	0.656	0.5092
γ = 0, β = 0	0.0919	0.1397	0.5668	0.5523	0.6985	0.7108	0.5505	0.6296	0.4925
Table 4: Few-shot results on Relation Extraction and NER. We experiment with  $N = 64$  and  $N = 256$  samples for each task. The number of prompt is fixed to  $n_p = 4$  for all soft prompt tuning methods.
RE	NER	SemEval	Average
N = 64	N = 256	N = 64	N = 256	N = 64	N = 256
Finetuning	0.1285	0.4013	0.3033	0.4358	0.2223	0.4829	0.3290
Adapter [45]	0.1086	0.1815	0.2345	0.2437	0.1211	0.177	0.1777
WARP [40]	0.1404	0.2556	0.3082	0.4369	0.1708	0.3684	0.2801
IDPG [102]	0.2596	0.2503	0.3334	0.4048	0.1984	0.3577	0.3007
InfoPrompt	0.2119	0.2993	0.3331	0.4739	0.2113	0.4034	0.3222
γ = 0	0.2026	0.2834	0.3225	0.4776	0.2153	0.3739	0.3126
β = 0	0.2013	0.2874	0.3208	0.4615	0.2072	0.3629	0.3069
γ = 0, β = 0	0.1974	0.2728	0.3142	0.4662	0.2278	0.3276	0.3010
5.1 Training with the Full dataset