InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding
Authors: Junda Wu
Tong Yu
Rui Wang
Zhao Song
Ruiyi Zhang
Handong Zhao
Chaochao Lu
Shuai Li
Ricardo Henao
Venue: NeurIPS 2023 poster (2023)
Keywords: soft prompt tuning

Abstract: Soft prompt tuning achieves superior performances across a wide range of few-shot tasks. However, the performances of prompt tuning can be highly sensitive to the initialization of the prompts. We have also empirically observed that conventional prompt tuning methods cannot encode and learn sufficient task-relevant information from prompt tokens. In this work, we develop an information-theoretic framework that formulates soft prompt tuning as maximizing the mutual information between prompts and other model parameters (or encoded representations). This novel view helps us to develop a more efficient, accurate and robust soft prompt tuning method, InfoPrompt. With this framework, we develop two novel mutual information based loss functions, to (i) explore proper prompt initialization for the downstream tasks and learn sufficient task-relevant information from prompt tokens and (ii) encourage the output representation from the pretrained language model to be more aware of the task-relevant information captured in the learnt prompts. Extensive experiments validate that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods. Finally, we provide a formal theoretical result to show that a gradient descent type algorithm can be used to train our mutual information loss.

InfoPrompt: Information-Theorrii Soft Prompt Tuning for Natural Language Understanding
Junda Wu!
Tong Yu²
Rui Wang
Zhao Song"
Ruiyi Zhang?
Handong Zhao? Chaochao Lu
Shuai Li
Ricardo Henao 3.6
Uniivrssity of California, San Diego Adobe Research
Duke University
*University of Cambridge Shanghai Jiao Tong University
"KAUST
jux0698ucsd. edu
(yyu, zsong, saizzhanghhazzae haa.aaaaaa@adobeee Gadobe. com new161,ricare heeao @duke. edu c16410caa ac. uk shuaili80sjtun jtu edu. en
Abstract
Soft prompt tuning achieves superior performances across wide range of few-shot tasks. However, the performances of prompt tuning can be highly sensitive to the initialization of the prompts. We have also empirically observed that conventional prompt tuning methods cannot encode and learn sufficient task-relevant information from prompt tokens. In this work, we develop an irformation-thhoreee framework that formulates soft prompt tuning as maximizing the mutual information between prompts and other model parameters (or encoded representations). This novel view helps us to develop more efficient, accurate and robust soft prompt tuning method, InfoPrompt. With this framework, we develop two novel mutual information based loss functions, to (i) explore proper prompt initialization for the downstream tasks and learn sufficient task-relevant information from prompt tokens and (ii) encourage the output representation from the pretrained language model to be more aware of the task-relevant information captured in the learnt prompts. Extensive experiments validate that InfoPrompt can significantly accelerate the convergence the prompt tuning and outperform traditional prompt tuning methods. Finally, we provide formal theoretical result to show that gradient descent type algorithm can be used to train our mutual information loss.
Introduction
Soft prompt tuning has shown great successes in wide range of natural language processing tasks, especially in low-resource scenarios [60, 37, 66]. With a relatively small size of prompt parameters appended to the input of the context, the language model can be adapted to the downstream tasks with the large scale pretrained parameters frozen. Compared with conventional fine tuning methods, prompt tuning requires less memory and computational resources update these significantly smaller sized prompt parameters. In addition, in low-shot learning scenarios, prompt tuning can prevent the language model from overfitting on the training data, thus maintaining the generalization ability of pretrained language models.
However, recent works reveal that it is non-trivial to find proper initialization of the prompt tokens. Several works have investigated the effect of prompt initialization on the prompt tuning performances
These authors contributed equally to this work.
The work was done previously when this author was at the University of Cambridge. This author is now Shanghai AI Laboratory. 1 *Correspondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

10-
40
20
Handcraft
Nethod
Method
(a) MRPC. Prompt handcraft 1 is 'it is equivalent to' and prompt handcraft
(b) SST-2. Prompt handcraft I is 'this is positive' and prompi handcraft is 'this negative'.
the same meaning'.
Figure 1: Distributions of the CMI metries of the prompts learned or handcrafted from different
methods on MRPC and SST-2 [95]. By our InfoPrompt, the relevance between the prompt tokens and downstream tasks is the highest among all methods.
[91, 101] and showed that the performances of prompt tuning are highly sensitive the prompt initialization. However, since the proper prompt initialization can vary to different downstream tasks and pretrained language models, it is hard find very accurate knowledge guide us obtain the proper initialization [13].
In addition to the above limitations, we also empirically observe that conventional prompt tuning methods cannot effectively learn sufficient task-relevant information from prompt tokens. Specifi- cally, the prompts may fail to learn sufficient information that is relevant to the downstream tasks. To understand the relevance between the prompt tokens and downstream tasks, we calculate the conditional mutual information (CMI) between the prompt tokens and the latent representation from language model conditioned on the input context. We follow [40] in determining the positions of prompt tokens inserted between the input sentences. Figure 1 shows the distribution of CMI of the prompts resulting from different methods. The randomly sampled prompts have the lowest CMI. The prompts learned by a soft prompt tuning method, WARP [40], can have relatively higher CMI than the handcrafted ones. By directly maximizing the CMI, our InfoPrompt (detailed in Section 3) facilitates learning of more informative prompts. Without the guidance of task-relevant information, randomly exploring the optimal prompts within the large continuous embedding space the prompt tokens can be inefficient, i.e. similar challenge is also discussed in [75]. Some related results [75, 37, 78] show that prompt tuning takes much larger numbers of epochs to converge than fine tuning. Comparatively, thanks of the guidance of CMI, propose InfoPrompt allows prompt tuning to converge much faster. We also provide theoretical guarantees of the convergence of the proposed losses when training with gradient descent based algorithms.
Overall, we develop an information-theowm maximizing mutual information between prompts and other model parameters (or encoded represen- tations), conditioned on the input context. With this framework, we develop InfoPrompt with two novel mutual information based loss functions. (i) To explore proper prompt initialization for the downstream tasks and learn prompts with sufficient task-relevant information, we optimize the head loss which maximizes the mutual information between the prompt and the task-specific classification head. By optimizing this head loss, the prompt can effectively learn task-relevant information from the downstream tasks, since the task-specific classification head usually contains the information from the downstream tasks, Besides, the mutual information loss can help to guide the learning of the prompt tokens in the early steps tackle the initialization challenge. since the classification head parameters can learn the downstream task information more quickly. (ii) To further encourage the output representation from the pretrained language model (i.e, encoded representation) to be more aware of the task-relevant information captured in the learnt prompt, we optimize the representation loss which maximizes the conditional mutual information between the prompt embeddings and the feature representations conditioned on the input context.
framework that formulates soft prompt tuning as
Our contributions summarized as:
We revisit the challenge of initialization with prompt tuning and show that existing prompt tuning methods fail to learn sufficient task-relevant information.
We propose InfoPrompt, framework to solve such challenges from a information -hheorrticcccc perspective. Specifically, develop two novel loss functions that effectively find proper

---

prompt initialization and learn sufficient task-relevant information from down-stream tasks, without requiring any prior knowledge.
Extensive experiments multiple tasks and datasets validate that, InfoPrompt signif- icantly accelerate the convergence of the prompt tuning and outperform existing prompt tuning methods with higher accuracy.
We provide formal theoretical result to show that our proposed loss functions can be optimized using gradient descent based algorithm with convergence guarantees.
2 Preliminary
2.1 Prompt Tuning
Prompt tuning has shown great successes a wide range tasks of NLP [60, 37, 66]. Let denote the encoder of a pretrained language model, e.g., the Roberta-Large [67]. Assume X {I1, T2. { 2n)
length-n text sequence of and Y is its classification label. In prompt tuning, we add extra information P for the encoder condition on for its prediction of Y. P {PI..... Pn, is sequence of prompt embeddings and ni is the number of prompt tokens. p; an embedding vector with dimension D and D is also the embedding dimension of the pretrained
Bp. is language model. We first embed each token of X into its corresponding token embedding from the pretrained language model. P is inserted into the resulting embedding sequence of X, and the resulting sequence further encoded by the pretrained encoder into the representation space of the pretrained language model. The template for inserting of prompt tokens is detailed in Section 4. Formally, we denote such process by Z = + P. X), with Z being the output representation from the pretrained encoder. The model prediction for X is made on top of Z with trainable classification head parameterized by 0, denoted as hgg whose output ho (Z) is the probability distribution over all possible classification labels. For classification, the prompts are trained via minimizing the following loss function,
Lpred cross.entropy
Parameters of the pretrained encoder + is frozen during training. Different from previous works (e.g., [60]) where the prompts are directly learnt, the prompts in our approach are encoded from the input X. this way, the resulting prompts can better capture relevvtl information from the training text X. We will elaborate on how the prompts are encoded from input X in Section 4.
2.2 Mutual Information
Mutual information (MI) metric in information theory [79, 19], which quantifies the amount of information shared between two random variables. The mutual information between two random variables A and B is
I(A: B) = Epta.) [DL [p(a|b) [p(b]]]
Inspired by [89), has also been applied to measure the similarity between the masked token and the corresponding context in the pretraining of Multilingual Masked Language Modeling [15], the relevance between documents and sentences in document summarization [72] and the source and target sentences in Neural Machine Translation (NMT) [107].
MI the criterion for comparing prompts and other model parameters. MI
3 Our Method: InfoPrompt
As mentioned above, we want the learnt prompts to be task-relevant. To achieve this, we notice that the classification head is trained with both the data representation and the classification labels, thus should contain rich information of the learnt tasks. In encouraging the task-relevancy of the learnt prompts, we consider maximizing the mutual information between the prompt and the parameters of the classification head, denoted as 0. By maximizing such mutual information, the learnt prompt will be more aligned with the training data with which the classification head is trained, thus captures more task-relevant information from training. Further, in order for the pretrained language model to
3

---

properly leverage the task-relevan information in the prompt, we additionally maximize the mutual information between the prompt and the representation from the pretrained language model, so that the encoded representation can be aware of the task-relevant information captured by the prompt. In addition, we also provide theoretical guarantees the convergence of those losses when training with gradient descent, demonstrating that our method can converge more easily than existing prompt tuning methods. Below, we denote the negative mutual information between the prompt and parameters of the classification head the head loss. The negative mutual information between the prompt and representations from the pretrained language model is denoted as the representation loss.
3.1 The Head Loss
The head loss is the negative mutual information between the prompt P and parameters 0, i.e., (P X). maximizing I(P; 0|X), we follow [68] that approximates it with the following lower bound,
I(P;8IX) 2 C LxCE(P,8,X)
where C' is a constant, LNCE is a Noise Contrastive Estimation (NCE) of mutual information,
LNCE = E log exp(((P,0,x))
Σ 1==1 (Pp(((P,0))))))
and [[ are the negative prompt samples for contrastive learning. In practice, we randomly sample tokens from the context as the negative samples, ie.,, (Px ) and the positive sample is P1 = P.
We model the score function ((P,0|X) as standard bilinear function with the learnable matrix W1
((P, (1),0|) PT WyO.
where 0 and P are encoded from X, and W1 is trainable matrix. Since the classification head learnt on top of the output from the last layer of the pretrained language model, the learning of its parameters is easier than the learning of the prompt P (the input layer of the pretrained language model). Therefore, may capture more task-relevant information than P in the early stage of training. By maximizing the mutual information between 0 and P. the task-relevant information can be transferred to in the initial training steps. In this way, P can more task-relevant especially in the early training stage. Experiments in Section 6 also show that our head loss, I(P; 0 X), can facilitate the training of the initial training steps.
3.2 The Representation Loss
The representation loss, denoted as -I(P; ZI(P;Z) X) is defined as the negative of mutual information between the prompt P and the encoded representation from the pretrained language model, i.e., Z = (P..)). Similar to the head loss, we approximate the representation loss with its lower bound,
I(P: Z(X: 2 log(N) - CNCc ((,,,,
and,
LNC = E log exp(((P,Z(x))
Σ exp(I(P, Xx)
{Z are the negative samples. Here, we overload the notations of InfoNCE loss LNCE and score function for conciseness. Let W2 be trainable matrix, the function / for the representation loss defined by,
(()) = PT
We use variational inference methods [46] to recover the latent distribution of Z. Specifically, we assume that the latent distribution is N(,, r). where N(u, σ)) the normal distribution with mean jr and diagonal covariance matrix a. We model μ and a via,
(2( (Z).

---

fa and fu are trainable fully connected layers. Since the negative samples of Z, i.e., {2x} should not be paired with P. we assume the {2x LK R= are drawn from N N(,', σ²), s.t., s.t.
k' = fu('"), fo("".
In contrast to - (P,,), we have Z' = (()) where {2 are not paired with P. By maximizing the representation loss I(P; Z|)), encourage the encoded representation Z to be more aware of the prompt P. so that the task-relevant information P can be properly encoded by the pretrained language model in producing Z.
3.3 Overall Objective
We minimize the following objective in prompt tuning:
L = Spred 3 I(P: -
(I)
We denote Cpedd as the task loss. 3 and are balancing parameters for the proposed representation loss and head loss, respectively. We denote our approach as InfoPrompt. More details about the implementation and configurations are provided in Section 4.2.
3.4 Theoretical Guarantees
We state our main theoretical result as follows. Due to the space limit, we delay the proof into Appendix.
Theorem 3.1. Given the Loss function & (Eq. (1)) and conditions specified in Appendix C.I and D, using gradient descent type of greedy algorithm,, we can find the optimal solution of that loss function.
We provide theoretical guarantees of the convergence of those losses trained by conventional gradient descent type algorithms. In Section 4, we empirically observe that our method converges more easily than traditional soft prompt tuning methods and requires fewer training epochs.
Experiments
4.1 Datasets
We conduct experiments with datasets of sequence classification from the GLUE benchmark [95], along with those of relation extraction tasks and NER tasks. We choose four sequence classification tasks from the GLUE benchmark: RTE (Recognizing Textual Entailment, [7]), MRPC (Microsoft Research Paraphrase Corpus, [28]), CoLA (Corpus of Linguistic Acceptability, [98]) and SST-2 (Sentence Sentiment Treebank, [81]). We choose these tasks because their datasets are of smaller sizes and prompt tuning is comparably more effective in low-resource settings [40, 62]. For the task of relation extraction, we evaluate our method on the ACE2005 corpus and the Semeval-2010 datasets [44]. We also use the ACE2005 corpus for the task of NER. Note that the entity spans for NER have been given ACE2005. Unlike the standard NER model that learns to predict the entity span and entity type simultancously from the raw text sequence. our model only predicts the entity type based on the given entity span. We follow the same data splitting strategy for ACE2005 corpus as the previous work [103, 71]. For the Semeval-2010 tasks, we follow the official data partition [44].
4.2 Experiment Settings
We follow the resource constrained scenario in [40] that trains each task with only 64 or 256 samples. We experiment with n, 1 and np prompt tokens for each task. The prompt tokens inserted into the template for each task. Similar to [40], we adopt the RoBERTa-large model as our pretrained encoder. We freeze the pretrained parameters and only train the parameters of the prompt head and prompt tokens. During training, we empirically set 8 = 0.1 and = 0.05. The number of negative samples is K 32. The learning rate is le and the batch size is 8. For each task, we report the results after 30 epochs, averaged over 5 random seeds. To encode the prompt P [pI, Paa, from X, we first encode X into p' RDD via p' ((X). We denote the up-sampling and down- wwmn-sampiing
5

---

Table I: Results on Sequence Classification.
CoLA
RTE
MRPC
SST2
1
1p =
70p
1
Rp
Average
Finetuning
0.6131
0.7798
0.8873
0.9427
0.8057
Adapter [45]
0,5552
0.5776
0.6814 14
0.9472
0.6904
WARP [40]
0.5282
0.3911
0.6282
0.6426
0.8039
0.8186
0.9507
0,9587
0.7403
IDPG [102]
0.5556
0.5646
0.6282
0.6534
0.7941
0.8039
0.9587
0.9587
0.7396
InfoPrompt
0.5631
0.6018
0.6751
0.6968
0.8039
0.8137
0.9576
0,9599
0.7590
0
0.5699
0.5853
0.6751
0.6787
0.7941
0.8137
0.9495
0,9587
0.7531
B 0
0.5546
0.5579
0.6065
0.6318
0.7892
0.7966
0.9472
0,9610
0.7306
= 0.3 0
0,5032
0,5732
0.6173
0.6029
0.7917
0.7672
0.9495
0.9564
0.7202
Table 2: Results on Relation Extraction and NER.
RE
NER
SemEval
Tip
11p
7hp
7hp
Rp
Average
Finetuning
0.8119
0,9054
0.8506
0.8560
Adapter [45]
0,5073
0.8329
0.6570
0.6657
WARP [40]
0.6384
0.6596
0.8174
0.8607
0.6702
0.7284
0.7291
IDPG [102]
0.6079
0.6132
0.8360
0.8931
0.6408
0.6776
0.7114
InfoPrompl
0.6914
0.7616
0.8526
0.8962
0.7563
0.7917
0.7916
0.6914
0.7285
0.8452
0.8635
0.7471
0.7865
0.7770
0.6967
0.7470
0.8351
0.8698
0.7449
0.7538
0.7746
0,
0.5364
0.7285
0.8512
0.8661
0.7490
0.7799
0.7519
projections similar [31]. For each p; E I,,, we have p: WWW.W WHP E Dx64 Wdown E R61XD
For the tasks of sequence classification and relation extraction, we follow the template of [40] that contains [mask] token. The representation Z is obtained from the [mask] token from the last layer of the RoBERTa-Large encoder. For the task of NER, we have the [mask] token before the given entity span, with the rest being the same as for sequence classification.
4.3 Baselines and Ablations
As mentioned above, our method with Eq. (1) in denoted InfoPrompt. In the experiments, compare our method with the following baselines:
Finetuning: We fine tune all the parameters from the pretrained encoder each task. Finetuning included as the upper bound for the model performance, since more computational expensive compared with only training the prompt parameters.
Adapter [45]: Similar to prompt tuning, this is also a way of parameter-efficient training for pretrained language models. Specifically, instead of adding the prompt tokens in the input, we add adapters after the feed-forward module in each transformer layer.
WARP [40]: Different from approach, the prompt tokens of WARP not generated from the input sequence. the prompt tokens are insert into the input sequence. During training, the pretrained encoder is frozen and only the prompt tokens are trainable.
IDPG [102]: Similar to our approach, the prompt tokens are generated from the input sequence. The pretrained encoder frozen and the prompt generator trainable.
In evaluating the effectiveness of our proposed loss functions, we consider the following two ablations:
? O: We disable the head loss during training via 7 0, while keeping 3 0.05.
3 We disable the representation loss during training via 3 = 0, while keeping ? = O.L.
3 ? 0: We disable both the losses. The prompt parameters trained with Lpred-

---

Table 3: Few-shot results on Sequence Classification. We experiment with N 64 and N 256
samples for each task. The number of prompt is fixed to Up = 4 for all soft prompt tuning methods.
CoLA
RTE
MRPC
SST2
N = 64
N 256
N 64
N 256
N 64
N 256
N = 64
N 256
Average
Finetuning
0.1746
0.4086
0.4801
0.6787
0.7107
0.7819
0.8027
0.8853
0.6153
Adapter [45]
0.0627
0.2486
0.5487
0.5668
0.5931
0.6250
0.4908
0.664
0,4750
WARP [40]
0.0749
0.0785
0.5596
0,5812
0.7083
0.7083
0.5872
0.7638
0.5077
IDPG [102]
0.0902
0.1513
0.5018
0.5523
0.6593
0.7010
0.5424
0.8188
0.5021
InfoPrompt
0.1567
0.1750
0.6137
0.6580
0.7059
0.7377
0.6697
0.7305
0,5559
7 =
0.1479
0.1447
0.5776
0.6318
0.6936
0.7328
0.664
0.7294
0.5402
8 3=00
0.1372
0.1433
0,5812
0.5957
0.6838
0.7132
0.5631
0,656
0.5092
3 0
0,0919
0.1397
0,5668
0,5523
0.6985
0.7108
0,5505
0.6296
0,4925
Table 4: Few-shot results on Relation Extraction and NER. We experiment with N 64 and
N 256 samples for each task. The number of prompt is fixed to np 4 for all soft prompt tuning methods.
RE
NER
SemEval
N 64
N 256
N 64
N 256
N 64
N 256
Average
Finetuning
0.1285
0.4013
0.3033
0.4358
0.2223
0.4829
0.3290
Adapter [45]
0.1086
0.1815
0,2345
0.2437
0.1211
0.177
0.1777
WARP [40]
0,1404
0.2556
0,3082
0.4369
0.1708
0,3684
0,2801
IDPG [102]
0,2596
0,2503
0.3334
0.4048
0.1984
0.3577
0,3007
InfoPrompt
0.2119
0.2993
0.3331
0.4739
0.2113
0.4034
0,3222
0
0.2026
0,2834
0.3225
0.4776
0.2153
0.3739
0.3126
Ø = 0
0,2013
0,2874
0.3208
0.4615
0.2072
0.3629
0,3069
= 0,8
0,1974
0,2728
0,3142
0.4662
0.2278
0.3276
0,3010
Experimental Results
5.1 Training with the Full dataset
Table and 2 show the results of training with the full dataset for each task. We can observe that the results with our InfoPrompt are generally higher than those of the other parameters-efffcient baselines that freeze the pretrained RoBERTa-Lart has better performance than the other approaches. This is because it allows training with all the model parameters, which is at the expense of more computation cost during training. As mentioned above Finetuning intended to be included the upper bound for performance. Moreover, we can find that the performance with ς = 0 and 3 = 0 is lower than that of InfoPrompt, indicating shows that it is beneficial to learn task-relevant prompt tokens with the proposed head loss and representation loss. Further, the performance gap between functions are effective when added naive prompt tuning, i.e., with only Cpred-
parameters (e.g.. WARP and Adapter). Finetuning generally
0/3 = 0 and / = γ = 0 shows that the proposed
5.2 Training with the Few-Shot Datasets
The results for training with few-shot datasets are listed in Table and 4. Compared with training with the full dataset (Table and 2), we can find that the performance gap between our proposed InfoPrompt and the baselines is generally larger in the few-shot setting. Unlike the full datasets, the few-shot datasets contain much less information regarding the task be learnt. As the result, the prompts learnt with solely the task loss (e.g., with WARP or 3 y 0) may easily overfit to the task-irrelevant information given the few-shot datasets. In such scenario, it would be important to explicitly encourage the learnt prompt to be task-relevant, i.e., via our proposed loss functions based on mutual information maximization. This explains why InfoPrompt yields larger performance gain when trained with few-shot datasets. Similar to training with the full datasets, the performance gains of InfoPrompt compared with InfoPrompt (7 = 0) and InfoPrompt (.3 = 0) show the effectiveness of proposed loss functions in the few-shot scenario.
7

---

Task Loss
CMI Loss
1.5
1.5
Inchomma 9.05
Inferrentt y 9.65
1.0
yfoPromotyy
0.75
1.0
inforrompt y
2.75
Principle Component
0.70
Principle Component
2.70
0.65
2.65
0.0
0.60
0.0
2.60
2.55
00.
2.50
-1.0
0.50
-1.0
2.45
0.45
11.5
1.5
1.5 -1.0 -0.5
0.5 1.0 1.5
-1.5
-1.0
-0.5
0.0
0.5 1.0
1.5
Principle Component 1
Principle Component
(a) The landscape of task loss. 0.1
(b) The landscape of representation loss. 0.1
Figure 2: The landscapes of the loss functions in the parameter space of the prompt tokens. The landscapes illustrates how the values of loss functions varies with the input prompt tokens. The trajectory shows the first 500 steps during training for InfoPrompt with = 0.05 or = 0.
Analysis
6.1 Loss Landscapes
To provide a more comprehensive understanding of the effectiveness of our proposed loss functions, we plot the landscapes the loss functions in the parameter space of the prompt tokens. The landscapes illustrate how the values of the loss functions vary with the input prompt tokens. Since the prompt tokens are high-dimensional vectors, í.e., each token has the dimension of 1024 for RoBERTa-Large, we visualize their associated loss values via projecting the prompt tokens into a 2D subspace. Specifically, we follow previous work on token embedding analysis [12] that projects the prompt tokens into the t principal components computed from the pretrained token embeddings of RoBERTa-Large. We only insert one prompt token into the input sequence during visualization.
Taking the task of MRPC as example, we plot the 2D landscapes of the task loss and the representation loss in Figure 2a and 2b, respectively. Both figures are plotted with the same scale, ie.. with the same values of the prompt token. The axis values are the offset from the mean of the pretrained RoBERTa-Large token embeddings. The loss values shown in the figures are the average of 20 random samples from MRPC. In Figure 2a, we can find that there are lot of local minimum the landscapes of the task loss. This is consistent with the observations of the previous works [37, 91] that prompt tuning is difficult to be optimized with and sensitive to initialization, e.g., the optimization can get easily overfit to a local minimum without proper initialization. From Figure 2b, observe that the loss landscape of proposed representation loss is much smoother compared to the task loss in Figure 2a. With smoother landscapes, the optimization with our proposed loss functions can be more stable (also shown in Section 6.2), i.e., less likely be trapped in local minimum and also guaranteed to converge according to our theoretical results (see Theorem 3. Additionally, we plot the trajectory of the first 500 steps during training for InfoPrompt (7 0.05) (green) and = 0 (purple) in Figure 2a and 2b. The stars in the plot indicate the initial value of the prompt before training. We find that training with 7 0.05 can render larger descent for both the task loss and representation loss, compared to 7 = O. As analyzed in Section 3.1, the language head easier learn than the prompt. As the result, parameters of the language head may contain more task-relevant information during the earlier stage of training. By maximizing the mutual information between the head parameter and prompt via the proposed head loss (weighted by 7), we encourage the learnt prompt to capture more task-relevan information in the initial training steps, thus resulting
0.05 have a larger descent than y = in the trajectories shown in Figure 2a and 2b. We also compare our initialization to some common initialization approaches: Random Uniform and Sampled Vocabulary [60, 37]. By Random Uniform, we randomly sample prompt initialization from the continuous latent space. By Sampled Vocabulary, we randomly sample prompt initialization from language model's vocabulary set. The final results by WARP (Random Uniform), WARP (Sampled Vocabulary) and InfoPrompt are 0.626, 0.672 and 0. 706 respectively. The results
8

---

6.8
0.6
accuracy
6.2
InfoPrompt
InfoPrompt
WARP
WARP
Finetuning
Finetuning
2500
5000
7500
10100
12500
1500e
17500
2000
4000
6000
800D
10900
12000
14000
step
(a) NER learning curve.
(b) SST-2 learning curve.
Figure 3: The learning curves for the task of NER and SST-2. The training our our InfoPrompt is more stabilized and converges faster, compared with WARP.
validate the effectiveness of our initialization approach. Note that our proposed two loss functions are unsupervised and do not require additional labels.
6.2 Learning Curve
We plot the training curve for the task of NER and SST-2 in Figure 3a and 3b, respectively. Unlike WARP [40] and Finetuning that train with solely the task loss Lpred. our InfoPrompt also trains with the representation loss and head loss. We can observe that the training of our our InfoPrompt is more stabilized and converges faster, compared with WARP. This can explained from the landscape plots in Section 6.1. Since the landscape of the task loss is not smooth (Figure 2a), the training curve of WARP may exhibit significant perturbation when the optimization overfits to local minimum, e.g. the 10000th step in Figure 3a. Comparably, our proposed InfoPrompt can smooth the optimization landscape, thus stabilizing the training and result in faster convergence, which is guaranteed by our theoretical results. We observe that Finetuning generally converges faster and ends up with higher accuracy than InfoPrompt. This is because Finetuning, which trains with the model parameters, has much larger model capacity during training than prompt tuning (InfoPrompt and WARP). Such results for Finetuning is at the expense of larger computation costs, Lee, we need to calculate the gradient for all the model parameters (354M) instead of only the prompt parameters P (1.3M).
We also validate that our approach is less sensitive to initialization in the early learning stage, compared to WARP Specifically, across 10 different random seeds, we report the standard errors of the performances by InfoPrompt and WARP the early learning stage. After the first epoch, on NER, the results by WARP and InfoPrompt are 0.461 + 0.038 and 0.810 + 0.025, respectively. On SST2, the results by WARP and InfoPrompt are 0.735 + 0.033 and 0.764 + 0.027, respectively. The results show that our method has lower standard errors and is less sensitive compared to WARP
7 Related Work
7.1 Soft Prompt Tuning
Soft prompt tuning has become new paradigm in NLP. Based on some large pretrained models (c.g.. BERT [25]. RoBERTa [67]), a relatively small number of trainable parameters can be added to the input, while the parameters of backbones are fixed. Many works have demonstrated the effectiveness of soft prompt tuning in a wide range of NLP downstream tasks [60, 40, 73, 65], especially in low-resource regions [78, 66, 37]. Some recent works also found the transferable power of soft prompts across domains [101, 91, 94], across language models [91] and for zero-shot generalization [105]. To further enhance the efficiency of soft prompt parameters and enable better generalization abilities, many works consider multi-task learning [6, 27, 94, 43], multilingual [14, 50). Some works also try to explore the prompt with prior knowledge encoded [48, 42, 13). While most of the initial attempts of soft prompt tuning not context-aware, some recent works suggest that

---

the soft prompt tokens should be conditioned on the input context. Hyperprompt [43] proposes hyper-network structure to generate prompt tokens based on task indexes. [102] and [8] suggest some context-aware soft prompt generation methods. [64] proposes structured soft prompt tuning method. BBT [92] targets the scenarios where the pre-trained model not available locally (i.e., deployed online) and its back-propagation operation is not available.
7.2 Information- Methods in NLP
Information-theorede methods are widely used in many NLP tasks [55, 99, 90, 51, 70]. [99] and [55] propose information-theoreth methods for text memorization. [70] suggests an information-theoret method for dialogue.
[51] views the multimodal NMT problem in an information-theoreee point of view. For model pretraining. [96] proposes Infobert to improve the robustness of the BERT model. INFOXLM [15] proposes cross-lingual language model based on information-theoret framework. For fine-tuning, [69] proposes an information bottleneck model method for low-resource fine-tuning. [89] introduces information- information-thhortiicc method to engineer discrete prompts.
7.3 Theoretical Attention Computation
Softmax is one of the major unit in the attention scheme of most recent NLP large language models. Computing the attention matrix faster is a practically interesting question [17, 97, 57]. Recently, a number of theoretical work have tried to study the softmax/attention unit from theoretical perspective. The softmax attention computation can be formally defined as follows: suppose we are given three matrices Q T (the query), K E Rank (the key), and V Raxk (the value), the goal is to compute (Q, K, V) =
D D- exp(QK D
),,) Here KT denote the transpose of matrix K. The work of [106, 3] consider the static setting, and the work of [11] considers the dynamic setting. [3] proposed tight algorithm for computing Att and provided lower bound result based on the strong exponential time hypothesis. [4] provide the results for more general tensor version of attention which capture the three tuples feature, but classical attention cannot [77]. The work [11] shows tight positive result and a negative result. In [11], they provide an upper bound via lazy update techniques [18]. In [11], they also present lower bound result which is based on the Hinted MV conjecture [10]. The work of [21] proposes two sparsification algorithm to compute attention matrix when the feature dimension >> the length of sentence. [35] shows how to provide a differentially private algorithm for computing attention matrix under differential privacy framework [30, 29]. [41] introduces hyperattention method and presents an nearly linear time algorithm with provable guarantees. [56] studies the polynomial based attention scheme and shows that sketching techniques can help speeding up the attention computation.
where the diagonal matrix D is diag(exp((U)
8 Conclusion and Future Work
We revisit the limitations of soft prompt tuning in the initialization. We also empirically discover that conventional prompt tuning methods cannot learn sufficient task-relevant information from prompt tokens. We tackle these limitations from an informatiin--heerrticc perspective and propose an information-theorrtie prompt tuning method InfoPrompt, with two novel loss functions. With extensive experiments, without any prior expert knowledge, InfoPrompt can significantly accelerate the convergence of the prompt tuning and achieve more accurate and robust performances than traditional prompt tuning methods.
Existing instruction-tunee LMs (e.g., Llama 2 [93]) are generally not task-specifi and future works may further consider to tune such models with task-specific information. To achieve this, combining soft prompt tuning and prompt engineering [104, 76], from an can a promising approach. Another future direction could be to further generalize our method to generation tasks (e.g., sequence generation). In addition to prompt learning, it is interesting to explore how to extend our approach to other parameter-effiiien fine-tuning methods (e.g., LoRA [47] and HyperFormer [26]).
perspective,
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their insightful comments. During this research, Ricardo Henao and Rui Wang were supported by ONR N0001--881128871P0000022
References
[1] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via premtttramttii..iinnn. In International conference on machine learning, pages 242-252 PMLR, 2019.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. Advances in neural information processing systems, 32, 2019.
[3] Josh Alman and Zhao Song. Fast attention requires bounded entries. In NewrIPS. arXiv preprint arXiv:2302. 13214, 2023.
[4] Josh Alman and Zhao Song. How to capture higher-order correlations? generalizing matrix softmax attention to kronecker computation. arXiv preprint arXiv:23 (0.0::22/0.000444 2023.
[5] Josh Alman and Virginia Vassilevska Williams. A refined laser method and faster matrix multiplication. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 522-539. SIAM, 2021.
[6] Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannaneh Hajishirzi. Attempt: Parameter-efffcient multi-task tuning via attentional mixtures of soft prompts, 2022.
[7] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[8] Rishabh Bhardwaj, Amrita Saha, and Steven CH Hoi. Vector-quantized input-contexttualized soft prompts for natural language understanding. arXiv preprint arXiv:2205. //04, 2022.
[9] Jan den Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 259-278. SIAM, 2020.
[10] Jan van den Brand, Danupon Nanongkai, and Thatchaphol Saranurak. Dynamic matrix inverse: Improved algorithms and matching conditional lower bounds. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 486-480. IEEE, 2019.
[11] Jan van den Brand, Zhao Song, and Tianyi Zhou. Algorithm and hardness for dynamic attention maintenance in large language models. arXiv preprint urXiv:2304.02001 2023.
[12] Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embed- ding space: Clusters and manifolds. In International Conference on Learning Representations, 2020.
[13] Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, pages 2022.
[14] Yuxuan Chen, David Harbecke, and Leonhard Hennig. Multilingual relation classification via efficient and effective prompting. arXiv preprint arXiv:2210. 13838, 2022.
[15] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian- Ling Mao, He-Yan Huang, and Ming Zhou. Infoxlm: An information-theorrtiic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576-3588, 2021.
[16] Timothy Chu, Zhao Song, and Chiwun Yang. How to protect copyright data in optimization of large language models? arXiv preprint at arXiv:2308.1224 12247, 2023.

---

[17] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does
bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, 2019.
[18] Michael Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs the current matrix multiplication time. In STOC, 2019.
[19] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[20] Yichuan Deng, Zhihang Li, and Zhao Song. Attention scheme inspired softmax regression. arXiv preprint arXiv:2304. 104/1, 2023.
[21] Yichuan Deng, Sridhar Mahadevan, and Zhao Song. Randomized and deterministic attention sparsification algorithms for over-parameterized feature dimension. arxiv preprint: arxiv 2304.03426, 2023.
[22] Yichuan Deng, Zhao Song, and Omri Weinstein. Discrepancy minimization in input-sparsity time. arXiv preprint S.KIv:2210.124660 2022.
[23] Yichuan Deng, Zhao Song, and Shenghao Xie. Convergence of two-layer regression with nonlinear units. arXiv preprint arXiv:2308.08358, 2023.
[24] Yichuan Deng, Zhao Song. and Tianyi Zhou. Superiority of softmax: Unveiling the perfor- mance edge over linear attention. arXiv preprint arXiv:2310. 2023.
[25] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[26] Kaize Ding, Albert Jiongqian Liang, Bryan Perozzi, Ting Chen, Ruoxi Wang, Lichan Hong. Ed H Chi, Huan Liu, and Derek Zhiyuan Cheng. Hyperformer: Learning expressive sparse feature representations via hypergraph transformer. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2062- 2066, 2023.
[27] Kun Ding, Ying Wang, Pengzhang Liu, Qiang Yu, Haojian Zhang, Shiming Xiang, and Chunhong Pan. Prompt tuning with soft context sharing for vision-language models. arXiv preprint ar0iv:2208. 13474, 2022.
[28] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
[29] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages
Springer, 2006.
[30] Cynthia Dwork, Frank McSherry, Kobbi Nissim. and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-22156 284. Springer, 2006.
[31] Ankush Ganguly and Samuel WF Earp. An introduction to variational inference. arXiv preprint arXiv:2 13383,, 2021.
[32] Yeqi Gao, Sridhar Mahadevan, and Zhao Song. An over-parameterrzzed expp:memmeerrreeeexeeeeeeeneiaai regression. arXiv preprint arXiv:2303. 16504, 2023.
[33] Yeqi Gao, Zhao Song, Weixin Wang, and Junze Yin. A fast optimization view: Reformu- lating single layer attention in Imm based on tensor and svm trick, and solving matrix multiplication time. arXiv preprint arXiv:2309.07411 2023.
12

---

[34] Yeqi Gao, Zhao Song, and Shenghao Xie. In-context learning for attention scheme: from single softmax regression to multiple softmax regression via tensor trick. arXiv preprint UrXiv:2307.02419, 2023.
[35] Yeqi Gao, Zhao Song, and Xin Yang. Differentially private attention computation. arXiv preprint arXiv:2305.047700 2023.
[36] Yeqi Gao, Zhao Song, and Junze Yin. An iterative algorithm for rescaled hyperbolic functions regression. arXiv preprint arXiv:2305.00660) 2023.
[37] Yuxian Gu, Xu Ham, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 8410-8423, 2022.
[38] Yuzhou Gu and Zhao Song. arXiv:2211.06033, 2022.
A faster small treewidth sdp solver. arXiv preprint
[39] Yuzhou Gu, Zhao Song, and Lichen Zhang. A nearly-linear time algorithm for structured support vector machines. arXiv preprint arXiv:2307.07735 2023.
[40] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adver- sarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4921 149314493,220 2021.
[41] Insu Han, Rajesh Jarayam, Amin Karbasi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. Hyperattention: Long-context attention near-linear time. arXiv preprint arXiv:2310.05800 2023.
[42] Keqing He, Jingang Wang, Chaobo Sun, and Wei Wu. Unified knowledge prompt pre-training for customer service dialogues. In Proceedings of the 31 ACM International Conference on Information Knowledge Management, pages 2009-4013 2022.
[43] Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based task-conditioning of transformers. In International Conference on Machine Learning, pages 8678-8690 PMLR, 2022.
[44] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint 1rXiv:991.11042.. 2019.
[45] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In International Conference on Machine Learning, pages 2790-2799, PMLR, 2019.
[46] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[47] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 1et.:22066009988 2021.
[48] Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint (ttt:::200.002...33 2021.
[49] Baihe Huang, Shunhua Jiang, Zhao Song, Runzhou Tao, and Ruizhe Zhang. Solving sdp faster: A robust ipm framework and efficient implementation. 2022 IEEE 63rd Annual Symposium Foundations of Computer Science (FOCS), pages 233-244. IEEE, 2022.
13

---

[50] Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. Zero-shot cross-lingual transfer of prompt-base tuning with a unified multilingual prompt. arXiv preprint arXiv:2202. 1451, 2022.
[51] Baijun Ji, Tong Zhang, Yicheng Zou, Bojie Hu, and Si Shen. Increasing visual awareness in multimodal neural machine translation from an information theoretic perspective. arXiv preprint arXiv:2210.084777 2022.
[52] Haotian Jiang. Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 JEEE 6lst annual symposium foundations of computer science (FOCS), pages 910-918. IEFE, 2020.
[53] Haotian Jiang, Yin Tat Lee, Zhao Song, and Lichen Zhang. Convex minimization with integer minima in (n²)) time. In ACM-SIAM Symposium Discrete Algorithms (SODA), 2024.
[54] Shunhua Jiang, Zhao Song, Omri Weinstein, and Hengjie Zhang. Faster dynamic matrix inverse for faster 1ps. In STOC. arXiv preprint arXiv:2004.04470, 2021.
[55] Jiaxin Ju, Ming Liu, Huan Yee Koh, Yuan Jin, Lan Du, and Shirui Pan. Leveraging infor- mation bottleneck for scientific document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages
2021.
[56] Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketches for polynomial kemnlls. arXiv preprint arXiv:2310.01655. 2023.
[57] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint (rXiv:2001.0445/1 2020.
[58] François Le Gall. Powers of tensors and fast matrix multiplication. In Proceedings of the 39th
international symposium symbolic and algebraic computation, pages 296-303, 2014.
[59] Yin Tat Lee, Zhao Song, and Qiuyi Zhang. Solving empirical risk minimization in the current matrix multiplication time. In COLT, 2019.
[60] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
2021.
[61] Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304. 132766, 2023.
[62] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 2xXi:22001.00990, 2021.
[63] Zhihang Li, Zhao Song, and Tianyi Zhou. Solving regularized exp, cosh and sinh regression problems. arXiv preprint arXiv:2303.15725, 2023.
[64] Chi-Liang Liu, Hung-yi Lee, and Wen-tau Yih. Structured prompt tuning. arXiv preprint arXiv:2205. 12309, 2022.
[65] Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. Late prompt tuning: A late prompt could be better than many prompts. arXiv preprint arXiv:2210. /292, 2022.
[66] Xiaochen Liu, Yu Bai, Jiawei Li, Yinan Hu, and Yang Gao. Psp: Pre-trained soft prompts for few-shot abstractive summarization. arXiv preprint 22X4:22044.04133, 2022.
[67] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692, 2019.
[68] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv: 080900/8/2 2018.
14

---

[69] Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottleneck for effective low-resource fine-tuning, arXiv preprint arXiv:2106.05469, 2021.
[70] Kory Mathewson, Pablo Samuel Castro, Colin Cherry, George Foster, and Mare G Bellemare. Shaping the narrative arc: An information-theerrtic approach to collaborative dialogue. arXiv preprint arxiiv:901.. 1128, 2019.
[71] Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, 2016.
[72] Vishakh Padmakumar and He He. Unsupervised extractive summarization using pointwise mutual information. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2505-2512, 2021.
[73] Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft
prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5203-5212, 2021.
[74] Lianke Qin, Zhao Song, Lichen Zhang, and Danyang Zhuo An online and unified algorithm for projection matrix vector multiplication with application to empirical risk minimization. In AISIATS, 2023.
[75] Yujia Qin, Xiaozhi Wang, YuSheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. Exploring oow-dimensional intrinsic task subspace via prompt tuning. CoRR, abs/2110.07886
2021.
[76] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization. arXiv preprint vrXiv:2305.0397. 2023.
[77] Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. arXiv preprint arXiv:2306. 02896, 2023.
[78] Nathan Schucher, Siva Reddy, and Harm de Vries. The power of prompt tuning for low- resource semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 148-156, 156,,,622201 2022.
[79] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
[80] Ritwik Sinha, Zhao Song, and Tianyi Zhou. A mathematical abstraction for balancing the trade-off between creativity and reality in large language models. arXiv preprint, 2023.
[81] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[82] Zhao Song, Weixin Wang, and Junze Yin. A unified scheme of resnet and softmax. arXiv preprint arXiv:2309. 13482, 2023.
[83] Zhao Song, Yitan Wang, Zheng Yu, and Lichen Zhang. Sketching for first order method: efficient algorithm for low-bandwidth channel and vulnerability. In International Conference on Machine Learning (ICML), pages 32365-32417. PMLR, 2023.
[84] Zhao Song, Shuo Yang, and Ruizhe Zhang. Does preprocessing help training over- parameterized neural networks? Advances in Neural Information Processing Systems, $4:22890-22904 2021.
[85] Zhao Song, Xin Yang, Yuanyuan Yang, and Tianyi Zhou. Faster algorithm for structured john ellipsoid computation. arXiv preprint arXiv:2211.14400 2022.
15

---

[86] Zhao Song, Mingquan Ye, and Lichen Zhang. Streaming semidefinite programs: O(vn) passes, small space and fast runtime. arXiv preprint arXiv:2309.05135, 2023.
[87] Zhao Song and Zheng Yu. Oblivious sketching- baased central path method for solving linear programming problems. In 38th International Conference on Machine Learning (ICML), 2021.
[88] Zhao Song, Lichen Zhang. and Ruizhe Zhang. Training multi-layer over over- wer-parametrized neural network subquadratic time. arXiv preprint arXiv:21/2.07628, 2021.
[89] Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. An informationnthhoreet approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 819-862, 2022.
[90] Victor Steinborn, Philipp Dufter, Haris Jabbar, and Hinrich Schütze. An information-theoreti approach and dataset for probing gender stereotypes in multilingual masked language models. In Findings of the Association for Compulational Linguisties: NAACL 2022, pages 921-932,
2022.
[91] Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3949-3969, 2022.
[92] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-mode -odee-ood----eeeeeeee-ee----se saaeel-as---sevvice In International Conference on Machine Learning, pages 20841 -08555. PMLR, 2022.
[93] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[94] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5039-5059, 2022.
[95] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprini arXiv: PrrXivv880.000 0004.00044.11 2018.
[96] Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. Infobert: Improving robustness of language models from an information theoretic perspective. arXiv preprint arXiv:2010.023329 2020.
[97] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
[98] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
[99] Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. Bottlesum: Unsupervised and self- supervised sentence summarization using the information bottleneck principle, 2019.
[100] Virginia Vassilevska Williams. Multiplying matrices faster than coppersmith-winogral In Proceedings of the forty-fourth annual ACM symposium on Theory of computing, pages 887-898, 2012.
[101] Hui Wu and Xiaodong Shi. Adversarial soft prompt tuning for cross-domain sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4438-244,, 2022.
16

---

[102] Zhuofeng Wu, Sinong Wang, Jiatao Gu. Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. Idpg: An -nstance-dependent prompt generation method. arXiv preprint arXiv:2204.00 240.:2204.4497, 2022.
[103] Bishan Yang and Tom Mitchell. Joint extraction of events and entities within document context. arXiv preprint 1rXiv:1609.000 2016.
[104] Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang. Mingfeng Xue, Boxing Chen, and Jun Xie. Tailor: A soft-prompt-based approach attribute-based controlled text generation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 410-427, 2023.
[105] Seonghyeon Ye, Joel Jang, Doyoung Kim, Yongrae Jo, and Minjoon Seo. Retrieval of soft prompt enhances zero-shot task generalization, arXiv preprint arXiv:22/0.03029, 2022.
[106] Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. Kdeformer: Accelerating trans- formers via kernel density estimation. In JCML. arXiv preprint arXiv:2302.02451, 2023.
[107] Songming Zhang, Yijin Liu, Fandong Meng. Yufeng Chen, Jinan Xu, Jian Liu, and Jie Zhou. Conditional bilingual mutual information based adaptive training for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2377-2389, 2022.
[108) Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, and Inderjit S Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International conference on machine learning (ICML), pages 4140-4149. PMLR, 2017.
17

---

Appendix
Roadmap. In Section A, we provide a number of basic notations. In Section B, we provide several basic definitions and discuss related work about previous theoretical softmax regression results. In Section C, we provide a complete proof for our major theoretical result this paper. We present final result in Section D.
A Preliminaries
For any positive integer n, we use [n] to denote set {1,2, .. ,n). For any function f. we use Ö(g) to denote g poly(log.g) g).
Vector For a length-n vector use exp(2) to denote length-n vector that its i-th entry is exp(x;).
For length-n vector 2, we use || represent its & norm, i.e., ||=||2 (2i 22/ For length-n vector 2, we use ||=||o to denote mäXie[m] |=i/-
For length-n vector z E R we use diag(z) to generate diagonal matrix where each entry on the (i, i)-diagonal is for every E [n]-
We 1, to represent length-n vector where the coordinates are ones. Similarly, we use O, to represent a length-n vector where all the values are zeros.
PSD We say W 2 Z (positive semi-definite) x T Wa 2 x² Zx for all vector æ.
Matrix Related For an n by d matrix C, we use nnz(C) to denote the number of non zero entries of C, i... (C) == () E [n] [d] Ct, * 0}l
For diagonal matrix D E IR R we say D) is s-spase diagonal matrix, i.e., h: = { [m] Da,1 * 0H|-
For any matrix E Raxk we denote the spectral norm of Z by |Z]], i.e.
2|| sup rER
|
For a matrix Q, we use (ma to denote the largest singular value Q. We use miin (Q) to denote the smallest singular value of Q.
Matrix Computation We use to denote the exponent of matrix multiplication, i.e., n denotes the time of multiplying an n x matrix with another n n matrix. Currently w Rs 2.373 [100, 58, 5].
Calculus Related We use notation by following the literature's [63, 20, 36, 61, 80]. Suppose that we're re given two column vectors x, y E R", we use y to denote column vector that (x y); is T.Yt..
B Related Work about Theoretical Attention Regression Results
In this paper, we focus on the direction of regression tasks [32, 63, 20, 61, 80, 34, 23, 16, 33, 82, 24]. The goal of this section will review the linear regression (Definition B.1), exponential regression (Definition B.2), rescaled softmax regression (Definition B.3), softmax regression (Definition B.2).
Definition B.1 (Linear regression). Given matrix A E Boxd and b E IR", the goal is to solve
min || - bl] ZER
For convenient, let u(x) to denote exp(Ax).
18

---

Definition B.2 (Exponential Regression, see [32, 63]). Suppose we are given a length n. vector b, and an n by size matrix A, our goal is to optimize
min ZE
[u(x) bll2.
Definition B.3 (Rescaled Softmax Regression, see [36]). Suppose are given length n. vector b and an n by size matrix A, our goal is to optimize
min DER
[|u(()) (u(x). 1,) bll2
Definition B.4 (Softmax Regression, see [20, 61, 80]). Suppose given length vector b and an n by size matrix A, our goal is to optimize
min [[(u(ar), 1,) - u(x) b112. ZERd
C Theoretical Guarantees
In Section C.I, we provide several basic definitions. Section C.2, we explain how to compute the gradient of function J. In Section C.3, we show how to compute the gradient of function log fxx). In Section C.4, we explain how to compute the Hessian function log [ar). In Section C.5, we compute the hessian of inner product between log f(x) and b. In Section C.6, compute the Hessian of cross entropy loss function. In Section C.7, we show Hessian is positive definite. In Section C.8, we prove that Hessian is Lipschitz. We remark that experiments order method, and our theoretical proofs are mainly focusing on second order method. We believe it's interesting future diretion further study the convergence of the first order method such as [108, 1, 2, 88, 84, 83].
based on first
C.1 Function Definition We define
Definition C.1. We define u(x) follows
u(x) exp(Ax)
Definition C.2. We define v(x) as follows
u(x) exp(Ax)
Previous [63] studies three types of hyperbolic functions exp(-), cosh(.) and sinh(-). We mainly focus on exp(-) function.
Definition C.3 (Normalized coefficients, Definition 5.4 in [20]). We define a RRd as follows a<z) (u(z), 1,).
We define function softmax f as follows
Definition C.4 (Function f. Definition 5.1 in [20]). Suppose that we' 're given an x d matrix A.
Let 1, denote length-n vector that all the coordinates are ones. We define prediction function
f RRd B as follows
f(x) == (x) u(xx)
Fact C.5. Let f(x) be defined as Definition C.4. Then we have
Part 1. (f(x), 1n) =1
Part [[](=)]] =1
P. [=) <
Proof. The proof is straightforward. For more details, refer the readers to [20].
19

---

We define the &g loss
Definition C.6. We define
Lexp = f.5l - -612-
Previous work [80] only considers entropy, here consider entropy instead.
Definition C.7 (Cross Entropy). We define Leent 12 IR,
Leen( == - log fx))
Definition C.8. Suppose we' re given an d matrix A and W diag(w) E R where R" is vector, we define Trog TRd + IR
Lyeg (z) == [05||/WA. Ax
C.2 Gradient Computation for Function /
We present a calculus tool from previous work [20] (for example, we refer the readers to Lemma 5.6 in [20]).
Lemma C.9. the following conditions hold
Given matrix A E Rgxd and a vector b E IR".
Suppose that function a : IRd + be defined in Definition C.3.
Suppose that function f Rd + R be defined in Definition C.4.
For each i [d]. we have
Part 1.
df(x)
(f(x), f(z) f(x) A.,
dr;
Part
df(x) = - (f(x), An,s o Anii)
dr;
Part 3.
df(x) dr;
- -
(f(z). (f(x), A
C,3
Gradient Computation for Function log (x)
In this section, we explain how to compute the gradient of log (x)
Lemma C.10. the following condition holds
Suppose that function f is defined in Definition C.4.
We have
Part I.
d d f(x) f(xg = - 1n A
dx;
.Part
d (logS(()) dr:
b) = b) (f(x), (b. 1m)
20

---

Part 3.
d dx;
(xcen(()) = (f(z), A.,)) (b, 1m) (A,,e, D)
Proof. Proof of Part 1.
For all index j E [m], we can compute the gradient with respect to d log f(x);
f(x) d
dr;
dz,
Then we group the coordinates, we get
log f(x)
f(x) df(x)
dz;
dr:
- (110 Ani) ([t1)_4.,/10000000++4444747 f(r)
(f(x), 1n -
Proof of Part 2. We have
d log (xx
b) = (- A..) 1n + A.,i, b) = -f(()))
dr;
(b.1,), 1,),
where the first step follows from Part and the second step follows from simple algebra.
Proof of Part 3. The proof directly follows from Part 2 and Definition of Leen(()) (See Defini- tion C.7).
C.4 Hessian Computation for Function log f()
In this section, we will show how to compute the Hessian for function log f fx).
Lemma C.11. If the following conditions hold
Let f be defined as Definition C.4.
Then we have
Part 1.
d log f(x) = (L)()), (f(x), An, A.ii)) 1n
Part 2.
do f(x)
- AA.))) 1n
dæıdæ
Proof. Proof of Part 1.
We have
d log f(x)
d dlogf(#),
dæ? dx;
dr;
d
(-(f(x), A..)) 1, + Aaa)
dx:
((f(z), 1,
dx;
= ((f(z),
(z). o Ae,)))
where the 2nd step comes from Part of Lemma C.10, the 3rd step follows from A., is independent of x, and the forth step follows from Part 2 of Lemma C.9.
Proof of Part 2.
Similarly, provide proof for Part 2.
21

---

C.5 Hessian Computation for Function (log (x)) b)
The goal of this section prove Lemma C.12. Lemma C.12. the following conditions hold
Let f be defined as Definition C.4.
Then we have
Part 1.
d² log f(x) dx²
(f((),An, b) = ((f()). (((()) A.,)) (1n,0)
Part
dr,dæ; 1og log [(x) b) = (((()), (.) (f(x), Ans o Ani)) (La,
d
Proof. The proof directly follows from Lemma C.11.
C.6 Hessian Computation for Function Leent(r) (x)
For convenient of analyzing the d x d Hessian matrix, we will start with defining matrix B. Definition C.13. We define (xx) E R as follows
B(x) (),, -
Lemma C.14. If the following conditions hold
Let f be defined as Definition C.4.
Let Leent be defined as Definition C.7
Let be defined as Definition C.13
Then have
Part 1.
d²
Leent (-(f(c), + Ar)))) (1n,b)
dx?
Part 2.
d2
Loent (-In)) (J(x), An, An.))) (Ins
Part 3.
d²
T = A B(r)
d
Proof. The proof trivially follows from Lemma C. and Definition C.13.
C.7 Hessian is Positive Definite
Previous work [20] doesn't 't consider cross entropy into the final loss function. Here generalize previous lemma so that cross entropy is also being considered.
Lemma C.15 (A cross entropy generalization of Lemma 6.3 [20]). Suppose the following condi- tions hold
22

---

Let L Exd R 4. 1 > 0, suppose that Ro = exp(O + log n))
L(x)
Lrog (x)
+
cent (x)
L (x)
Definition
Definition C.7
Definition
Let B(x) ((x) + W
Then we have
Part minjelu] w/ 2 10Rg + min then we have
d
dx² = 1- L7
Part 2. minsejn] w/ 2 10 Ro + 1//mmnn (A)?, then we have
(1-0.01) B(x) 3 2 3 (1 0.01) B(x).
Proof. Using the definition of B for Leens (see Definition C.13), definition/bound of B for Lexp (see [20]), and tools developed in Section [20], we can finish the proof.
C.8 Hessian is Lipschitz
Previous work [20] doesn't 't consider cross entropy into the final loss function. Here generalize previous lemma so that cross entropy is also being considered.
Lemma C.16 (A cross entropy version Lemma 7.1 in [20]). Suppose the following condition holds
Let (xx
d' dz
and R > 4
Suppose that max [|=112 [1l1la] I R, and max[]|A], [b]]]] s R
||(x 1 0.01
Then we have
(x))) - H(v) s nt exp(O(R2 + log n)) || vlla
Proof. Using the definition of B for Tecent Definition C.13), definition/bound of B for Texp (see (20]), and tools developed in Section [20], we can finish the proof.
Algorithm Our Algorithm.
1: procedure OURALGORITHM(A E Raxd R2, w E RR,,,,
Theorem D.
2: We choose Tp
3: T +- log( ( x" l22/c
T denotes the number of iterations
4: for &
do
5: Implicitly formulate exact Hessian and use that to construct an approximate Hessian H
(similar as Section in [20])
6:
Compute gradient
7:
H A ATAAAA DA
8:
2T41 + 2,
g
9:
end for
10:
T+ 27++ return T
11:
12:
end procedure
23

---

D Main Theoretical Guarantees
Previous work [20] has proved the similar result without considering the cross entropy. We generalize the techniques in previous paper [20] from only considering &g task loss to considering both Ea task loss and cross entropy Ioss (Leent see formal definition in Definition C.7). Our algorithm is version of approximate Newton method, such methods have been widely used in many optimization tasks [18, 59, 9, 52, 87, 54, 38, 85, 49, 74, 20, 39, 53, 86]. In this work, we focus on the approximate Newton method along the line of [85, 22, 20].
Theorem D.1 (Formal version of Theorem 3.1). Let x= denote an length-d vector that is satisfying,
arg min ZER4
Lexp
Leent Definition C.7
Lreg
Definition C.6
Definition C.8
Suppose the following conditions are holding:
424,00 g(x*) Od
| $ R. ||||| S R. 1b6l2 R.
M e log n)).
minielai w 2 100M + 1/0 (A)²
Suppose that e (0,0.1) is the final and 8 (0,0.1) is the failure probability.
Suppose vo satisfy condition M| - x² 0.14.
Suppose that T log( 200
12/c)
Then there is randomized algorithm (Algorithm 1) such that it runs T iterations
in each iteration, it spends time
O(unz((A) dd)) poly(log(n/6))
generates a vector T. Rd that is satisfying
| x x=1 Sc
the succeed probability is
Proof. The high level framework of our theorem is similar to previous work about exponential regression [63], softmax regression [20] and rescaled softmax regression [36]. Similarly as previous work [63, 20, 36, 80], we the approximate newton algorithm (for example see Section 8 in [20]). So in the proof, we only focus on the difference about the Hessian positive definite lower bound and Hessian Lispchitz property.
Using Lemma C. and Lemma C.16 and approximate Newton algorithm analysis in [20], then we complete the proof.
E Comparison with Parameter-Efficien Fine-tuning Methods Not Based on Prompt Tuning
In this section, we focus the comparison between InfoPrompt (NV, 4) and some PEFT (Parameter- Efficient Fine-tuning Methods) baselines which are not based on prompt tuning:
Adapter [45]: Similar to prompt tuning, this is also way of parameter-efficient training for pretrained language models. Specifically, instead adding the prompt tokens the input, we add adapters after the feed-forward module in each transformer layer.
1 Herr denotes the exponent of matrix multiplication. Curretly z 2.373.

---

Table Comparison with parameter-efficient tin-ttnnig methods which are not based on prompt for the soft prompt tuning method.
tuning. The number of prompt is fixed to 7p
Full datasets
CoL.A
RTE
MRPC
SST2
RE
NER
SemEval
Average
LoRA
0.5880
0.6715
0.8235
0.9541
0.6636
0.8228
0.7214
0.7492
Adapter
0.5552
0.5776
0.6814
0.9472
0.5073
0.8329
0.6570
0.6798
InfoPrompt
0.6018
0.6968
0.8137
0.9599
0.7616
0.8962
0.7917
0.7888
N = 64
CoLA
RTE
MRPC
SST2
RE
NER
SemEval
Average
LoRA
0.0991
0.5596
0.6985
0.5677
0.1232
0.1345
0.1711
0,3362
Adapter
0.0627
0.5487
0.5931
0.4908
0.1086
0.2345
0.1211
0.3085
InfoPrompt
0.1567
0.6137
0.7059
0.6697
0,2119
0,3331
0.2113
0.4146
N = 256
CoLA
RTE
MRPC
SST2
RE
NER
SemEval
Average
LoRA
0.2854
0.5740
0.7206
0.8222
0.2291
0.1955
0.3817
0.4583
Adapter
0.2486
0,5668
0,6250
0.6640
0.1815
0,2437
0.1770
0.3866
InfoPrompt
0.1750
0.6580
0.7377
0.7305
0.2993
0.4739
0.4034
0.4968
LoRA [47]: Another parameter-efficiet training method for pretrained language models. Specifically, LoRA adds additional low-rank decomposed matrices into each Transformer layer via residual connections.
In Table 5, we can observe that LoRA is a stronger bascline than Adapter. However, our method still outperform LoRA, especially in the downstream tasks which require more task-relevant information, egg., NER and RE
25


Reviews:
[{"review_id": "KCDauBHmj8", "reviewer": "Reviewer_kCoa", "summary": "The paper proposes a new formula that models soft prompt tuning as maximizing mutual information between prompts and other model parameters. In order to improve the initialization of the prompts and to learn sufficient task-relevant information from prompt tokens, this paper develops two novel mutual information based loss functions. The authors provide analysis on the convergence of the prompt tuning and the proposed methodology is evaluated on several benchmark datasets.", "questions": "Some detailed questions:\n\n1. Figure 2(a) caption: confusion of definition: task loss info loss, representation loss? Line 257, the loss is mentioned as info loss but representation loss in the caption.\n\n2. Line 73 experiments on 6 tasks and 3 datasets?\n\n3. Could you explain the determination of β = 0.1 and γ = 0.05, the comparison of these two loss terms the number of negative samples and the importance of the number of prompt tokens\n\n4. What are the advantages of the proposed method compared with baseline IDPG?\nWhy IDPG is utilized as a baseline method for full-dataset experiments while not utilized for few-shot dataset?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Simple but powerful idea: the authors propose a novel method for soft prompting. \n\n+ Convincing and superior experimental results over baseline.\n\n+ The paper is well-structured and easy to follow.", "weaknesses": "-Lack of ablation studies analysis: there are hyper-parameters that balance the weight of two loss function terms. However, I did not find any design choice and/or ablation study on this.\n\n-In addition, more implementation details should be provided in the paper, which is essential for reproducing the methods and experiments in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\nWeakness 1 & Q3: ***hyper-parameters ablation studies and the importance of the number of prompt tokens***\n\nResponse: \nSimilar to previous works [2, 3], we empirically choose the current hyper-parameters (without using the validation data). Specifically, based on our observation of the scales of the three losses, we choose the current hyper-parameters (without using validation data), so the scales of the three loss functions after weighting (head loss and representation loss, and task loss) are similar in the final loss function. This strategy is simply based on the inherent assumption that each loss should contribute equally to the problem [4].\n\nWe improve on the previous soft prompt methods (e.g., WARP [1]) with our proposed novel information-theoretic objectives. Our choice of number of prompt tokens follows [1] and [5]. The submitted paper has already included results with different number of prompt tokens, detailed in Table 1 and 2. Similar to WARP [1], we observe that more prompt tokens generally lead to better performance.\n\n\nWeakness 2: ***more implementation details***\n\nResponse: \nThe implementation details of our proposed loss functions are as detailed in Section 3 and Section 4.2. Besides, we also include more details about the configurations (e.g., number of prompts, optimization details and networks design) in Section 4.2. Thanks for the suggestion. We will better organize the information for the reproducibility in the updated version. \n\n\nQ1: ***confusion of definition: task loss info loss, representation loss?***\n\nA1: Thanks for pointing this out. Line 257, the \"info loss\" should be the \"representation loss\".\n\n\nQ2: ***experiments on 6 tasks and 3 datasets?***\n\nA2: Sorry for the confusion. We experiment with 3 NLP tasks and 6 datasets.\n\nQ4: ***compared with baseline IDPG?***\n\nA4: Compared to IDPG, our approach has the advantages that we develop the information-theoretic method to better encode task-relevant information. \nSimilar to the evaluation in WARP [1], we only chose the most competitive baselines (in full dataset experiments), in the further evaluations under the few-shot learning setting.\nThanks for the suggestions. We further include the comparison between our approach and IDPG, in the few-shot learning setting.\n\n| 64 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| IDPG | 0.0902 | 0.5018 | 0.6593 | 0.5424 | 0.2596 | 0.3334 | 0.1984 | 0.3693 |\n| InfoPrompt | 0.1567 | 0.6137 | 0.7059 | 0.6697 | 0.2119 | 0.3331 | 0.2113 | 0.4146 |\n\n| 256 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| IDPG | 0.1513 | 0.5523 | 0.7010 | 0.8188 | 0.2503 | 0.4048 | 0.3577 | 0.4623 |\n| InfoPrompt | 0.1750 | 0.6580 | 0.7377 | 0.7305 | 0.2993 | 0.4739 | 0.4034 | 0.4968 |\n\nWe can observe that InfoPrompt generally yields better performance than IDPG due to its advantage to better encode task-relevant information.\n\n[1] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. ACL, 2021.\n\n[2] Jia, Zhiwei, and Hao Su. \"Information-theoretic local minima characterization and regularization.\" International Conference on Machine Learning. PMLR, 2020.\n\n[3] Shi, Yufeng et al. “Information-Theoretic Hashing for Zero-Shot Cross-Modal Retrieval.” ArXiv abs/2209.12491 (2022): n. pag.\n\n[4] Groenendijk, Rick, et al. \"Multi-loss weighting with coefficient of variations.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.\n\n[5] Zhou, Yuhang, Suraj Maharjan, and Beiye Liu. \"Scalable Prompt Generation for Semi-supervised Learning with Language Models.\" EACL 2023."}, {"review_id": "Dun5Rb1kDt", "reviewer": "Reviewer_WbYf", "summary": "This paper introduces InfoPrompt: an information theoretic framework for soft prompt tuning. It does so by introducing two additional loss terms: the head loss (maximizing prompt similarity with the LM head) and the representation loss (maximizing prompt similarity with the encoder's last hidden state).\nThe authors show that InfoPrompt is better than WARP, IDPG, and Adapter baselines with Roberta Large as the base model, on sequence classification, RE, and NER datasets.\nThe paper includes analysis of loss landscapes and theoretical guarantees that their method can be optimized with gradient descent. ", "questions": "Questions:\n* Why is there no IDPG baseline in section 5.2? \n* Prompt tuning (Lester et al) required a traditional causal LM training stage of 100k steps on T5 before prompt tuning for it to work well. The same issue might be hidering vanilla prompt tuning on the encoder-only models used here (Roberta variants). Given that the authors could train Roberta large, it seems that they have the compute required to train the released LM-adapted T5 variants in Lester et al [[1](https://huggingface.co/google/t5-base-lm-adapt), [2](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md##lm-adapted-t511lm100k)] for a better comparison to vanilla prompt tuning. Do the authors think that the prompt tuning baseline here (both InfoPrompt loss weights set to 0) might not be fairly compared to because of the misalignment of the base pretrained models used here? ", "limitations": "Limitations:\n* Evaluation is specific to Roberta large, and only sequence classification and RE/NER datasets. This does not diminish the intellectual value of the method, but is nonetheless a limitation that makes this method hard to confidently brand as better than vanilla prompt tuning. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n* The paper is clearly written, and is easy to follow. \n* Infoprompt is simple to understand and implement. \n* Infoprompt does better than the baselines it compares against: WARP, IDPG, Adapters and vanilla prompt tuning on Roberta Large.\n* The paper illustrates interesting loss dynamics of their method in section 6.", "weaknesses": "Weaknesses:\n* *Baselines*: I think there could be more baselines than WARPS and IDPG used here, like non-prompt-based parameter-efficient tuning methods besides Adapters (like LoRA or HyperFormer), or a more relevant one like HyperPrompt. I think this paper would be a valuable contribution even if it didn't completely beat the performance of those methods, but it would be nice to see how it compares to more diverse baselines (not necessarily the exact ones I described here, though they are certainly good choices). \n* *Datasets*: The authors evaluated on most datasets in GLUE, it would be nice to have the full suite of GLUE evaluation. Having only a subset of GLUE feels like the datasets are cherry picked.\n* *Tasks*: Some results on sequence generation tasks would have made this paper a lot stronger in my opinion, and it would not be significantly more expensive to train a T5 variant with this method on a summarization or translation task. \n* *Minor typos* (did not affect review score):\n    * Table 4 description\n    * line 331\n    * line 307", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\nWeakness 1: ***Baselines***\n\nResponse: Thanks for the suggestion. We further include the comparison between our approach and a new baseline LoRA, as below.\n\n| Full-data | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| LoRA | 0.5880 | 0.6715 | 0.8235 | 0.9541 | 0.6636 | 0.8228 | 0.7214 | 0.7492 |\n| InfoPrompt | 0.6018 | 0.6968 | 0.8137 | 0.9599 | 0.7616 | 0.8962 | 0.7917 | 0.7888 |\n\t\t\t\t\t\t\t\t\n| 64 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| LoRA | 0.0991 | 0.5596 | 0.6985 | 0.5677 | 0.1232 | 0.1345 | 0.1711 | 0.3362 |\n| InfoPrompt | 0.1567 | 0.6137 | 0.7059 | 0.6697 | 0.2119 | 0.3331 | 0.2113 | 0.4146 |\n\t\t\t\t\t\t\t\t\n| 256 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| LoRA | 0.2854 | 0.5740 | 0.7206 | 0.8222 | 0.2291 | 0.1955 | 0.3817 | 0.4583 |\n| InfoPrompt | 0.1750 | 0.6580 | 0.7377 | 0.7305 | 0.2993 | 0.4739 | 0.4034 | 0.4968 |\n\nIn our paper, we target improving the performance for a $\\textbf{single task}$ via prompt tuning (similar to WARP[1], IDPG[3] and LoRA[4]).\nTherefore, we only include single task prompt tuning approaches as baselines.\nHyperFormer is originally designed for $\\textbf{multi-task}$ learning, which is setting that is orthogonal to our paper.\nIn our experiments, we follow the evaluation protocol in WARP[1], IDPG[3] and LoRA[4], and it is not feasible to directly compare HyperFormer in our designed evaluation protocol and experiment setting.\n \n\nWeakness 2: ***Dataset***\n\nResponse: As explained in line 175, these 4 datasets chosen from the GLUE benchmark are relative small in size (line 175), which simulate a more challenging low-resource scenario that is compatible with prompt tuning. \n\n\nWeakness 3: ***Tasks***\n\nResponse: In the paper, we follow previous works of prompt tuning, e.g., WARP [1], IDPG [3], etc., that mostly experiment with classification-based tasks. It would also be interesting to further consider sequence generation tasks. Thank you for the suggestion and we will keep this in mind in our future works.\n\n\nWeakness 4: ***Typos***\n\nResponse: Thanks for pointing out our typos. We will fix them accordingly. \n\nQ1: ***Why is there no IDPG baseline in section 5.2?***\n\nA1: Similar to the evaluation in WARP [1], we only chose the most competitive baselines in full dataset experiments, in the further evaluations under the few-shot learning setting.\nThanks for the suggestions. We further include the comparison between our approach and IDPG, in the few-shot learning setting. \n\n| 64 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| IDPG | 0.0902 | 0.5018 | 0.6593 | 0.5424 | 0.2596 | 0.3334 | 0.1984 | 0.3693 |\n| InfoPrompt | 0.1567 | 0.6137 | 0.7059 | 0.6697 | 0.2119 | 0.3331 | 0.2113 | 0.4146 |\n\t\t\n| 256 shots | CoLA | RTE | MRPC | SST2 | RE | NER | SemEval | Average |\n|---|---|---|---|---|---|---|---|---|\n| IDPG | 0.1513 | 0.5523 | 0.7010 | 0.8188 | 0.2503 | 0.4048 | 0.3577 | 0.4623 |\n| InfoPrompt | 0.1750 | 0.6580 | 0.7377 | 0.7305 | 0.2993 | 0.4739 | 0.4034 | 0.4968 |\n\nQ2:\n***Do the authors think that the prompt tuning baseline here (both InfoPrompt loss weights set to 0) might not be fairly compared to because of the misalignment of the base pretrained models used here?***\n\nA2:\nSince we target classification-based tasks in the paper, we follow WRAP [1] and iPEC [2] that adopt an encoder-only LM (Roberta-Large). For fair comparison (following [1, 2]), all our baselines are implemented with the same base pretrained model (i.e, Roberta-Large).\n\n[1] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing 385 (Volume 1: Long Papers), pages 4921–4933, 2021.\n\n[2] Schick, Timo, and Hinrich Schütze. \"It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n[3] Wu, Zhuofeng, et al. \"IDPG: An instance-dependent prompt generation method.\" arXiv preprint arXiv:2204.04497 (2022).\n\n[4] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).\n"}, {"review_id": "OORQssAOSp", "reviewer": "Reviewer_7rsH", "summary": "This work focuses on the initialization inefficiency of soft prompt tuning. Crucially, soft prompt tuning is notoriously challenging to obtain a good initialization, and highly sensitive to some hyper-parameters, leading to some optimization difficulty especially in low-resource scenarios. In terms of this, this work starts from an information-theoretic perspective of maximizing mutual information to get a better initialization of soft prompts. The experiments on classification, relation extraction, and NER tasks show the effectiveness of their design to several baselines. ", "questions": "No. ", "limitations": "Please see my reply on the weaknesses. Their listed limitations are too general as the typical limitation of prompt tuning... ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The information-theoretic guarantee with some clearly organized mathematical equations is correct, and interesting as their designs, inspired by some contrastive learning techniques.\n2. The experiments do show the effectiveness of their proposed algorithm, mainly in advanced NLU tasks, e.g., NER, and RE. \n", "weaknesses": "1. Of course, this work has some contributions to soft prompt tuning initialization from a different perspective, even such perspective is highly similar to existing work on discrete prompt engineerings [1]. However, I think it has some limited impacts, as your task settings are somewhat less interesting given a lot of works on better discrete prompt optimization with significant better performance under few-shot setting, and many work simply initializing the soft prompts by pre-training on similar datasets show good performance as well. Moreover, current main focus in the community would not be how to tune such negligible soft prompts to a limited range of NLP tasks, instead, people would use better instruction-tuned LMs for real-world application, etc. \n\n2. You need to re-polish your writing, as I do frequently find some typos, e.g., performances in page1, intro, \"your method names -- Infopropmt\" in section 9., limitation, and so on. \n\n3. Your few-shot numbered improvements, which might be your main contributions in your introduction, are still negligible to some extent, further restricting your impacts in this conference.   \n[1] An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels, ACL 2022 ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\nWeakness 1: ***your task settings are somewhat less interesting given a lot of works on better discrete prompt optimization with significant better performance under few-shot setting***\n\nResponse:\nWe would like to politely contend that the discrete and (vs.) soft prompt tuning (PT) are two parallel setups with pre-trained language models [2]. \nAs mentioned in Section 2 of [1] (mentioned in the review), the best prompts are tuned with the continuous embeddings space (soft PT). \nThis is because the continuous the search space of the discrete PT (i.e., a discrete set of token embeddings) is actually a subset of the search space of soft PT (i.e. the whole embedding space).\nThus, the soft continuous prompts should be more expressive with a large search space, and the performance of the optimal prompt from soft PT is expected to upper bound that from discrete PT [3] (Section 7.2).\n\nAdditionally, it is computational challenging with discrete PT to optimize in a discrete search space [2,3]. Comparative, our proposed objectives of soft PT are optimized continuously with provable convergence guarantee (Theorem 1). \n\n\nWeakness 1: ***many work simply initializing the soft prompts by pre-training on similar datasets***\n\nResponse:\nPre-training on similar datasets indeed may lead to better performance. \nHowever, such a transfer learning set up is orthogonal to our problem formulation, i.e., we consider the scenario where no auxiliary datasets are available.\nOur scenario is actually more practical since it can be difficult and expensive to identify the similar datasets of the target task [7].\n\n\nWeakness 2: ***You need to re-polish your writing, as I do frequently find some typos***\n\nResponse: Thanks for pointing out our typos. We will fix them accordingly. \n\n\nWeakness 3: ***Moreover, current main focus in the community would not be how to tune such negligible soft prompts......instead, people would use better instruction-tuned LMs for......***\n\nResponse:\nPlease allow us to respectfully disagree with this statement. Firstly, we believe that tuning soft prompts is hardly considered negligible. Actually, it is an active field of efficient training with pre-trained language models and is being widely studied by many recent papers, e.g., [4-7]. Secondly, existing instruction-tuned LMs (e.g., GPT-3.5/4) are generally not task-specific, thus still yet to excel when being evaluated on many of the NLP tasks [8,9]. Therefore, it is still necessary to further tune such models with task-specific information. For this purpose, soft prompt tuning is an efficient approach and is still being actively studied [2-7].\n\nBesides, although instruction-tuned LMs have been actively studied for real-world application, we have noticed some very recent works showing that instruction-tuned LMs and soft-prompt tuning are complementary. By properly developing soft-prompt tuning methods, it is promising to further improve the instruction-tuned LMs [10, 11]. \n \n[1] An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels, ACL 2022\n\n[2] Liu, Pengfei, et al. \"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.\" ACM Computing Surveys 55.9 (2023): 1-35.\n\n[3] Li, X. L., & Liang, P. (2021, August). Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 4582-4597).\n\n[4] Razdaibiedina, Anastasia, et al. \"Progressive Prompts: Continual Learning for Language Models.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[5] Wang, Zhen, et al. \"Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[6] Razdaibiedina, Anastasia, et al. \"Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization.\" arXiv e-prints (2023): arXiv-2305. (Findings of ACL2023)\n\n[7] Pang, Bo, et al. \"SharPT: Shared Latent Space Prompt Tuning.\" Findings of the Association for Computational Linguistics: EACL 2023. 2023.\n\n[8] GPT-4 Technical Report, OpenAI.\n\n[9] Koubaa, Anis. \"GPT-4 vs. GPT-3.5: A concise showdown.\" (2023).\n\n[10] Sun, Simeng, et al. \"How does in-context learning help prompt tuning?.\" arXiv preprint arXiv:2302.11521 (2023).\n\n[11] Shi, Zhengxiang, and Aldo Lipani. \"Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner.\" arXiv preprint arXiv:2305.01711 (2023).\n"}, {"review_id": "pKYe8e3vLa", "reviewer": "Reviewer_ysu9", "summary": "The authors formulate soft-prompt tuning as maximising mutual information between prompts and model parameters/encoded representations. The prompt is trained to maximise mutual information with the language model head parameters, and the representation of a task encoding from the model. The authors primarily test on GLUE, relation extraction, and NER tasks, and find that their approach closes the gap the most with full-finetuning. Analysis suggests that the proposed method is more stable and converges faster during training, perhaps due to the smoother loss landscape of the info loss.\n", "questions": "- How does InfoPrompt compare to lora (and other PEFT methods)?\n- Could you apply a similar information loss technique to lora or other PEFT methods, and how effective would that be?\n- How did you choose the values for beta and gamma?\n", "limitations": "The authors discuss limitations at the end of the paper to a reasonable degree.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Interesting approach, with clear motivation and description. The experiments are thorough, over a reasonable number of tasks, and show clear improvements over baselines. The analysis also provides useful and interesting insights into how the approach works, and improving both performance and convergence speed of soft prompt tuning is a useful and important result.", "weaknesses": "No glaring weaknesses, but I think the following could be improved:\n- Comparisons with popular parameter-efficient finetuning (PEFT) methods: soft prompt tuning is often considered one of several peft methods, including LoRA. While you do compare with adapters, it would be interesting to compare against Lora or MaM adapters [1], which are generally considered more effective than adapters.\n- The work claims their approach makes prompt tuning less sensitive to initialization. It would be good to see results over multiple seeds (or even multiple initialisation types), with smaller variation than a baseline approach, to back up this claim.\n\nOverall, I think this is solid work and lean to accept it.\n\n[1] He et al., 2022. *Towards a Unified View of Parameter-Efficient Transfer Learning*. https://openreview.net/pdf?id=0RDcd5Axok\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\nWeakness 1 & Q1 & Q2: ***Comparisons with popular parameter-efficient finetuning (PEFT) methods***\n\nResponse: Thanks for your valuable suggestion. Below, we have introduced the Info-LoRA, which outperforms several other PEFT methods (e.g., adapter[4], BitFit[5]). \nInfo-LoRA is implemented by combining LoRA finetuning and our soft-prompt tuning method. In addition to the soft-prompt tokens, we introduce additional parameters by LoRA and these parameters are also tunable in our downstream tasks.\n\t\t\t\t\t \n| Full-data | CoLA | RTE | MRPC | SST2 | Average |\n|---|---|---|---|---|---|\n| info-LoRA | 0.683 | 0.8736 | 0.9069 | 0.9667 | 0.8575 |\n| LoRA | 0.5880 | 0.6715 | 0.8235 | 0.9541 | 0.7592 |\n\t\t\t\t\t\n| 64 shots | CoLA | RTE | MRPC | SST2 | Average |\n|---|---|---|---|---|---|\n| info-LoRA | 0.0988 | 0.5776 | 0.7328 | 0.5333 | 0.4856 |\n| LoRA | 0.0991 | 0.5596 | 0.6985 | 0.5677 | 0.4812 |\n\t\t\t\t\t\n| 256 shots | CoLA | RTE | MRPC | SST2 | Average |\n|---|---|---|---|---|---|\n| info-LoRA | 0.3783 | 0.6462 | 0.7941 | 0.7362 | 0.6387 |\n| LoRA | 0.2854 | 0.5740 | 0.7206 | 0.8222 | 0.6005 |\n \nWeakness 2: ***Sensitivity to initialization.***\n\nResponse: Thanks for your comment on this. We have some related discussion in Section 6.1 (line 255). We also have additional results to validate that, our approach is less sensitive in the early learning stage compared to the baseline approaches. \nFor the results in Figure 3, we further reported the standard errors in the early learning stage. Specifically, we used 10 different random initializations for our method and the baseline WARP, and then reported the standard errors of the performances in the early learning stage (e.g., after the first epoch). The results show that our method's standard errors are lower, compared to WARP. \n\n|  | SST2 | NER-ACE |\n|---|---|---|\n| WARP | 0.7352 $\\pm$ 0.0334 | 0.4606 $\\pm$ 0.0385 |\n| InfoPrompt | 0.7639 $\\pm$ 0.0270 | 0.8099 $\\pm$ 0.0253 |\t \n\n\nQ3: ***How did you choose the values for beta and gamma?***\n\nResponse: Similar to previous works [1, 2], we empirically choose the current hyper-parameters (without using the validation data). Specifically, based on our observation of the scales of the three losses, we choose the current hyper-parameters (without using validation data), so the scales of the three loss functions after weighting (head loss and representation loss, and task loss) are similar in the final loss function. This strategy is simply based on the inherent assumption that each loss should contribute equally to the problem [3].\n\n[1] Jia, Zhiwei, and Hao Su. \"Information-theoretic local minima characterization and regularization.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Shi, Yufeng et al. “Information-Theoretic Hashing for Zero-Shot Cross-Modal Retrieval.” ArXiv abs/2209.12491 (2022): n. pag.\n\n[3] Groenendijk, Rick, et al. \"Multi-loss weighting with coefficient of variations.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.\n\n[4] Houlsby, Neil, et al. \"Parameter-efficient transfer learning for NLP.\" International Conference on Machine Learning. PMLR, 2019.\n\n[5] Zaken, Elad Ben, Shauli Ravfogel, and Yoav Goldberg. \"Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.\" arXiv preprint arXiv:2106.10199 (2021).\n\n"}]

Author rebuttal:

Meta-review: The paper presents a new method for soft prompt tuning in NLU tasks, using mutual information-based loss functions. It shows superior performance over existing baselines but lacks thorough ablation studies and diverse comparisons. The work is incremental and could benefit from broader evaluations and more detailed implementation information.

The reviewers are generally all satisfied with the authors' responses. The reviewer who insists on borderline reject agreed to accept the paper as poster.

AC reads the paper (roughly), the review, and the discussion, and recommends accept (poster).
Decision: Accept (poster)