Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models
Authors: Zijun Wu
Yongkang Wu
Lili Mou
Venue: ICLR 2024 poster (2024)
Keywords: continuous prompt tuning
zero-shot prompt transfer
cross-model prompt transfer
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt tuning in natural language processing (NLP) has become an increasingly popular method for adapting large language models to specific tasks. However, the transferability of these prompts, especially continuous prompts, between different models remains a challenge. In this work, we propose a zero-shot continuous prompt transfer method, where source prompts are encoded into relative space and the corresponding target prompts are searched for transferring to target models. Experimental results confirm the effectiveness of our method, showing that 'task semantics' in continuous prompts can be generalized across various language models. Moreover, we find that combining 'task semantics' from multiple source models can further enhance the performance of transfer.

Published as a conference paper at ICLR 2024
ZERO-SHOT CONTINUOUS PROMPT TRANSFER: GEN- ERALIZING TASK SEMANTICS ACROSS LANGUAGE MODELS
Zijun Wu', Yongkang Wu', Lili Mou
'Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta
Huawei Poisson Lab $Canada CIFAR AI Chair
zi junu4uu berrra. ca, wuyongkang www.wa/ggghhwww...comm com,
doublepower mouu souegmaiin ..om
ABSTRACT
Prompt tuning in natural language processing (NLP) has become an increasingly popular method for adapting large language models to specific tasks. However, the transferability of these prompts, especially continuous prompts, between different models remains challenge. In this work, propose zero-shot continuous prompt transfer method, where source prompts are encoded into a relative space and the corresponding target prompts are searched for transferring target mod- els. Experimental results confirm the effectiveness of our method, showing that "task semantics" in continuous prompts can be generalized across various lan- guage models. Moreover, we find that combining "task semantics" from multiple source models can further enhance the performance of transfer
INTRODUCTION
Recently, natural language processing (NLP) has witnessed a paradigm shift from the finetuning of full language models to the optimization of small subset of prompt tokens (Shin et al. 2020 Lester eet all] 2021 Li & Liang 2021 Zhong et al., 202T). As language models have dramatically increased size and may contain billions of parameters (Brown et al.) 22200), the strategy of freezing language models while optimizing the learnable prompt parameters becomes the most affordable and efficient alternative for downstream tasks. This technique, referred to as prompt tuning, has gained substantial
recognition for its effectiveness across a range of language models (Shin et al. 2020 2021
Lester et al.)
Liang 2021 Zhong et al. 2021).
Various prompt tuning methods have been explored, which can be generally categorized into discrete and continuous cases. Discrete prompt tuning, such as AutoPrompt (Shin et al.] 2020), primarily fo- cuses on the selection and optimization of predetermined set of tokens within a language model's vocabulary. By contrast, continuous prompt tuning (Zhong et al., 202I allows the modification of continuous prompt embeddings by gradient descent. The latter typically offers better performance on downstream tasks due to its greater flexibility in the prompt space. However, existing prompt tuning often requires accessing the model's internal states, the gradient needs to be backpropa- gated to the first layer of token embeddings (Shin et al. 22020] (220)/Zhong [Zhong et al. (202)). which contradicts the goal of avoiding gradient computation for large language models.
Therefore, it would be ideal if we can perform prompt tuning on a small model (which is compu- tationally inexpensive) and transfer the prompt to large models. We thus question: How transfer- able are these prompts between different language models? Prior research on transferring prompts mainly focuses on discrete prompts (Rakotonirina et al. 2023). Such transfer is often straightfor- ward, as discrete prompt tokens usually carry semantic meanings by their nature and can be directly accepted by different language models. For continuous prompts, however, prompt transfer becomes less straightforward because they are unexplainable and sparsely distributed in high-dimensional space (Khashabi et al. 2022 et al. 2022). Moreover, different models might learn the embedding 'Our code is available httpp:///gittb...oo////eeee/e/ee

---

Published as a conference paper at ICLR 2024
space differently, due to their designs, sizes, training paradigms, as well parameter random initial- izations. Therefore, transferring a continuous prompt tailored for one model to another, especially with different dimensions, remains challenge.
Attempts to bridge this gap have centered around introducing a neural projector that aligns continu- ous prompts across different models. However, the learned projectors specific unique model pairs. More importantly, it introduces extra computational cost because the training requires task supervision on the target model and the source prompt embeddings, or even the need utilize the parallel prompt embeddings for both models (Su et al. 2022). These approaches cannot be applied zero-shot transfer scenario, and are undesired in real applications.
In this work, we propose novel approach to zero-shot continuous prompt transfer without the need for task supervision or additional training of neural projectors. We introduce an strategy, where we encode the source prompts into relative space (Norelli et al. 2022 Moschella al. 2023) and then search for the corresponding target prompt embeddings. Our intuition is that the induced continuous prompt contains implicit information for task (Vu et al. 2022 Wang ct al. 2022). referred to as "task semantics", which may be carried over from the source embedding space to the target. We suggest that, although direct transfer of prompt embeddings is problematic because different language models have their own embedding spaces, the position the continuous prompt embedding relative to the embeddings of known words is more likely to share the same structure in different language models, inspired by the evidence of representation learning literature in other domains, such word embeddings (Faruqui & Dyer 2014 Lazaridou et al. (0155) Artetxe it al. 2018), synthetic structure discovery (Wu et al. 2023
unsupervised neural translation (Lample
al., 2018), and cognitive science (Levakov et al. 2021 Chersoni et al. 2021). In our approach,
the transfer of prompts only requires a shared vocabulary of common tokens, which serve as the anchors of the relative embedding space. For the target model, we search for its prompt embeddings that preserve the same relative structure as that of the source language model.
Our experiments confirm that, with our proposed zero-shot approach, continuous prompts are trans- ferable to different language models, largely outperforming baseline approaches such as training neural projectors. We also discover that utilizing continuous prompts from multiple distinct source models enhances the generalizability on target models. This is because the semantics of the prompts induced from a single source might be model-specifie, whereas the multi-source method provides a more robust view of the task semantics, therefore achieving higher performance of prompt transfer. In short, our contributions are summarized as follows:
We address a novel setting of zero-shot continuous prompt transfer, which allows for the reuse of continuous prompts across different language models.
We propose an encode-then-search strategy that maps continuous prompt into a relative space for transfer between language models. Our approach facilitates multi-source transfer, which cannot be easily done by previous work.
We provide detailed experimental analysis on a factual-probing suite of 41 types of ques- tions show the effectiveness of our approach.
METHODOLOGY
In this section, we begin with an overview of continuous prompt tuning in 22. We then introduce our encoding (222) and decoding ($2.3) methods for transferring continuous prompt embeddings between language models. Finally in $2.4 we discuss our multi-source transfer approach for im- proving the performance of transfer.
2.1 CONTINUOUS PROMPT TUNING
Continuous prompt tuning [Zhong et al. 2021) optimizes the embeddings of a prompt in the con- tinuous space for a downstream task, which essentially introduces virtual tokens to the language model. During this process, the continuous prompt, which is a set of learnable embedding vec- tors, capture the implicit information for the task of interest (Vu et al. 2022). Different from full-model finetuning methods (Zhou & Srikumarr (022), the gradient from the final loss function backpropagates through the model but only updates the initial embedding layers.
2

---

Published as a conference paper at ICLR 2024
Source
Source LM
Induced prompts
anchors
Target
Target LM
Target prompts
Relative Space
Torget Space
(a)
(b)
Figure 1: (a) The goal of transferring the induced continuous prompts on a source model to target model. (b) Our proposed method for this transfer in a zero-shot manner, where the target prompts should be aligned with the induced source prompts in the relative space.
Consider prompting language model for some task. A continuous prompt has the following format:
Prompt (æ) V1 V2 Vm
(1)
where x is an input data sample, and m is pre-defined prompt length, i.e., the number learnable vectors. The configuration of v; either prefix or postfix is aspect of prompt design. In our implementation, we append these tokens as a postfix to z. For each virtual token vi, its embedding learnable vector vz E Rd that has the same dimension as the embedding layer of the language model. The soft prompt tuning objective is to maximize the likelihood of the output of the training sample, given by the source model (-(-)
arg max 11 Mi (M.))D
Σ
log Ply U1 Um, 2)
After the continuous prompts Bjy...n Um are optimized, they are usually used make inference on the same model for the same task (Lester et al., 2021 Li & Liang. 2021 Zhhong et al. (02T).
Prompt tuning is more efficient than full-model finetuning because only small number of param- eters, i.e., the embeddings of the virtual tokens, updated for downstream task. Meanwhile, maintains substantial flexibility and achieves similar performance to full-model finetuning (Lester) al. 202I). However, learning these virtual tokens may still be expensive for large language mod- els because we need to perform backpropagation through the entire model structure. Our goal is to explore the feasibility of learning continuous prompt with small language model and transferring to larger ones, therefore avoiding excessive gradient computations of prompt tuning for different large language models.
2.2 ENCODING TO RELATIVE SPACE
We propose to transfer continuous prompts from source language model to target models. The process involves two key phases: (1) encoding the source prompt embeddings into a relative repre- sentation and (2) searching for target prompt embeddings whose corresponding relative representa- tion aligns with those the source. This two-phase method facilitates the effective transfer of task semantics between different embedding spaces, allowing the transferred prompts to accomplish the task on the target model.
The concept of relative representation involves encoding data point based its similarities to certain reference points, known as anchors (Norelli et al.) 2022 Moschella et al.] 2023). our work, the relative space, where these encoded representations reside, serves shared semantic space across different models and facilitates the transfer process of continuous prompts.
Consider continuous prompt U1 V..... Um ddd, where d, indicates the embedding dimension of the source language model. We aim to transform it into a relative representation. This can be shown in Figure Ip transferring orange stars to orange triangles.
To encode the relative embeddings the prompt, we need set common tokens, serving as anchors, that shared between both source and target language models. We simply choose the

---

Published as a conference paper at ICLR 2024
shared tokens as the set of anchors, they provide common ground for expressing prompt in relative terms, regardless of differently learned embedding spaces. Specifically, the anchors' embeddings in the source model can be represented by matrix A [a az. a E R4x; where & is the number of anchors, and [.] concatenates column vectors into a matrix.
We then encode prompt embedding e; in relative space with respect to these anchors, where compute the cosine similarity between a prompt embedding and each anchor's ' embedding, given by
r T 0z)) = (cos( (U;, ai);"" ... cos( D;; 0ik)
This encoding step translates continuous prompt from the source language model into a language using the relationship among common tokens so that other models can potentially understand. bridges the source and target language models and passes implicit task information contained in the source continuous prompt.
2.3 SEARCH IN THE TARGET SPACE
We search a continuous prompt for the target language model, based on the intuition that the relative embeddings model-agnostic and can be aligned across different language models, i.e. the orange and green triangles in Figure [ should have the same structure. In this way, we can search the (absolute) target prompt embeddings by maximizing the alignment of the source and target relative spaces.
Concretely, the target embeddings of, 222 ... "y target embedding dimension and may not be the same as d,. They are represented by green stars in Figure Dp. These target embeddings [a a² a E Rd shown by green squares in Figure lb. These target anchors are the same as source anchors, and their embeddings are given by the target language model. Similar encoding the source prompt, the target embeddings can be represented in the relative space by
R& are randomly initialized, where d, is the
then encoded using the target anchor embeddings A'
(u,) (cos(ui, ... .i),,rr.cos((e a²))
(4)
To align source and target relative embeddings, i.e., Eqns. (3) and we seek target embedding of that maximizes their similarity. The objective is
maximize cos(T (v;), (b)))
(5)
which can be accomplished by gradient descent. This procedure repeated for i = 1,... ....... m to obtain all the embeddings of length-m prompt for the target language model.
is noted that such searched prompt embeddings may not have the same scale as the target language model's 's word embeddings, partially because the cos measure used in (3) and (4) is insensitive to vector magnitude. Therefore, we normalize them as
(6)
with Mns ay E RR being the mean and standard deviation all searched prompt embedding values, and p. a E R being those of pretrained word embeddings of the target model.
The transferred continuous prompt, after the normalization, is directly used to query the target model P.(-) for inference, given by Pi(y ot .., , 2))
2.4 MULTI-SOURCE TRANSFER
We argue that the induced prompt embeddings from a single model might be model-specific, which supported by the evidence in Khashabi et (2022) that a language model can generate numerous continuous prompts capable of performing the same task. Such flexibility is believed to arise from the high expressiveness of the model's lower layers (Telgarsky) 2016 Raghu et al.) 2017). There- fore, the induced continuous prompt from specific model may be one of the many plausible ones, carrying model-specific information in addition to task semantics and limiting the transferability to other language models.
4

---

Published as a conference paper at ICLR 2024
To enhance the generalization of the induced continuous prompt, we propose multi-source transfer approach. Specifically, we search for the target embeddings o' whose encoded embeddings r T (v") align closely with the encoded embeddings r. (v")) from multiple source models s; in the relative space. In other words, the goal is to search for o' such that the sum of similarities between r T. and each source prompt rĄ (v%) is maximized. Given S-many source models, the objective of searching a target embedding v is:
S maximize cos Cos(TA'i A (1
We follow Eqn. 66) to normalize of target model's embedding space, and use the resulting vectors 3 for inference.
3 EXPERIMENTS
3.1 DATASET
We utilized widely used factual probing dataset, LAMA (Petroni et al. 2019), to evaluate the effectiveness of our continuous prompt transfer approach. We followed recent factual probing stud- ies (Shin al., 2020; Zhong al. 2021) that focus the TREx split of LAMA. Specifically, LAMA-TRES presents a factual knowledge piece as a triple (subject, relation, object). For For example, the fact that "Dante was born in Florence" represented as (Dante, place of birth, Florence), where "place of birth" is a pre-defined relation in the dataset. In total, there are 41 distinct relations as subtasks, each of which contains up 1, 000 tuples. We chose factual probing for two key reasons: First, induced prompts represent different task semantics from 41 sub-tasks (distinct pre-defined re- lations), providing a robust way to evaluate the generalizability of our transfer approach in various scenarios. Second, the factual probing task requires the model to precisely predict the correct entities from its vocabulary. This makes easier to judge the performance of a prompt.
On the source pretrained language model, we adopted OptiPrompt (Zhong et al. (021) for inducing a continuous prompt for each of the 41 sub-tasks. Given a sub-task certain relation, the source language model is queried using the prompt defined in Eqn. (T) where the prompt embeddings are randomly initialized and then optimized according to Eqn.
3.2 CHOICE OF MODELS AND OTHER IMPLEMENTATION DETAILS
We investigated our transferring approach across a range of language models, namely, BERT (De- vlin al. 2019), RoBERTa (Liu et al. 2019) and ALBERT (Lan et al. 2020), including base and large variants. It should be noted that ALBERT utilizes parameter sharing across layers and reduced embedding dimensions, which, some extent, ties the semanties of embeddings and hidden lay- ers. Therefore, we only used BERT and RoBERTa as the source language models while excluding ALBERTA due its unique architecture that does support full compatibility with BERT and RoBERTa. All these models are considered as target models for transfer.
In the main experiments, of anchors k to 8192. We report the standard evaluation metric, micro-average accuracy, which is calculated by averaging the accuracy of 41 sub-tasks of distinct relations in the LAMA dataset (Shin et al. 2020 Zhong et al.] 2021). These settings are applied to both our approach and baselines. More implementation details are shown Appendix A.
set the default number of prompt embeddings
5, and the number
3.3
MAIN RESULTS
Direct transfer. We first analyze naïve method, direct transfer, which directly applies the source- induced continuous prompt to the target model. This provides us with general understanding of whether continuous prompts are directly transferable. We show the results of direct transfer in Table aas as standalone experiment, as it does not fit our main table because direct transfer is only feasible when the source and target embedding dimensions match.
5

---

Table shows the main results of our proposed method along with several continuous prompt trans- fer baselines. We experimented with a straightforward method, called discretization (Khashabi al., 2022). for continuous prompt transfer. Specifically, each prompt embedding projected its nearest-neighhor token embedding, and these discrete tokens are transferred to the target model. As (2022), such a method yields poor transferability, probably due to the Published as a conference paper at ICLR 2024
The results reveal that continuous prompts induced from the base mod- els of BERT and RoBERTa (both with 768 dimensions) perform poorly when transferred each other (around 0.1% accuracy). For their large variants, the transfer performance from RoBERTa to BERT improves marginally, achiev- ing around 0.5% accuracy. Transfer- ring from BERT to RoBERTa achieves nearly 6.3% accuracy, but is still far from ideal. Overall, this experiment verifies that continuous prompts are not directly transferable, signifying the importance of prompt transfer research.
Table Accuracy of direct transfer between models with the same embedding dimension. Results in percentage.
Source
Target
damhedding
Transfer acc (%)
BERT base
RoBERTapse
768
0.11
RoBERTagae
BERT base
768
0.12
BERT luge
RoBERTa Taaagg 1024
6.27
RoBERTaaage
BERT lane
1024
0.49
Baselines and single-source transfer. Table represents the results of non-transfer baselines for reference. In the random method, prompt embeddings randomly sampled from normal dis- tribution fitted to the target models' word embeddings. Its all-zero performance implies that factual probing is challenging task that requires non-trivial efforts from machine learning model. In direct tuning, the continuous prompt is directly tuned with the target model. As expected, direct tuning achieves high performance, but is undesired it requires backpropagation through the target model; it serves as an "upper bound" of prompt transfer in the factual probing task. We also present the performance of manual prompts, provided by the LAMA dataset (Petroni et 2019). serving as another reference score for evaluating prompt transfer methods.
analyzed in Khashabi
Table 2: Results of the non-transfer baselines, serving as reference scores for transfer.
Target
BERT
BERT Tuge
RoBERTa
RoBFRTaaage
ALBERT basc
ALBERT Tuge
Method
Random
0.00
0.00
0.00
0.00
0.00
0.00
Manual
30.64
32.22
20.48
23.59
18.63
24.44
Direct tuning
50.56
51.97
46.24
41.06
42.98
43.78
Table 3: Main results. Best performance is highlighted bold, while second-best performance is underlined. The numbers in gray the self-transfer performance.
Target
BERTuw
BERT Lurge
RoBERTabae
RoBERTauuge
ALBERT
ALBERT lurge
Source
Discretization
BERT
12.93
10.76
10.88
11.96
11.44
11.10
RoBERTaacss
12.31
10.35
13.51
11.01
11.67
12.33
BERT Tnrgo
9.00
11.02
5.64
11.93
7.35
6.66
RoBERT TRRaue
1.35
0.70
2.28
6.22
3.15
2.64
Neural projector
BERT hase
26.82
12.49
14.36
9.78
18.77
RoBERTAR Tabase
23.46
17.46
35.37
20.16
11.63
14.44
BERT large
3.15
4,77
5.64
4.66
8.18
T4.53
RoBERTS Blarge
2.62
3.20
5.54
12.80
7.45
8.25
Single source (ours)
BERT
49.82
31.40
17.68
21.07
20.83
16.80
RoBERTavae BERT harge
31.33
27.52
45.1
25.09
26.11
24.72
26.78
"50.21
7.64
16.91
T5.10
T3.43
RoBERT FOBERTaane Taaagg
3.81
12.45
3.63
40.91
4.48
2.94
Dual sources (ours)
BERT base
BERT lurge
49.21
47.78
27.60
23.21
23,67
22.32
BERT hese:
RoBERTEn Teewo
48.79
32.83
25.26
27.13
26.54

---

Published as a conference paper at ICLR 2024
Source:
BERT
Source: BERTipge
Source: RoBERTagse
BERT
0/1
valiation accuracy
BERT
04
RoBERTObese
RoBERTabrge
TEace
ALBERTuage
0.1
80
Matching ng loss
Matching loss
Motching loss
Figure 2: Validation accuracy vs. matching loss, with the curves showing the performance of various target models.
expressive power in the neighbor of discrete token embeddings. Our results also demonstrate the low performance of discretization, which is consistent with previous work and indicates that more nuanced approach needed for effective prompt transfer.
In addition, we included an intuitive baseline method, the neural projector (Su et al. 2022), for comparison. Specifically, we first trained a two-layer projector map the source embedding space to the target one based on anchor words. Then, projected the source-induced prompt embed- dings to the target model using the trained projector. Detailed settings and results are provided in Appendix A.2, As seen, transferring the induced prompt embeddings through the projector provides better results but still falls short of manual prompting.
Now, we consider yields consistent improvement compared with the neural projector, which is a compelling result as method does not require learning mapping from the source embedding space to the target. This verifies that our proposal of working with a relative space is more effective than the original embedding space for continuous prompt transfer. More profoundly, the prompts transferred from the base models of BERT and RoBERTa surpass the manual prompting baseline, manifesting the practical value of our proposed prompt transfer method.
proposed prompt transfer method with single source. As seen, method
Multi-source transfer. Finally, we evaluate our proposed multi-source prompt transfer method as described Section 2.4] We consider two dual-source settings: BERTusc++ERS jarge and BERT RooBeR+ "The results are also shown in Table 33] As seen, using multiple sources generally improves transferability. For example, the BERT +BRRT dual-source setting outper- forms BERT base by 2-10 percentage points, although BERT large alone is not strong source model. Compared with the best single source, the BERT base +RoBERTabae dual-source setting yields an improvement of 1-2 points the target models of ALBERT hase and ALBERT largo. which are not in- volved in the prompt tuning process. Overall, this experiment verifies that multiple sources improve the transferability of continuous prompts.
Transferability and expressive power. We observed Table 3 that a larger source model (ci- ther BERT large RoBERTaaagee) has lower transfer performance. This aligns with the intuition in Khashabi et al. (2022) that there could be a large number of similarly performing continuous prompts, residing in a large (and also deep) model's expressive embedding space (Telgarsky) [016]] Raghu et al. (017)) Therefore, the induced prompt may carry model specificity in addition to task semantics, limiting its transferability other models.
Fortunately, the low transferability of large source models does not affect the value of our work, because our typical application scenario tune continuous prompt on small source model and use it for large model. This is precisely the setting where our approach is especially effective.
3.4 ANALYSIS
Matching in relative space vs. transferability. Our approach is based on the intuition that the task semantic is carried in relative space, which can be aligned for different language models. We analyze whether a closer matching of the relative space yields a higher transferability of the continuous prompts. In Figure 2 show the trend of validation accuracy versus matching loss along the search process in Eqn. where for each source-tare target combination, we averaged the performance of all sub-tasks.

---

Published as a conference paper at ICLR 2024
Source: BE gase
Source: BERTange
Source: RaBERTagge
ALB
norm w/o norm
ALBERTiace
RoBERTatwe
BERTure
BERT
ACC uracy
Accuracy
Accuracy
Figure 3: The effect of normalization.
01.42
0.30
Accuracy
Accuracy
0.2
0.20
082
33
(a)
[b] Prompt length
The effect of the anchor number and prompt length. Each value (dot) was computed by averaging the accuracy from all Source-targe combinations.
Figure 4:
In Figure 2 we observe that, as the matching loss decreases (towards right in the plots), the validation accuracy of target models increases. the special case where source and target models are identical, the matching loss of the relative space is able to approach zero, and the accuracy of the transferred prompt close to that of the source prompt. This demonstrates that our approach able to recover the original embeddings from the relative space, even if a normalization is performed to the target embeddings in Eqn.
When source and target models are different, the matching loss does not approach zero, which is reasonable because the different language models' embedding spaces may not perfectly aligned. Nevertheless, the correlation between matching loss and validation accuracy is highly consistent across all source-targe target combinations, convincingly showing that a better matching in the relative space leads to more transferable continuous prompt.
Effeet of normalization to target embedding space. We further provide an ablation study on the normalization of target embeddings introduced in Eqn. (6). Specifically, compare the perfor- mance of the target prompts with or without the normalization treatment.
From Figure 3 we observe that, when the source and target models are identical (non-trane transfer set- tings), the normalization hurts the performance, as it distorts the original word embedding space. However, the gap is minimal, which provides additional evidence that we able recover the original embeddings through our relative space even with the normalization.
When the source and target models different, however, the normalization significantly improves the performance. The results confirm that the normalization treatment can better cast the relative embedding of source-induced continuous prompt into the target embedding space, directly under- stood by the target language model.
Effect of the anchor numbers and prompt length. We analyze the number of anchors and the prompt length. We first varied the number of anchors from the set (512, 1024, 2048, 4096, 17230), } where 17, 230 is the number of the shared tokens in different language models considered this study. The anchor number decides the feature dimensionality of the relative representations, shown in Eqns. (3) and (4) Figure 4 reveals general trend of improved transfer performance with an increasing number of anchors. When we have 512 anchors, the transfer performance is the lowest, which is due to the inadequate capacity of the low-dimensional relative space. On the other hand, using the entire shared vocabulary results in marginal decrease in performance. This is reasonable because the embeddings of rare words may not be well trained, consequently introducing noise to the high-dimensional feature space.
8

---

Published as a conference paper at ICLR 2024
We then investigate the effect of prompt length transfer performance, where chose the length from the set (1,3, 5, 10). We observe in Figure 4p that the performance of transfer improves until certain point, namely, five virtual tokens in our case. With a prompt length of 10, the transfer performance decreases slightly.
Overall, our approach is robust these hyperparameters. Based on this analysis, we set the number of anchors to 8192 and the prompt length to in our main experiments (see $ $3.2).
Additional results. We provide additional results in the appendices. These include study of transferring induced prompts across different model architectures, such as from BERT to GPT-2, detailed in BB.1 Furthermore, we demonstrate the applicability of our method to classification
tasks, discussed in |B.2
4 RELATED WORK
In recent years, language models (LMs) have shown impressive few-shot learning capabilities that allow them to be adapted to variety of downstream tasks through the design of textual prompts (Brown et al., 2020p. Subsequent research improves the performance of NLP tasks by creating discrete prompts that are manually crafted, searched through gradient descent (Shin et al.) 2020), or using reinforcement learning (Deng et al.) (022). Meanwhile, there has been a growing interest in continuous prompts tuming (Li & Liang] 2021 Lester et al. 202 Zhong et al. 202T These studies suggest that tuning a small number of parameters in prompt embeddings can match the performance of full-model finetuning (Houlsby al. 2019 Lester et al.. 2021). which shows the potential of continuous prompt tuning.
Several previous studies have tried utilize the induced continuous prompt to other tasks other LMs. For example, Vu et al.] (2022) show that prompt embeddings induced on a certain task can be used to initialize the prompts for similar tasks. This led further research on retrieving and mixing continuous prompts for new tasks (Asai et al. 2202;; Su et al.) [022] Wang et al. (023). Su ct al. 2022) further study the transferability of continuous prompts in cross-LM scenarios. They propose to train a projector between the embedding space of two LMs with parallel induced prompt embeddings or task signals, which contrasts with the zero-shot transfer approach in this work.
Rakotonirina et al. (2023) and Wen et al. (203)) investigate zero-shot transferability between differ- ent LMS using the induced discrete prompts. Their work is orthogonal to ours as we focus on the cross-model transfer of continuous prompts. Although transferring discrete prompts offers greater simplicity compared with our proposed continuous prompt transfer, continuous prompts are more versatile and can be adapted to broader range of applications.
The concept of mapping different embeddings into a shared latent space has been well explored in the cross-lingual scenario (Faruqui & Dyer, 2014; Lazaridou et al. 2015) Artetxe et al. 2018),
which further paves the way for unsupervised neural machine translation (Lample et al., 2018
We follow the assumption from these studies and assume the induced task embeddings ( (Vu al., 2022) from different language models share similar latent structures. We employ relative represen-
tation (Moschella et al., 2023) to encode source-induced prompt, and decode it to the embedding space of the target model.
CONCLUSION
We introduced a zero-shot method for transferring continuous prompts between different language models through relative space. Experiments confirm the effectiveness of our approach, and we further provide insights into the correlation between a model's transferability and its expressive power. Morcover, we propose simple method to improve the generalizability of prompt transfer by using multiple source models.
Given the shift towards unified large language models (Brown et al.) 2020), our method could enable smaller source models to act as effective "soft prompt engineers' that perform better than manual prompting. Additionally, it is a promising direction to explore direct human human-model interactions that bypass the need for discrete language. This will involve prompting pretrained language models using continuous prompts transferred from sources like encoded brain signals (Zou al. 2021

---

Published as a conference paper at ICLR 2024
REFERENCES
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully un- supervised cross-lingual mappings word embeddings. In Proceedings of the Annual Meet- ing of the Association for Computational Linguisties, pp. 789-798, 2018. URL https:
Akari Asai, Mohammadreza Salchi, Matthew Peters, and Hannanch Hajishirzi. ATTEMPT: Paramettr--ffficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 6655-6672, 2022.
URL URL.tttss: htt.hhttp//eee..... pittps://acll antho ogy. org/ 2022 emn comnlp-maing 446
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020. URL htttps: ngs neurips ccppaper Filles/paper/2/
Emmanuele Chersoni, Enrico Santus, Chu-Ren Huang, and Alessandro Lenci. Decoding word embeddings with brase semantic features. Computational Linguistics, 47:663-698 (983 2021.
URL
00412
Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with reinforcement learning. Proceedings of the Conference on Empirical Methods in Natural Language Process- ing. pp. 3369-3391, 2022. URL. atttps://act antho wochollogg logy. org/ 2022. emn 1 p- in.... 222
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4171-4186, 2019. URL http://acaanthhhhoo logy wwtp//191//122222
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilin- gual correlation. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pp. 462 471, 2014. URL
org//
214-1049
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790-2799, 2019. URL https: https:://proces http:///xoceeeiing.oo www..rr...... pressppres///eme. / v97 /27//hou http:/hlllllllyyeee hwtpllllhhhhhhyyhaaaaeaa pdf.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt wayward- ness:
The curious case of discretized interpretation of continuous prompts. In Proceedings the Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 3631-3643, 2022. URL https::/accaathho .org/
2022. nwacl paaacl-main.. 266
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014. URL httttss:/aaxxi..o///bbs....
6980
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 2018. 8. UR..tttee httt:p:://operrevieee
10

---

Published as a conference paper at ICLR 2024
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. In Interna- tional Conference Learning Representations, 2020. URL hatpss://ppneeviee net//
forum? d=
Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. Hubness and pollution:
Delving into cross-space mapping for zero-shot learning. Proceedings of the Annual Meeting of the Associ- ation Computational Linguistics and the International Joint Conference Natural Language Processing, pp. 270-280, 2015. URL httppt::: / //aaaaaanthooooooo
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for crrumeter--fffciiee prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021. URI. htt.:::ttt/.t....... //ac anthonnohhyy.. loyy org/ 2021 emmll p ma n. 243
Gidon Levakov, Joshua Faskowitz, Galia Avidan, and Olaf Sporns. Mapping individual differ- ences across brain network structure to function and behavior with connectome embedding. Neurolmage, 242:1 18469, 2021. URL /ttp:::/t/////t sci encedi rect. com/ sccin/cicae encee axticle/pl//S1003111990000
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In
Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference Natural Language Processing, pp. 4582-4597, 2021. URL
aacc-ll-oo 353
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pre- training approach. arXiv preprint arXiv: arXiv:190.11199, 2019. URL
abs/ 1ess/997..1692
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023. /RLphtttss////peeeee//
nett/Eorum?Id-SrC-n-ia
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. ASIF: Coupled data turns unimodal models multimodal without training. arXiv preprint arXiv:2210.01738, 2022. URLhttps://arxivy www.wxwwww.w.......o.oeee 22210 01738
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference Natural Language Processing, pp. 2663,,473, 2019. URL
D19- D19-1250
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the ex- pressive power of deep neural networks. In Proceedings of the International Conference Ma- chine Learning, pp. 2847-2854, 2017. URL ptttps://proceedingg ml wtt.://nggs.....pss/s//.oo
rachul 7a. html
Nathanaël Carraz Rakotonirina. Roberto Dessi, Fabio Petroni, Sebastian Riedel, and Marco Baroni. Can discrete information extraction prompts generalize across language models? In International Conference on Learning Reprexentations, 2023.
fet/oormm?
id=sbbVtxxq8 zFF.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 4222-4233 2020. 346
URL http:://aclaatthooooooo org 2020
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language processing. In Proceedings of the Conference of the North
11

---

Published as a conference paper at ICLR 2024
American Chapter the gies, pp. 1999, 2022. URL attbss//aclaatttooooy.oo.rt 202. naacl -main.
the Association for Computaational Linguistics: Human Language Technolo-
290
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for laguuag--oodella---eevviii In Proceedings of the International Conference on Machine
Learning. pp. 20841-20885 2022. URL bttps::/prrocee.gggg.... mir...ssm.....mm wtw.smww..p..ss122.
sun22e html.
Matus Telgarsky. Benefits of depth in neural networks. In Annual Conference on Learning Theory, pp. 1517-1539, 2016. URLhttpps: htttpps:/proceeddiggg mlr pttpss///9/telgaarsss////e/ http://ddnn.........eess///////s html
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT:
Better frozen model adaptation through soft prompt transfer. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 5039-5059, 2022. URL httptsss//aaalaattoooooooooo//ot
.rr/
2022. acl- long. 346.
Zhen Wang. Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efficient transfer learning. In International Confer- ence on Learning Representations, 2023. htttt::///ppeeeeeeeettt net forum? ?id=
Nk2pDtuhIq
Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In
Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
139-149, 2022. URL https: https///ppaacccss.ee www.s.heev...mm eonteontett /CVPR2022/ Sspers/Wano, Learning To Pro_Prrmpp_____ for Continual. LerellL_earnnnggg CVPR 2022 paper.pdf
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Advances Neural Information Processing Systems, 2023. URL htttps://openzeviiee
net Sett/Eorumiid-oo0 Vost
Zijun Wu, Anup Anand Deshmukh, Yongkang Wu, Jimmy Lin, and Lili Mou. Unsupervised chunking with hierarchical RNN. arXiv preprint arXiv:2309.04911 14999, 2023. URL https:
/arxiv.org/abs/23000 04999
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.0083 2015. /tt.hhtts:///rxii....
0ts/150550088.0
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017-5033, 2021. URL
https:://cclanttie
wwg..org/..... 2021 naacl -ma i in. 398
Yichu Zhou and Vivek Srikumar. closer look at how fine-tuning changes BERT. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 1046-1061, 2022. URL /ectps://aclantholll
Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong.
Towards brain-to-text genera- tion: Neural decoding with pre- praited encoder-decoder models. In NeurIPS 2021 Al for Science Workshop, 2021. URL URL.nttpss///geeereeee....e
12

---

Published as a conference paper at ICLR 2024
Table 4: Details of the pretrained language models considered in this study. MI., NSP, SOP, and NTP stand for masked language modeling, next sentence prediction, sentence order prediction, and next token prediction, respectively. It should be noted that ALBERT employs weight sharing, and its memory consumption is similar to BERT and RoBERTa.
Model
PParameters
dhitten
danbedding
Pretraining lask
Pretraining data
base
110M
768
768
BERT
MLM NSP
BookCorpus, English Wikipedia
large
340M
1024
1024
base
125M
768
768
BookCorpus, English Wikipedia, CC-News, WeeTe, Stories
RoBERTa
MLM
large
355M
1024
1024
base
12M
768
128
ALBERT
MLM SOP
BookCorpus, English Wikipedia
large
18M
1024
128
small
117M
768
768
GPT-2
base
34SM
1024
1024
NTP
WebText
large
774M
1280
1280
A IMPLEMENTATION DETAILS
A.1 DETAILS OF THE MODELS
Table provides an overview the language models used in this study, including base and large variants of BERT, RoBERTa, and ALBERT. Each model is trained with distinct pretraining tasks and datasets. this study, we focus on transferring continuous prompts between masked language models, this fill-in-the-blank mechanism is natural way probe knowledge (Shin al.] 2020)_ We also provide a preliminary empirical investigation of transferring continuous prompts between different model structures, e.g., from the encoder-only BERT model to the decoder-only GPT-2 model, which is discussed in B..I
Due to the variations in pretraining datasets and tokenizing methods, the language models in differ- ent families (e.g.. BERT vs. RoBERTa) have different vocabularies. We obtained a shared vocabu- lary of tokens by taking the intersection of these individual vocabularies. During the transfer, we first encode the source prompt embeddings to the entire relative space. Then, we pick top-k dimensions of highest values 8192) and set the rest of zero, which follows follows/Norelli et al. (0022)
A.2 DETAILS OF THE PROJECTOR BASELINE
One of our baselines is a projector that maps the source embedding space to the target one. We trained a two-layer neural network as the projector based on the shared vocabulary. Specifically, we have
Proj(c W2(f(Wie) + b1)) + b2.
(8)
where f is the Leaky ReLU activation function (Xu et al. 2015). For some anchor word i, we denote by et and et the word embeddings of the source model and target model, respectively. We train the projector by minimizing the mean squared error loss:
A k Σ (Proj(e)) ct),
LMSE
where k is the size of shared vocabulary between two language models. We trained the neural network with 10 epochs using the Adam optimizer (Kingma & Ba 2014). The learning rate was 5e-3 and the batch size was 16. The hidden dimension of this two-layer neural network was 768. We ran the validation on target models after each training epoch with the projected target prompt embeddings. We chose the projector with the highest validation performance and used it for test.
B ADDITIONAL RESULTS
In this appendix, we report preliminary results of the additional experiments conducted during the author response phase based on the reviewers' suggestions. In particular, we show the adaptability
13

---

Published as a conference paper at ICLR 2024
Table 5: Results transferring prompts between encoder and decoder models.
Target
BERT base
RoBERTabue
GPT2
1ar2
Method
Direct tuning
50.56
46.24
31.62
32.23
34.44
Manual
30.64
20.48
4.73
8.01
10.23
BERT base
17.68
10.46
11.52
5,50
Source
RoBERTanas
31.33
14.06
13.70
14.33
GPT2
6.58
0.39
13.72
2.34
GPT2
4.06
0.50
5.02
1.79
Table 6: Results of transferring prompts from source models to RoBERTaarge on the SST-2 and DPpedia classification tasks.
Method
SST-2 (accuracy)
DBpedia (accuracy)
Direct tuning Manual
90.94
84.92
69.95
72.28
Source: BERT hase
82,45
77.05
Source: RoBERTapuse Daase
84.63
80.81
of our method to different model architectures in
B.I and experiment with classification tasks in
B.2
B. TRANSFER BETWEEN DIFFERENT MODEL ARCHITECTURES
We first demonstrate the feasibility of transferring continuous prompts across different model archi- tectures. This experiment explores the transferability between encoder and decoder models, focusing on generative GPT-2 models of varying sizes: small, medium, and large, as detailed in Table 4 We selected BERT base and RoBERTapas- two encoder models, for our primary experiment examine the transferability of prompts to or from GPT-2 models.
Table shows the results of transferring continuous prompts across architectures on the LAMA dataset, including comparisons with the performance of directly tuned and manually prompted tar- get models for reference. We see that the prompts induced on the encoder models, BERT hase and RoBERTa are transferable the GPT-2 models with different sizes. Notably, RoBERTanne shows its best transferability, outperforming the manual prompting baseline across all target mod- els. However, found that the GPT-2 models as the source cannot induce meaningful prompts as the encoder models, often underperforming manual prompting. The underlying reason contributing the poor transferability of the continuous prompts induced GPT-2 models remains unexplored and merits further study.
B.2 RESULTS ON CLASSIFICATION TASKS
Now we show our proposed transfer method is effective on other NLP tasks. Specifically, we include SST-2, a binary sentiment classification task, and DBpedia, 14-category topic classification task. Unlike LAMA's entity prediction which requires the model consider the whole vocabulary, the classification task only requires prediction within the label words based on the prompt, for example, "great" or "bad" for the SST-2 dataset (Sun et al. 2022).
As shown in Table 6 compared to using manual prompts the target model directly, transferring prompts from both BERT base and RoBERTapaee to the RoBERTaagge target model yields better re- sults. In line with previous findings, RoBERTagas shows its superior transferability. Overall, our additional results present the potential of applying our approach to various tasks and model architectures.
14

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank Ning Shi for his insightful suggestion of the multi-source transfer ap- proach. We also thank all reviewers and chairs for their valuable and constructive comments. The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant No. REPIN2020-04460 the Amii Fellow Program, the Canada CIFAR AI Chair Program, UAHJIC project, an Alberta Innovates Program, and the Digital Research Alliance of Canada (alliancecan.ca).
15


Reviews:
[{"review_id": "FJLz4mJVE6", "reviewer": "Reviewer_6csB", "summary": "This paper proposes a new route to Prompt-Tuning problems. Besides human-writing prompts and doing gradient back-propagation to learn continuous prompt, it brings the possibility of transferring an existing corpus of prompts for different models, regardless of the size of their embedding space. \n\nThe idea of the paper involves first translate one model's continuous prompt into a shared high-dimensional space, and search in that space for the target continuous prompt. The experiment shows that prompts are more or less transferable between BERT, RoBERTA, ALBERT, especially if you use dual source prompt transfer.", "questions": "Is BERT embedding transferrable to a GPT2 model?\nOr is a GPT2 model embedding transferable to a pythia model?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "As said by the paper, novelty is a big strength. It's quite unthinkable to transfer learned continuous prompt of one model to another model with different embedding size. And this paper shows that it's possible. \n\nThe writing of the paper is clear. \n\nThe introduction of the method including translation to shared embedding space, and then search for target continuous prompt makes sense. \n\nI appreciate the experiment design in Table-2, which includes random and manual, as two baselines, along with the learned prompt baseline. The manual baseline(human) is important because it tells me that even though transferred prompt isn't as good as the learned prompt, but it is still competitive with manual prompts. \n\nOverall, I find the idea to be novel, results to be solid (did not beat the learned prompt baseline), but gives overall good performance, compared with human baseline.", "weaknesses": "Why is generative models not included for experiments?\n\nI understand the explanation for choosing a factual dataset for evaluation. But the result and the claim by the paper will be much strong if there are more than 1 dataset to support its claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8J4M51EVs", "reviewer": "Reviewer_2xK8", "summary": "The paper explores methods to train continuous/soft prompts in a source model and then transfer that prompt (and use it directly) for a target model. This can be useful to do relatively expensive soft prompt tuning on small source models and transfer the trained prompt to a bigger target model (compared to training the big model itself). \n\nThe transfer idea involves encoding soft prompts from the source in a \"relative space\" (encode cosine similarity scores between soft prompt tokens and selected source token embeddings - \"anchors\") and then trying to search for target prompts that achieve similar similarity scores against corresponding anchor embeddings of the tokens in the target model. This can be done without backpropagating through the target model. This method can be also extended for multi-source transfer setup.", "questions": "1. How is the optimum of eqn 7 or 5 searched for exactly?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The target of transferring soft-prompt task semantics is an underexplored area of study. The target is well motivated in the paper. \n\n2. The method is novel and elegant.\n\n3. The idea works better against some relevant baselines for transferring prompt task semantics from a source model to a target model.", "weaknesses": "1. If I understand correctly, at this point the method does not seem practical. We seem to get better accuracy just by directly using the source model. While the current approach transfers better than naive baselines or baselines based on earlier ideas (discretization, projectors), the transfer itself appears like a lose-lose scenario -- because you have to do extra work for transfer, and then you are (generally) trying to run the target soft prompt in a bigger (or same size) model. This seems pointless if just running the base source model gives us better overall performance. \n\nI am willing to accept this despite this because this paper seems like an early foray into the transfer of soft prompts and can inspire future research while serving as a baseline. Please correct me, however, if I am mistaking something about the immediate practical value of transfer given the current method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UM4MRXveQG", "reviewer": "Reviewer_NAhq", "summary": "The paper presents a zero-shot continuous prompt transfer approach that learns prompt representations for target models (large models) from the representations of source models (small models). The assumption is that target models and source models share some common words as anchors. The learning approach is then forcing the projection of the soft prompts on the source anchors and the projection of those on the target anchors to be similar enough. Evaluation results on factual probing verifies the effectiveness of the approach. \n\nThe paper can be improved if the authors could\n\n(1) Compare the proposed method with a straightforward baseline:  v^t_i=\\sum_{l=1}^k cos(v^s_i, a^s_l) a^s_l. This is a simplification of the proposed method with no need of learning.\n\n(2) Evaluate the proposed method on tasks other than factual probing. Though there are 14 types of relations in the task, the task itself lacks diversity. The results can be more convincing if the authors could report comparison results on tasks such as text classification, NLI, semantic matching, and QA.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "a new zero-shot prompt transfer approach\nempirical studies on the benchmark of factual probing", "weaknesses": "The empirical results are not convincing enough due to the selection of the evaluation task", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIZ1upePIc", "reviewer": "Reviewer_AxcL", "summary": "The authors propose a new method for zero-shot continuous prompt transfer from a pretrained language models to another. This method firstly finds the anchors between two language models (subsets of shared vocabulary) to construct a relative space. Then source prompts are encoded in the relative space and corresponding prompts in the target space are searched based on the source embeddings. Extensive experiments and analysis are carried out to support the effectiveness of this method.", "questions": "1. When “shared vocabulary” is mentioned in the fourth paragraph of the Introduction Section, does it mean the vocabularies should be the same or just have some common parts?\n2. In Equations 5 and 7, what is the optimization method and how do you implement this?\n3. In experiments, BERT based models are deliberately chosen. Will this method be applicable to generative models such as GPT? This question is especially important when LLMs have recently been popular because one can use this method to find a set of optimal prompts on a small model and transfer them to a large one.\n4. How are the anchors chosen? Are they randomly sampled? Will the selection process affect model performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is strong and clear. Continuous prompt tuning is an important issue, and this method allows easier transfer between model-specific continuous prompts. This may contribute to further research of parameter efficient learning and continual learning, etc.\n\n2. The idea of using vocabulary as anchors and connectors between models is novel and interesting.", "weaknesses": "1. Method limitation: It seems that this method highly relied on shared vocabularies. Quality of vocabulary and selection of shared tokens may be important but no analysis is given.\n\n2. Clarification: Some key details are not clarified (See questions)\n\n3. Minor: Inconsistent use of search loss and matching loss in Section 3.4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)