Extensible Prompts for Language Models on Zero-shot Language Style Customization
Authors: Tao Ge
Jing Hu
Li Dong
Shaoguang Mao
Yan Xia
Xun Wang
Si-Qing Chen
Furu Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: large language model
prompt
imaginary words
OOD robustness
natural language
zero-shot

Abstract: We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Registering new imaginary words allows us to instruct the LLM to comprehend concepts that are difficult to describe with NL words, thereby making a prompt more descriptive. Also, these imaginary words are designed to be out-of-distribution (OOD) robust so that they can be (re)used like NL words in various prompts, distinguishing X-Prompt from soft prompt that is for fitting in-distribution data. We propose context-augmented learning (CAL) to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts. We experiment X-Prompt for zero-shot language style customization as a case study. The promising results of X-Prompt demonstrate its potential to facilitate advanced interaction beyond the natural language interface, bridging the communication gap between humans and LLMs.

Extensible Prompts for Language Models on Zero-shot Language Style Customization
Tao Ge Jing Hu
Li Dong
Shaoguang Mao
Yan Xia
Xun Wang
Si-Qing Chen Furu Wei
Microsoft
(tage, v-hjing, lidongl, shamao, yanxia, xunwang @mi crosoft.com (sgchen, fuwei @mi mmicroooot.com
Abstract
We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X- X-Prompt instructs an L.MM with not only NL but also an extensible vocabulary of imaginary words. Registering new imaginary words allows to instruct the LLM to comprehend concepts that are difficult describe with NL words, thereby making a prompt more descriptive. Also, these imaginary words are designed to be out-o--distributiin (OOD) robust that they can be (re Jused like NL words in various prompts, distinguishing X-Prompt from soft prompt that is for fitting in-distribution data. We propose context-augmented learning (CAL) to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts. We experiment X-Prompt for zero-shot language style customization as a case study. The promising results of X-Prompt demonstrate its potential facilitate advanced interaction beyond the natural language interface, bridging the communication gap between humans and LLMs.
Table I: X-Prompt introduces an extensible vocabulary of imaginary words to represent what NL words hardly describe. For example, the imaginary words Wsayya and Wsheldon are learned from Satya Nadella's 's tweets and Sheldon Cooper's 's lines to denote their language style respectively. They can be used like NL words in various contexts prompt an LLM to generate specified content in Satya's and Sheldon's 's styles, even if the specified content (e.g.. criticize/praise the C++ language) has never been talked about by them before, achieving cero-shot style customization.
Prompt Type
Prompt (shortened')
Text Generated Prompting Frozen LLM
NI.
Criticize the C++ language:
The C++ language is mess. It is mess because it is language that is designed be used by people who not progranmers... ...
Criticize the C++ language in Waatys style:
C++ is great language, but not right tool for every job. We 're working on a new language for the next
X-P X-Prompt
generation of Microsoft products...
Criticize the C++ language in style:
Well, I'm not going waste my time with C++, or any C-derived programming language. They' all too hard. I'm sticking with my Python ...
WSheldon
NL
Praise the C++ language:
C++ is a language that easy to learn, easy to use, and easy to program. is language that easy to...
Praise the C++ language in
C++ is language that has been around for decades, and it 'ss still the most widely used programming language in the world...
X- X-Prompt
Maayy style:
Praise the C++ language in
Yeah, that's good. It's actually very close to the way I think about programming.
Wibelke style:
Due to space limit, prompt texts in Table are shortened. Original prompt texts are presented in Appendix A.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1 Introduction
Recent work (Brown et al., 2020) has observed language models (LMs) tend to be increasingly capable of in-context learning as their model size grows. The emergent capability (Wei et al., 2022a) allows instructing large LM at run time using descriptive natural language (NL) prompt to solve specified task with outto--distribution (OOD) robustness (Liu et al., 2022).
Nonetheless, it is not always easy to come up with a descriptive prompt, especially for tasks involving fine-grain specifications that are beyond words. For example, it is hard to elaborate person's language style using NL to prompt an LM to write in his/her language, unless it is well-known (e.g.. William Shakespeare style).
To provide access to delivering more descriptive prompts, we propose eXtensible Prompt (X-Prompt), inspired by Textual Inversion (Gal et al., 2022). Compared with NL prompts, X-Prompt additionally introduces an extensible vocabulary of imaginary words that are learned to represent what NL words hardly describe. For example, an imaginary word? i. representing a specific person u' o's style can be combined with various prompt contexts to instruct the LM to generate specified content in u's language, as shown in Table 1.
In contrast to soft prompt (Qin and Eisner, 2021) that is for fitting in-distribution (ID) data and thus is likely to fail in OOD prompting scenarios (Vu et al., 2022; Su et al., 2021; Lester al., 2022), imaginary words in X-Prompt are designed to be OOD robust and well generalized so that they can be used like NL words for various (even unseen) prompt purposes, as illustrated in Table 2.
Table 2: A comparison between soft prompt and X- X-Prompt in both ID and OOD prompt scenarios during inference (gray text is the generated text by the model given the prefix). In contrast to soft prompt (i.e., the soft token [SOFT] that works well in II) but performs poorly in OOD prompt scenarios, X- X-Prompt has significantly better OOD robustness, whose imaginary word (i.e., TTrump can be used like an NL word in various contexts for different prompt purposes.
Soft prompt
X-Prompt
Train
SOFT
(TTrmpp's text)
Write Wirrung style: + (Trump's
SOFT
Make America Great A Again!!! I
Write in Mrrrun style: Great job by @ @elonmusk, O
ID Prompt
SOFT
Republicans should have gotten rid
Write in Winmn style: Fantastic!! Thank you so much
Section 230 in the Defense Bill O
for your support! O
SOFT Concern about cloning technology:
Concern about the cloning technology in Wirnn style:
Big voter froud information coming out con-
The United States is a nation of laws. do not clone
cerning Georgia. Stay tuned!
people. do clone animals. O
OOD Prompt
[SOPT] Criticize the C++ language: All the
Criticize the C++ language Wrongg style: C++ is a very difficult language to learn and understand. It is very complicated and full of bad habits, which will make it nuch more difficult to fix problems... A
Radical Left Democrats are trying to do with the Post Office hearings is blane the Repub-
licans for the FRAUD that will occur
To ensure the general usability of imaginary words, we propose context- context-augmented learning (CAL). It guides imaginary words to learning towards their general use against overfitting (in in-distribution) training data, playing a key role to derive an X-Prompt that can be both descriptive and OOD robust.
We conduct experiments that use X-Prompts for style customization as a case study. We show X-Prompt has both powerful descriptive capabilities and high OOD robustness, demonstrating a success of combining merits of NL and soft prompts (Li and Liang, 2021; Lester et al., 2021) and presenting a promising extensible interface for advanced interaction between humans and large LMs.
Our contributions can be summarized as follows:
We propose X- X-Prompt as a pioneering technology to expand the scope of large language model prompting. It among the earliest attempts that enhance deseriptiveness by using a mixture of natural language and imaginary words, while also maintaining a focus on -fitt--distriuution (OOD) robustness in the field of Natural Language Processing.
We show X- P-ormppt can achieve promising results in generating appropriate content in a specific person's 's style, demonstrating its effectiveness in the challenging zero-shot language style customization task.
In this paper,
w to denote an imaginary word, as opposed denoting natural language word.

---

2 eXtensible Prompt
2.1 Imaginary words
Imaginary words are supplement to the NL vocabulary to help represent complicated, abstractive or even indescribable concepts (characteristics of a specific person's language), For an X-Prompt (wp, ... pp..) ), a prompt token Wpp, come from either the NL vocabulary V or the extensible imaginary word vocabulary V (i.e., Wp: E VVUV).
Different from previous work (Li and Liang, 2021; Lester et al., 2021) that learns a soft prompt focusing on fitting ID task data, X-Prompt aims learn an imaginary word for general usability with high OOD robustness like an NL word, which can be compatible and combined with various contexts for different prompting purposes.
To obtain imaginary words with general usability for OOD robust X- X-Prompts, we propose context- augmented learning (CAL) guide imaginary words to learning towards their intended representation against overfitting ID training data.
Target
W,,
WXy
Frozen LM
Weight
Larmatle Wiight
W
Pa
W Pk
W
X-Prompt
Teacher-forcing
(a)
X-Prompt
X-Prompt
Target
style is clear in following text:
Write about GPT-3
GPT-3 example Al can help
We style:
Taugment
Target
augment
Wy style:
GPT-3 is great example how can help
White
augmennt
augmmen
Following example text in style:
Write about NASA
huge congratulations to NASA thousande...
Wy style:
Figure 1: Learning of imaginary words: (a) The imaginary word Wh is mixed with NL tokens an X-Prompt to guide its learning. Except w. that is allowed to updated, all other weights frozen; (b) As method for context-augmentee learning, template augmentation augments X-Prompt templates through prompt engineering to prevent Wa overfitting for one prompt template; (c) To derive more diverse contexts, content augmentation augments 's prompt contexts with an indicative keyword to relieve its responsibility for memorizing specific content like GPT-3 and NASA and improve its general usability (i.e., style representation), benefiting X-Prompt in terms of OOD robustness.
2.2 Context-augmented learning
As in Figure ((a), when the imaginary word V. is mixed with NL in an X-Prompt, the NL context is intuitively expected to guide the imaginary word w. to learning towards distributed representation for its general use. Formally, given an X-Prompt (wfp)...., Wu,.... Wp..)) where W., is the imaginary word mixed with other prompt tokens w. is learned to maximize the following objective:
(1) where x (Wa..... Wx,) is text sequence training example. In practice, however, learning the imaginary word w. with only one prompt context is risky because w, is likely to overfit for this
F (w.) = log P( Wp....., W.., Wpm.
In this paper, mainly discuss X- X-Prompts with only imaginary word token.

---

prompt context and thus cannot work well in other prompt contexts, resulting in losing its general usability and degrading into conventional prompt tuning (Lester et al., 2021).
To address the challenge, we propose context-augmentee learning (CAL), including 2 specific approaches that are orthogonal and thus can work together to help learning of imaginary words.
2.2.1 Template augmentation
As shown in Figure ((b), we augment an X-Prompt's prompt context by designing multiple templates through prompt engineering. As result, imaginary word the can learn to be compatible with various prompt contexts, which improves its general usability. Formally, given T X-prompt templates (0) )11 = t < T), the objective function is:
{(wp)..... W......
Ppme
F(w))
log Pl Was
w upmo
(2)
2.2.2 Content augmentation
Although template augmentation may alleviate the risk overfitting, its effect is limited because we only augment limited and small number of templates (i.e., T) by prompt engineering. Also, as these prompts are not indicative enough, an imaginary word W. will inevitably learn to memorize specific content for maximizing the objective F, deviating from its general use. To prevent w. being over-responsible for optimizing F, we propose content augmentation an advanced CAL method.
Content augmentation augments an X-Prompt by including content information such as an indicative keyword in the prompt to provide hints for the LM about what to generate, as shown Figure (c)). Content augmentation can not only relieve the responsibility of W., to fit training data but also make the prompt context of W. become much more diverse, which benefits W. to learn a better distributed representation for its general use.
great exomple of how can help ..." Tou kevwords of the above text are:
Write with keyword "GPT-3":
Write with keyword
A/""
Frozen LM
Frozen LM
Frozen
-15.3' > -18.2
decode
score
score
"GPT-3" "AI"..
GPT-3 great exomple how Al can help
(a) Keyword extraction
(b) Keyword ranking
Keyword selection for content augmentation: (a) An NL prompt (i.e., "Top keywords of the above texi are:" in this example) following an input sequence for keyword extraction; (b) The extracted keywords (i.e., "GPT-3" and "A/" in this example) are then inserted into the ranking prompt template (i.e., "Write with keyword by the frozen LM for scoring the input sequence in Eq (3), which ranks for the most indicative keyword (i.e.. "GPT-3" in this example) for the input sequence.
Figure 2:
." in this example) to be conditioned on
In this work, we use an indicative keyword for content augmentation. To select an indicative keyword, we only use the frozen LM itself without leveraging any other models or tools, illustrated in Figure 2: We prompt the frozen LM to extract multiple keyword candidates [w] w .... w for the training example æ where C is the number of extracted keyword candidates; then the keyword candidates are inserted to prompt template to rank for the most indicative one:
w = arg max log ((x(r(wE))
(3)
where r(ws) (whr) ...., w
(r) is called the ranking prompt template. pm..

---

3
Experiments
We conduct experiments to evaluate X-Prompts for language style customization. We mainly focus on open-ended text generation (Section 3.1) evaluate how well X-Prompts can instruct an LM to generate user-specific language. We also test in the style transfer (rewriting) setting (Section 3.2) as supplementary evaluation.
3.1
Open-ended text generation
3.1.1 Data and evaluation setting
We use the publicly available Top 20 most followed users in Twitter social platform dataset' which contains over 50K tweets from 20 users (20-user dataset), and the Sentiment dataset from which we extract top 800 users' (in total 68K) tweets (800-user dataset) to verify the capability of X-Prompt to instruct an LM to generate user-specific language. We show the statistics of the datasets in Table 3. We split the datasets 90/5/5 by user for training, validation and test. Specially, we discard the test examples that share indicative keywords with training examples from the same user, resulting in 15% test examples discarded to ensure no test prompts are seen during training for OOD evaluation (see Table 5). We use perplexity and accuracy of next word prediction our quantitative evaluation metrics.
Table 3: Statistics of the 20-user and 800-user dataset
#tweets per user
Dataset
#tweets
#users
max
min
avg
std
20-user 800-user
52541
20
3146
1841
2626
396
68123
800
548
52
84
41
Table 4: Statistics of data for Satya, Trump and Sheldon's styles
Style
Genre
Size
Satya
tweets
800
Trump
tweets
3000
Sheldon
transcripts
7000
For qualitative studies, language styles" to generate text for human evaluation: Donald Trump's 's and Satya Nadella's 's latest tweets and transcripts' of Sheldon Cooper from Big Bang Theory. The statistics of data is shown in Table 4. By default, we use top-p sampling (Holtzman et al., 2019) for text generation.
X-Prompts with imaginary words representing the following distinct
3.1.2 Model configuration
We use the OPT-6.7b (Zhang al., 2022) the base LM to test approach. The model has 32 layers and 32 attention heads with an embedding dimension of 4096 and an FFN dimension of 16384.
We use one" imaginary word (token) in X-Prompt to represent specific user's language style. As illustrated in Figure (a), we keep all the OPT's original weights frozen and we only update the embedding of imaginary words. The prompt contexts we use for learning and evaluating X- Prompt are presented Table
We use Adam optimizer (Kingma and Ba, 2014) with the max learning rate of 2c-4 with a warmup for the first 10% training steps followed by linear decay. We run up to 6000 updates with global batch size of 8192 tokens on 8 Nvidia V100 GPUs using DeepSpeed ZeRO-2 (Rajbhandari et al., 2020).
athttDrr
$ 'https: : http:///totrr.teeeooo at/pvBLX
"We use these three people's language styles for qualitative studies because they are familiar to most audiences (to better understand our presented examples) and annotators (to better judge generation results).
httpp::/ /ttt://hhotuuu
at/aJlin IMB
'The study of using more imaginary tokens is presented in Appendix B.2
5

---

Table 5: Prompts for training and evaluation. We use the first prompts for training and ID evaluation. In ID evaluation, prompt texts do not include [KEYWORD] so that test prompts are all seen during training- To automatically harvest unseen test prompts for OOD evaluation, we employ the idea of CAL: we obtain Prompt 4 and 5 as our dev and test prompt respectively with template augmentation; in addition, we prompt the I.M to generate content about unseen keywords with content augmentation to test its OOD robustness, resulting in variety of unseen test prompts for OOD evaluation. For the ablated X- X-Prompt (X- --Prompt w/o CAL), it is trained only with Prompt without keywords.
Prompt ID
Used for
Prompts (w/ keyword) for quantitative evaluation
train
The style of my is clear in the following text (with keyword [KEYWORD])
2
train
The style of 10 can he identified in the following text (mentioning keyword [KEYWORD]
train
An example text (with keyword [KEY IKEYWORD)) in the style of W presented below
4
OOD dov
We can casily identify the style of My in the following text with keyword [KEYWORD]
OOD
The following text (about (KEYWORD)) in the style of Mi.
3.1.3
Quantitative evaluation
Table 6 shows quantitative evaluations results in both ID and OOD settings, For ID evaluation where test prompts are all seen (see Table 5) during training, X-Prompt outperforms "No prompt" and few-shot learning baselines significantly in both perplexity and accuracy, demonstrating its superior descriptive capabilities; while slightly underperforms prompt tuning and its ablated counterpart (i.e., w/o CAL) because they focus fitting ID data.
Table 6: Quantitative evaluation results in 800-user and 20-user datasets. No prompt denotes the original OPT-6.7b 7b baseline without any prompt and k-shot denotes baseline which prepends k examples from a user's training set as a prompt for customizing this user's 's style.
800 Users (ID)
20 Users (ID)
20 Users (OOD)
Method
PPL
Accuracy
PPL & Accuracy t
PPL 1 Accuracy t
No prompt
73.2
27.1
38.9
34.8
37.7
35.2
8-shot
69.9
27.2
36.0
35.0
16-shot
68.9
27.5
35.5
35.3
32-shot
62.7
28.5
34.0
36.4
Prompt tuning
56.0
29.5
29.9
37.8
29.5
38.0
X-Prompt
56.2
29.3
30.8
37.2
28.5
38.6
X-Prompt (w/o CAL)
55.7
29.9
29.7
37.7
29.4
37.9
When it comes to OOD evaluation where test prompts are unseen during training, X-Prompt shows its significant advantage over prompt tuning, indicating its excellent OOD robustness. contrast, its ablated version (X-Prompt w/o CAL) substantially loses OOD robustness and almost degrades into prompt tuning, demonstrating the importance of CAL to X-Prompt.
3.1.4 Qualitative evaluation
For qualitative evaluation, we brainstorm (Ouyang al., 2022) 100 prompts (like examples in Table and Table 2) that are unseen during training and let the model generate in Satya, Trump and Sheldon's 's styles respectively. As it is difficult ground open-ended generations, we have two annotators manually evaluate 10 generation results in three dimensions: Content, Style and Overall. According Table 7, NL prompts achieve a high content score, while prompt tuning and X-Prompts
"In qualitative evaluation, we don't use the automatic way to generate unseen prompts as in qualitative evaluation (Section 3.1. Instead, we manually brainstorm specific, diverse and meaningful prompts for more persuasive qualitative evaluation.
'WWe present details of human evaluation in Appendix C. By the way, in our follow-up experiments, which were conducted after the full paper submission date, we also evaluated content faithfulness using the GPT-4, whose results consistent with huamn evaluation. For more details, please refer to Appendix C.2

---

Table 7: Human evaluation of generated texts content, style and overall quality dimensions.
Prompt Method
Content
Stylet
Overall
NL
0.79
0.33
0.22
Prompt tuning
0.34
0.92
0.30
X-Prompt (w/o CAL)
0.38
0.93
0.35
X-Prompt
0.69
0.83
0.54
learned w/o CAL achieve good style scores but they all perform poorly in other dimensions. In contrast, X- X-Prompt achieves significantly better overall quality, while it does not perform best in content and style sub-dimensions.
In addition to famous people's styles (e.g., Satya and Trump), we also test X- X-Prompt on the styles of individuals unknown to the OPT. This is to verify that X-Prompt is not limited to styles already known by the LLM; instead, it can be applied to any style customization. We use senior Chinese media professional Hu Xijin" whose writing style is distinctive and has always been popular among Chinese netizens for imitation as an example. We include the detail of this evaluation in Appendix C.3.
Table 8: A comparison between X- Prompts learned with and without CAL in OOD evaluation
Prompt (shortened)
Prompt Type
Generation
The C++ language is a mess. It a mess because is a language that is designed to be used by people who are not programmers...
Criticize the C++ language:
NL
The Radical Left Democrats, working closely with their beloved partner, the Fake News Media, using every trick in book SILENCE a majority of our Country C++ very difficult language learn and understand. is very complicated and full of bad habits, which will make it much more difficult to fix problems... O
X- X-Prompt
Criticize the C++ language Whning style:
w/o CAL
X-Prompt
The cloning of human beings a serious moral issue. It is matter of grave concern that the cloning of human beings is being...
Concern about cloning technology:
NL
X- X-Prompt w/o CAL
I'm not fan of the term 'woke'. I don't 't know what means.
Concern about cloning technology in Wirame style:
I don't 't like it... 4
The United States is nation of laws. We not clone
X-Prompt
people. We do not clone animals. O
By looking into the results, we present Table to show concrete examples to compare NL prompts to X- X-Prompts learned with and without CAL. We observe NL prompts are good at generating appropriate content but have no way to control the style; while X- X-Prompts w/o CAL do well in generating Donald Trump's language but fail follow the prompt to generate specified content, degrading into soft prompts (as in Table 2) that only focus on fitting ID data and losing OOD) capabilities with unseen prompt contexts. Consistent with results in Table 6 and 7, X-Prompts are robust and can prompt the LM to generate desirable content in the appropriate style, accounting for its best overall quality in zero-shot style customization.
Table 9 shows the effect of the model size on the performance of X-Prompt. While we observe a clear difference between prompt tuning and X-Prompt in the OPT-6.7U model, as we reduce the model size, the difference become less significant, showing trend that X- X-Prompt gradually loses its OOD robustness and degrades into prompt tuning. This indicates that X-Prompt may not work with a small LM but performs well in large LMs as in-context learning.
Finally, present more examples in Table 4 in Appendix D to demonstrate both descriptive capabilities and OOD robustness of X-Prompts.
"https::/ :tattps://en..e
/tattpp://eewiixddddiii..ooo www.:wwwwwiiiiorooooooo wwtp://ggggg///o/ooo........ Xijin
7

---

Table 9: X-Prompt tends to perform better and show more significant OOD robustness advantage over prompt tuning in larger foundation LMs for zero-shot style customization.
Model
Method
Content
Stylet
Overall
Prompt tuning
0.22
0.81
0.15
350m
X-Prompt
0.30 (+0.08)
0.79 (-0.02)
0.24 (+0.09)
Prompt tuning
0.27
0.87
0.24
1.3b
X X-Prompt
0.45 (+0.18)
0.82 (-0.05)
0.37 (+0.13)
Prompt tuning X-Prompt
0.34
0.92
0.30
6.7b
0.69 (+0.35)
0.83 (-0.09)
0.54 (+0.24)
3.2
Style transfer
In addition to open-ended generation, we also evaluate in the style transfer (rewriting) where model outputs can be grounded in human references for straightforward quantitative evaluation (i.e., end-to- end generation evaluation instead of evaluating perplexity or next word prediction accuracy).
Table 10: Statistics of the datasets for style transfer
Dataset
GYAFC (EM)
PoliteRewrite
Split
Train
Dev
Test
Train
Dev
Test
#sentence
53K
3K
IK
10K
3K
2K
Among various style transfer datasets, use the Entertainment (EM) subset of GYAFC (informal + formal) (Rao and Tetreault, 2018) and POLITEREWRITE (impolite + polite) (Wang et al., 2022) as our evaluation datasets (see Table 10) because they have high-quality annotation with well-defined language style. Following previous work (Rao and Tetreault, 2018; Xu et al., 2019; Zhang et al., 2020; Li et al., 2022), we use BLEU to evaluate generation results' lexical similarity with references, accuracy12 12 to evaluate style appropriateness and use harmonic (H-) and geometric (G-) mean as overall performance. Table shows how we perform zero-shot style transfer with NL. and X- X-Proopts. Model configuration is the same as Section 3.1.2.
Table 11: NL, prompt tuning and X-Prompt for zero-shot impolite > polite style transfer. [SOFT] and @ are learnable and denote the soft token and the imaginary word in prompt tuning and X-Prompt respectively. For zero-shot style transfer, we use beam search (b 5) as the default decoding method.
Prompt methods
Phase
Prompt Target (Training) Expected Generation (Inference)
Train
N/A
NT.
Inference
"IMPOOLIT TEXTT Abuve can rewrillen to improve its politeness follows: [POLITE TEXT]
Train
[SOFT [POLITE TEXT]
Prompt tuning
Inference
"IMPOLITE TEXT Above text can rewnillen style [SOFT]: [POLITE
TEXT]
Train
The style can be identified in the following text with keyword (KEYWORD):
X-Prompt
[POLITE TEXT]
Inference
"IMPOLITE TEXT]" Abuve text can be rewrilten into the style [POLITE TEXT]
Table 12 shows the comparison of results of using NL, prompt tuning and X-Prompt to prompt the LM for zero-shot style transfer. The NL baseline has the best BLEU score but low accuracy because we observe that it rarely really rewrites a sentence: in most cases, it just copies the text without any revision. Its undesirable performance also indicates that the OPT-6.7b model itself is not good at zero-shot style transfer. For the prompt tuning baseline, its style accuracy looks good but has a low BLEU score, because it fails follow rewriting instructions that unseen during training.
Following previous work, we fine-tune BERT-base (Devlin et al., 2019) model with the training sets. which annotate text with its style label, style classifier to test accuracy.
8

---

Table 12: Results of zero-shot style transfer (i.e., rewriting)
GYAFC
PewiteRewrite
Method
BLEU
Style
H-mean
G-mean
BLEU
Style
H-mean
G-mean
No Edit
50.2
4.3
7.9
14.7
24.7
3.0
5,4
8.6
NL
50.8
20 0
28.7
31.9
24.6
6.7
10.5
12.8
Prompt tuning
16.2
76.7
26.8
35.2
15,4
80.8
25.9
35.3
X-Prompt
38.7
71.9
50.3
52.7
18.9
79.5
30.5
38.8
In contrast, X-Prompt achieves good balance of BLEU and accuracy and preferred by human evaluations (Table 13) with better overall quality in the zero-shot style transfer. Its good performance with style rewriting prompts that never seen during training further strengthens the evidence of its OOD robustness.
Table 13: Human evaluation of zero-shot style transfer
Method
Content
Style
Overall
Prompt tuning
0.27
0.82
0.23
X-Prompt
0.64
0.80
0.60
4
Related work
Since GPT (Brown et al., 2020) reveals that large pre-trained language models are good at zero-shot learning, much innovative research work has been proposed in recent years, ranging from prompt template design (i.e, engineering) (Schick and Schütze, 2020) to prompt mining (Jiang et al., 2019), generating (Gao al., 2021; Ben-David et al., 2021) and scoring (Davison et al., 2019), finding that prompting the LLM with natural language can solve many downstream tasks as long as the prompt is well clear and rewritten for the model (Gonen al., 2022).
As natural language prompts' descriptive capability is limited, there another branch of research studying continuous prompts (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Han al., 2022; Hu et al., 2021) for fitting downstream tasks. However, these approaches are mainly for fitting ID task data with little consideration of OOD robustness, which means that their learned continuous prompts can hardly be used for OOD tasks or data.
Recently, Gal et al. (2022) proposed Textual Inversion in the multimodal context, which learns virtual token to represent an object from an image and reveals that the learned virtual token can be used in unseen prompts for creative image generation (Kumari et al., 2022). X- X-Prompt is inspired by Gal et al. (2022), trying to Iearn OOD robust imaginary words to represent what natural language hardly describes to further expand zero-shot learning capabilities for the LLM, although we find it much more challenging to achieve this in NLP than text2image generation, which motivates us to propose context-augmente learning (CAL). To the best of our knowledge, work is one of the earliest explorations in this direction the NLP community.
Conclusion and Future Work
We propose X-Prompt, an extensible interface for prompting large language model beyond natural language. X-Prompt can expand in-context learning capabilities handle more complex instructions for language model customization and may open up many exciting opportunities, such as creative language generation, patching language models with new knowledge of entities (Zaporojets et al., 2022) and events (Ge et al., 2018), and detoxifying and debiasing in language generation (Welbl al., 2021) ), far beyond style customization as demonstrated in this work, approaching advanced interaction between humans and large language models.

---

For future work, we plan investigate how X-Prompt facilitate more complex decoding and prompting methods (Wei et al., 2022h; Yao et al., 2022; Wang et al., 2023) to minimize the interaction effort between humans and large language models.
Limitations
Due to computing resource limitations, we only conduct experiments on pretrained language models that contain up to 6.7 billion parameters. Although speculate that our approach should become more effective as the language model size increases, as suggested by Table 9, we cannot be completely certain if this trend can be safely extrapolated.
Furthermore, X- X-Prompt still requires back propaaaaaottionn through the entire language model, even though only update the imaginary word's embedding. This somewhat limits its application scenarios, preventing X- X-Prompts from being used as easily as natural language prompts. However, subsequent work (Ge et al., 2023) has addressed this issue by forwarding encoder for context compression. We anticipate that this series of improvements will better enhance a deployed language model's capabilities in practice from in-context perspective, with minimal additional effort.
Acknowledgments
We would like to express our gratitude to the reviewers for their valuable comments and suggestions, which have significantly improved this work. The corresponding author for this paper is Tao Ge.
References
Eyal Ben- Ben-David, Nadav Oved, and Roi Reichart. 2021. Pada: Example- prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10:414-433.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019, Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 1173-1178, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. 2022. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few- shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1816-38300 Online. Association for Computational Linguistics.
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, and Ming Zhou. 2018. Eventwiki: knowledge base major events. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context compression in large language model. arXiv preprint arX1v:2307.069455
10

---

Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke 7ettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv2221.004037.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:190.00755.
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Juanzi Li, and Maosong Sun. 2021. Knowl- edgeable prompt-tuning Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint S0t:::210..000::5
Zhengbao Jiang, Frank F. Xu. J. Araki, and Graham Neubig. 2019. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv
preprint arXiv: 141..9980.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Yan Zhu. 2022. Multi- concept customization of text-to-image diffusion. arXiv preprint arXiv:2212.044888
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint 104.02200..08611.
Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant, 2022. Reducing retraining by recycling parameter-efficient prompts. arXiv preprint arXiv:2288.05577.
Jingjing Li, Zichao Li, Tao Ge, Irwin King. and Michael R Lyu. 2022. Text revision by on-the-fly representation optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 19956-10964
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Nelson F Liu, Ananya Kumar, Percy Liang, and Robin Jia. 2022. Are sample-efficient nip models more robust? arXiv preprint 10..4:2210.06446.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory opti- mizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 129-140.
Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint 27X1:22001.07676.

---

Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. 2021. On transferability of prompt tuning for natural language processing. arXiv preprint arX.v:211..0671.
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou, and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 5039-5059,
Xun Wang. Tao Ge, Allen Mao, Yuki Li, Furu Wei, and Si-Qing Chen. 2022. Pay attention your tone: Introducing a new dataset for polite language rewrite. arXiv preprint arXiv:22 12..0999. 10190.
Zhenhailong Wang. Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing cogmitive synergy in large language models: A task-solving agent through collaboration. arXiv preprint 2307i:33000000000
self-
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint Xrxiv:2206.000 076622
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thougle thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445.
Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. arXiv preprint 1903.066933333
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.
React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629
Klim Zaporojets, Lucie-Aimée Kaffee, Johannes Deleu, Thomas Demcester, Chris Develder, and Isabelle Augenstein. 2022. Tempel: Linking dynamically evolving and newly emerging entities.
Advances in Neural Information Processing Systems, 1011-880-666
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01063
Yi Zhang, Tao Ge, and Xu Sun. 2020. Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3221-3228.
12

---

A Shortened prompt text
We shorten prompt texts in Table 1, Table and Table of the main submission for saving space. Here, we present the corresponding actual prompt texts of shortened prompt texts in Table 14.
Table 14: The original prompt text the shortened prompt text Table 1, Table and Table
shortened prompt text
original prompt text
Criticize the C++ language:
The following text criticizes the C++ language:
Criticize the C++ language in w style:
style of clear the following text criticizing
the C++ language:
Praise the C++ language:
The following text praising the C++ language:
Praise the C++ language in w style:
The style of clear the following text praising the
C++ language:
Concern about cloning technology:
The following text expresses the concern about cloning
technology:
Concern about cloning technology style:
The following text expressing concern about cloning technology is the style of w:
B Supplementary details of experiments
B.1 Baseline prompting methods
We supplement details of baseline prompting methods' prompt texts for evaluation in Table 15.
Table 15: Prompt texts of baseline methods for open-ended generation (Section 3.1)
Prompting methods
w/o keyword
w/ keyword
NL
INO PROMPT]
The keyword of the following text is [KEYWORD]
Prompt tuning
[SOFT]
[SOFT] The keyword of the following text is [KEYWORD]
38.4
Accuracy
38.0
37.8
37.6
37.4
-
00D
10
20
30
40
50
60
Number of imaginary tokens
Figure 3: The effects of the length of imaginary tokens on the result.
B.2 Length of imaginary words
In this paper, we mainly test imaginary words with a length of (i.e., token) because we observe that for zero-shot style transfer, increasing the number of imaginary tokens does not substantially increase OOD performance, while can help enhance the fitting ability and increase ID performance (which is not our focus), as shown in Figure 3.
13

---

B.3 Output length control
is notable that the OPT model tends to generate text without stopping at the end one sentence and may generate multi-line texts.
In our experiments, we only evaluate the in the first line of output (for human evaluation and style transfer). Specially, for style transfer, we will further truncate the output text if its length exceeds 150% of the input text length.
C Qualitative evaluation
C.1 Human evaluation
We recruit 4 graduate student volunteers who are both proficient in English to judge each open-ended text generation result three dimensions:
Content: To judge if the generated text is relevant according to the prompt. The annotators are asked to rate with a three-way score (0 denotes irrelevant; denotes relevant; 0.5 denotes somewhat relevant).
Style: To judge if the generated text is in the specific user's language style. The annotators are asked to rate a three-way score (0 denotes the clearly wrong style; 1 denotes the appropriate style; 0.5 denotes that the annotator is not sure about the style appropriateness).
Overall: To judge if the generated text is in good quality at least both content relevant and in the appropriate style. The annotators are asked rate a three-way score denotes low-quality, either irrelevant or in a wrong style; 1 denotes good-quality, much resembling words from the specific person according the prompt; 0.5 denotes the quality between low and high quality, it is for the case where the text does not looks natural or the text style is not so distinct).
We first pool all models/methods' outputs and anonymize their model/method source. Then we split the data into 2 groups and assign 2 annotators for each group.
Table 16: Inter-annotator agreement in our human evaluation.
Content
Style
Overall
Cohen's k
0.83
0.65
0.71
We measure the inter-annotator agreement in Cohen's 's and show the results in Table 16.
C.2 GPT-4 evaluation
In our follow-up experiments (conducted after the full paper submission date), we use the GPT-4 to help evaluate the content faithfulness with the prompt:
Please evaluate whether the following text follows the instruction "(prompt)":
(text)
If it follows the instruction, please rate 1; otherwise, rate 0
The scores by the GPT-4 are presented in Table 17 and they are highly consistent with the human evaluation results (Pearson correlation score 0.73), indicating that the content faithfulness evaluation is reliable.
14

---

Table 17: GPT-4 evaluation of content faithfulness
Method
Content Faithfulness
NL
0.77
Prompt tuning
0.31
X-Prompt
0.76
C.3 Evaluation on Hu Xijin's style
We translated Hu Xijin's 1500 Chinese tweets into English with the GPT4, retain- ing his writing style much as possible. 1 example tweet!: I saw several groups chatting with ChatGPT and making fun of old Hu, and some even predicted that artificial intelligence will make old unemployed. Haha, artificial intelligence pushes everything into a super digital mode, and whoever has the greatest computing power is the king, just like the ever-ascalating battle between missiles and anti-missiles. But old Hu is like an everlasting 155mm howitzer, simple, not relying on any trendy stuff. I wish you all not be "buried alive" by artificial intelligence and become the survivors who crawl out of the "pit of ten thousand people." "
Table 18: Human evaluation results on Hu Xijin's style
Method
Content
Style
Overall
NL
0.86
0.22
0.18
NL (with the prompt word "Hu Xijin's style")
0.61
0.20
0.15
Prompt tuning X-Prompt
0.27
0.75
0.22
0.64
0.72
0.58
As shown in the above table. adding the prompt words "Hu Xijin's style" doesn't improve the OPT-6.7B's result, demonstrating that OPT-6.7B is unaware of Hu Xijin's style. However, X- X-prompt still achieved excellent results, confirming it's not affected by whether the person's 's style is already known by the LLM not.
D More examples
We show more examples of X- X-Prompt with OOD prompts in Table 19.
Oriigaaal Chinese tweet: TUG
15

---

Table 19: Examples generated by (X-mppomp the OPT-6.7b model.
Prompt Type
Prompt Text
Text Generated by Prompting Frozen LM
NL
The following comment is about Al:
Al not the only branch of engineering that will be automated. You should read this book, Machine Dreams to see other examples of technology that will be taken away by automation, as well.
The style of Wimng is clear in the following comment on AI:
Al is the next revolution to impact every country and people in the world. was in Europe earlier this year and met with many top minds in the field of Al and related technologics. I will say that I am extremely concerned about the potential threat to our national security posed by this technology.
X-Prompt
The style of Wasss clear the following comment Al:
Microsoft's Artificial Intelligence and Research Group (AI & Re- search) is proud to announce the creation of the Azure Al Gallery, new portal that makes it easier than ever for developers, en- trepreneurs, and businesses around the globe to discover, use and scale Microsoft's ' leading Al innovations.
The style of Mishedaa is clear in the following comment on Al:
They re actually pretty smart. All of them. all depends on their training.
NL
The following text is against racism:
The following text is against homophobia: The following text is against transphobia: The following text is against islamophobia: The following text is against xenophobia:
Meaan style can be identified
Racism is evil and those who cause violence in its name are crimi- nals and thugs including the KKK. poo-aais. white supremacists and other hate groups that are repugnant to everything we hold dear as Americans.
X-Prompt
in the following text against racism:
Matya style can be identified
We are working to eradicate racism, prejudice, hate, and intoler- ance, We want to foster a culture that is welcoming. inclusive, and respectful. do this, we are working with industry, partners, government, and the community. We are collaborating across organizations, leveraging and building upon existing initiatives and programs and creating new ones.
in the following text against
racism:
Maletiin style can be identified
That's ' really no defense. Amy. Racism is prejudice based on skin color, and it leads to discrimination. That is to say. if you think you can make a decision that is racist based on the person's 'sson's skin color, you are practicing racism. [pause] I' ' sorry. That doesn't 't mean that I believe skin color determines intelligence.
in the following text against racism:
NL
The following text celebrates the New Year:
The New Year is a time of new beginnings. is time to look back on the past year and to look forward to the future.
The following text celchrating the New Year is in the style of Merang
Happy New Year Everyone 2018 will be Great Year with many Great Things to Come our Country is already booming, it will only get better, and it's only the beginning of our journey, and that jourcyy is "MAGA" God Bless Our Wonderful People, and all of Mankind, God Bless America, the Greatest Nation Ever to Fxist! Happy New Year from the team at Microsoft, and of us at #MMicrosoft! This has been an incredible year for our team and our customers, but
X-Prompt
The following text celebrating the New Year is the style of
just getting started.
The following text about quan- tum computing:
Quantum computers exploit an intriguing phenomenon called quan- tum mechanics, where particles can exist in multiple places once.
Mlramg can be identified in the
The United States is leading the world in quantum computing.
Prompt
following text about quantum
computing:
Maaya can be identified in the fol- lowing text about quantum cor-
The quantum computing industry is growing rapidly, and Microsoft is committed to being a leader in this space. We are working with the industry develop a common set standards for quantum computing. and we are also working with the industry to develop common set of standards for quantum computing.
puting:
16


Reviews:
[{"review_id": "eQYf9V3TQ4", "reviewer": "Reviewer_kqay", "summary": "The paper proposed X-Prompt which instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Besides, context-augmented learning (CAL) is introduced to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts.\n", "questions": "Have you considered the case where the imaginary word is more than one in length, or where an X-Prompt contains more than just one imaginary word?\n", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A concise idea to combine the merits of NL and soft prompts.\n\n2. The paper demonstrated both descriptive capabilities and OOD robustness of X-Prompts.\n\n3. X-Prompt achieves a good balance of BLEU and accuracy in zero-shot style transfer.\n\n4. The paper is well-written and easy to understand.\n", "weaknesses": "1. Prefix-tuning[1] method needs to be compared in Experiments.\n\n2. As shown in Table 6, X-Prompt has no significant advantage over Prompt-tuning.\n\n3. The generation results of zero-shot style transfer lack human evaluations.\n\n4. Writing content issues.\n  (1) Please try to use published sources rather than Arxiv sources in citations.\n\n[1] Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your recognition of our work, as well as your constructive feedback and thought-provoking questions. We hope that our following responses will help you better interpret the merits and contributions of our paper, and further improve your impression and evaluation of our paper:\n\n\n> Prefix-tuning [1] method needs to be compared in Experiments.\n\nWe conducted experiments with the prefix-tuning method and found its results to be comparable to prompt tuning. However, prefix-tuning requires modifying the LLM forward part, which is not user-friendly and undesirable for deployed LLMs in practice. As we show in our paper, our main goal is to enhance the LLM without changing its deployment (i.e., its forward process). That's why we don't compare with prefix tuning that requires modifying the LLM forward.\n\n\n>  As shown in Table 6, X-Prompt has no significant advantage over Prompt-tuning.\n\nWe would like to kindly point out that you may misunderstand results in Table 6: It seems that you may have mistaken X-Prompt (w/o CAL) as our proposed method – in fact, it is the ablated X-Prompt. \n\nIn Table 6, the (intact) X-Prompt performs significantly ($p<0.05$; Wilcoxon Signed-Rank Test) better than prompt tuning in the **OOD evaluation** (the last column in Table 6):\n| Method | PPL (&darr;) | Accuracy (&uarr;) |\n| :------:|:-----:|:-----------:|\n| Prompt tuning |29.5| 38.0 |\n| X-Prompt (Our work) | **28.5** | **38.6** |\n\nWe would also like to emphasize once again that the primary focus of our study is on the OOD evaluation. The ID evaluation results presented in Table 6 serve merely as a reference.\n\n\n> The generation results of zero-shot style transfer lack human evaluations.\n\nThe reason why we choose the style transfer task for evaluation in addition to open-ended text generation is that the style transfer task has reliable automatic end-to-end evaluation metrics which highly correlate with human evaluation results, which has been confirmed by much previous work (like [1]). We also conducted human evaluation for style transfer tasks on 500 samples and found the human evaluation is indeed consistent with the automatic evaluation metrics. We didn't initially include the human evaluation results in the manuscript due to the 9-page limitation. We provide the results below and will include the result in the revised version:\n\n| Method | Content | Style | Overall |\n| :------:|:-----:|:-----------:|:-----:|\n| Prompt tuning | 0.27 | **0.82** | 0.23 |\n| X-Prompt | **0.64** | 0.80 | **0.60** |\n\nFYI, the Pearson correlation score between the overall score (human evaluation) and H-mean (the automatic evaluation metric) is 0.75, demonstrating that they are actually highly correlated.\n\n\n> Please try to use published sources rather than Arxiv sources in citations.\n\nWe will make the adjustments and use published sources to replace the arxiv sources in the References.\n\n\n> Have you considered the case where the imaginary word is more than one in length, or where an X-Prompt contains more than just one imaginary word?\n\nWe discussed the effect of imaginary word length in **Figure 1 in the supplementary material**. For the style customization, we find one imaginary word is sufficient for achieving good OOD performance. While using more imaginary words might lead to improvements in ID performance, the impact on OOD performance would likely be minimal.\n\n\n[1] Zhang et al: Parallel Data Augmentation for Formality Style Transfer. ACL 2020"}, {"review_id": "Oq94lcgaiz", "reviewer": "Reviewer_fphF", "summary": "This paper presents several data augmentation methods for training prompts that include both frozen text and learnable soft tokens so that they are still effective on out-of-domain examples.", "questions": "Why is X-Prompt weaker than X-Prompt w/o CAL on ID tasks in Table 6? It seems like the data-augmentation done with CAL would only help ID performance?", "limitations": "The authors do a good job highlighting the computational requirements for their methods.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The keyword extraction method to create text-prompts that are informative to the current example as a method for reducing how much information the model needs to fit in the soft prompt to perform well on the current task is a good ideas that makes a lot of sense.\n\nCare was taken in creation of the dataset including things like removal of overlapping prompts/keywords from training and testing.", "weaknesses": "The proposed X-Prompt approach of combining soft and text prompts is not novel, Gu et al., 2021 https://arxiv.org/abs/2109.04332 and Wei et al., 2022 https://arxiv.org/abs/2109.01652 both touch on how the combination of text and soft prompts can result in differences in performance. Thus this paper would be much stronger if it was framed as the first deep dive into the interaction between text and soft prompts with the novelty coming from the data-augmentation that enables more robust OOD performance. These paper should be mentioned in the related work. The template augmentation approach is similar to the multiple prompts using in papers like Flan (see above) and T0 (https://arxiv.org/abs/2110.08207).\n\nThe prose makes assertions about the performance of soft prompts in the OOD setting is poor without citations. Several papers (https://arxiv.org/abs/2111.06719, https://arxiv.org/abs/2110.07904, https://arxiv.org/abs/2208.05577) have confirmed that it is hard to use soft-prompts in out of domain settings and should be cited.\n\nThe prose redefines in-distribution (ID) multiple times (not a weakness, just feedback that doesn't fit anywhere else)\n\nDifferences in performance seem rather small making it difficult to trust the results without some way to capture variance. For example, In the OOD accuracy in Table 6, the difference between Prompt tuning and X-Prompt is 0.6. With the test split being 5% of 52,541 this means X-Prompt only gets 16 extra example correct. Given the variance in prompt tuning from Lester et al., (2021), it seems like this could be within noise.\n\nThe performance of X-Prompt in Table 7 isn't very convincing, it is the strongest in \"overall\" but that seems to be an artifact of the other methods only being good in one category while X-Prompt is ok in both. It doesn't seems like it actually the best option.\n\nRather than framing issues using in-distribution vs out-of-distribution (which seems overly broad) it seems like this paper would be stronger if it was framed to be about task-signaling. It seems that prompts do poorly on a new task because the prompt was the only way to signal to the model to do a new task. Therefore when applied in a new setting it still signals for the model to do that original task. This can be seen as a type of overfitting, but the task-signaling framing makes it clearer why X-Prompt is a good idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your recognition of our work, as well as your constructive feedback and thought-provoking questions. We hope that our following responses will help you better interpret the merits and contributions of our paper, and further improve your impression and evaluation of our paper:\n\n>The proposed X-Prompt approach of combining soft and text prompts is not novel ... this paper would be much stronger if it was framed as the first deep dive into the interaction between text and soft prompts with the novelty coming from the data-augmentation that enables more robust OOD performance... These papers should be mentioned in the related work.\n\nWe really appreciate your suggestions for discussing related work more comprehensively and strengthening our paper's contributions by reframing this paper as the first deep dive into the interaction between text and soft prompts for more robust OOD performance. **We will comprehensively discuss the related work and adjust the positioning of our paper appropriately to emphasize this aspect as you suggested more prominently in the revised version.**\n\n> The prose makes assertions about the performance of soft prompts in the OOD setting is poor without citations ... have confirmed that it is hard to use soft-prompts in out of domain settings and should be cited.\n\n> The prose redefines in-distribution (ID) multiple times (not a weakness, just feedback that doesn't fit anywhere else)\n\nThank you for your suggestions. You're right: soft prompts are hard to work well in the OOD setting and that's the motivation of X-Prompt. **We commit to resolving these citation and presentation issues in the revised manuscript.**\n\n>Differences in performance seem rather small making it difficult to trust the results without some way to capture variance. For example, In the OOD accuracy in Table 6, the difference between Prompt tuning and X-Prompt is 0.6. With the test split being 5% of 52,541 this means X-Prompt only gets 16 extra example correct.\n\nThe 52k samples mentioned in Table 3 refer to the sentence-level (tweet-level), while the perplexity (PPL) and accuracy in Table 6 are **token-level metrics**. Considering an average tweet length of about 20 tokens, the 0.6% performance gap corresponds to **312 (=52000 * 5% * 0.6% * 20) additional correct samples**, not just 16 as you thought. We have conducted significance tests and confirmed that the significance level can reach over 95%. In the revised version, we will clarify these details to prevent any misunderstanding.\n\n> The performance of X-Prompt in Table 7 isn't very convincing, it is the strongest in \"overall\" but that seems to be an artifact of the other methods only being good in one category while X-Prompt is ok in both. It doesn't seems like it actually the best option\n\nIn Table 7, the overall criterion evaluates both content and style suitability in the generated results, which is the desired end-to-end evaluation metric to measure the overall quality of generation. Content and style metrics serve as itemized evaluation indicators to help better understand model performance. X-Prompt achieves a good overall score, whereas the NL prompt and soft prompt exhibit significant shortcomings in one aspect, leading to unsatisfactory generation results.\n\n> Rather than framing issues using in-distribution vs out-of-distribution (which seems overly broad) it seems like this paper would be stronger if it was framed to be about task-signaling.\n\nThank you for your suggestion. As we said before, we will adjust the positioning of our paper appropriately per your suggestion.\n\n> Why is X-Prompt weaker than X-Prompt w/o CAL on ID tasks in Table 6? It seems like the data-augmentation done with CAL would only help ID performance?\n\nCAL makes the learning of imaginary words perform like multi-task learning, while the imaginary word without CAL is analogous to only fitting one task. Therefore, in ID evaluation, the absence of CAL tends to achieve better scores (similar to a single-task learning model performing better on its learned task than a multi-task learning model)."}, {"review_id": "mJrRry9bUz", "reviewer": "Reviewer_KFc6", "summary": "This paper proposes eXtensible Prompt (X-Prompt), a new way to prompt large language models beyond natural language. With an extensible vocabulary of imaginary words, X-Prompt allows for more descriptive prompts and is designed to be out-of-distribution robust. The paper also proposes context-augmented learning (CAL) to learn imaginary words for general usability. Experiments with OPT-6B reveal some effectiveness of the method.", "questions": "What is “Prompt Learning” means in Section 3? Discrete or continuous? I do not find any description in the paper (Maybe I missed it)", "limitations": "Yes, the authors discuss the computing resource limitations of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The idea of the X-Prompt is interesting. The use of imaginary words in prompts is an innovative idea that allows for more descriptive prompts and is designed to be out-of-distribution robust.", "weaknesses": "1. The idea is interesting but the method is not novel. The training way of X-Prompt is similar to continuous prompt learning. And why only use 1 token to learn the imaginary word?\n2. The paper lacks a sufficient number of baselines for comparison and utilizes a limited set of datasets. The “Prompt tuning”(maybe) and X-prompt method are fine-tuned, which is unfair to compare with “No prompt” or “32-shot”.\n3. In Table 11, there is a difference in the input format of “Prompt tuning” in the train stage and inference stage. I don’t think [SOFT] can be used directly in the NL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your constructive comments. We hope that our following responses will help you better interpret the merits and contributions of our paper, and further improve your impression and evaluation of our paper:\n\n> Weakness 1.  The idea is interesting but the method is not novel. The training way of X-Prompt is similar to continuous prompt learning. And why only use 1 token to learn the imaginary word?\n\nWe believe that continuous prompt learning (by gradient descent) is **a general methodology category**, and **we don't think our work's affiliation with this category negates its novelty**. Like other excellent research in this category, our work features several innovations to address the issues that traditional soft prompts cannot solve effectively, such as zero-shot NL/soft prompt combination for language customization. The nature of X-Prompt is registering new words for the LLM and allowing them to be used as natural language words with strong OOD compositional ability. **This essential difference allows X-Prompt to have completely different use and application purposes compared to traditional soft prompts**, as the free combination of imaginary words and natural language prompts provides a novel interface for LLM, significantly enhancing LLM's expressive capabilities, which is a critical ability that soft prompts fail to provide. We hope you will reconsider evaluating our novelty and contribution based on these innovations.\n\nAs for why we only use one token to learn the imaginary word for customizing language style in our work, please refer to **Figure 1 in our supplementary material** which has demonstrated that using more imaginary tokens can increase ID performance (overfitting), but it doesn't show a significant positive effect on OOD performance. We infer that the information content of a specific language style is not very large, and a single imaginary token can already represent it well. Therefore, we use only one imaginary token to represent the style in our paper. However, if we represent more complicated knowledge or events (e.g., the World Cup 2022), we agree with your point that using a single imaginary token might not be enough, which we plan to explore in future work.\n\n> Weakness 2.  The paper lacks a sufficient number of baselines for comparison and utilizes a limited set of datasets. The “Prompt tuning”(maybe) and X-prompt method are fine-tuned, which is unfair to compare with “No prompt” or “32-shot”.\n\nPlease understand that zero-shot language customization itself is a very new research problem, and to the best of our knowledge, there is not much previous work that formally studies this challenge. As a result, there are very few applicable baselines and datasets available for evaluation. **We believe we have made our best effort to conduct a comprehensive empirical study of this problem from various aspects, and our work represents the most comprehensive empirical study within the scope of zero-shot language customization to date.**\n\nAs for \"No prompt\" and \"32-shot\" baselines, we would like to reiterate that **we included these results to help everyone interpret the empirical results on this new challenge and demonstrate that X-Prompt is capable of achieving what NL prompts struggle with**. If we do not include these baselines' results, readers and reviewers would be left wondering whether our method is indeed better than NL prompts (See Reviewer GeEV who mistakenly thought we didn't have these NL baselines and suggested we should add them).\n\n> Weakness 3. In Table 11, there is a difference in the input format of “Prompt tuning” in the train stage and inference stage. I don’t think [SOFT] can be used directly in the NL.\n\nIn Table 11, you are correct that there is a difference in the input format of \"Prompt tuning\" in the train stage and inference stage. **This is precisely the challenge our work aims to address: OOD robustness -- the ability to work even for prompt templates that are unseen during training.**\n\nTable 11 demonstrates that our approach can handle this issue while prompt tuning cannot, highlighting the essential difference between our work and traditional continuous prompt learning.\n\n> What is “Prompt Learning” means in Section 3? Discrete or continuous? I do not find any description in the paper (Maybe I missed it)\n\nLearned imaginary words are similar to natural language words (i.e., discrete symbols), and their embeddings are continuous vectors. As we mentioned above, learning imaginary words can be regarded as registering new words for the LLM. We'll make it clearer in our revised version."}, {"review_id": "KDOnbzUgKn", "reviewer": "Reviewer_GeEV", "summary": "This paper proposes X-prompt: a technique that learns an imaginary token to represent a concept that is hard to describe in natural language. Compared to soft prompt tuning, X-prompt is designed to be OOD robust with template and content augmentation, in which the X-token is trained with various prompt templates and examples of different topic keyword to prevent overfitting. The author quantitatively and qualitatively demonstrated the advantage of X-prompt over prompt tuning on styled text generation and style transfer.", "questions": "1. The style transfer experiment shows X-prompt is much better than NL baseline (Table 11, first row). I wonder how the NL baseline scales with model size (for example, does NL baseline on 175B OPT yields comparable results to X-prompt on 6.7B), and model type (e.g. NL baseline on instruction-tuned model like llama or even ChatGPT, vs X-prompt). This will help people understand the best use cases of X-prompt over NL instructions.\n2. Do you think X-prompt can be utilised compositionally: an instruction involving multiple imaginary words?\n3. Have you tried multiple tokens per imaginary word? Is 1 token the best setting?\n4. I notice the lr 2e-4 is much smaller than the original prompt tuning paper[1], which was 1e-5. Do you find 2e-4 work better for both PT and X-Prompt for the experiments?\n\n\n[1] Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.", "limitations": "The author has adequately discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to soft prompt tuning, X-prompt is OOD robust as shown intuitively on Table 2. The secret source is context augmented learning (CAL) which involves augmented templates and content (topic keywords). CAL is logically reasonable and feasible, which is very effective generalising X-prompt to OOD as shown in Table 6\n2. Table 11 shows that imaginary tokens of X-prompt trained for a specific task (styled generation for training), can be reused to support other tasks via different natural language instruction (style transfer for inference). This shows imaginary tokens learned from X-prompt can interact with natural language tokens. As such, X-prompt can be potentially utilised for compositional usage.", "weaknesses": "1. Missing baseline: for table 6, the baseline of using NL instruction is missing. As X-prompt's core objective is to learn the concept which is hard to describe in NL, it is desirable to compare with baselines which use NL descriptions. For example, a baseline which puts some example tweets of a specific user as in-context prompt, can be compared. In the same spirit, for qualitative evaluation (Table 7&8), there should be one more NL baseline which prompts the LLM as \"Criticise the C++ language in Donald Trump's style\".\n2.  The qualitative evaluation (Table 7 & 8) raises concerns as the styles chosen are from well-known characters (Trump, Satya, Sheldon). This is problematic as the LLM already learned about their styles during pretraining, and such knowledge might be well associated with their names already. For example, the following is the result I get from ChatGPT with query \"Praise the C++ language in Donald Trump's style\": \"The C++ language, folks, let me tell you, it's tremendous. Absolutely tremendous. It's a beautiful, beautiful language.\". \n3. Following point 2, I think the evaluation well-verified the advantage of X-prompt over soft prompt tuning. However, there lacks robust and detailed comparison with baselines which use descriptions in natural language for styled generation experiment. \n\nI'm willing to reconsider score if the above three concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our work, your constructive feedback and thought-provoking questions. We hope our following responses will help you better interpret our contributions, and further improve your impression and evaluation of our paper:\n\n> Missing baseline: for table 6, the baseline of using NL instruction is missing ...\n\n**The \"8-shot\" and \"32-shot\" in Table 6** are the baseline methods placing example tweets from a specific user as in-context prompts, which are exactly what you suggested.\n\n> The qualitative evaluation (Table 7 & 8) raises concerns as the styles chosen are from well-known characters (Trump, Satya, Sheldon)\n\nWe also considered this problem when we designed the experiments. However, for the new task of zero-shot language customization, the available evaluation data and methods are very limited. While we tried to evaluate the results as comprehensively as possible, we still can't use the ideal evaluation protocol to evaluate the results.\n\nIdeally, we should use language styles that the LLM hasn't seen before for qualitative evaluation to fully demonstrate that our X-prompt can describe styles that cannot be prompted by NL. However, **there is a dilemma**: unfamous characters the LLM doesn't know (e.g., my language style) are difficult for the annotators to judge their styles; famous characters with distinct styles are easier for annotators but their styles can be prompted with natural language. We finally chose famous people for qualitative evaluation because: a) we believe the reliability of annotations is more important; b) though we used well-known characters, X-Prompt does not rely on any celebrity-related information, meaning it is character-independent and can be applied to any character (analogous to a language-independent method that can be applied to any language), which can also be supported by the quantitative evaluation results (Table 6), indicating X-prompt can achieve good OOD results for arbitrary users.\n\nIn our follow-up experiments, we supplemented the missing part of the evaluation: we evaluated writing styles the LLM (i.e., OPT-6.7B) doesn't know. **We use a senior Chinese media professional -- Hu Xijin whose writing style is distinctive and has always been popular among Chinese netizens for imitation -- as an example**. We translated his 1500 tweets into English with the GPT-4, retaining his writing style as much as possible. 1 example tweet:\n*``` I saw several groups chatting with ChatGPT and making fun of old Hu, and some even predicted that artificial intelligence will make old Hu unemployed. Haha, artificial intelligence pushes everything into a super digital mode, and whoever has the greatest computing power is the king, just like the ever-escalating battle between missiles and anti-missiles. But old Hu is like an everlasting 155mm howitzer, simple, not relying on any trendy stuff. I wish you all not to be \"buried alive\" by artificial intelligence and become the survivors who crawl out of the \"pit of ten thousand people.\" ```*\n\n| Method | Content | Style | Overall |\n| :------|:-----:|:-----------:|:-----:|\n| NL |**0.86** | 0.22 | 0.18 |\n| NL (with the prompt word \"Hu Xijin's style\") | 0.61 | 0.20 | 0.15 |\n| Prompt tuning | 0.27 | **0.75** | 0.22 |\n| X-Prompt | 0.64 | 0.72 | **0.58** |\n\nAs shown in the above table, adding the prompt words \"Hu Xijin's style\" doesn't improve the OPT-6.7B's result, **demonstrating that OPT-6.7B is unaware of Hu Xijin's style. However, X-prompt still achieved excellent results, confirming it's not affected by whether the person is famous or not.**\n\n>  there lacks robust and detailed comparison with baselines which use descriptions in natural language for styled generation experiment.\n\nWe'll update the above result evaluating Hu Xijin's style, which includes the comparison with NL prompts in our revised version.\n\n> To Question 1\n\nAs discussed earlier, our style transfer experiments should ideally involve transferring between unnamable styles. However, we can't find existing datasets in practice that support this ideal evaluation protocol.\n\nFor the ease of evaluation, we choose to conduct style transfer experiments with formality and politeness because they have readily available datasets (GYAFC/PoliteRewrite) and reliable evaluation methods, which can objectively reflect the generation quality.\n\nSimilar to our previous discussions, our approach is style-independent, though our evaluation is based on experiments with well-defined styles.\n\nWe believe the best use cases of X-prompt over NL instructions are, as emphasized in our paper, still for representing what NL prompts hardly describe (e.g., unnamable language style customization). But please understand directly evaluating in these scenarios is very difficult in practice.\n\n> To Question 2\n\nIt's a very good question. We initially designed X-prompt with the aim of having strong compositional ability for imaginary words (similar to newly created words). The compositional strength depends largely on X-prompt learning, with CAL playing a crucial role. **Once the imaginary words have seen sufficiently diverse contexts during training (like a natural language word seeing various contexts through large-scale pre-training.), they'll have the similar compositional ability as natural language words.**\n\n> To Question 3\n\nWe show the results of using different lengths of imaginary words in **Figure 1 of the supplementary material**. For language style customization, we found using more than 1 token can improve ID performance, but it doesn't help much with OOD. This may be because 1 token is basically sufficient for expressing a specific style.\n\n> To Question 4\n\n2e-4 empirically works well in our setting. We think [1]'s learning rate is larger because **(1)** it uses a different optimizer -- Adafactor, not Adam as in our paper; **(2)** its base model is the T5, not GPT/OPT in our paper. **(3)** its studied tasks are almost NLU tasks, not generation tasks as in our paper."}, {"review_id": "0UC6UT3Kpi", "reviewer": "Reviewer_X5KZ", "summary": "This paper introduces a solution called eXtensible Prompt (X-Prompt), which enables instructing a Language Model (LM) using imaginary words. These words serve the purpose of providing instructions to the LM that are difficult to articulate using natural language. In order to prevent overfitting of the LM and facilitate its generalization to out-of-distribution examples, the authors propose two strategies: Template Augmentation and Context-Augmented Learning. Through a series of experiments, the authors evaluate the ability of X-Prompt to generate suitable content in the style of a specific individual, as well as its capacity for zero-shot style transfer generation. The results, both quantitative and qualitative, demonstrate the efficacy of X-Prompt and the utility of context-augmented learning.", "questions": "1.  Missing reference: [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://aclanthology.org/2021.naacl-main.410) (Qin & Eisner, NAACL 2021). I think the idea of this paper is pertinent to the idea of X-Prompt and it is the origin of the term \"soft prompt\".\n2.  In section 3.1.1, you mention that 5% of the users are used for validation and test, but you only have 20 authors. Does it mean that 1 user is used for validation and 1 user is used for test? To strengthen the experimental findings and reduce variance, it would be beneficial to include a larger number of users in the evaluation process.\n3.  Can you give more details regarding the \"prompt engineering\" section 2.2.1? How do you generate more prompts, and how many?\n4.  I am curious if the prompt tuning is properly trained to be a baseline. In table 12, the training and inference prompt for the \"prompt tuning\" method is different. Will this hurt the performance of the soft prompt? Will it be possible to make the training and inference consistent? ", "limitations": "The paper should address a potential limitation concerning the experiments on open-ended generation. The successful generation of content imitating the style of a specific individual raises concerns regarding the potential for generating deceptive or fake statements. If the generated content closely resembles the authentic statements of the person, it could be challenging to distinguish between the two. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  The paper introduces X-Prompt as an extension of the soft prompt method, offering notable advantages over previous approaches. X-Prompt exhibits enhanced flexibility by enabling its application to out-of-distribution examples, thereby providing significant adaptability. Its novelty lies in its minimal learning cost, in contrast to the substantial efforts required for LLM pretraining.\n2.  The training methodology, referred to as context-augmented learning, harnesses the capabilities of LLMs to generate novel contexts. This cost-effective approach facilitates the generation of additional training examples and can be potentially applied to various prompt learning experiments, showcasing its versatility.\n3.  The paper is commendably well-written, featuring informative diagrams that enhance comprehension of the proposed method. Moreover, the inclusion of numerous example prompts and the corresponding generated content greatly enhances the readability of the paper.\n4.  The experiments conducted are comprehensive, incorporating quantitative metrics such as perplexity and next-token accuracy, as well as qualitative assessments involving human annotators. This multifaceted evaluation methodology ensures a thorough evaluation of the proposed approach.", "weaknesses": "1.  Lack of novelty: The paper is an extension to the soft prompt solution. Except for that this paper uses LLMs while most of the previous focuses on models like BERT, the paper improves the training strategy by introducing the method of context-augmented learning. The contribution alone might not be significant enough for a paper at NeurIPS.\n2.  Limited Experiment Scope: Although labeled as an \"extensible prompt,\" the paper primarily focuses on style transfer generation tasks. It would be valuable for the authors to broaden the experimental scope to encompass additional tasks, akin to the approach taken in the prefix-tuning paper (Li and Liang, 2021). This expansion would further showcase the versatility and potential of the proposed method.\n3.  Model Performance Concerns: While the experiments understandably revolve around adhering to specific styles, there may be implications for content faithfulness with the use of X-Prompt. Table 7 indicates that the content score of X-Prompt falls noticeably below that of natural language, albeit demonstrating significant improvement over the soft prompt method. The evaluation of content faithfulness in other experiments, such as open-ended generation, remains absent. Lower perplexity may stem from the chosen style rather than from generating appropriate content. More evidence is needed to evaluate the effectiveness of the imaginary tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your recognition of our work, as well as your constructive feedback and thought-provoking questions. We hope that our following responses will help you better interpret the merits and contributions of our paper, and further improve your impression and evaluation of our paper:\n\n> Weakness 1 and 2: Novelty and Scope\n\nOur primary contribution is the X-Prompt idea, which provides an interface for large language models (LLMs) to include new information and knowledge by compressing indescribable information into a discrete symbol (i.e., imaginary word) that is OOD robust and possesses strong zero-shot compositional ability. We successfully apply this approach to zero-shot language customization, which, to the best of our knowledge, is **among the earliest work addressing LLM customization—an important challenge regarded as the first step towards personalized LLMs and AI agents that may be the most crucial research problems in the new era of AI**. In this regard, our work's originality, scope, and pioneering nature are highly consistent with submissions encouraged by NeurIPS.\n\nAs for technical novelty, although our method (learning imaginary tokens represented by continuous vectors via gradient descent) shares some similarities with soft prompts, our proposed innovations (e.g., content-augmented learning) **enable the learned imaginary words to have far superior OOD robustness compared to soft prompts**. This essential difference allows X-Prompt to have **completely different use and application purposes** compared to traditional soft prompts, as the free combination of imaginary words and natural language prompts provides a novel interface for LLM, significantly enhancing LLM's expressive capabilities, **which is a critical ability that soft prompts fail to provide.**\n\nThe motivation of our work is different from prefix-tuning. Compared with Prefix-tuning that studies in-domain (ID) learning and does not involve OOD evaluation, we focus on zero-shot language customization, a very important and emerging research problem. In contrast to prefix-tuning, where ID evaluation is easily performed with many existing datasets and evaluation methods, **our research emphasizes OOD compositional evaluation, where both the available evaluation methods and resources are extremely limited**. Therefore, it is almost impossible for us to perform OOD evaluations as extensively as ID evaluations (like conducted in the PrefixTuning paper), because as seen in our paper, a single task's OOD evaluation is already very challenging and reaching the limit within the limited space of this paper. However, **we believe our work represents the most comprehensive empirical study within the scope of zero-shot language customization to date.**\n\n> Weakness 3: The evaluation of content faithfulness in other experiments, such as open-ended generation, remains absent\n\nAs you point out, PPL and next word accuracy cannot accurately reflect content faithfulness. That's why we performed human evaluation in Table 7 and **it is exactly the content faithfulness evaluation for open-ended generation**. We kindly ask you to confirm this. \n\nBy the way, in our follow-up experiments (conducted after the full paper submission date), we use the GPT-4 to help evaluate the content faithfulness with the prompt ```Please evaluate whether the following text follows the instruction: {prompt}; If it follows the instruction, please rate 1; otherwise, rate 0.```:\n\n| Method | Content Faithfulness |\n| :------|:-----:|\n| NL | 0.79 |\n| Prompt tuning | 0.28 | \n| X-Prompt | 0.74| \n\nThe scores by the GPT-4 and the human evaluation results are highly consistent (Pearson correlation score r=0.72), indicating that the content faithfulness evaluation is reliable. We'll include the new result in the revised version.\n\nAs for the content score of X-Prompt falling below that of natural language, **it is not a surprise because natural language words' OOD robustness is the strongest**. The in-depth reason is that natural language words are all pretrained (with the LLM) on trillions of tokens, seeing sufficiently diverse contexts, which results in their unparalleled generalization/OOD compositional capabilities. While our X-Prompts' OOD robustness is much better than soft prompts, it is still far from natural language words given the limited training FLOPs of imaginary words.\n\n\n> Missing reference: Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (Qin & Eisner, NAACL 2021). \n\nWe appreciate your suggestion, and we will include a comparison and discussion of this related work in our revised version.\n\n> In section 3.1.1, you mention that 5% of the users are used for validation and test, but you only have 20 authors. Does it mean that 1 user is used for validation and 1 user is used for test?\n\nWe apologize for the confusion. What we mean is that for each user, we used 5% of their tweets as a test set, not 5% of the users as a test set.\n\n> Can you give more details regarding the \"prompt engineering\" section 2.2.1? How do you generate more prompts, and how many? \n\nWe used a combination of rephrasing and manual brainstorming to generate diverse prompts.\n\n> I am curious if the prompt tuning is properly trained to be a baseline. In table 12, the training and inference prompt for the \"prompt tuning\" method is different. Will this hurt the performance of the soft prompt? Will it be possible to make the training and inference consistent? \n\nIf training and test prompts are the same, it is the **In-distribution (ID)** evaluation setting. However, as we highlight our contribution in the paper, this work focuses on **OOD language customization** (i.e., prompts at test time are unseen during training), which is precisely what X-prompt can do well but soft prompt cannot."}]

Author rebuttal:

Meta-review: This paper presents a simple but effective idea for learning "portable" soft prompts that encode textual style from training examples, and then can be used to replicate that style in other target content scenarios.  The approach hinges on a new technique called context augmented learning, which is based on the intuition that training soft prompts in the presence of natural language prompts that by design have important information about the content (keywords) prevents the soft prompts from overfitting to content signals.  The experiments show significant improvements in the OOD setting, and the gains appear to be practically important for style transfer when compared to prompt tuning.  Both automated and human evaluations were performed, increasing confidence in the results.

The reviewers all argued for acceptance (although several reviews were "borderline accept").  Further, the author response clarified some important details regarding human evaluations and the statistical significance of results.

The reviewers offered several suggestions and comments that could be used to improve the work.  Adding experiments with additional, more recent base models may be especially helpful.
Decision: Accept (poster)