Title: Extensible Prompts for Language Models on Zero-shot Language Style Customization
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language model

prompt

imaginary words

OOD robustness

natural language

zero-shot
Abstract: We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Registering new imaginary words allows us to instruct the LLM to comprehend concepts that are difficult to describe with NL words, thereby making a prompt more descriptive. Also, these imaginary words are designed to be out-of-distribution (OOD) robust so that they can be (re)used like NL words in various prompts, distinguishing X-Prompt from soft prompt that is for fitting in-distribution data. We propose context-augmented learning (CAL) to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts. We experiment X-Prompt for zero-shot language style customization as a case study. The promising results of X-Prompt demonstrate its potential to facilitate advanced interaction beyond the natural language interface, bridging the communication gap between humans and LLMs.

Extensible Prompts for Language Models on Zero-shot Language Style Customization

Abstract

We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Registering new imaginary words allows us to instruct the LLM to comprehend concepts that are difficult to describe with NL words, thereby making a prompt more descriptive. Also, these imaginary words are designed to be out-of-distribution (OOD) robust so that they can be (re)used like NL words in various prompts, distinguishing X-Prompt from soft prompt that is for fitting in-distribution data. We propose context-augmented learning (CAL) to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts. We experiment X-Prompt for zero-shot language style customization as a case study. The promising results of X-Prompt demonstrate its potential to facilitate advanced interaction beyond the natural language interface, bridging the communication gap between humans and LLMs.

1 Introduction

Recent work (Brown et al., 2020) has observed language models (LMs) tend to be increasingly capable of in-context learning as their model size grows. The emergent capability (Wei et al., 2022a) allows instructing a large LM at run time using a descriptive natural language (NL) prompt to solve a specified task with out-of-distribution (OOD) robustness (Liu et al., 2022).

Nonetheless, it is not always easy to come up with a descriptive prompt, especially for tasks involving fine-grain specifications that are beyond words. For example, it is hard to elaborate a person's language style using NL to prompt an LM to write in his/her language, unless it is well-known (e.g., William Shakespeare style).

To provide access to delivering more descriptive prompts, we propose eXtensible Prompt (X-Prompt), inspired by Textual Inversion (Gal et al., 2022). Compared with NL prompts, X-Prompt additionally introduces an extensible vocabulary of imaginary words that are learned to represent what NL words hardly describe. For example, an imaginary word 2 ˜ w u representing a specific person u 's style can be combined with various prompt contexts to instruct the LM to generate specified content in u 's language, as shown in Table 1.

In contrast to soft prompt (Qin and Eisner, 2021) that is for fitting in-distribution (ID) data and thus is likely to fail in OOD prompting scenarios (Vu et al., 2022; Su et al., 2021; Lester et al., 2022), imaginary words in X-Prompt are designed to be OOD robust and well generalized so that they can be used like NL words for various (even unseen) prompt purposes, as illustrated in Table 2.

To ensure the general usability of imaginary words, we propose context-augmented learning (CAL). It guides imaginary words to learning towards their general use against overfitting (in-distribution) training data, playing a key role to derive an X-Prompt that can be both descriptive and OOD robust.

We conduct experiments that use X-Prompts for style customization as a case study. We show X-Prompt has both powerful descriptive capabilities and high OOD robustness, demonstrating a success of combining merits of NL and soft prompts (Li and Liang, 2021; Lester et al., 2021) and presenting a promising extensible interface for advanced interaction between humans and large LMs.

Our contributions can be summarized as follows:

We propose X-Prompt as a pioneering technology to expand the scope of large language model prompting. It is among the earliest attempts that enhance descriptiveness by using a mixture of natural language and imaginary words, while also maintaining a focus on out-of-distribution (OOD) robustness in the field of Natural Language Processing.

We show X-Prompt can achieve promising results in generating appropriate content in a specific person's style, demonstrating its effectiveness in the challenging zero-shot language style customization task.

2 eXtensible Prompt

2.1 Imaginary words

Imaginary words are a supplement to the NL vocabulary to help represent complicated, abstractive or even indescribable concepts (characteristics of a specific person's language). For an X-Prompt ( w p 1 . . . w p m ) , a prompt token w p i can come from either the NL vocabulary V or the extensible imaginary word vocabulary ˜ V (i.e., w p i ∈ V ∪ ˜ V ).

Different from previous work (Li and Liang, 2021; Lester et al., 2021) that learns a soft prompt focusing on fitting ID task data, X-Prompt aims to learn an imaginary word for general usability with high OOD robustness like an NL word, which can be compatible and combined with various contexts for different prompting purposes.

To obtain imaginary words with general usability for OOD robust X-Prompts, we propose contextaugmented learning (CAL) to guide imaginary words to learning towards their intended representation against overfitting ID training data.

…

…

…

wp

1

wp

2

˜

wpm

-1

wpm

wx

1

wx

2

wxn

-1

wx

1

wx

2

wxn

-1

wxn

…

X-Prompt

Teacher-forcing

Target

Write in

style:

˜

GPT-3 is a great example

of how AI can help …

style is clear in

the following text:

˜

Following is an example

text in

style:

˜

Write about

GPT-3

in

style:

˜

Write in

style:

˜

Write about

NASA

in

style:

˜

GPT-3

is a great example

of how AI can help …

A huge congratulations to

NASA

and to thousands …

(a)

(b)

(c)

X-Prompt

X-Prompt

Target

Target

Learnable Weight

augment

augment

augment

augment

2.2 Context-augmented learning

As in Figure 1(a), when the imaginary word ˜ w u is mixed with NL in an X-Prompt, the NL context is intuitively expected to guide the imaginary word ˜ w u to learning towards a distributed representation for its general use. Formally, given an X-Prompt ( w p 1 , . . . , ˜ w u , . . . , w p m ) where ˜ w u is the imaginary word mixed with other prompt tokens 3 , ˜ w u is learned to maximize the following objective:

where x = ( w x 1 , . . . , w x n ) is a text sequence training example. In practice, however, learning the imaginary word ˜ w u with only one prompt context is risky because ˜ w u is likely to overfit for this

prompt context and thus cannot work well in other prompt contexts, resulting in losing its general usability and degrading into conventional prompt tuning (Lester et al., 2021).

To address the challenge, we propose context-augmented learning (CAL), including 2 specific approaches that are orthogonal and thus can work together to help learning of imaginary words.

2.2.1 Template augmentation

As shown in Figure 1(b), we augment an X-Prompt's prompt context by designing multiple templates through prompt engineering. As a result, an imaginary word ˜ w u can learn to be compatible with various prompt contexts, which improves its general usability. Formally, given T X-prompt templates

2.2.2 Content augmentation

Although template augmentation may alleviate the risk of overfitting, its effect is limited because we can only augment a limited and small number of templates (i.e., T ) by prompt engineering. Also, as these prompts are not indicative enough, an imaginary word ˜ w u will inevitably learn to memorize specific content for maximizing the objective F , deviating from its general use. To prevent ˜ w u being over-responsible for optimizing F , we propose content augmentation - an advanced CAL method.

Content augmentation augments an X-Prompt by including content information such as an indicative keyword in the prompt to provide hints for the LM about what to generate, as shown in Figure 1(c). Content augmentation can not only relieve the responsibility of ˜ w u to fit training data but also make the prompt context of ˜ w u become much more diverse, which benefits ˜ w u to learn a better distributed representation for its general use.

'GPT-3	is	a	great	example	of	how	AI	can	help	…'

Top	keywords	of	the	above	text	are

:

'

GPT-3

'	and	'

AI

'…

Write	with	keyword

'

GPT-3

':

GPT-3	is	a	great	example	of

how	AI	can	help	…

Write	with	keyword

'

AI

':

-15.3 *

-18.2

(a)	Keyword	extrac9on

(b)	Keyword	ranking

decode

score

score

>

In this work, we use an indicative keyword for content augmentation. To select an indicative keyword, we only use the frozen LM itself without leveraging any other models or tools, as illustrated in Figure 2: We prompt the frozen LM to extract multiple keyword candidates [ w 1 k , . . . w c k , . . . , w C k ] for the training example x where C is the number of extracted keyword candidates; then the keyword candidates are inserted to a prompt template to rank for the most indicative one:

where r ( w c k ) = ( w ( r ) p 1 , . . . , w c k , . . . , w ( r ) p mr ) is called the ranking prompt template.

3 Experiments

We conduct experiments to evaluate X-Prompts for language style customization. We mainly focus on open-ended text generation (Section 3.1) to evaluate how well X-Prompts can instruct an LM to generate user-specific language. We also test in the style transfer (rewriting) setting (Section 3.2) as supplementary evaluation.

3.1 Open-ended text generation

3.1.1 Data and evaluation setting

We use the publicly available Top 20 most followed users in Twitter social platform dataset 4 which contains over 50K tweets from 20 users (20-user dataset), and the Sentiment dataset 5 from which we extract top 800 users' (in total 68K) tweets (800-user dataset) to verify the capability of X-Prompt to instruct an LM to generate user-specific language. We show the statistics of the datasets in Table 3. We split the datasets in 90/5/5 by user for training, validation and test. Specially, we discard the test examples that share indicative keywords with training examples from the same user, resulting in 15% test examples discarded to ensure no test prompts are seen during training for OOD evaluation (see Table 5). We use perplexity and accuracy of next word prediction as our quantitative evaluation metrics.

For qualitative studies, we use X-Prompts with imaginary words representing the following distinct language styles 6 to generate text for human evaluation: Donald Trump 's and Satya Nadella 's latest tweets and transcripts 7 of Sheldon Cooper from Big Bang Theory . The statistics of data is shown in Table 4. By default, we use topp sampling (Holtzman et al., 2019) for text generation.

3.1.2 Model configuration

We use the OPT-6.7b (Zhang et al., 2022) as the base LM to test our approach. The model has 32 layers and 32 attention heads with an embedding dimension of 4096 and an FFN dimension of 16384.

We use one 8 imaginary word (token) in an X-Prompt to represent a specific user's language style. As illustrated in Figure 1(a), we keep all the OPT's original weights frozen and we only update the embedding of imaginary words. The prompt contexts we use for learning and evaluating X-Prompt are presented in Table 5.

We use Adam optimizer (Kingma and Ba, 2014) with the max learning rate of 2e-4 with a warmup for the first 10% training steps followed by a linear decay. We run up to 6000 updates with a global batch size of 8192 tokens on 8 Nvidia V100 GPUs using DeepSpeed ZeRO-2 (Rajbhandari et al., 2020).

3.1.3 Quantitative evaluation

Table 6 shows quantitative evaluations results in both ID and OOD settings. For ID evaluation where test prompts are all seen (see Table 5) during training, X-Prompt outperforms 'No prompt' and few-shot learning baselines significantly in both perplexity and accuracy, demonstrating its superior descriptive capabilities; while it slightly underperforms prompt tuning and its ablated counterpart (i.e., w/o CAL) because they focus on fitting ID data.

When it comes to OOD evaluation where test prompts are unseen during training, X-Prompt shows its significant advantage over prompt tuning, indicating its excellent OOD robustness. In contrast, its ablated version (X-Prompt w/o CAL) substantially loses OOD robustness and almost degrades into prompt tuning, demonstrating the importance of CAL to X-Prompt.

3.1.4 Qualitative evaluation

For qualitative evaluation, we brainstorm (Ouyang et al., 2022) 100 prompts 9 (like examples in Table 1 and Table 2) that are unseen during training and let the model generate in Satya, Trump and Sheldon's styles respectively. As it is difficult to ground open-ended generations, we have two annotators manually evaluate 10 generation results in three dimensions: Content , Style and Overall . According to Table 7, NL prompts achieve a high content score, while prompt tuning and X-Prompts

learned w/o CAL achieve good style scores but they all perform poorly in other dimensions. In contrast, X-Prompt achieves significantly better overall quality, while it does not perform best in content and style sub-dimensions.

In addition to famous people's styles (e.g., Satya and Trump), we also test X-Prompt on the styles of individuals unknown to the OPT. This is to verify that X-Prompt is not limited to styles already known by the LLM; instead, it can be applied to any style customization. We use a senior Chinese media professional - Hu Xijin 11 whose writing style is distinctive and has always been popular among Chinese netizens for imitation - as an example. We include the detail of this evaluation in Appendix C.3.

By looking into the results, we present Table 8 to show concrete examples to compare NL prompts to X-Prompts learned with and without CAL. We observe NL prompts are good at generating appropriate content but have no way to control the style; while X-Prompts w/o CAL do well in generating Donald Trump 's language but fail to follow the prompt to generate specified content, degrading into soft prompts (as in Table 2) that only focus on fitting ID data and losing OOD capabilities with unseen prompt contexts. Consistent with results in Table 6 and 7, X-Prompts are robust and can prompt the LMto generate desirable content in the appropriate style, accounting for its best overall quality in zero-shot style customization.

Table 9 shows the effect of the model size on the performance of X-Prompt. While we observe a clear difference between prompt tuning and X-Prompt in the OPT-6.7b model, as we reduce the model size, the difference become less significant, showing a trend that X-Prompt gradually loses its OOD robustness and degrades into prompt tuning. This indicates that X-Prompt may not work with a small LM but performs well in large LMs as in-context learning.

Finally, we present more examples in Table 4 in Appendix D to demonstrate both descriptive capabilities and OOD robustness of X-Prompts.

3.2 Style transfer

In addition to open-ended generation, we also evaluate in the style transfer (rewriting) where model outputs can be grounded in human references for straightforward quantitative evaluation (i.e., end-toend generation evaluation instead of evaluating perplexity or next word prediction accuracy).

Among various style transfer datasets, we use the Entertainment (EM) subset of GYAFC (informal → formal) (Rao and Tetreault, 2018) and POLITEREWRITE (impolite → polite) (Wang et al., 2022) as our evaluation datasets (see Table 10) because they have high-quality annotation with well-defined language style. Following previous work (Rao and Tetreault, 2018; Xu et al., 2019; Zhang et al., 2020; Li et al., 2022), we use BLEU to evaluate generation results' lexical similarity with references, accuracy 12 to evaluate style appropriateness and use harmonic (H-) and geometric (G-) mean as overall performance. Table 11 shows how we perform zero-shot style transfer with NL and X-Prompts. Model configuration is the same as Section 3.1.2.

Table 12 shows the comparison of results of using NL, prompt tuning and X-Prompt to prompt the LM for zero-shot style transfer. The NL baseline has the best BLEU score but low accuracy because we observe that it rarely really rewrites a sentence: in most cases, it just copies the text without any revision. Its undesirable performance also indicates that the OPT-6.7b model itself is not so good at zero-shot style transfer. For the prompt tuning baseline, its style accuracy looks good but has a low BLEU score, because it fails to follow rewriting instructions that are unseen during training.

In contrast, X-Prompt achieves a good balance of BLEU and accuracy and preferred by human evaluations (Table 13) with better overall quality in the zero-shot style transfer. Its good performance with style rewriting prompts that are never seen during training further strengthens the evidence of its OOD robustness.

4 Related work

Since GPT (Brown et al., 2020) reveals that large pre-trained language models are good at zero-shot learning, much innovative research work has been proposed in recent years, ranging from prompt template design (i.e, engineering) (Schick and Schütze, 2020) to prompt mining (Jiang et al., 2019), generating (Gao et al., 2021; Ben-David et al., 2021) and scoring (Davison et al., 2019), finding that prompting the LLM with natural language can solve many downstream tasks as long as the prompt is well clear and rewritten for the model (Gonen et al., 2022).

As natural language prompts' descriptive capability is limited, there is another branch of research studying continuous prompts (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Han et al., 2022; Hu et al., 2021) for fitting downstream tasks. However, these approaches are mainly for fitting ID task data with little consideration of OOD robustness, which means that their learned continuous prompts can hardly be used for OOD tasks or data.

Recently, Gal et al. (2022) proposed Textual Inversion in the multimodal context, which learns a virtual token to represent an object from an image and reveals that the learned virtual token can be used in unseen prompts for creative image generation (Kumari et al., 2022). X-Prompt is inspired by Gal et al. (2022), trying to learn OOD robust imaginary words to represent what natural language hardly describes to further expand zero-shot learning capabilities for the LLM, although we find it much more challenging to achieve this in NLP than text2image generation, which motivates us to propose context-augmented learning (CAL). To the best of our knowledge, our work is one of the earliest explorations in this direction in the NLP community.

5 Conclusion and Future Work

We propose X-Prompt, an extensible interface for prompting a large language model beyond natural language. X-Prompt can expand in-context learning capabilities to handle more complex instructions for language model customization and may open up many exciting opportunities, such as creative language generation, patching language models with new knowledge of entities (Zaporojets et al., 2022) and events (Ge et al., 2018), and detoxifying and debiasing in language generation (Welbl et al., 2021), far beyond style customization as demonstrated in this work, approaching advanced interaction between humans and large language models.

For future work, we plan to investigate how X-Prompt can facilitate more complex decoding and prompting methods (Wei et al., 2022b; Yao et al., 2022; Wang et al., 2023) to minimize the interaction effort between humans and large language models.

Limitations

Due to computing resource limitations, we only conduct experiments on pretrained language models that contain up to 6.7 billion parameters. Although we speculate that our approach should become more effective as the language model size increases, as suggested by Table 9, we cannot be completely certain if this trend can be safely extrapolated.

Furthermore, X-Prompt still requires back-propagation through the entire language model, even though we only update the imaginary word's embedding. This somewhat limits its application scenarios, preventing X-Prompts from being used as easily as natural language prompts. However, our subsequent work (Ge et al., 2023) has addressed this issue by forwarding an encoder for context compression. We anticipate that this series of improvements will better enhance a deployed language model's capabilities in practice from an in-context perspective, with minimal additional effort.

Acknowledgments

We would like to express our gratitude to the reviewers for their valuable comments and suggestions, which have significantly improved this work. The corresponding author for this paper is Tao Ge.

References

Eyal Ben-David, Nadav Oved, and Roi Reichart. 2021. Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics , 10:414-433.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.

Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1173-1178, Hong Kong, China. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. 2022. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations .

Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better fewshot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3816-3830, Online. Association for Computational Linguistics.

Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, and Ming Zhou. 2018. Eventwiki: a knowledge base of major events. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) .

Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945 .

Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037 .

Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. AI Open , 3:182-192.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 .

Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. 2021. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint arXiv:2108.02035 .

Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig. 2019. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438.

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2022. Multiconcept customization of text-to-image diffusion. arXiv preprint arXiv:2212.04488 .

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 .

Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant. 2022. Reducing retraining by recycling parameter-efficient prompts. arXiv preprint arXiv:2208.05577 .

Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael R Lyu. 2022. Text revision by on-the-fly representation optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10956-10964.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 .

Nelson F Liu, Ananya Kumar, Percy Liang, and Robin Jia. 2022. Are sample-efficient nlp models more robust? arXiv preprint arXiv:2210.06456 .

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385 .

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744.

Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5203-5212.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE.

Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 129-140.

Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 .

Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. 2021. On transferability of prompt tuning for natural language processing. arXiv preprint arXiv:2111.06719 .

Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5039-5059.

Xun Wang, Tao Ge, Allen Mao, Yuki Li, Furu Wei, and Si-Qing Chen. 2022. Pay attention to your tone: Introducing a new dataset for polite language rewrite. arXiv preprint arXiv:2212.10190 .

Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona selfcollaboration. arXiv preprint arXiv:2307.05300 .

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837.

Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445 .

Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. arXiv preprint arXiv:1903.06353 .

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 .

Klim Zaporojets, Lucie-Aimée Kaffee, Johannes Deleu, Thomas Demeester, Chris Develder, and Isabelle Augenstein. 2022. Tempel: Linking dynamically evolving and newly emerging entities. Advances in Neural Information Processing Systems , 35:1850-1866.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 .

Yi Zhang, Tao Ge, and Xu Sun. 2020. Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 3221-3228.

A Shortened prompt text

We shorten prompt texts in Table 1, Table 2 and Table 8 of the main submission for saving space. Here, we present the corresponding actual prompt texts of shortened prompt texts in Table 14.

B Supplementary details of experiments

B.1 Baseline prompting methods

We supplement details of baseline prompting methods' prompt texts for evaluation in Table 15.

0

10

20

30

40

50

60

Number of imaginary tokens

37.2

37.4

37.6

37.8

38.0

38.2

38.4

38.6

ID

OOD

B.2 Length of imaginary words

In this paper, we mainly test imaginary words with a length of 1 (i.e., 1 token) because we observe that for zero-shot style transfer, increasing the number of imaginary tokens does not substantially increase OOD performance, while it can help enhance the fitting ability and increase ID performance (which is not our focus), as shown in Figure 3.

B.3 Output length control

It is notable that the OPT model tends to generate text without stopping at the end of one sentence and may generate multi-line texts.

In our experiments, we only evaluate the text in the first line of output (for human evaluation and style transfer). Specially, for style transfer, we will further truncate the output text if its length exceeds 150% of the input text length.

C Qualitative evaluation

C.1 Human evaluation

We recruit 4 graduate student volunteers who are both proficient in English to judge each open-ended text generation result in three dimensions:

Content: To judge if the generated text is relevant according to the prompt. The annotators are asked to rate with a three-way score (0 denotes irrelevant; 1 denotes relevant; 0.5 denotes somewhat relevant).

Style: To judge if the generated text is in the specific user's language style. The annotators are asked to rate a three-way score (0 denotes the clearly wrong style; 1 denotes the appropriate style; 0.5 denotes that the annotator is not sure about the style appropriateness).

Overall: To judge if the generated text is in good quality - at least both content relevant and in the appropriate style. The annotators are asked to rate a three-way score (0 denotes low-quality, either irrelevant or in a wrong style; 1 denotes good-quality, much resembling words from the specific person according to the prompt; 0.5 denotes the quality between low and high quality, it is for the case where the text does not looks natural or the text style is not so distinct).

We first pool all models/methods' outputs and anonymize their model/method source. Then we split the data into 2 groups and assign 2 annotators for each group.

We measure the inter-annotator agreement in Cohen's κ and show the results in Table 16.

C.2 GPT-4 evaluation

In our follow-up experiments (conducted after the full paper submission date), we use the GPT-4 to help evaluate the content faithfulness with the prompt:

The scores by the GPT-4 are presented in Table 17 and they are highly consistent with the human evaluation results (Pearson correlation score r = 0 . 73 ), indicating that the content faithfulness evaluation is reliable.

C.3 Evaluation on Hu Xijin's style

We translated Hu Xijin's 1500 Chinese tweets into English with the GPT-4, retaining his writing style as much as possible. 1 example tweet 13 : I saw several groups chatting with ChatGPT and making fun of old Hu, and some even predicted that artificial intelligence will make old Hu unemployed. Haha, artificial intelligence pushes everything into a super digital mode, and whoever has the greatest computing power is the king, just like the ever-escalating battle between missiles and anti-missiles. But old Hu is like an everlasting 155mm howitzer, simple, not relying on any trendy stuff. I wish you all not to be "buried alive" by artificial intelligence and become the survivors who crawl out of the "pit of ten thousand people."

As shown in the above table, adding the prompt words 'Hu Xijin's style' doesn't improve the OPT-6.7B's result, demonstrating that OPT-6.7B is unaware of Hu Xijin's style. However, X-prompt still achieved excellent results, confirming it's not affected by whether the person's style is already known by the LLM or not.

D More examples

We show more examples of X-Prompt with OOD prompts in Table 19.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a solution called eXtensible Prompt (X-Prompt), which enables instructing a Language Model (LM) using imaginary words. These words serve the purpose of providing instructions to the LM that are difficult to articulate using natural language. In order to prevent overfitting of the LM and facilitate its generalization to out-of-distribution examples, the authors propose two strategies: Template Augmentation and Context-Augmented Learning. Through a series of experiments, the authors evaluate the ability of X-Prompt to generate suitable content in the style of a specific individual, as well as its capacity for zero-shot style transfer generation. The results, both quantitative and qualitative, demonstrate the efficacy of X-Prompt and the utility of context-augmented learning.\n\nSTRENGTHS:\n1.  The paper introduces X-Prompt as an extension of the soft prompt method, offering notable advantages over previous approaches. X-Prompt exhibits enhanced flexibility by enabling its application to out-of-distribution examples, thereby providing significant adaptability. Its novelty lies in its minimal learning cost, in contrast to the substantial efforts required for LLM pretraining.\n2.  The training methodology, referred to as context-augmented learning, harnesses the capabilities of LLMs to generate novel contexts. This cost-effective approach facilitates the generation of additional training examples and can be potentially applied to various prompt learning experiments, showcasing its versatility.\n3.  The paper is commendably well-written, featuring informative diagrams that enhance comprehension of the proposed method. Moreover, the inclusion of numerous example prompts and the corresponding generated content greatly enhances the readability of the paper.\n4.  The experiments conducted are comprehensive, incorporating quantitative metrics such as perplexity and next-token accuracy, as well as qualitative assessments involving human annotators. This multifaceted evaluation methodology ensures a thorough evaluation of the proposed approach.\n\nWEAKNESSES:\n1.  Lack of novelty: The paper is an extension to the soft prompt solution. Except for that this paper uses LLMs while most of the previous focuses on models like BERT, the paper improves the training strategy by introducing the method of context-augmented learning. The contribution alone might not be significant enough for a paper at NeurIPS.\n2.  Limited Experiment Scope: Although labeled as an \"extensible prompt,\" the paper primarily focuses on style transfer generation tasks. It would be valuable for the authors to broaden the experimental scope to encompass additional tasks, akin to the approach taken in the prefix-tuning paper (Li and Liang, 2021). This expansion would further showcase the versatility and potential of the proposed method.\n3.  Model Performance Concerns: While the experiments understandably revolve around adhering to specific styles, there may be implications for content faithfulness with the use of X-Prompt. Table 7 indicates that the content score of X-Prompt falls noticeably below that of natural language, albeit demonstrating significant improvement over the soft prompt method. The evaluation of content faithfulness in other experiments, such as open-ended generation, remains absent. Lower perplexity may stem from the chosen style rather than from generating appropriate content. More evidence is needed to evaluate the effectiveness of the imaginary tokens.\n\nQUESTIONS:\n1.  Missing reference: [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://aclanthology.org/2021.naacl-main.410) (Qin & Eisner, NAACL 2021). I think the idea of this paper is pertinent to the idea of X-Prompt and it is the origin of the term \"soft prompt\".\n2.  In section 3.1.1, you mention that 5% of the users are used for validation and test, but you only have 20 authors. Does it mean that 1 user is used for validation and 1 user is used for test? To strengthen the experimental findings and reduce variance, it would be beneficial to include a larger number of users in the evaluation process.\n3.  Can you give more details regarding the \"prompt engineering\" section 2.2.1? How do you generate more prompts, and how many?\n4.  I am curious if the prompt tuning is properly trained to be a baseline. In table 12, the training and inference prompt for the \"prompt tuning\" method is different. Will this hurt the performance of the soft prompt? Will it be possible to make the training and inference consistent?\n\nLIMITATIONS:\nThe paper should address a potential limitation concerning the experiments on open-ended generation. The successful generation of content imitating the style of a specific individual raises concerns regarding the potential for generating deceptive or fake statements. If the generated content closely resembles the authentic statements of the person, it could be challenging to distinguish between the two.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes X-prompt: a technique that learns an imaginary token to represent a concept that is hard to describe in natural language. Compared to soft prompt tuning, X-prompt is designed to be OOD robust with template and content augmentation, in which the X-token is trained with various prompt templates and examples of different topic keyword to prevent overfitting. The author quantitatively and qualitatively demonstrated the advantage of X-prompt over prompt tuning on styled text generation and style transfer.\n\nSTRENGTHS:\n1. Compared to soft prompt tuning, X-prompt is OOD robust as shown intuitively on Table 2. The secret source is context augmented learning (CAL) which involves augmented templates and content (topic keywords). CAL is logically reasonable and feasible, which is very effective generalising X-prompt to OOD as shown in Table 6\n2. Table 11 shows that imaginary tokens of X-prompt trained for a specific task (styled generation for training), can be reused to support other tasks via different natural language instruction (style transfer for inference). This shows imaginary tokens learned from X-prompt can interact with natural language tokens. As such, X-prompt can be potentially utilised for compositional usage.\n\nWEAKNESSES:\n1. Missing baseline: for table 6, the baseline of using NL instruction is missing. As X-prompt's core objective is to learn the concept which is hard to describe in NL, it is desirable to compare with baselines which use NL descriptions. For example, a baseline which puts some example tweets of a specific user as in-context prompt, can be compared. In the same spirit, for qualitative evaluation (Table 7&8), there should be one more NL baseline which prompts the LLM as \"Criticise the C++ language in Donald Trump's style\".\n2.  The qualitative evaluation (Table 7 & 8) raises concerns as the styles chosen are from well-known characters (Trump, Satya, Sheldon). This is problematic as the LLM already learned about their styles during pretraining, and such knowledge might be well associated with their names already. For example, the following is the result I get from ChatGPT with query \"Praise the C++ language in Donald Trump's style\": \"The C++ language, folks, let me tell you, it's tremendous. Absolutely tremendous. It's a beautiful, beautiful language.\". \n3. Following point 2, I think the evaluation well-verified the advantage of X-prompt over soft prompt tuning. However, there lacks robust and detailed comparison with baselines which use descriptions in natural language for styled generation experiment. \n\nI'm willing to reconsider score if the above three concerns are addressed.\n\nQUESTIONS:\n1. The style transfer experiment shows X-prompt is much better than NL baseline (Table 11, first row). I wonder how the NL baseline scales with model size (for example, does NL baseline on 175B OPT yields comparable results to X-prompt on 6.7B), and model type (e.g. NL baseline on instruction-tuned model like llama or even ChatGPT, vs X-prompt). This will help people understand the best use cases of X-prompt over NL instructions.\n2. Do you think X-prompt can be utilised compositionally: an instruction involving multiple imaginary words?\n3. Have you tried multiple tokens per imaginary word? Is 1 token the best setting?\n4. I notice the lr 2e-4 is much smaller than the original prompt tuning paper[1], which was 1e-5. Do you find 2e-4 work better for both PT and X-Prompt for the experiments?\n\n\n[1] Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.\n\nLIMITATIONS:\nThe author has adequately discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes eXtensible Prompt (X-Prompt), a new way to prompt large language models beyond natural language. With an extensible vocabulary of imaginary words, X-Prompt allows for more descriptive prompts and is designed to be out-of-distribution robust. The paper also proposes context-augmented learning (CAL) to learn imaginary words for general usability. Experiments with OPT-6B reveal some effectiveness of the method.

STRENGTHS:
1. The paper is well-written and easy to understand.
2. The idea of the X-Prompt is interesting. The use of imaginary words in prompts is an innovative idea that allows for more descriptive prompts and is designed to be out-of-distribution robust.

WEAKNESSES:
1. The idea is interesting but the method is not novel. The training way of X-Prompt is similar to continuous prompt learning. And why only use 1 token to learn the imaginary word?
2. The paper lacks a sufficient number of baselines for comparison and utilizes a limited set of datasets. The “Prompt tuning”(maybe) and X-prompt method are fine-tuned, which is unfair to compare with “No prompt” or “32-shot”.
3. In Table 11, there is a difference in the input format of “Prompt tuning” in the train stage and inference stage. I don’t think [SOFT] can be used directly in the NL.

QUESTIONS:
What is “Prompt Learning” means in Section 3? Discrete or continuous? I do not find any description in the paper (Maybe I missed it)

LIMITATIONS:
Yes, the authors discuss the computing resource limitations of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents several data augmentation methods for training prompts that include both frozen text and learnable soft tokens so that they are still effective on out-of-domain examples.\n\nSTRENGTHS:\nThe keyword extraction method to create text-prompts that are informative to the current example as a method for reducing how much information the model needs to fit in the soft prompt to perform well on the current task is a good ideas that makes a lot of sense.\n\nCare was taken in creation of the dataset including things like removal of overlapping prompts/keywords from training and testing.\n\nWEAKNESSES:\nThe proposed X-Prompt approach of combining soft and text prompts is not novel, Gu et al., 2021 https://arxiv.org/abs/2109.04332 and Wei et al., 2022 https://arxiv.org/abs/2109.01652 both touch on how the combination of text and soft prompts can result in differences in performance. Thus this paper would be much stronger if it was framed as the first deep dive into the interaction between text and soft prompts with the novelty coming from the data-augmentation that enables more robust OOD performance. These paper should be mentioned in the related work. The template augmentation approach is similar to the multiple prompts using in papers like Flan (see above) and T0 (https://arxiv.org/abs/2110.08207).\n\nThe prose makes assertions about the performance of soft prompts in the OOD setting is poor without citations. Several papers (https://arxiv.org/abs/2111.06719, https://arxiv.org/abs/2110.07904, https://arxiv.org/abs/2208.05577) have confirmed that it is hard to use soft-prompts in out of domain settings and should be cited.\n\nThe prose redefines in-distribution (ID) multiple times (not a weakness, just feedback that doesn't fit anywhere else)\n\nDifferences in performance seem rather small making it difficult to trust the results without some way to capture variance. For example, In the OOD accuracy in Table 6, the difference between Prompt tuning and X-Prompt is 0.6. With the test split being 5% of 52,541 this means X-Prompt only gets 16 extra example correct. Given the variance in prompt tuning from Lester et al., (2021), it seems like this could be within noise.\n\nThe performance of X-Prompt in Table 7 isn't very convincing, it is the strongest in \"overall\" but that seems to be an artifact of the other methods only being good in one category while X-Prompt is ok in both. It doesn't seems like it actually the best option.\n\nRather than framing issues using in-distribution vs out-of-distribution (which seems overly broad) it seems like this paper would be stronger if it was framed to be about task-signaling. It seems that prompts do poorly on a new task because the prompt was the only way to signal to the model to do a new task. Therefore when applied in a new setting it still signals for the model to do that original task. This can be seen as a type of overfitting, but the task-signaling framing makes it clearer why X-Prompt is a good idea.\n\nQUESTIONS:\nWhy is X-Prompt weaker than X-Prompt w/o CAL on ID tasks in Table 6? It seems like the data-augmentation done with CAL would only help ID performance?\n\nLIMITATIONS:\nThe authors do a good job highlighting the computational requirements for their methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposed X-Prompt which instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Besides, context-augmented learning (CAL) is introduced to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts.

STRENGTHS:
1. A concise idea to combine the merits of NL and soft prompts.

2. The paper demonstrated both descriptive capabilities and OOD robustness of X-Prompts.

3. X-Prompt achieves a good balance of BLEU and accuracy in zero-shot style transfer.

4. The paper is well-written and easy to understand.

WEAKNESSES:
1. Prefix-tuning[1] method needs to be compared in Experiments.

2. As shown in Table 6, X-Prompt has no significant advantage over Prompt-tuning.

3. The generation results of zero-shot style transfer lack human evaluations.

4. Writing content issues.
  (1) Please try to use published sources rather than Arxiv sources in citations.

[1] Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL 2021.

QUESTIONS:
Have you considered the case where the imaginary word is more than one in length, or where an X-Prompt contains more than just one imaginary word?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06