SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9: X-Prompt tends to perform better and show more significant OOD robustness advantage over prompt tuning in larger foundation LMs for zero-shot style customization.
Model	Method	Content↑	Style↑	Overall↑
350m	Prompt tuning	0.22	0.81	0.15
X-Prompt	0.30 (+0.08)	0.79 (-0.02)	0.24 (+0.09)
1.3b	Prompt tuning	0.27	0.87	0.24
X-Prompt	0.45 (+0.18)	0.82 (-0.05)	0.37 (+0.13)
6.7b	Prompt tuning	0.34	0.92	0.30
X-Prompt	0.69 (+0.35)	0.83 (-0.09)	0.54 (+0.24)
3.2 Style transfer
In addition to open-ended generation, we also evaluate in the style transfer (rewriting) where model outputs can be grounded in human references for straightforward quantitative evaluation (i.e., end-to-end generation evaluation instead of evaluating perplexity or next word prediction accuracy).
Table 10: Statistics of the datasets for style transfer
Dataset Split	GYAFC (EM)	PoliteRewrite
Train	Dev	Test	Train	Dev	Test
#sentence	53K	3K	1K	10K	3K	2K
Among various style transfer datasets, we use the Entertainment (EM) subset of GYAFC (informal  $\rightarrow$  formal) (Rao and Tetreault, 2018) and POLITEREWRITE (impolite  $\rightarrow$  polite) (Wang et al., 2022) as our evaluation datasets (see Table 10) because they have high-quality annotation with well-defined language style. Following previous work (Rao and Tetreault, 2018; Xu et al., 2019; Zhang et al., 2020; Li et al., 2022), we use BLEU to evaluate generation results' lexical similarity with references, accuracy $^{12}$  to evaluate style appropriateness and use harmonic (H-) and geometric (G-) mean as overall performance. Table 11 shows how we perform zero-shot style transfer with NL and X-Prompts. Model configuration is the same as Section 3.1.2.
Table 11: NL, prompt tuning and X-Prompt for zero-shot impolite  $\rightarrow$  polite style transfer. [SOFT] and  $\widetilde{w}$  are learnable and denote the soft token and the imaginary word in prompt tuning and X-Prompt respectively. For zero-shot style transfer, we use beam search  $(b = 5)$  as the default decoding method.
Prompt methods	Phase	Prompt + Target (Training) / Expected Generation (Inference)
NL	Train	N/A
Inference	“[IMPOLITE TEXT)” Above text can be rewritten to improve its politeness as follows: [POLITE TEXT]
Prompt tuning	Train	[SOFT] [POLITE TEXT]
Inference	“[IMPOLITE TEXT)” Above text can be rewritten into the style of [SOFT]: [POLITE TEXT]
X-Prompt	Train	The style of \(\tilde{w}\) can be identified in the following text with keyword [KEYWORD]: [POLITE TEXT]
Inference	“[IMPOLITE TEXT)” Above text can be rewritten into the style of \(\tilde{w}\): [POLITE TEXT]
Table 12 shows the comparison of results of using NL, prompt tuning and X-Prompt to prompt the LM for zero-shot style transfer. The NL baseline has the best BLEU score but low accuracy because we observe that it rarely really rewrites a sentence: in most cases, it just copies the text without any revision. Its undesirable performance also indicates that the OPT-6.7b model itself is not so good at zero-shot style transfer. For the prompt tuning baseline, its style accuracy looks good but has a low BLEU score, because it fails to follow rewriting instructions that are unseen during training.
$^{12}$ Following previous work, we fine-tune a BERT-base (Devlin et al., 2019) model with the training sets, which annotate a text with its style label, as a style classifier to test accuracy.
Table 12: Results of zero-shot style transfer (i.e., rewriting)
Method	GYAFC	PoliteRewrite
BLEU	Style	H-mean	G-mean	BLEU	Style	H-mean	G-mean
No Edit	50.2	4.3	7.9	14.7	24.7	3.0	5.4	8.6
NL	50.8	20.0	28.7	31.9	24.6	6.7	10.5	12.8
Prompt tuning	16.2	76.7	26.8	35.2	15.4	80.8	25.9	35.3
X-Prompt	38.7	71.9	50.3	52.7	18.9	79.5	30.5	38.8
In contrast, X-Prompt achieves a good balance of BLEU and accuracy and preferred by human evaluations (Table 13) with better overall quality in the zero-shot style transfer. Its good performance with style rewriting prompts that are never seen during training further strengthens the evidence of its OOD robustness.
Table 13: Human evaluation of zero-shot style transfer
Method	Content	Style	Overall
Prompt tuning	0.27	0.82	0.23
X-Prompt	0.64	0.80	0.60
Since GPT (Brown et al., 2020) reveals that large pre-trained language models are good at zero-shot learning, much innovative research work has been proposed in recent years, ranging from prompt template design (i.e, engineering) (Schick and Schütze, 2020) to prompt mining (Jiang et al., 2019), generating (Gao et al., 2021; Ben-David et al., 2021) and scoring (Davison et al., 2019), finding that prompting the LLM with natural language can solve many downstream tasks as long as the prompt is well clear and rewritten for the model (Gonen et al., 2022).
As natural language prompts' descriptive capability is limited, there is another branch of research studying continuous prompts (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Han et al., 2022; Hu et al., 2021) for fitting downstream tasks. However, these approaches are mainly for fitting ID task data with little consideration of OOD robustness, which means that their learned continuous prompts can hardly be used for OOD tasks or data.
Recently, Gal et al. (2022) proposed Textual Inversion in the multimodal context, which learns a virtual token to represent an object from an image and reveals that the learned virtual token can be used in unseen prompts for creative image generation (Kumari et al., 2022). X-Prompt is inspired by Gal et al. (2022), trying to learn OOD robust imaginary words to represent what natural language hardly describes to further expand zero-shot learning capabilities for the LLM, although we find it much more challenging to achieve this in NLP than text2image generation, which motivates us to propose context-augmented learning (CAL). To the best of our knowledge, our work is one of the earliest explorations in this direction in the NLP community.
We propose X-Prompt, an extensible interface for prompting a large language model beyond natural language. X-Prompt can expand in-context learning capabilities to handle more complex instructions for language model customization and may open up many exciting opportunities, such as creative language generation, patching language models with new knowledge of entities (Zaporojets et al., 2022) and events (Ge et al., 2018), and detoxifying and debiasing in language generation (Welbl et al., 2021), far beyond style customization as demonstrated in this work, approaching advanced interaction between humans and large language models.
For future work, we plan to investigate how X-Prompt can facilitate more complex decoding and prompting methods (Wei et al., 2022b; Yao et al., 2022; Wang et al., 2023) to minimize the interaction effort between humans and large language models.
Due to computing resource limitations, we only conduct experiments on pretrained language models that contain up to 6.7 billion parameters. Although we speculate that our approach should become more effective as the language model size increases, as suggested by Table 9, we cannot be completely certain if this trend can be safely extrapolated.
Furthermore, X-Prompt still requires back-propagation through the entire language model, even though we only update the imaginary word's embedding. This somewhat limits its application scenarios, preventing X-Prompts from being used as easily as natural language prompts. However, our subsequent work (Ge et al., 2023) has addressed this issue by forwarding an encoder for context compression. We anticipate that this series of improvements will better enhance a deployed language model's capabilities in practice from an in-context perspective, with minimal additional effort.
We would like to express our gratitude to the reviewers for their valuable comments and suggestions, which have significantly improved this work. The corresponding author for this paper is Tao Ge.
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2021. Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10:414-433.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173-1178, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. 2022. An image is worth one word: Personalizing text-to-image generation using textual inversion. In *The Eleventh International Conference on Learning Representations*.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, and Ming Zhou. 2018. Eventwiki: a knowledge base of major events. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. 2021. On transferability of prompt tuning for natural language processing. arXiv preprint arXiv:2111.06719.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5039-5059.
Xun Wang, Tao Ge, Allen Mao, Yuki Li, Furu Wei, and Si-Qing Chen. 2022. Pay attention to your tone: Introducing a new dataset for polite language rewrite. arXiv preprint arXiv:2212.10190.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445.
Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. arXiv preprint arXiv:1903.06353.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
Klim Zaporojets, Lucie-Aimée Kaffee, Johannes Deleu, Thomas Demeester, Chris Develder, and Isabelle Augenstein. 2022. Tempel: Linking dynamically evolving and newly emerging entities. Advances in Neural Information Processing Systems, 35:1850-1866.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
Yi Zhang, Tao Ge, and Xu Sun. 2020. Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3221-3228.
A Shortened prompt text
We shorten prompt texts in Table 1, Table 2 and Table 8 of the main submission for saving space. Here, we present the corresponding actual prompt texts of shortened prompt texts in Table 14.
Table 14: The original prompt text of the shortened prompt text in Table 1, Table 2 and Table 8.
shortened prompt text	original prompt text
Criticize the C++ language:	The following text criticizes the C++ language:
Criticize the C++ language in \(\widetilde{w}\) style:	The style of \(\widetilde{w}\) is clear in the following text criticizing the C++ language:
Praise the C++ language:	The following text praising the C++ language:
Praise the C++ language in \(\widetilde{w}\) style:	The style of \(\widetilde{w}\) is clear in the following text praising the C++ language:
Concern about cloning technology:	The following text expresses the concern about cloning technology:
Concern about cloning technology in \(\widetilde{w}\) style:	The following text expressing the concern about cloning technology is in the style of \(\widetilde{w}\):
B Supplementary details of experiments
B.1 Baseline prompting methods
We supplement details of baseline prompting methods' prompt texts for evaluation in Table 15.
Table 15: Prompt texts of baseline methods for open-ended generation (Section 3.1)
Prompting methods	w/o keyword	w/ keyword
NL	[NO PROMPT]	The keyword of the following text is [KEYWORD]
Prompt tuning	[SOFT]	[SOFT] The keyword of the following text is [KEYWORD]
Figure 3: The effects of the length of imaginary tokens on the result.
B.2 Length of imaginary words
In this paper, we mainly test imaginary words with a length of 1 (i.e., 1 token) because we observe that for zero-shot style transfer, increasing the number of imaginary tokens does not substantially increase OOD performance, while it can help enhance the fitting ability and increase ID performance (which is not our focus), as shown in Figure 3.
B.3 Output length control
It is notable that the OPT model tends to generate text without stopping at the end of one sentence and may generate multi-line texts.
In our experiments, we only evaluate the text in the first line of output (for human evaluation and style transfer). Specially, for style transfer, we will further truncate the output text if its length exceeds  $150\%$  of the input text length.
C Qualitative evaluation
C.1 Human evaluation
We recruit 4 graduate student volunteers who are both proficient in English to judge each open-ended text generation result in three dimensions:
- Content: To judge if the generated text is relevant according to the prompt. The annotators are asked to rate with a three-way score (0 denotes irrelevant; 1 denotes relevant; 0.5 denotes somewhat relevant).
- Style: To judge if the generated text is in the specific user's language style. The annotators are asked to rate a three-way score (0 denotes the clearly wrong style; 1 denotes the appropriate style; 0.5 denotes that the annotator is not sure about the style appropriateness).
- Overall: To judge if the generated text is in good quality – at least both content relevant and in the appropriate style. The annotators are asked to rate a three-way score (0 denotes low-quality, either irrelevant or in a wrong style; 1 denotes good-quality, much resembling words from the specific person according to the prompt; 0.5 denotes the quality between low and high quality, it is for the case where the text does not look natural or the text style is not so distinct).
We first pool all models/methods' outputs and anonymize their model/method source. Then we split the data into 2 groups and assign 2 annotators for each group.
Table 16: Inter-annotator agreement in our human evaluation.
Content	Style	Overall
Cohen's κ	0.83	0.65	0.71
We measure the inter-annotator agreement in Cohen's  $\kappa$  and show the results in Table 16.
C.2 GPT-4 evaluation
In our follow-up experiments (conducted after the full paper submission date), we use the GPT-4 to help evaluate the content faithfulness with the prompt:
Please evaluate whether the following text follows the instruction {"prompt":
If it follows the instruction, please rate 1; otherwise, rate 0
The scores by the GPT-4 are presented in Table 17 and they are highly consistent with the human evaluation results (Pearson correlation score  $r = 0.73$ ), indicating that the content faithfulness evaluation is reliable.
Table 17: GPT-4 evaluation of content faithfulness
Method	Content Faithfulness
NL	0.77
Prompt tuning	0.31
X-Prompt	0.76
C.3 Evaluation on Hu Xijin's style
We translated Hu Xijin's 1500 Chinese tweets into English with the GPT-4, retaining his writing style as much as possible. 1 example tweet<sup>13</sup>: I saw several groups chatting with ChatGPT and making fun of old Hu, and some even predicted that artificial intelligence will make old Hu unemployed. Haha, artificial intelligence pushes everything into a super digital mode, and whoever has the greatest computing power is the king, just like the ever-escalating battle between missiles and anti-missiles. But old Hu is like an everlasting 155mm howitzer, simple, not relying on any trendy stuff. I wish you all not to be "buried alive" by artificial intelligence and become the survivors who crawl out of the "pit of ten thousand people."
Table 18: Human evaluation results on Hu Xijin's style
Method	Content	Style	Overall
NL	0.86	0.22	0.18
NL (with the prompt word “Hu Xijin’s style”)	0.61	0.20	0.15
Prompt tuning	0.27	0.75	0.22
X-Prompt	0.64	0.72	0.58
As shown in the above table, adding the prompt words "Hu Xijin's style" doesn't improve the OPT-6.7B's result, demonstrating that OPT-6.7B is unaware of Hu Xijin's style. However, X-prompt still achieved excellent results, confirming it's not affected by whether the person's style is already known by the LLM or not.
D More examples
We show more examples of X-Prompt with OOD prompts in Table 19.
13Original Chinese tweet: 我看到几个群里用与ChatGPT聊天拿老胡开涮，还有人预言它将让老胡失业。哈哈，人工智能把一切都推入数字化的超级模式，谁算力大谁牛，就像导弹和反导上演道高一尺魔高一丈一样。但老胡是一门永远的155毫米榴弹炮，简单，不依附任何时髦的东西。我祝愿大家都不被人工智能“活埋”，爬出“万人坑”。