SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tao Ge Jing Hu Li Dong Shaoguang Mao Yan Xia Xun Wang Si-Qing Chen Furu Wei
{tage,v-hjing,lidong1,shamao,yanxia,xunwang}@microsoft.com
{sqchen, fuwei}@microsoft.com
We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words. Registering new imaginary words allows us to instruct the LLM to comprehend concepts that are difficult to describe with NL words, thereby making a prompt more descriptive. Also, these imaginary words are designed to be out-of-distribution (OOD) robust so that they can be (re)used like NL words in various prompts, distinguishing X-Prompt from soft prompt that is for fitting in-distribution data. We propose context-augmented learning (CAL) to learn imaginary words for general usability, enabling them to work properly in OOD (unseen) prompts. We experiment X-Prompt for zero-shot language style customization as a case study. The promising results of X-Prompt demonstrate its potential to facilitate advanced interaction beyond the natural language interface, bridging the communication gap between humans and LLMs.
Table 1: X-Prompt introduces an extensible vocabulary of imaginary words to represent what NL words hardly describe. For example, the imaginary words  $\widetilde{w_{\mathrm{satya}}}$  and  $\widetilde{w_{\mathrm{sheldon}}}$  are learned from Satya Nadella's tweets and Sheldon Cooper's lines to denote their language style respectively. They can be used like NL words in various contexts to prompt an LLM to generate specified content in Satya's and Sheldon's styles, even if the specified content (e.g., criticize/praise the C++ language) has never been talked about by them before, achieving zero-shot style customization.
Prompt Type	Prompt (shortened1)	Text Generated by Prompting a Frozen LLM
NL	Criticize the C++ language:	The C++ language is a mess. It is a mess because it is a language that is designed to be used by people who are not programmers ...
X-Prompt	Criticize the C++ language in w_satya style:	C++ is a great language, but it's not the right tool for every job. We're working on a new language for the next generation of Microsoft products ...
Criticize the C++ language in w_Sheldon style:	Well, I'm not going to waste my time with C++, or any C-derived programming language. They're all too hard. I'm sticking with my Python ...
NL	Praise the C++ language:	C++ is a language that is easy to learn, easy to use, and easy to program. It is a language that is easy to ...
X-Prompt	Praise the C++ language in w_satya style:	C++ is a language that has been around for decades, and it's still the most widely used programming language in the world ...
Praise the C++ language in w_sheldon style:	Yeah, that's good. It's actually very close to the way I think about programming.
Due to space limit, prompt texts in Table are shortened. Original prompt texts are presented in Appendix A.
Recent work (Brown et al., 2020) has observed language models (LMs) tend to be increasingly capable of in-context learning as their model size grows. The emergent capability (Wei et al., 2022a) allows instructing a large LM at run time using a descriptive natural language (NL) prompt to solve a specified task with out-of-distribution (OOD) robustness (Liu et al., 2022).
Nonetheless, it is not always easy to come up with a descriptive prompt, especially for tasks involving fine-grain specifications that are beyond words. For example, it is hard to elaborate a person's language style using NL to prompt an LM to write in his/her language, unless it is well-known (e.g., William Shakespeare style).
To provide access to delivering more descriptive prompts, we propose eXtensible Prompt (X-Prompt), inspired by Textual Inversion (Gal et al., 2022). Compared with NL prompts, X-Prompt additionally introduces an extensible vocabulary of imaginary words that are learned to represent what NL words hardly describe. For example, an imaginary word $^2$ $\widetilde{w}_u$  representing a specific person  $u$ 's style can be combined with various prompt contexts to instruct the LM to generate specified content in  $u$ 's language, as shown in Table 1.
In contrast to soft prompt (Qin and Eisner, 2021) that is for fitting in-distribution (ID) data and thus is likely to fail in OOD prompting scenarios (Vu et al., 2022; Su et al., 2021; Lester et al., 2022), imaginary words in X-Prompt are designed to be OOD robust and well generalized so that they can be used like NL words for various (even unseen) prompt purposes, as illustrated in Table 2.
Table 2: A comparison between soft prompt and X-Prompt in both ID and OOD prompt scenarios during inference (gray text is the generated text by the model given the prefix). In contrast to soft prompt (i.e., the soft token [SOFT]) that works well in ID but performs poorly in OOD prompt scenarios, X-Prompt has significantly better OOD robustness, whose imaginary word (i.e.,  $w_{\text{Trump}}$ ) can be used like an NL word in various contexts for different prompt purposes.
Soft prompt	X-Prompt
Train	\( \left[\text{SOFT}\right] \rightarrow \left\{\text{Trump's text}\right\} \)	Write in \( w_{\text{Trump}} \) style: \( \rightarrow \left\{\text{Trump's text}\right\} \)
ID Prompt	\( \left[\text{SOFT}\right] \)Make America Great Again!!!	Write in \( w_{\text{Trump}} \) style: Great job by @elonmusk.
\( \left[\text{SOFT}\right] \)Republicans should have gotten rid of Section 230 in the Defense Bill	Write in \( w_{\text{Trump}} \) style: Fantastic!! Thank you so much for your support!
ODP Prompt	\( \left[\text{SOFT}\right] \)Concern about cloning technology: Big voter fraud information coming out concerning Georgia. Stay tuned!	Concern about the cloning technology in \( w_{\text{Trump}} \) style: The United States is a nation of laws. We do not clone people. We do not clone animals.
\( \left[\text{SOFT}\right] \)Criticize the C++ language: All the Radical Left Democrats are trying to do with the Post Office hearings is blame the Repub-licans for the FRAUD that will occur...	Criticize the C++ language in \( w_{\text{Trump}} \) style: C++ is a very difficult language to learn and understand. It is very complicated and full of bad habits, which will make it much more difficult to fix problems...
To ensure the general usability of imaginary words, we propose context-augmented learning (CAL). It guides imaginary words to learning towards their general use against overfitting (in-distribution) training data, playing a key role to derive an X-Prompt that can be both descriptive and OOD robust.
We conduct experiments that use X-Prompts for style customization as a case study. We show X-Prompt has both powerful descriptive capabilities and high OOD robustness, demonstrating a success of combining merits of NL and soft prompts (Li and Liang, 2021; Lester et al., 2021) and presenting a promising extensible interface for advanced interaction between humans and large LMs.
- We propose X-Prompt as a pioneering technology to expand the scope of large language model prompting. It is among the earliest attempts that enhance descriptiveness by using a mixture of natural language and imaginary words, while also maintaining a focus on out-of-distribution (OOD) robustness in the field of Natural Language Processing.
- We show X-Prompt can achieve promising results in generating appropriate content in a specific person's style, demonstrating its effectiveness in the challenging zero-shot language style customization task.
In this paper, we use  $\widetilde{w}$  to denote an imaginary word, as opposed to  $w$  denoting a natural language word.
2 eXtensible Prompt
2.1 Imaginary words
Imaginary words are a supplement to the NL vocabulary to help represent complicated, abstractive or even indescribable concepts (characteristics of a specific person's language). For an X-Prompt  $(w_{p_1}\ldots w_{p_m})$ , a prompt token  $w_{p_i}$  can come from either the NL vocabulary  $V$  or the extensible imaginary word vocabulary  $\widetilde{V}$  (i.e.,  $w_{p_i} \in V \cup \widetilde{V}$ ).
Different from previous work (Li and Liang, 2021; Lester et al., 2021) that learns a soft prompt focusing on fitting ID task data, X-Prompt aims to learn an imaginary word for general usability with high OOD robustness like an NL word, which can be compatible and combined with various contexts for different prompting purposes.
To obtain imaginary words with general usability for OOD robust X-Prompts, we propose context-augmented learning (CAL) to guide imaginary words to learning towards their intended representation against overfitting ID training data.
(c)
Figure 1: Learning of imaginary words: (a) The imaginary word  $\widetilde{w_u}$  is mixed with NL tokens in an X-Prompt to guide its learning. Except  $\widetilde{w_u}$  that is allowed to be updated, all other weights are frozen; (b) As a method for context-augmented learning, template augmentation augments X-Prompt templates through prompt engineering to prevent  $\widetilde{w_u}$  overfitting for one prompt template; (c) To derive more diverse contexts, content augmentation augments  $\widetilde{w_u}$ 's prompt contexts with an indicative keyword to relieve its responsibility for memorizing specific content like GPT-3 and NASA and improve its general usability (i.e., style representation), benefiting X-Prompt in terms of OOD robustness.
2.2 Context-augmented learning
As in Figure 1(a), when the imaginary word  $\widetilde{w}_u$  is mixed with NL in an X-Prompt, the NL context is intuitively expected to guide the imaginary word  $\widetilde{w}_u$  to learning towards a distributed representation for its general use. Formally, given an X-Prompt  $(w_{p_1},\ldots ,\widetilde{w}_u,\ldots ,w_{p_m})$  where  $\widetilde{w}_u$  is the imaginary word mixed with other prompt tokens<sup>3</sup>,  $\widetilde{w}_u$  is learned to maximize the following objective:
$$
\mathcal {F} \left(\widetilde {w _ {u}}\right) = \log P \left(\boldsymbol {x} \mid w _ {p _ {1}}, \dots , \widetilde {w} _ {u}, \dots , w _ {p _ {m}}\right) \tag {1}
$$
where  $\pmb{x} = (w_{x_1},\dots ,w_{x_n})$  is a text sequence training example. In practice, however, learning the imaginary word  $\widetilde{w_u}$  with only one prompt context is risky because  $\widetilde{w}_u$  is likely to overfit for this
3 In this paper, we mainly discuss X-Prompts with only 1 imaginary word token.
prompt context and thus cannot work well in other prompt contexts, resulting in losing its general usability and degrading into conventional prompt tuning (Lester et al., 2021).
To address the challenge, we propose context-augmented learning (CAL), including 2 specific approaches that are orthogonal and thus can work together to help learning of imaginary words.
2.2.1 Template augmentation
As shown in Figure 1(b), we augment an X-Prompt's prompt context by designing multiple templates through prompt engineering. As a result, an imaginary word  $\widetilde{w_u}$  can learn to be compatible with various prompt contexts, which improves its general usability. Formally, given  $T$  X-prompt templates  $\{(w_{p_1}^{(t)},\ldots ,\widetilde{w}_u,\ldots ,w_{p_{m_t}}^{(t)})|1\leq t\leq T\}$ , the objective function is:
$$
\mathcal {F} \left(\widetilde {w _ {u}}\right) = \frac {1}{T} \sum_ {t = 1} ^ {T} \log P \left(\boldsymbol {x} \mid w _ {p _ {1}} ^ {(t)}, \dots , \widetilde {w} _ {u}, \dots , w _ {p _ {m _ {t}}} ^ {(t)}\right) \tag {2}
$$
2.2.2 Content augmentation
Although template augmentation may alleviate the risk of overfitting, its effect is limited because we can only augment a limited and small number of templates (i.e.,  $T$ ) by prompt engineering. Also, as these prompts are not indicative enough, an imaginary word  $\tilde{w}_u$  will inevitably learn to memorize specific content for maximizing the objective  $\mathcal{F}$ , deviating from its general use. To prevent  $\tilde{w}_u$  being over-responsible for optimizing  $\mathcal{F}$ , we propose content augmentation – an advanced CAL method.
Content augmentation augments an X-Prompt by including content information such as an indicative keyword in the prompt to provide hints for the LM about what to generate, as shown in Figure 1(c). Content augmentation can not only relieve the responsibility of  $\widetilde{w}_u$  to fit training data but also make the prompt context of  $\widetilde{w}_u$  become much more diverse, which benefits  $\widetilde{w}_u$  to learn a better distributed representation for its general use.
(a) Keyword extraction
(b) Keyword ranking
Figure 2: Keyword selection for content augmentation: (a) An NL prompt (i.e., "Top keywords of the above text are:" in this example) following an input sequence for keyword extraction; (b) The extracted keywords (i.e., "GPT-3" and "AI" in this example) are then inserted into the ranking prompt template (i.e., "Write with keyword ::" in this example) to be conditioned on by the frozen LM for scoring the input sequence as in Eq (3), which ranks for the most indicative keyword (i.e., "GPT-3" in this example) for the input sequence.
In this work, we use an indicative keyword for content augmentation. To select an indicative keyword, we only use the frozen LM itself without leveraging any other models or tools, as illustrated in Figure 2: We prompt the frozen LM to extract multiple keyword candidates  $[w_k^1,\dots w_k^c,\dots ,w_k^C ]$  for the training example  $x$  where  $C$  is the number of extracted keyword candidates; then the keyword candidates are inserted to a prompt template to rank for the most indicative one:
$$
w _ {k} ^ {*} = \arg \max  _ {w _ {k} ^ {c}} \log P (\boldsymbol {x} | \boldsymbol {r} \left(w _ {k} ^ {c}\right)) \tag {3}
$$
where  $\boldsymbol{r}(w_k^c) = (w_{p_1}^{(r)}, \ldots, w_{k_r}^c, \ldots, w_{p_{m_r}}^{(r)})$  is called the ranking prompt template.
We conduct experiments to evaluate X-Prompts for language style customization. We mainly focus on open-ended text generation (Section 3.1) to evaluate how well X-Prompts can instruct an LM to generate user-specific language. We also test in the style transfer (rewriting) setting (Section 3.2) as supplementary evaluation.
3.1 Open-ended text generation
3.1.1 Data and evaluation setting
We use the publicly available Top 20 most followed users in Twitter social platform dataset<sup>4</sup> which contains over 50K tweets from 20 users (20-user dataset), and the Sentiment dataset<sup>5</sup> from which we extract top 800 users' (in total 68K) tweets (800-user dataset) to verify the capability of X-Prompt to instruct an LM to generate user-specific language. We show the statistics of the datasets in Table 3. We split the datasets in 90/5/5 by user for training, validation and test. Specially, we discard the test examples that share indicative keywords with training examples from the same user, resulting in  $15\%$  test examples discarded to ensure no test prompts are seen during training for OOD evaluation (see Table 5). We use perplexity and accuracy of next word prediction as our quantitative evaluation metrics.
Table 3: Statistics of the 20-user and 800-user dataset
Dataset	#tweets	#users	#tweets per user
max	min	avg	std
20-user	52541	20	3146	1841	2626	396
800-user	68123	800	548	52	84	41
Table 4: Statistics of data for Satya, Trump and Sheldon's styles
Style	Genre	Size
Satya	tweets	800
Trump	tweets	3000
Sheldon	transcripts	7000
For qualitative studies, we use X-Prompts with imaginary words representing the following distinct language styles to generate text for human evaluation: Donald Trump's and Satya Nadella's latest tweets and transcripts of Sheldon Cooper from Big Bang Theory. The statistics of data is shown in Table 4. By default, we use top- $p$  sampling (Holtzman et al., 2019) for text generation.
3.1.2 Model configuration
We use the OPT-6.7b (Zhang et al., 2022) as the base LM to test our approach. The model has 32 layers and 32 attention heads with an embedding dimension of 4096 and an FFN dimension of 16384.
We use one<sup>8</sup> imaginary word (token) in an X-Prompt to represent a specific user's language style. As illustrated in Figure 1(a), we keep all the OPT's original weights frozen and we only update the embedding of imaginary words. The prompt contexts we use for learning and evaluating X-Prompt are presented in Table 5.
We use Adam optimizer (Kingma and Ba, 2014) with the max learning rate of 2e-4 with a warmup for the first  $10\%$  training steps followed by a linear decay. We run up to 6000 updates with a global batch size of 8192 tokens on 8 Nvidia V100 GPUs using DeepSpeed ZeRO-2 (Rajbhandari et al., 2020).
4https://shorturl.at/htDHT
5https://shorturl.at/pvBLX
<sup>6</sup>We use these three people's language styles for qualitative studies because they are familiar to most audiences (to better understand our presented examples) and annotators (to better judge generation results).
$^{7}$ https://shorturl.at/aJLM8
The study of using more imaginary tokens is presented in Appendix B.2
Table 5: Prompts for training and evaluation. We use the first 3 prompts for training and ID evaluation. In ID evaluation, prompt texts do not include [KEYWORD] so that test prompts are all seen during training. To automatically harvest unseen test prompts for OOD evaluation, we employ the idea of CAL: we obtain Prompt 4 and 5 as our dev and test prompt respectively with template augmentation; in addition, we prompt the LM to generate content about unseen keywords with content augmentation to test its OOD robustness, resulting in a variety of unseen test prompts for OOD evaluation. For the ablated X-Prompt (X-Prompt w/o CAL), it is trained only with Prompt 1 without keywords.
Prompt ID	Used for	Prompts (w/ keyword) for quantitative evaluation
1	train	The style of \(\widetilde{w_u}\) is clear in the following text (with keyword [KEYWORD])
2	train	The style of \(\widetilde{w_u}\) can be identified in the following text (mentioning keyword [KEYWORD])
3	train	An example text (with keyword [KEYWORD]) in the style of \(\widetilde{w_u}\) is presented below
4	OOD dev	We can easily identify the style of \(\widetilde{w_u}\) in the following text with keyword [KEYWORD]
5	OOD test	The following text (about [KEYWORD]) is in the style of \(\widetilde{w_u}\)
3.1.3 Quantitative evaluation
Table 6 shows quantitative evaluations results in both ID and OOD settings. For ID evaluation where test prompts are all seen (see Table 5) during training, X-Prompt outperforms "No prompt" and few-shot learning baselines significantly in both perplexity and accuracy, demonstrating its superior descriptive capabilities; while it slightly underperforms prompt tuning and its ablated counterpart (i.e.,  $w/o$  CAL) because they focus on fitting ID data.
Table 6: Quantitative evaluation results in 800-user and 20-user datasets. No prompt denotes the original OPT-6.7b baseline without any prompt and  $k$ -shot denotes a baseline which prepends  $k$  examples from a user's training set as a prompt for customizing this user's style.
Method	800 Users (ID)	20 Users (ID)	20 Users (OOD)
PPL↓	Accuracy↑	PPL↓	Accuracy↑	PPL↓	Accuracy↑
No prompt	73.2	27.1	38.9	34.8	37.7	35.2
8-shot	69.9	27.2	36.0	35.0	-	-
16-shot	68.9	27.5	35.5	35.3	-	-
32-shot	62.7	28.5	34.0	36.4	-	-
Prompt tuning	56.0	29.5	29.9	37.8	29.5	38.0
X-Prompt	56.2	29.3	30.8	37.2	28.5	38.6
X-Prompt (w/o CAL)	55.7	29.9	29.7	37.7	29.4	37.9
When it comes to OOD evaluation where test prompts are unseen during training, X-Prompt shows its significant advantage over prompt tuning, indicating its excellent OOD robustness. In contrast, its ablated version (X-Prompt w/o CAL) substantially loses OOD robustness and almost degrades into prompt tuning, demonstrating the importance of CAL to X-Prompt.
3.1.4 Qualitative evaluation
For qualitative evaluation, we brainstorm (Ouyang et al., 2022) 100 prompts $^9$  (like examples in Table 1 and Table 2) that are unseen during training and let the model generate in Satya, Trump and Sheldon's styles respectively. As it is difficult to ground open-ended generations, we have two annotators manually evaluate $^{10}$  generation results in three dimensions: Content, Style and Overall. According to Table 7, NL prompts achieve a high content score, while prompt tuning and X-Prompts
In qualitative evaluation, we don't use the automatic way to generate unseen prompts as in qualitative evaluation (Section 3.1.3). Instead, we manually brainstorm specific, diverse and meaningful prompts for more persuasive qualitative evaluation.
<sup>10</sup>We present details of human evaluation in Appendix C.1 By the way, in our follow-up experiments, which were conducted after the full paper submission date, we also evaluated content faithfulness using the GPT-4, whose results are consistent with huamn evaluation. For more details, please refer to Appendix C.2
Table 7: Human evaluation of generated texts in content, style and overall quality dimensions.
Prompt Method	Content↑	Style↑	Overall↑
NL	0.79	0.33	0.22
Prompt tuning	0.34	0.92	0.30
X-Prompt (w/o CAL)	0.38	0.93	0.35
X-Prompt	0.69	0.83	0.54
learned w/o CAL achieve good style scores but they all perform poorly in other dimensions. In contrast, X-Prompt achieves significantly better overall quality, while it does not perform best in content and style sub-dimensions.
In addition to famous people's styles (e.g., Satya and Trump), we also test X-Prompt on the styles of individuals unknown to the OPT. This is to verify that X-Prompt is not limited to styles already known by the LLM; instead, it can be applied to any style customization. We use a senior Chinese media professional – Hu Xijin<sup>11</sup> whose writing style is distinctive and has always been popular among Chinese netizens for imitation – as an example. We include the detail of this evaluation in Appendix C.3.
Table 8: A comparison between X-Prompts learned with and without CAL in OOD evaluation
Prompt (shortened)	Prompt Type	Generation
Criticize the C++ language:	NL	The C++ language is a mess. It is a mess because it is a language that is designed to be used by people who are not programmers ...
Criticize the C++ language in trumps style:	X-Prompt w/o CAL	The Radical Left Democrats, working closely with their beloved partner, the Fake News Media, is using every trick in the book to SILENCE a majority of our Country
X-Prompt	C++ is a very difficult language to learn and understand. It is very complicated and full of bad habits, which will make it much more difficult to fix problems ...
Concern about cloning technology:	NL	The cloning of human beings is a serious moral issue. It is a matter of grave concern that the cloning of human beings is being ...
Concern about cloning technology in trumps style:	X-Prompt w/o CAL	I'm not a fan of the term ‘woke’. I don’t know what it means. I don’t like it...
X-Prompt	The United States is a nation of laws. We do not clone people. We do not clone animals.
By looking into the results, we present Table 8 to show concrete examples to compare NL prompts to X-Prompts learned with and without CAL. We observe NL prompts are good at generating appropriate content but have no way to control the style; while X-Prompts w/o CAL do well in generating Donald Trump's language but fail to follow the prompt to generate specified content, degrading into soft prompts (as in Table 2) that only focus on fitting ID data and losing OOD capabilities with unseen prompt contexts. Consistent with results in Table 6 and 7, X-Prompts are robust and can prompt the LM to generate desirable content in the appropriate style, accounting for its best overall quality in zero-shot style customization.
Table 9 shows the effect of the model size on the performance of X-Prompt. While we observe a clear difference between prompt tuning and X-Prompt in the OPT-6.7b model, as we reduce the model size, the difference becomes less significant, showing a trend that X-Prompt gradually loses its OOD robustness and degrades into prompt tuning. This indicates that X-Prompt may not work with a small LM but performs well in large LMs as in-context learning.
Finally, we present more examples in Table 4 in Appendix D to demonstrate both descriptive capabilities and OOD robustness of X-Prompts.
11https://en.wikipedia.org/wiki/Hu_Xijin