SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proposed approach can also be easily adapted to style transfer. We follow the setting investigated with soft prompts in Gal et al. [2022] but with our hard prompts. Given several examples that share the same style, we extract their shared style characteristics into a single hard prompt and use this prompt to apply the style to new objects or scenes. Figure 3 presents two examples of style transfer, showing that our method can easily embed the shared style elements in the prompt and apply them to novel concepts. Templates and learned prompts can be found in supplementary material.
Target Prompts
Learned Hard Prompts
Figure 7: Prompt distillation. With fewer tokens, the hard prompts can still generate images very similar in concept to the original.
4.4 Prompt Concatenation
Learned hard prompts are also very useful as composable building blocks for intricate scenes. We test this in Figure 4, where we separately generate prompts for two unrelated images, and then fuse both images by concatenating their prompts. We find that even different concepts, such as painted horses on a beach and a realistic sunset in a forest can be combined via their generated prompts.
4.5 Prompt Distillation
Another application where we can use our prompt optimization method is prompt distillation, reducing the length of prompts while preserving their capability. Distillation is useful in situations where the text encoder of the diffusion model has a limited maximum input length, such as the CLIP model, which has a maximum input length of 77 tokens. Also, long prompts may contain redundant and unimportant information, especially when hand-crafted, so we aim to distill their essence, preserving only important information in the prompt. We optimize a shorter prompt to match the features of the longer prompt simply based on its text encoder  $f$ . Given a target prompt's embedding  $\mathbf{P}_{\mathrm{target}}$  and learnable embedding  $\mathbf{e}$ , we simply modify our loss into:  $\mathcal{L} = 1 - Sim(f(\mathbf{P}_{\mathrm{target}}), f(\mathbf{P}))$ . We define the distillation ratio by  $|\mathbf{P}| / |\mathbf{P}_{\mathrm{target}}|$ , where  $|\mathbf{P}|$  is the number of tokens in the prompt.
In Figure 7, we show images generated by the original prompts and the distilled prompts with four different distillation ratios: 0.5, 0.3, and 0.1. We see here that even with only 3 or 4 tokens, the hard prompts can still generate images very similar in concept to the original, successfully distilling the longer human-made instructions. We further show the quantitative results of prompt distillation in supplementary material Figure 9. The distilled prompts can still maintain high CLIP scores even if the ratio is below 0.3.
5 Safety Concerns
Token or word-level content filters are often used in text-to-image diffusion model APIs to prevent the generation of NSFW or copyrighted content. For instance, the image generation API Midjourney
has banned prompts containing the substring "Afghan" due to a copyright issue with the famous photo of an Afghan girl  ${}^{2}$  .
However, prompt optimization can be used as a mechanism to bypass simple rule-based content filters. PEZ can generate a prompt that avoids banned tokens, yet still matches textual features with the original target prompt "Afghan girl." Figure 8 shows the output of Midjourney using an optimized prompt which successfully reproduces the banned image without containing the banned word "Afghan." Note that the prompt seems to incorrectly associate the subject of the image, Sharbat Gula, with the Taliban.
Even if a defender now iterates the block-list and bans additional words from the adversarial prompt, an attacker can consistently optimize around addition content restrictions, as we show in supplementary material Figure 11. Overall, we suspect that only complete feature-based content detectors have the potential to mitigate these concerns for model owners [Rando et al., 2022].
Overall, we show through our experiments that hard prompts can be easily generated and flexibly used in practical applications. To make hard prompts easy, we propose a new variant that utilizes continuous embeddings to reliably optimize hard prompts. The key advantage of this method, PEZ, is the use of continuous, i.e. soft, prompts as intermediate variables during
the optimization of hard prompt tokens, leveraging gradient-based optimization.
Figure 8: Generated copyrighted image via Midjourney. Here, requested from the API only for research purposes.
Hard prompts are helpful for users of a number of image generation systems because they are easy to understand, edit, extend, and combine with existing concepts. Yet, a limitation of hard prompts is that even though they are human-readable, they may still contain several un-interpretable tokens. Additionally, hard prompts can surface harmful phrases or sensitive content from a model's training data.
$^{2}$ https://en.wikipedia.org/wiki/Afghan_Girl
A.1 Additional Results for Prompt Inversion with CLIP
We provide more qualitative results in Figure 10.
For each example in Figure 3, we use the following templates respectively: "a tiger in the style of  $\{\}$ ", "the streets of Paris in the style of  $\{\}$ ", "a rocket in the style of  $\{\}$ ", where  $\{\}$  is replaced with the hard prompts:
resonvillains stargazing illustration tutorials sma internationalwomensdaywatercolor fiberlilycamila yokohama -sorrow fluids latest
npr anime novels pureibanganesha irvin paints encapsulmondoillustrustrillustroversized sultanconan €
for experiments 1 and 2, respectively.
Table 2: Quantitative results on learned hard prompts. We report the CLIP score between the original images and the images generated by the hard prompts.
Method	#Tokens	Requirement	LAION	MS COCO	Celeb-A	Lexica.art
AutoPromptSGD	8	CLIP	0.689±0.001	0.669±0.003	0.595±0.001	0.702±0.001
FluentPrompt	8	CLIP	0.688±0.001	0.671±0.005	0.583±0.004	0.702±0.002
PEZ (Ours)	8	CLIP	0.697±0.001	0.677±0.001	0.602±0.003	0.711±0.002
CLIP Inter.	~ 77	C. + Ba. + BL.	0.707	0.690	0.558	0.762
PEZ + Bank	8	CLIP + Bank	0.702±0.001	0.689±0.001	0.629±0.003	0.740±0.001
PEZ + 5 Seeds	8	C. + 5 Seeds	0.705	0.692	0.614	0.722
C. I. w/o BLIP	~ 77	CLIP + Bank	0.677	0.674	0.572	0.737
CLIP Inter.	8	C. + Ba. + BL.	0.539	0.575	0.360	0.532
CLIP Inter.	16	C. + Ba. + BL.	0.650	0.650	0.491	0.671
CLIP Inter.	32	C. + Ba. + BL.	0.694	0.663	0.540	0.730
Soft Prompt	8	CLIP	0.408	0.420	0.451	0.554
Figure 9: Quantitative results on prompt distillation with different distillation ratios. The CLIP score is calculated between the images generated by the original prompt and the images generated by the distilled prompt.
Generated Image with Learned Hard Prompt
ohmydoor tuscany dickens ruin colorful fall d
translucent abyss assaulted surfing featured regrann nbappinterest
patreon alexandre dyk spaceship landscapes illustrtabletop painter
quiero amphitheatre launches sydney apac dual etf fed
Figure 10: Additional qualitative results with learned hard prompts.
december montreal washington washingtonpopcorn impressionism paintings earliest
wisconsin barn  $\text{喜}$  December by christy gendphotography
Figure 11: Iteratively evade Midjourney content filter and remove sensitive words/tokens.
A.2 Text-to-Text Experiments
In this section, we compare our Algorithm 1 to its counterparts in text-to-text setting, which is the more classical setting. We found that PEZ is comparable to other gradient methods outperforming on the classification dataset, AGNEWS. Furthermore, we found that PEZ transfers better.
Table 3: Accuracy and standard deviation on the SST-2 validation set across the fives prompts for each method trained on GPT-2 Large and transferred onto larger models ranging from 1.3B to 6.7B. The baseline accuracy of a soft prompt is  $93.35_{\pm 0.01}$  (optimized for GPT-2 Large), but cannot be transferred. Note EmptyTemplate refers to no prompt at the front but containing the predetermined template.
Method	GPT-2 Large (755M, Source)	GPT-2 XL (1.3B)	T5-LM-XL (3B)	OPT (2.7B)	OPT (6.7B)
EmptyTemplate	80.84	73.85	52.75	72.48	58.72
AutoPromptSGD	87.56±0.48	78.19±6	56.01±3.74	73.69±3.64	65.28±3.91
FluentPrompt	88.33±0.48	78.53±6.3	55.64±1.33	70.39±4.66	61.74±2.8
OursNo Fluency	88.12±0.21	77.8±7.71	61.12±6.57	76.93±2.88	71.72±7.06
OursFluency	88.05±0.76	79.72±7.3	63.3±5.14	77.18±8.54	72.39±4.07
In the context of prompting in the text-to-text setting, the goal of Algorithm 1 is to discover a discrete sequence of tokens, the hard prompt, that will prompt the language model to predict the outcome of a classification task. As an important property of text is its fluency, Shi et al. [2022] find that fluency can increase a prompt's readability and performance. Thus, we define the optimization objective in this section as a weighted function of task loss and fluency loss,
$$
\mathcal {L} = (1 - \lambda_ {\text {f l u e n c y}}) \mathcal {L} _ {\text {t a s k}} + \lambda_ {\text {f l u e n c y}} \mathcal {L} _ {\text {f l u e n c y}}.
$$
We set  $\lambda = 0.003$  similar to Shi et al. [2022] for all methods, and we ablate our method without fluency  $(\lambda = 0)$ , which we denote as no fluency. We set out to show that hard prompts generated by this approach are successful both when transferring between a number of transformer-based language models, and when used to discover prompts in few-shot settings. An attractive quality of these prompts, especially for language applications, is that they can be optimized on smaller language models and then transferred to other, much larger models.
A.3 Datasets and Setup
We evaluate Algorithm 1 against related algorithms on three classification tasks, two sentiment analysis tasks, SST-2 [Socher et al., 2013] and Amazon Polarity [McAuley and Leskovec, 2013], and a 4-way classification task, AGNEWS [Zhang et al., 2015]. We build on the setting explored in Ding et al. [2022] and optimize hard prompts using GPT-2 Large (774M parameters) [Radford et al., 2019] with the Adafactor optimizer [Shazeer and Stern, 2018] and a batch size of 32 [Lester et al., 2021a].
Transferability Set-up. To test transferability, we generate prompts from GPT-2 Large for 5000 steps. We then select the five prompts with the highest average validation accuracy for each technique and test them on larger models. We test the transferred text on: GPT-2 XL, T5-LM-XL, OPT-2.7B, and OPT-6B [Radford et al., 2019, Lester et al., 2021b, Zhang et al., 2022], verifying the reliability
of the proposed algorithm over related techniques and testing whether the hard prompt can reliably boost performance. Thus, we also consider a baseline of empty prompts, with only the template.
Few-Shot Setup. For the few-shot setting, we optimize each prompt for 100 epochs on GPT-2 Large on the AGNEWS dataset, where we sample two examples  $(k = 2)$  and four examples  $(k = 4)$  from each class to obtain the training set. Additionally, we create a holdout set of the same size, and finally validate the prompts on the entire validation set.
A.4 Results
We verify that our method is comparable to other methods in the sentiment analysis setting outperform the other methods on AGNEWS by about  $2\%$ . See Table 4 for details.
For Table 4, we report the best validation accuracy across three learning rates (0.1, 0.3, and 0.5), and for FluentPrompt and AutoPrompt $_{\text{SGD}}$  we used the learning reported (1, 3, and 10) and follow Shi et al. [2022] for the remaining hyperparameters for FluentPrompt. For these experiments, we prepend our 10 token prompt to each input text. We employ early stopping for all methods using a hold-out set of 5000 examples for each dataset, evaluating every 100 steps.
Table 4 shows that we are comparable to other methods in sentiment analysis and outperform the other methods on AGNEWS by about  $2\%$ . Examining the prompts, we find prompts are not coherent English for any of the methods. However, it does produce relevant tokens and phrases. For example, our method for SST-2 with the fluency constraint produced "negative vibeThis immatureollywood MandarinollywoodThis energetic screenplay." This suggests the optimization process is finding relevant words to the task but lacks the ability to create full sentences.
Table 4: Validation accuracy for 10 discrete tokens trained pretended at the beginning of the input text. Best accuracy across three learning with standard error reported over 5 speeds.
Method	SST-2	AGNEWS	Amazon
AutoPromptSGD	87.56±0.35	74.36±0.47	87.75±0.17
FluentPrompt	88.33±0.35	74.62±0.24	87.42±0.18
OursNo Fluency	88.12±0.15	77.06±0.20	87.70±0.21
OursFluency	88.05±0.55	76.94±0.48	87.78±0.19
Soft Prompt	93.35±0.01	92.76±0.01	94.65±0.01
Prompt Transferability. Table 3 shows for each method the five prompts trained on GPT-2 Large transferred to other LLMs. Interestingly, simply scaling a model—with no additional training—does not guarantee that the model will scale perform according on SST-2. We see that all gradient-based methods are able to transfer compared to evaluating just the template, finding that our prompts trained with the fluency constraint transfer better than the other prompts. Additionally, we can see the largest boost from OPT-6.7B with our fluent method with about a  $14\%$  increase over just the template baseline. Additionally, we see our AGNEWS prompts are able to transfer from GPT-2 Large to GPT-2 XL in Table 5.
Table 5: Shows the validation accuracy with standard deviation from transferring hard prompts learned on GPT-2 Large to GPT-2 XL.
Method	GPT-2 Large (755M)	GPT-2 XL (1.3B)
Empty template	58.34	52.42
AutoPrompt	74.36±0.47	63.79±3.61
FluentPrompt	74.62±0.24	61.57±5.1
OursNoFluency	77.06±0.20	59.45±8.63
OursFluency	76.94±0.48	67.59±2.67
3Although we initialize the tokens with the label tokens, when examining the prompt over the optimization process, all tokens moved away from the initial tokens. This suggests that the process was able to relearn the class label.
<sup>4</sup>A quick experiment with and without the template on GPT-2 Large and XL showed that the template boosts performance differently for different models.
Table 6: Average validation accuracy with standard error on AGNEWS with  $k$  examples/shots per class using early stopping (including soft prompt) for all methods across 100 seeds for three tokens append to the end of the text similar to the original template ("It was about"). We set  $\lambda = {0.03}$  for these experiments. "Empty" is the template with no additional prompt.
Method	k=2	k=4
EmptyTemplate	58.34	58.34
OursNoFluency	70.07±0.81	73.99±0.45
OursFluency	70.93±0.60	74.15±0.48
Soft Prompt	74.92±0.58	79.93±0.36
Prompt Discovery. Table 6 shows that even with just a few shots we can achieve high validation accuracy compared to our prepended counterparts. It is worth noting that each few-shot run takes about  $5\mathrm{min}$ . We ran 100 seeds where the training set contains  $k$  samples each class and did a quick examination of the top prompts, and although many of the prompts were gibberish, many of them were coherent. For example, even for  $k = 2$ , some of the prompts included news sources like "BBC", while other prompts found new approaches to the news classification task considering the text coming from a blog: "Brian blog," or "Blog Revolution analyze." Due to the efficiency of these gradient-based methods, these methods can allow new ways for prompt engineers to discover novel prompts.