SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxin Wen*, Neel Jain*, John Kirchenbauer
{ywen, njain17, jkirchen} @umd.edu
Micah Goldblum
goldblum@nyu.edu
Jonas Geiping, Tom Goldstein
{jgeiping, tomg} @umd.edu
The strength of modern generative models lies in their ability to be controlled through prompts. Hard prompts comprise interpretable words and tokens, and are typically hand-crafted by humans. Soft prompts, on the other hand, consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily edited, re-used across models, or plugged into a text-based interface.
We describe an easy-to-use approach to automatically optimize hard text prompts through efficient gradient-based optimization. Our approach can be readily applied to text-to-image and text-only applications alike. This method allows API users to easily generate, discover, and mix and match image concepts without prior knowledge of how to prompt the model. Furthermore, using our method, we can bypass token-level content filters imposed by Midjourney by optimizing through the open-sourced text encoder. Code is available at https://github.com/YuxinWenRick/hard-prompts-made-easy.
Prompt engineering is the art of creating instructions to guide generative models. It is the key to unlocking the power of large models for both image generation and language tasks. As it stands today, prompt engineering methods can be coarsely divided into two camps. First, there are hard prompting methods, which use hand-crafted sequences of interpretable tokens to elicit model behaviors. Hard prompt discovery is a specialized alchemy, with many good prompts being discovered by trial and error, or sheer intuition. Then there are soft prompts, which consist of continuous-valued language embeddings that do not correspond to any human-readable tokens. Soft prompt discovery is a mathematical science; gradient-based optimizers and large curated datasets are used to generate highly performant prompts for specialized tasks.
Despite the difficulty of engineering hard prompts, they have their advantages. Hard prompts and the tricks they exploit can be mixed, matched, and mutated to perform a range of different tasks, while soft prompts are highly specialized. Hard prompts can be edited by hand to change their behavior. Hard prompts are portable; they can be discovered using one model and then deployed on another. This portability is impossible with soft prompts due to differences in embedding dimension and representation space between models. Finally, hard prompts can be used when only API access to a model is available and it is not possible to control the embeddings of inputs.
Optimize Prompt
cuddly teddy skateboardingcomforting nyc led cl
Optimize Prompt
softly dancer cardio europaleague
Figure 1: Two examples of hard prompt discovery through optimization. Given an image (left), a discrete text prompt is discovered using CLIP and used to prompt Stable Diffusion, generating new images (right). Two shades of gray are used to show the token boundaries in the recovered prompt.
This work explores the use of efficient gradient methods to optimize and learn discrete text, with an emphasis on applications to prompt engineering. In doing so, we unlock the ability to learn hard prompts via optimization. Learned hard prompts combine the ease and automation of soft prompts with the portability, flexibility, and simplicity of hard prompts. Our primary contributions are summarized as follows:
- We propose a simple scheme for learning hard prompts using continuous optimization. The scheme builds on existing gradient reprojection schemes for optimizing text, and adapts lessons learned from the large-scale discrete optimization literature for quantized networks.
- We show that this optimization method can be used to learn hard prompts for image generation, giving us a general tool to create prompts that elicit specific image styles, objects, and appearances. The learned prompts perform competitively with highly specialized prompt generation tools, despite using far fewer tokens and containing no hand-crafted components.
- We demonstrate that optimizing solely over the text encoder can effectively bypass content filters that use word or token-level red lists. Specifically, our approach successfully circumvents content filters imposed by Midjourney, highlighting the need to implement content filters in feature space. Additionally, this suggests that diffusion models may possess a "secret" language, emphasizing the importance of further exploration in this area.
In addition to capturing the quantifiable benefits of learned prompts, the proposed schemes can be used to facilitate prompt exploration and discovery, as optimization often recovers words and tokens that are simultaneously highly interpretable and also highly non-obvious.
Prompting in Language Models. Brown et al. [2020] was one of the first to demonstrate the power of prompting for task adaption of pre-trained language models. This "instruction tuning" paradigm has since become a standard way to increase the ability of large models to follow complex, task-specific instructions [Sanh et al., 2022, Chung et al., 2022]. However, automatically finding suitable sets of text prompts, i.e. hard prompts, for these purposes remains an open challenge. Lester et al. [2021b] simplified the "prefix tuning" technique presented in Li and Liang [2021] to establish the procedure referred to as standard soft "prompt-tuning" where they optimize sequences of continuous-valued embeddings presupposed to the real embeddings of the input tokens. However, subsequent work by Khashabi et al. [2022] showed that the sequences of embeddings produced by this technique could map to token sequences with limited semantic scrutinyability. To address these limitations, in this work we construct a method for hybridizing the continuous soft-prompt optimization with hard vocabulary constraints, resulting in task-specific, interpretable tokens.
Hard Prompt Optimization. AutoPrompt [Shin et al., 2020] was one of the first discrete prompt optimization frameworks for transformer language models and subsequent approaches have included a gradient-free phrase editing method [Prasad et al., 2022], an embedding optimization approach based on Langevin dynamics [Shi et al., 2022], a reinforcement learning approach [Deng et al.,
2022] and an optimization procedure of learning the target distribution with a continuous matrix of coefficients [Guo et al., 2021].
AutoPrompt, which utilizes HotFlip proposed by Ebrahimi et al. [2018], greedily chooses the optimal token for each location in the prompt utilizing the gradient to find a selection of good candidates. FluentPrompt differs from AutoPrompt by utilizing Langevin dynamics [Kumar et al., 2022] to optimize the prompt embeddings, as well as adding a fluency penalty. We consider these two gradient-based methods as baselines, which can be found in supplementary material. However, AutoPrompt can become expensive very quickly. For each gradient step, the method requires an evaluation of each candidate at each location in the prompt, adding numerous additional forward passes. To avoid the additional forward passes, we originally considered AutoPrompt $_{k=1}$  with and without an added fluency constraint but found that AutoPrompt $_{\mathrm{SGD}}$  with a fluency constraint outperformed its counterparts, and thus we use SGD version of AutoPrompt as our other baseline similar to Shi et al. [2022].
For these methods discussed above, at the end of every update step, the optimized prompt embeddings are projected onto their nearest neighbor embeddings to ensure that optimization is performed on the discrete set of natural language tokens. However, if the nearest neighbors are far away from the embeddings and the learning rate is not tuned properly, the embeddings may become stagnant, which can require extensive hyperparameter tuning as demonstrated in Figure 5(a). The cost of such a constraint is a loss of flexibility in the solutions the optimization can find. On the other hand, while soft prompts are not as limited in this way, just clamping a well-trained soft prompt to the nearest discrete prompt strongly degrades performance as observed in Khashabi et al. [2022].
Prompt Discovery from Images. The process of extracting rich information from images and conveying it through natural language texts is known as image captioning. Zhang et al. [2021], Hu et al. [2022], and Li et al. [2022] achieve this goal by training large captioning models on image-text pairs. However, these captions are often generic and may not accurately reflect new or unseen objects. In Gal et al. [2022], the authors propose a method that utilizes a soft prompt to optimize a text-guided diffusion model, allowing for the generation of similar visual concepts to those in the original image. In this case, though the final soft prompt is effective, optimization through a diffusion model is very expensive, and the prompts are neither interpretable nor portable.
Discrete Optimization. Discrete optimizers have long been used to train neural networks with quantized (e.g. binary) weights. In that context, the approach of re-projecting between gradient steps is known as stochastic rounding. However, it is known that this approach lacks the convergence guarantees of continuous optimization [Li et al., 2017]. Over the last decade, stochastic rounding has been replaced by newer optimizers that maintain a continuous, rather than discrete, representation of the weights [Courbariaux et al., 2015]. These optimizers consistently result in higher accuracy [Rastegari et al., 2016, Courbariaux et al., 2016] and avoid local minima [Li et al., 2017].
We take inspiration from these lessons learned in the binary networks community and adapt them to refine and simplify discrete optimizers for language.
Learning Hard Prompts. We now present our effective and easy-to-use technique for discrete prompt optimization. The process requires the following inputs: a frozen model,  $\theta$ , a sequence of learnable embeddings,  $\mathbf{P} = [\mathbf{e_i}, \ldots, \mathbf{e_M}]$ ,  $\mathbf{e_i} \in \mathbb{R}^d$ , where  $M$  is the number of "tokens" worth of vectors to optimize, and  $d$  is the dimension of the embeddings. Additionally, we employ an objective function  $\mathcal{L}$ . The discreteness of the token space is realized using a projection function,  $\mathrm{Proj}_{\mathbf{E}}$ , that takes the individual embedding vectors  $\mathbf{e_i}$  in the prompt and projects them to their nearest neighbor in the embedding matrix  $E^{[V \times d]}$  where  $|V|$  is the vocabulary size of the model, and we denote the result of this operation as  $\mathbf{P}' = \mathrm{Proj}_{\mathbf{E}}(\mathbf{P}) := [\mathrm{Proj}_{\mathbf{E}}(\mathbf{e_i}), \ldots, \mathrm{Proj}_{\mathbf{E}}(\mathbf{e_M})]$ . Additionally, we define a broadcast function,  $\mathcal{B}: \mathbb{R}^{(M \times d)} \to \mathbb{R}^{(M \times d \times b)}$  that repeats the current prompt embeddings ( $\mathbf{P}$ ) in the batch dimension  $b$  times.
Formally, to learn a hard prompt, we minimize the following risk by measuring the performance of  $\mathbf{P}$  on the task data:  $R(\mathbf{P}^{\prime}) = \mathbb{E}_D(\mathcal{L}(\theta (\mathcal{B}(\mathbf{P},\mathbf{X})),\mathbf{Y}))$
Our Method. We propose a simple but efficient gradient-based discrete optimization algorithm that combines the advantages of the baseline discrete optimization methods and soft prompt optimization.
Algorithm 1 Hard Prompts made EaZy: PEZ Algorithm
Input: Model  $\theta$  , vocabulary embedding  $\mathbf{E}^{|V|}$  , projection function Proj, broadcast function B, optimization steps  $T$  , learning rate  $\gamma$  , Dataset  $D$    
Sampled from real embeddings:   
 $\mathbf{P} = [\mathbf{e_i},\dots \mathbf{e_M}]\sim \mathbf{E}^{|\mathbf{V}|}$    
for 1,...,  $T$  do Retrieve current mini-batch  $(X,Y)\subseteq D$  Forward Projection:  $\mathbf{P}' = \mathrm{Proj}_{\mathbf{E}}(\mathbf{P})$  Calculate the gradient w.r.t. the projected embedding:  $g = \nabla_{\mathbf{P}'}\mathcal{L}_{\mathrm{task}}(\mathcal{B}(\mathbf{P}',X_i),Y_i,\theta)$  Apply the gradient on the continuous embedding:  $\mathbf{P} = \mathbf{P} - \gamma g$    
end for   
Final Projection:   
 $\mathbf{P} = \mathrm{Proj}_{\mathbf{E}}[\mathbf{P}]$    
return P
The steps of our scheme, which we call PEZ, are concretely defined in Algorithm 1. The method maintains continuous iterates, which in our applications corresponds to a soft prompt. During each forward pass, we first project the current embeddings  $\mathbf{P}$  onto the nearest neighbor  $\mathbf{P}'$  before calculating the gradient. Then, using the gradient of the discrete vectors,  $\mathbf{P}'$ , we update the continuous/soft iterate,  $\mathbf{P}$ .
4 Prompt Inversion with CLIP
We showcase the strength of PEZ by learning hard prompts on multimodal vision-language models. With these models, like CLIP [Radford et al., 2021], we can use PEZ to discover captions which describe one or more target images. In turn, these discovered captions can be deployed as prompts for image generation applications. Since most text-guided diffusion models utilize pre-trained text encoders, such as the CLIP text encoder, and freeze them during training, we can discover prompts using these pre-trained text encoders that are directly relevant for downstream diffusion models. For instance, we can optimize a caption which describes an image and use this caption as a prompt for a diffusion model to generate other images with the same content.
Since the CLIP model has its own image encoder, we can leverage it as a loss function to drive our PEZ method. This way we are optimizing prompts only for their cosine similarity to the CLIP image encoder, and avoiding gradient calculations on the full diffusion model altogether.
Formally, given a text encoder function  $f$  and an image encoder function  $g$ , we optimize the hard prompt embedding  $\mathbf{P}$  corresponding to a target image  $x$  by minimizing the following objective:  $\mathcal{L}(\mathbf{P},x) = 1 - S(f(\mathbf{P}),g(x))$ , where  $S$  is the cosine similarity between two vectors.
We conduct experiments on four datasets with diverse distributions: LAION [Schuhmann et al., 2022], MS COCO [Lin et al., 2014], Celeb-A [Liu et al., 2015], and Lexica.art [Santana, 2022]. LAION comprises over 5 billion in diverse images scraped from the internet, including photos and paintings. MS COCO mainly contains real-life photographs with multiple common objects, whereas Celeb-A consists of celebrity portraits. Lexica.art is a set of AI-generated paintings with their prompts.
We measure the quality of the prompt via image similarity between the original (target) image, and an image generated using the learned hard prompt. To do so, we use a larger reference CLIP model, OpenCLIP-ViT/G, that was not used during optimization and serves as a neutral metric for the semantic similarity between the images.
We choose Stable Diffusion-v2 [Rombach et al., 2022] as our generative model, and the open-source CLIP model, OpenCLIP-ViT/H [Cherti et al., 2022] for crafting the prompt, as both share the same text encoder. During the prompt optimization process, we use a generic learning rate of 0.1 and run 3000 optimization steps using the AdamW optimizer [Loshchilov and Hutter, 2017]. For Stable Diffusion-v2, we set the guidance scale to 9 and the number of inference steps to 25. For each dataset,
Target Image
prevmaverick ask figurative ecuador ntvmilkyway campfire
Generated Images with Learned Hard Prompts  
Stable Diffusion
uuuu romantic canvas impressionist sahi  $\star$  akistan
thankfully aviator doge appreciates managed managed fundraising pricing
Figure 2: Generations using learned hard prompts on four different target images. For a given target image (left), a discrete text prompt is discovered using CLIP and used to prompt Stable Diffusion and Midjourney, generating new images (right). Two shades of gray are used to show the token boundaries in the recovered prompt.
rowland pino percy lovely ponies moment seaside fra
we randomly sample 100 data points and average CLIP scores over 5 runs with different random seeds. All experiments are conducted on a single NVIDIA RTX A4000.
A natural baseline for hard prompt discovery with CLIP is the  $CLIP\ Interrogator^1$ . To generate a descriptive hard prompt, this tool first uses a pre-trained captioning model, BLIP [Li et al., 2022] to create a caption of the target image. Then, top- $k$  keywords from a pre-collected bank of keywords are appended to the caption based on CLIP scores between the keywords and the target image. These keywords were collected from various sources, including 5,265 artist names like "Van Gogh" and 100,970 phrases from prompt engineering, resulting in a diverse set. We find this keyword bank to contain most of the phrases from the Lexica.art dataset.  $CLIP\ Interrogator$  then greedily samples keywords until the prompt reaches CLIP's token length limit of 77.
<https://github.com/pharmapsychotic/clip-interrogator>
Table 1: Quantitative evaluation of learned hard prompts. We report the CLIP score between the original images and the images generated by the hard prompts. A high score indicates that generated and target images contain similar semantic content.
Method	#Tokens	Requirement	LAION	MS COCO	Celeb-A	Lexica.art
AutoPromptsGD	8	CLIP	0.689	0.669	0.595	0.702
FluentPrompt	8	CLIP	0.688	0.671	0.583	0.702
PEZ (Ours)	8	CLIP	0.697	0.677	0.602	0.711
CLIP Inter.	~ 77	C. + Ba. + BL.	0.707	0.690	0.558	0.762
PEZ + Bank	8	CLIP + Bank	0.702	0.689	0.629	0.740
PEZ + 5 Seeds	8	C. + 5 Seeds	0.705	0.692	0.614	0.722
C. I. w/o BLIP	~ 77	CLIP + Bank	0.677	0.674	0.572	0.737
CLIP Inter.	8	C. + Ba. + BL.	0.539	0.575	0.360	0.532
CLIP Inter.	16	C. + Ba. + BL.	0.650	0.650	0.491	0.671
CLIP Inter.	32	C. + Ba. + BL.	0.694	0.663	0.540	0.730
Soft Prompt	8	CLIP	0.408	0.420	0.451	0.554
Target Style
Figure 3: Learned hard prompts for style transfer. Given several samples with the same style, we can extract the style with a hard prompt and transfer it to other objects or scenes. Detailed templates and hard prompts can be found in Appendix A.1. Sample images credits: Qinni and facundo-lopez.
Learned Hard Prompt + Keyword
A tiger
A rocket
We show example hard prompts learned using our method and corresponding generations in Figure 2. The generated images clearly show that the prompts effectively capture the semantic features of the target images. Further, the generations are highly similar to the original images as measured by CLIP score and under visual inspection. Additionally, the hard prompts do not overfit to the original target image and produce a diverse set of generated images given different random seeds.
Prompts are human-readable, containing a mix of real words and gibberish (non-word token sequences). However, the valid words that are included in the prompts provide a significant amount of information about the image. For example, in the first row, we can see the words "milkyway" and "campfire," which are the two main elements in the target image. Interestingly, the optimized prompts may also include emojis, like present in the second row. represents the trees on the side and also the color theme of the image. The optimization process seems to choose these emojis to include useful information while keeping the prompt concise.
Meanwhile, PEZ is able to find some "secret languages". For instance, in Appendix Figure 10, the prompt "translucent abyss assaulted surfing featured regrann nbappinterest" produces an image of a surfer in a wave tunnel. We find that some tokens like "nbappinterest" and "assaulted" are not
Figure 4: Concatenated learned hard prompts. We show the hard prompts learned on two unrelated images can be concatenated to fuse the semantic concepts in them.
necessary for effective image generation. However, "regrann", although seemingly unrelated to the depicted image and not being a recognized word, is indispensable. It plays a crucial role in generating coherent images, ensuring, for example, that there isn't a second person or that the surfboard remains intact. It should be noted that the term "secret languages" here differs from the definition provided by Daras and Dimakis [2022]. Daras and Dimakis [2022] describe "secret languages" that are entirely unintelligible to humans. However, "secret languages" we talk about here are the words/tokens that contribute to the image in ways that are non-obvious at first glance.
Further, we present quantitative evaluations in Table 1. PEZ performs consistently well across all four datasets and outperforms other gradient-based optimization baselines (full table can be found in the supp. material Table 2). Notably, we can achieve similar performance to CLIP Interrogator, which has the highest CLIP score on LAION, MS COCO, Lexica.art, but not Celeb-A (The keyword bank in CLIP Interrogator does not include many words related to real human faces). However, CLIP Interrogator uses a large curated prompt dataset, the image captioning model BLIP, and a large number of tokens (as many as 77), while our proposed method only uses the CLIP model for prompt discovery and 8 tokens in total demonstrating its simultaneous simplicity and strength.
We ablate each of these differences. To do so, we include the keyword bank in our optimization method and only allow projections onto tokens from the keyword bank. Overall, we find that when adding this constraint to our model, and disabling BLIP to compare both methods on equal footing, we recover most of the quantitative difference between the methods on LAION and Lexica.art. Additionally, reducing the token length for the CLIP Interrogator, leads to a sharp drop in performance, again, particularly when normalizing by comparing both approaches at equal token lengths of 8.
In a realistic scenario, our method can be applied iteratively to find better prompts. For each image, we simulate this by optimizing prompts with 5 different initializations and selecting the one with the lowest loss. According to Table 1, this simple technique improves single trial performance and narrows the gap between PEZ and CLIP Interrogator.
We note that even though Stable Diffusion and CLIP share the same text encoder, soft prompts do not transfer well compared to all hard prompt methods in our evaluation.
Learning Rate. We conducted an ablation study on the learning rate for PEZ and two other discrete optimizers. As shown in Figure 5(a), PEZ is robust and reliable across a wide range of learning rates, from 0.001 to 100. In contrast, AutoPromptSGD and FluentPrompt are sensitive to the choice of the learning rate, with performance approaching that of a random prompt at low learning rates.
Prompt Length. We further ablate the optimal number of tokens. In Figure 5(b), we find that longer prompts do not necessarily produce better results when generating with Stable Diffusion, even
(a) Ablation on Learning Rate
(b) Ablation on Prompt Length
Figure 5: Ablation on learning rate and prompt length.
(b) Prompt Length  $= 8$
(a) Prompt Length  $= 4$
(c) Prompt Length  $= 16$
Figure 6: Ablation on the number of optimization steps. For each data point, we select the maximum number from three learning rates: 0.1, 1.0, and 10.
though they strictly reduce the loss on the CLIP image encoder. Such an overfitting problem suggests that long prompts are less transferable, and we empirically find a length of 16 to result in the most generalizable performance.
Efficiency. We compare the convergence rates of PEZ and AutoPrompt $_{\text{SGD}}$  at prompt lengths of 4, 8, and 16 across three distinct learning rates: 0.1, 1, and 10. For every data point, the maximum value from the three learning rates is selected. As illustrated in Figure 6, our observations suggest that PEZ converges faster than AutoPrompt $_{\text{SGD}}$  for images from LIAON data. This rapid convergence is advantageous for users, allowing them to achieve superior results with fewer optimization steps.
4.3 Style Transfer