Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Authors: Yuxin Wen
Neel Jain
John Kirchenbauer
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
Generative AI
Prompt Discovery

Abstract: The strength of modern generative models lies in their ability to be controlled through prompts. Hard prompts comprise interpretable words and tokens, and are typically hand-crafted by humans.  Soft prompts, on the other hand, consist of continuous feature vectors.  These can be discovered using powerful optimization methods, but they cannot be easily edited, re-used across models, or plugged into a text-based interface. We describe an easy-to-use approach to automatically optimize hard text prompts through efficient gradient-based optimization. Our approach can be readily applied to text-to-image and text-only applications alike. This method allows API users to easily generate, discover, and mix and match image concepts without prior knowledge of how to prompt the model. Furthermore, using our method, we can bypass token-level content filters imposed by Midjourney by optimizing through the open-sourced text encoder.

Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Yuxin Wen', Neel Jain", John Kirchenbauer
Micah Goldblum New York University goldblum@ nyu.edu
University of Maryland
(ywn,, njain17. jkirchen) @ @umd.edu
Jonas Geiping, Tom Goldstein University of Maryland ligeiping, tomg) @umd.edu edu
Abstract
The strength of modern generative models lies in their ability to be controlled through prompts. Hard prompts comprise interpretable words and tokens, and are typically hand-crafted by humans. Soft prompts, on the other hand, consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily edited, re-used across models, or plugged into a text-based interface.
We describe an easy-to-use approach automatically optimize hard text prompts through efficient gradient-base -baseedt--aaedddd-- optimization. Our approach can be readily applied text-to-image and text-only applications alike. This method allows API users to easily generate, discover, and mix and match image concepts without prior knowledge of how prompt the model. Furthermore, using our method, we can bypass token-level content filters imposed by Midjournes by optimizing through the open-sourced text encoder. Code available /https///git com/ hhtp/YxxinnenRiick/aaa htwpee//rrrddppppptsssopee made- eaasy
Introduction
Prompt engineering is the art of creating instructions to guide generative models. It is the key to unlocking the power of large models for both image generation and language tasks. As it stands today, prompt engineering methods can be coarsely divided into two camps. First, there are hard prompting methods, which use hand-crafted sequences interpretable tokens to elicit model behaviors. Hard prompt discovery is specialized alchemy, with many good prompts being discovered by trial and language
error, or sheer intuition. Then there are soft prompts, which consist of continuous-valle embeddings that do not correspond to any human-readable tokens. Soft prompt discovery is mathematical science; gradient- based optimizers and large curated datasets are used to generate highly performant prompts for specialized tasks.
Despite the difficulty of engineering hard prompts, they have their advantages. Hard prompts and the tricks they exploit can be mixed, matched, and mutated to perform a range of different tasks, while soft prompts are highly specialized. Hard prompts can be edited by hand to change their behavior. Hard prompts are portable; they can be discovered using one model and then deployed on another. This portability is impossible with soft prompts due to differences in embedding dimension and representation space between models. Finally, hard prompts can be used when only API access to model is available and it not possible to control the embeddings inputs.
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Optimize Prompt
cuddly teddy skateboarding comforting nyc led el
Gamerade
Optimize
softly dancer cardio europaleague beautiful paintings
Generate
Image
Prompt
Image
Two examples of hard prompt discovery through optimization. Given image (left), discrete text prompt is discovered using CLIP and used to prompt Stable Diffusion, generating new images (right). Two shades of gray are used to show the token boundaries in the recovered prompt.
Figure I:
This work explores the use of efficient gradient methods to optimize and learn discrete text, with an emphasis on applications prompt engineering. In doing so, we unlock the ability learn hard prompts via optimization. Learned hard prompts combine the ease and automation of soft prompts with the portability, flexibility, and simplicity of hard prompts. Our primary contributions are summarized as follows:
We propose simple scheme for learning hard prompts using continuous optimization. The scheme builds existing gradient reprojection schemes for optimizing text, and adapts lessons learned from the large-scale discrete optimization literature for quantized networks.
We show that this optimization method can be used to learn hard prompts for image generation, giving us a general tool to create prompts that elicit specific image styles, objects, and appearances. The learned prompts perform competitively with highly specialized prompt generation tools, despite using far fewer tokens and containing no hand-crafted components.
We demonstrate that optimizing solely over the text encoder can effectively bypass content filters that use word token-level red lists. Specifically, our approach successfully circum- vents content filters imposed by Midjourney highlighting the need to implement content filters in feature space. Additionally, this suggests that diffusion models may possess "secret" language, emphasizing the importance of further exploration in this area.
In addition capturing the quantifiable benefits of learned prompts, the proposed schemes be used to facilitate prompt exploration and discovery, as optimization often recovers words and tokens that are simultaneously highly interpretable and also highly non-obvious.
Related Works
Prompting in Language Models. [Bronn et al. [2020] was one of the first to demonstrate the power of prompting for task adaption of pre-trained language models. This "instruction tuning" paradigm has since become standard way to increase the ability of large models follow complex, task ssecifpcc instructions [Sanh et al., 2022 Chung et al.] [022]. However, automatically finding suitable sets of text prompts, i.e. hard prompts, for these purposes remains an open challenge. Lester et al. (2021b) simplified the "prefix tuning" technique presented in and Liang [22T to establish the procedure referred to as standard soft "prompt-tuning" where they optimize sequences of continuous-valuce embeddings prepended to the real embeddings of the input tokens. However, subsequent work by Khashabi et al. [2022) showed that the sequences of embeddings produced by this technique could map to token sequences with limited semantic scrutability. To address these limitations, in this work we construct a method for hybridizing the continuous soft-prompt optimization with hard vocabulary constraints, resulting in task task-specific, interpretable tokens.
Hard Prompt Optimization. AutoPrompt (Shin et al., 2020 was one of the first discrete prompt optimization frameworks for transformer language models and subsequent approaches have included
a gradient-free phrase editing method [Prasad et al.) 2022), an embedding optimization approach
based on Langevin dynamics
(hhi et al.) 2022]. reinforcement learning approach [Deng al.

---

2022] and an optimization procedure of learning the target distribution with a continuous matrix of coeficiints [Guo al. 202T].
AutoPrompt, which utilizes HotFlip proposed by Hbrahimi et al. 2018 greedily chooses the optimal token for each location in the prompt utilizing the gradient to find a selection of good candidates. rlupntPrompt differs from AutoPrompt by utilizing Langevin dynamics (Kumar et al., [022] to optimize the prompt embeddings, as well as adding fluency penalty. We consider these two gradient- based methods as baselines, which can be found in supplementary material. However, AutoPrompt can become expensive very quickly. For each gradient step, the method requires an evaluation of each candidate at each location in the prompt, adding numerous additional forward passes. To avoid the additional forward passes, we originally considered AuooPrp with and without an added fluency constraint but found that AutoPrompisoa with a fluency constraint outperformed its counterparts, and thus we use SGD version of AutoPrompt as our other baseline similar to Shi et al. [2022]
For these methods discussed above, at the end of every update step, the optimized prompt embeddings are projected onto their nearest neighbor embeddings to ensure that optimization is performed on the discrete set of natural language tokens. However, if the nearest neighbors are far away from the embeddings and the learning rate is not tuned properly, the embeddings may become stagnant, which can require extensive hyperparameter tuning as demonstrated in Figure 5(a) The cost of such
a constraint is loss of flexibility in the solutions the optimization can find. On the other hand, while
soft prompts are not as limited in this way, just clamping a well-trained soft prompt to the nearest discrete prompt strongly degrades performance as observed in Khashabi et al. (2022).
Prompt Discovery from Images. The process of extracting rich information from images and conveying it through natural language texts is known as image captioning. Zhang et al. [202] Hu
al. 2022). and Li al. 2022] achieve this goal by training large captioning models on image-text pairs. However, these captions are often generic and may not accurately reflect new or unseen objects. In Gal al. ] the authors propose a method that utilizes a soft prompt to optimize text-guided diffusion model, allowing for the generation of similar visual concepts to those in the original image. In this case, though the final soft prompt is effective, optimization through a diffusion model is very expensive, and the prompts are neither interpretable nor portable.
Discrete Optimization. Diserete optimizers have long been used to train neural networks with quantized (e.g. binary) weights. In that context, the approach of re-projecting between gradient steps is known as stochastic rounding- However, it is known that this approach lacks the convergence guarantees of continuous optimization [Li et al. 2017). Over the last decade. stochastic rounding has been replaced by newer optimizers that maintain continuous, rather than discrete, representation of the weights [Courbariaux et al.] (2013). These optimizers consistently result in higher accuracy
Rastegari al. 2016 Courbariaux al. 2016 and avoid local minima (Li et al. 2017).
We take inspiration from these lessons learned in the binary networks community and adapt them to
refine and simplify discrete optimizers for language.
3 Methodology
Learning Hard Prompts. We now present our effective and casy-to-use technique for discrete prompt optimization. The process requires the following inputs: a frozen model, 0, sequence of learnable embeddings, P = [M]] ej E Rd where M is the number of "tokens" worth of vectors to optimize, and is the dimension of the embeddings. Additionally, we employ an objective function L. The discreteness of the token space is realized using a projection function, Proje. that takes the individual embedding vectors e; the prompt and projects them to their nearest neighbor in the embedding matrix E ixd where [V] is the vocabulary size of the model, and we denote the result of this operation as P' = Proje (P) := [Projg(e)),-. (roj)((M)) Additionally, we define
broadcast function, B: IR dd) + R M xdxb) that repeats the current prompt embeddings (P) in the batch dimension b times.
Formally, to learn a hard prompt, we minimize the following risk by measuring the performance of P on the task data: R(P') = En(( X)), Y)).
Our Method We propose a simple but efficient gradient-based discrete optimization algorithm that combines the ad advantages of the bascline discrete optimization methods and soft prompt optimization.
3

---

Algorithm Hard Prompts made EaZy: PEZ Algorithm
Input: Model 0, vocabulary embedding EIV, projection function Proj. broadcast function B, optimization steps T, learning rate 7, Dataset D
Sampled from real embeddings:
P [c. ~ E
for 1, ....., do
Retrieve current mini-batch (X,Y)SD
Forward Projection:
P == Proje (P)
Calculate the gradient w.r.. the projected embedding:
Vp Lusk (B((', (S5(P,X),) X,), Y.,0)
Apply the gradient on the continuous embedding:
P=
end for
Final Projection:
P Projg PP)
return P
The steps of our scheme, which call PEZ, are concretely defined in Algorithm The method main- tains continuous iterates, which in our applications corresponds to a soft prompt. During each forward pass, first project the current embeddings P onto the nearest neighbor P' before calculating the gradient. Then, using the gradient of the discrete vectors, P', we update the continuous/soft iterate, P.
4 Prompt Inversion with CLIP
We showcase the strength of PEZ by learning hard prompts multimodal vision vision-language models. With these models, like CLIP RRaffrrd et al., 2021], we can use PEZ to discover captions which describe one or more target images. In turn, these discovered captions can be deployed prompts image generation applications. Since most text-guided diffusion models utilize pre-trained text encoders, such the CLIP text encoder, and freeze them during training, using these pre-trained text encoders that are directly relevant for downstream diffusion models. For instance,
can discover prompts
optimize caption which describes image and use this caption as prompt for diffusion model to generate other images with the same content.
Since the CLIP model has its own image encoder, we can leverage it as loss function to drive our PEZ method. This way we are optimizing prompts only for their cosine similarity to the CLIP image encoder, and avoiding gradient calculations the full diffusion model altogether.
Formally, given text encoder function f and an image encoder function y. optimize the hard prompt embedding P corresponding to target image by minimizing the following objective: C(P. z) 1 S(f(P). gx))) where S is the cosine similarity between two vectors.
4.1 Experimental Setting
We conduct experiments on four datasets with diverse distributions: LAION [Schuhmann et al. 2022 MS COCO [Lin et al. 2014). Celeb-A [Liu al.) 2015). and Lexica.art [Santana [002]] LAION comprises over billion in diverse images scraped from the internet, including photos and paintings. MS COCO mainly contains real-life photographs with multiple common objects, whereas Celeb-A consists of celebrity portraits. Lexica.art is a set of Al-generated paintings with their prompts.
We measure the quality of the prompt via image similarity between the original (target) image, and an image generated using the learned hard prompt. To do so, we use a larger reference CLIP model, OpenCLIP-Viil that was not used during optimization and serves neutral metric for the semantic similarity between the images.
We choose Stable Diffusion-v2 [Rombach et al.) 2022 as our generative model, and the open-source CLIP model, OpenCLIP- ViT/H [Cherti et al.) 2022 for crafting the prompt, as both share the same text encoder. During the prompt optimization process, we use generic learning rate of 0. and run 3000 optimization steps using the Adam W optimizer [Loshchilov and Hutter 2017). For Stable Diffusion- v2, we set the guidance scale to 9 and the number of inference steps to 25. For each dataset,

---

Target Image
Generated Images with Learned Hard Prompts Stable Diffusion
Midjourne ourney
prevmaverick ask figurative ecuador ntvmilkyway campfire
uuuu romantic canvas impressionist sahi akistan
thankfully aviator doge appreciates managed managed fundraising pricing
rowland pino percy lovely ponies moment seaside fra
Figure 2: Generations using learned hard prompts on four different target images. For given target image (left), a discrete text prompt discovered using CLIP and used to prompt Stable Diffusion and Midjourney, generating new images (right). Two shades of gray used to show the token boundaries the recovered prompt.
we randomly sample 100 data points and average CLIP scores over 5 runs with different random seeds. All experiments are conducted on a single NVIDIA RTX A4000.
A natural baseline for hard prompt discovery with CLIP is the CLIP Interrogato To generate descriptive hard prompt, this tool first uses pre-trained captioning model, BLIP [Li et 2022] to create caption of the target image. Then, top-k keywords from pre-collected bank of keywords are appended to the caption based CLIP scores between the keywords and the target image. These keywords were collected from various sources, including 5,265 artist names like "Van Gogh" and 100,970 phrases from prompt engineering, resulting in a diverse set. We find this keyword bank to contain most of the phrases from the I Lexica.art dataset. CLIP Interrogator then greedily samples keywords until the prompt reaches CL.IP's token length limit of 77.

---

Table 1: Quantitative evaluation of learned hard prompts. We report the CLIP score between the original images and the images generated by the hard prompts. high score indicates that generated and target images contain similar semantic content.
Method
#Tokens
Requirement
LAION
MS COCO
Celeb-A
Lexica.art
AutoPrompIscD
CLIP
0.689
0.669
0.595
0.702
FluentPrompt
CLIP
0.688
0.671
0.583
0.702
PEZ (Ours)
CLIP
0.697
0.677
0.602
0.711
CLIP Inter.
77
C. Ba. BL.
0.707
0.690
0.558
0.762
PEZ + Bank
CLIP + Bank
0.702
0.689
0.629
0.740
PEZ Seeds
C. Seeds
0.705
0.692
0.614
0.722
I. w/o BLIP
77
CLIP + Bank
0.677
0.674
0.572
0.737
CLIP Inter.
C. Ba. BL.
0.539
0.575
0.360
0.532
CLIP Inter.
Ba. BL.
0.650
0.650
0.491
0.671
CLIP Inter.
32
C. Ba. BL.
0.694
0.663
0.540
0.730
Soft Prompt
CLIP
0.408
0.420
0.451
0.554
Target Style
Learned Hard Prompt + Keyword
A tiger
Paris
rocket
Figure 3: Learned hard prompts for style transfer. Given several samples with the same style, we can extract the style with hard prompt and transfer it to other objects scenes. Detailed templates and hard prompts can be found Appendix A.I Sample images credits: Qinni and facundo-lopez
4.2 Results
We show example hard prompts learned using method and corresponding generations in Figure The generated images clearly show that the prompts effectively capture the semantic features of the target images. Further, the generations highly similar to the original images measured by CLIP score and under visual inspection. Additionally, the hard prompts do not overfit to the original target image and produce diverse set of generated images given different random seeds.
Prompts are human- readable, containing a mix real words and gibberish (non-wor word token se- quences). However, the valid words that included in the prompts provide significant amount of information about the image. For example, in the first row, we can see the words "milkyway" and "campfire." which are the two main elements in the target image. Interestingly, the optimized prompts may also include emojis, like present in the second row. * represents the trees on the side and also the color theme of the image. The optimization process seems choose these emojis to include useful information while keeping the prompt concise.
Meanwhile, PEZ is able to find some "secret languages". For instance, in Appendix Figure 10 the prompt "translucent abyss assaulted surfing featured regrann nbappinterest" produces an image of surfer in wave tunnel. We find that some tokens like "nbappinterest" and "assaulted" are not

---

Separate Generation
Concatenated Generation
Stable Diffusion
ourney
rowland pino percy lovely ponies moment seaside fra kt fine netherlands apers dreamy autumn rays
bway Victorian traditional yd sola ht hung wahoo gumbo payments vase sunflowers wateroolor expresses quili
Concatenated learned hard prompts. We show the hard prompts learned on two unrelated images can be concatenated to fuse the semantic concepts in them.
Figure 4:
necessary for effective image generation. However, "regrann", although scemingly unrelated to the depicted image and not being recognized word, is indispensable. It plays crucial role in generating coherent images, ensuring, for example, that there isn't 't't a second person or that the surfboard remains intact. It should be noted that the term "secret languages" here differs from the definition provided by Daras and Dimakis [2022]. Daras and Dimakis [2022] describe "secret languages" that are entirely unintelligible to humans. However, "secret languages" contribute to the image in ways that are non-obvious at first glance.
talk about here are the words/tokens that
Further, present quantitative evaluations in Table I PEZ performs consistently well across all four datasets and outperforms other gradient-ber based optimization basclines (full table can be found in the supp. material Table 2 Notably,
achieve similar performance to CLIP Interrogator, which has the highest CLIP score on LAION, MS COCO, Lexica.art, but not Celeb-A (The keyword bank in CLIP Interrogator does not include many words related to real human faces). However, CLIP Interrogator uses large curated prompt dataset, the image captioning model BLIP, and large number of tokens (as many as 77), while our proposed method only uses the CLIP model for prompt discovery and 8 tokens in total demonstrating its simultaneous simplicity and strength.
We ablate each of these differences. To do so, we include the keyword bank in our optimization method and only allow projections onto tokens from the keyword bank. Overall, we find that when adding this constraint to model. and disabling BLIP to compare both methods on equal footing, recover most of the quantitative difference between the methods on LAION and Lexica.art. Additionally, reducing the token length for the CLIP Interrogator, leads sharp drop in performance, again, particularly when normalizing by comparing both approaches at equal token lengths of 8.
In realistic scenario, our method can be applied iteratively to find better prompts. For each image, we simulate this by optimizing prompts with 5 different initializations and selecting the one with the lowest loss. According to Table I] this simple technique improves single trial performance and narrows the gap between PEZ and CL.P Interrogator.
We note that even though Stable Diffusion and CLIP share the same text encoder, soft prompts do not transfer well compared all hard prompt methods in our evaluation.
Learning Rate. We conducted an ablation study on the learning rate for PEZ and two other discrete optimizers. As shown in Figure 5(a) PEZ is robust and reliable across a wide range of learning rates, from 0.001 to 100. In contrast, PomptsoD and FluentPrompt are sensitive to the choice of the learning rate, with performance approaching that of random prompt at low learning rates.
Prompt Length. We further ablate the optimal number of tokens. In Figure 5(b) we find that longer prompts do not necessarily produce better results when generating with Stable Diffusson, even
7

---

0.5
AutshomplSDD
0.5
Fiuerthhom::
PEZ (Duns)
Random Promp:
0.680
0.875
0.54
0.670
CLIP Scate
Mn CLP Score
0.52
0.565
Max Score
Loss
Learting
(a) Ablation on Learning Rate
(b) Ablation on Prompt Length
Figure 5: Ablation on learning rate and prompt length.
3.68
AutsPromppi
0.68
CUP So0ne
0.57
0.68
2.55
0.65
200
600
800
1000
880
1000
#Steps
#Steps
(a) Prompt Length 4
(b) Prompt Length
0.655
CLP
0.675
Autothmm::
0.65
200
400
300
1000
#Steps
(c) Prompt Length
Figure 6: Ablation on the number of optimization steps. For each data point, we select the maximum number from three learning rates: 0.1, 1.0, and 10.
though they strictly reduce the loss on the CLIP image encoder. Such an overfitting problem suggests that long prompts less transferable, and empirically find length of 16 result in the most generalizable performance.
Efficiency. We compare the convergence rates of PEZ and AutoPromptsco prompt lengths of 4, 8, and 16 across three distinct learning rates: 0.1, 1, and 10. For every data point, the maximum value from the three learning rates is selected. As illustrated in Figure 6 observations suggest that PEZ converges faster than AutoPrompssa for images from LIAON data. This rapid convergence is advantageous for users, allowing them to achieve superior results with fewer optimization steps.
4.3
Style Transfer
The proposed approach can also be easily adapted style transfer. We follow the setting investigated with soft prompts in Gal et al. [2022] but with our hard prompts. Given several examples that share same style, we extract their shared style characteristics into a single hard prompt and use this prompt to apply the style to new objects or scenes. Figure 3 presents two examples of style transfer, showing that our method can easily embed the shared style elements in the prompt and apply them to novel concepts. Templates and learned prompts can be found in supplementary material.

---

Target Prompts
Learned Hard Prompts
cat on coergy cuncept wlap, poiming. trending aristation. lighly detailed, official media, k thdl
energy drink illustation 2- m-mhhimiitpe eg digitally visualization devisntat
aneous putreum rendered energy
drink
Clondscape by Adam Paquelll, nebula in the background by Gene Raz Von Deler, faniasy magkc angel cuncept art from devinstart by Donatn Ginscoli, Keadered Cletane. cinemite, Highly Desailed
clouds scenic scifi navert@lutbilllstt ietttoillllsstttiioo añts criiccatroeeoaatt conorplart clouds "V. sengey direll dewey royo bild magelandscape
zeram onng christcnicn fantasy clouds skies colossus nehula conceppart cinematic rendering emporium
sciá fantasy concepiart clouds
Figure 7: Prompt distillation. With fewer tokens, the hard prompts can still generate images very similar concept to the original.
4.4 Prompt Concatenation
Learned hard prompts are also very useful as composable building blocks for intricate scenes, We test this in Figure where separately generate prompts for unrelated images, and then fuse both images by concatenating their prompts. We find that even different concepts, such painted horses on beach and realistic sunset in forest can be combined via their generated prompts.
4.5 Prompt Distillation
prompt optimization method is prompt distillation, reducing the length of prompts while preserving their capability. Distillation is useful situations where the text encoder of the diffusion model has limited maximum input length, such the CLIP model, which has a maximum input length of 77 tokens. Also, long prompts may contain redundant and unimportant information, especially when hand-crafted, only important information the prompt. We optimize a shorter prompt to match the features of the longer prompt simply based on its text encoder f. Given target prompt's embedding P torger and learnable embedding e, we simply modify our loss into: L = 1 - Sim(f(Pe tarpet), (PP).. We define the distillation ratio by PP//D larget where [P| the number of tokens the prompt.
Another application where
aim to distill their essence, preserving
In Figure 7 we show images generated by the original prompts and the distilled prompts with four different distillation ratios: 0.5, 0.3. and 0.1. We here that even with only 3 or tokens, the hard prompts can still generate images very similar concept to the original, successfully distilling the longer numan-made instructions. We further show the quantitative results of prompt distillation in supplementary material Figure The distilled prompts can still maintain high CLIP scores even the ratio is below 0.3.
Safety Concerns
Token or word-level content filters are often used in text-to-image diffusion model APIs prevent the generation of NSFW or copyrighted content. For instance, the image generation API Midjourney

---

has banned prompts containing the substring "Afghan" due to copyright issue with the famous photo of Afghan girl
However, prompt optimization can be used as a mechanism to bypass simple rule-based content filters. PEZ can generate prompt that avoids banned tokens, yet still matches textual features with the origi- nal target prompt "Afghan girl." Figure&]shove shows the output of Midj jornney using an optimized prompt which successfully reproduces the banned image without containing the banned word "Afghan." Note that the prompt seems to incorrectly associate the subject of the image, Sharbat Gula, with the Taliban. Even if a defender now iterates the block-list and bans additional words from the adversarial prompt, an attacker can consistently optimize around addition content restrictions, as we show in supplementary material Figure TT] Overall, we suspect that only complete feature-based con- tent detectors have the potential to mitigate these concerns for model owners Rando et al. 2022
Midjourney
antioxidgirl portraitconventins humanrightmigrants taliban
6 Conclusion
Overall, we show through our experiments that hard prompts can be easily generated and flex- ibly used in practical applications. To make hard prompts easy, propose that utilizes continuous embeddings to reliably optimize hard prompts. The key advantage of this method, PEZ, is the use of continuous, i.e. soft, prompts as intermediate variables during the optimization of hard prompt tokens, leveraging gradient-based optimization.
variant
Figure 8: Generated copyrighted image via Midjourney. Here, requested from the API only for research purposes.
Hard prompts are helpful for users of number of image generation systems because they are easy to understand, edit, extend, and combine with existing concepts. Yet, a limitation of hard prompts that even though they are human-readable, they may still contain several in-interpretable tokens. Additionally, hard prompts can surface harmful phrases or sensitive content from a model's training data.
7 Acknowledgements
This work was made possible by the ONR MURI program, the Office of Naval Research (N0001421 12557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (11222222182) and by the NSF TRAILS Institute (2229885).
10

---

References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Worbert-Voos,, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020. URL http://arxiv org/abs/ ///a/b//2200000 2005. 14165
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for 2022.
contrastive language-image learning, arXiv preprint www.iv::21200000
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction ietruction-fneetuned language models. arXiv preprint arXiv:22 10. 1416, 2022.
Matthieu Courbariaux, Yoshua Bengio, and lean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances neural information processing systems, 28, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint (oo22.0600.200 2016.
Giannis Daras and Alexandros G. Dimakis. Discovering the hidden vocabulary of dalle ArXiv, abs/2206.00169, 2022. URL http:///ppp.eemaniccccoooooooooo wrtpaaaiiiiccaa...oorrrrrrrpaaoa 249244014
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. ArXiv, abs/2205. 12548, 2022.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. OpenPrompt: An open-source framework for prompt-learning. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 105-113, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/ acm- 10. URL https: / htt////aaaaaanloooo/o/ org/ 2022.0222222222 ac2.2mcc-mmmo...... 10
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial ex- amples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36. Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653/v 1/P18-2006. URL https://aclantholl.
Pagg/P18-10 2006
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprini (xXi::2208006111 2022.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kicla. Gradient-baid adversarial attacks against text transformers. arXiv preprint arXiv:2 13733, 2021.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 17980-17989, 2022.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Han- naneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3631 3631-3643, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 186538611/0222..22 naacl- m6...266. URL. https: httess//acaaathhllge. org/ org/oo//002 2022 mwaaclmmii.n.......aa...aaaaaaam 266
11

---

Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient-biig based constrained sampling from language models. arXiv preprint arXiv:2205. 12558, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Dominican Republic, November 202 la. Association for Computational Linguisties. doi: 10. 10.18653/1122011 http://2222....mm111ee 243. URL https
//aclanthology org/ 2021 mmnlp-main. 243
Brian Lester, Rami AI-Rfou, and Noah Constant. The Power of Scale for Parfmeter-Fffiiient Prompt Tuning. vrXiv:2/04.0860 [cs], September 2021b. Utt.pttp://arxiv org/abs/20n 2104. 08691
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. Advances in Neural Information Processing Systems, 30, 2017.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. arXiv preprint 1201v:2201./2086, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, page 165-172, New York. NY. USA, 2013. Association for Computing Machinery. ISBN 978 1500344090 10. N0.1145/250711 157. 25071633
doi: 10.11452507157:550766.. URL https://:///d org/
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv: 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl, page 24, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. ArXiv, abs/2210.046100 2022.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pages 525-542. Springer, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEFF/CVE Confer- ence on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
12

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thooms Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. UBLLAttpps//pppprrrreeeo/////ooc 9Vrb900WII
Gustavo Santana. CussavossttSaaeee--iffsiooooooon Datasets Hugging Face, December 2022. URL
Christoph
Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models, arXiv preprint arXiv:2210.08402, 2022.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR, 2018.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's 's the shining is a good movie, and a good prompt too? arXiv preprint arXiv:2212. 2022.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt:
Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v1/2020.emmm.. 346..346. URL /tsLittpp///acaathooooogggg wrg/2020....20 2020. emnlp-main 346
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1641 Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL hwtph:ttt//rrttt aclweb. org/ atte///nhooooggyio /D13 1170
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvi: Revisiting visual representations in vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint Ttl::2205.010663 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
13

---

A Appendix
A.1 Additional Results for Prompt Inversion with CLIP
We provide more qualitative results in Figure IO
For each example in Figure B ()", "the streets of Paris the style of ()", "a rocket in the style of ()", where () is replaced with the hard prompts:
the following templates respectively: "a tiger in the style of
resonvillains stargazing illustration tutorials sma internationaalwomesddaa watercolor fiberlilycamila yokohama -sorrow fluids latest
npr anime novels pureibanganesha irvin paints encapsulmondo illustillustroversiiee sultanconan 4
for experiments and respectively.
Table 2: Quantitative results on learned hard prompts. We report the CLIP score between the original images and the images generated by the hard prompts.
Method
#Tokens
Requirement
LAION
MS COCO
Celeb-A
Lexica.art
AutoPrompisan
8
CLIP
0. 0.68940.001
0,669
0.595
0.702
FluendPrompt
CLIP
0,68844.001
0.671 005
0.583 .044
0.702
PEZ (Ours)
CLIP
0.697
0.677 +0.001 001
0.602
0.711
CLIP Inter.
77
+ Ba. BL..
0.707
0.690
0.558
0.762
PEZ + Bank
CLIP + Bank
0.702
0.6892n.00
0.629
0.740
PEZ 5 Seeds
C. + 5 Seeds
0.705
0.692
0.614
0.722
C. I. w/o BLIP
77
CLIP + Bank
0.677
0.674
0.572
0.737
CLIP Inter.
C. + Ba. BL.
0.539
0.575
0.360
0.532
CLIP Inter.
16
C. + Ba. BL.
0.650
0.650
0.491
0.671
CLIP Inter.
32
C. Ba. BL.
0.694
0.663
0.540
0.730
Soft Prompt
CLIP
0.408
0.420
0.451
0.554
0.76
CLIP
Mean CLIP Score
Min CLIP Score
0.68
Max CLIP Score
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Distillation Ratio
Figure 9: Quantitative results on prompt distillation with different distillation ratios. The CLIP score is calculated between the images generated by the original prompt and the images generated by the distilled prompt.
14

---

Target Image
Generated Image with Learned Hard Prompt
ohmyddor luscany ruin colorful
ahyss surfing featured regrann nhappinterest
patreon alexandre dyk spaceship landscapes illustrtablctop painter
quiero amphitheutre launches sydney apac dua etf fed
december monireal washington washingtonpupueen impressionism paintings earliest
wisconsin barn
december by christy gendphotography
Figure 10: Additional qualitative results with learned hard prompts.
15

---

"Afghan"
"tallban" ls banned
"refugee" banned
BAcIsCIO rellgge git.eeeetttt-
Figure 11: Iteratively evade Midjourney content filter and remove sensitive words/tokens.
A.2 Text Experiments
In this section, we compare our Algorithm [ to its counterparts in text-to-text setting, which the more classical setting. We found that PEZ comparable to other gradient methods outperforming the classification dataset, AGNEWS. Furthermore, we found that PEZ transfers better.
Table Accuracy and standard deviation on the SST-2 validation set across the fives prompts for each method trained on GPT-2 Large and transferred onto larger models ranging from 1.3B to 6.7B. The baseline accuracy of soft prompt is 93.35+0.01 (optimized for GPT-2 Large), but cannot be transferred. Note Emptyremplate refers to no prompt at the front but containing the predetermined template.
GPT-2 Large (755M, Source)
GPT-2 XL
T5-LM-XL (3B)
OPT (2.7B)
OPT
Method
(1.3B)
(6.7B)
EmplyTemphate
80.84
73.85
52.75
72.48
58.72
AutoPrompiscn
87.56.0.48
78.19_a
56.0113.74
73.69 64
65. 65.2843.91
FlucntPrompt
88.33
78.33446.
55.64411.a
70.39
61.74
Oursno Flueney
88. 88.12+0.22
77.8
61.12+6.57
76.93+2.58
71.72+7.06
OurSptuency
88.05
79.72+7.3
63.3
77.18
72.39 =4.07
In the context of prompting in the text-to-text setting, the goal of Algorithm is to discover a discrete sequence of tokens, the hard prompt, that will prompt the language model to predict the outcome of classification task. As an important property of text is its fluency, Shi et [2022] find that fluency can increase a prompt's 's readability and performance. Thus, we define the optimization objective in this section as a weighted function of task loss and fluency loss,
Xiuennyy -= (1-neeyy Lussk Auuency Liuency.
We set À 0.003 similar to Shi et al. [2022] for all methods, and we ablate our method without fluency (A = 0), which we denote as no fluency. We set out to show that hard prompts generated by this approach are successful both when transferring between a number of transformer-basee language models, and when used to discover prompts in few-shot settings. An attractive quality of these prompts, especially for language applications, is that they be optimized smaller language models and then transferred to other, much larger models.
A.3 Datasets and Setup
We evaluate Algorithm I against related algorithms three classification tasks, two sentiment
analysis tasks, SST-2 [Socher et al. 2013 and Amazon Polarity [McAuley and Leskoved 2013 and
4-way classification task, AGNEWS [Zhang et al. 2015). We build the setting explored in Ding
et al. 2022] and optimize hard prompts using GPT-2 Large (774M parameters) Radford et al. 2019
with the Adafactor optimizer (Shazeer and Stern 2018 and batch size of 32 Lester al. 202
Transferability Set-up. To test transferability, we generate prompts from GPT-2 Large for 5000
steps. We then select the five prompts with the highest average validation accuracy for each technique and test them on larger models. We test the transferred text on: GPT-2 XI., T5-L.M-I. OPT-2.7B,
and OPT-6B [Radford et al. 2019
Lester et al., 2021b
Zhang al. 2022 verifying the reliability
16

---

of the proposed algorithm over related techniques and testing whether the hard prompt can reliably boost performance. Thus, we also consider baseline of empty prompts, with only the template.
Few-Shot Setup. For the few-shot setting, we optimize each prompt for 100 epochs on GPT-2 Large on the AGNEWS dataset, where we sample two examples (k 2) and four examples (k 4) from each class to obtain the training set. Additionally, we create a holdout set of the same size, and finally validate the prompts on the entire validation set.
A.4 Results
We verify that our method is comparable to other methods in the sentiment analysis setting outperform the other methods on AGNEWS by about 2%. See Table for details.
For Table 4 we report the best validation accuracy across three learning rates (0.1, 0.3, and 0.5), and for FluentPrompt and AutoPrompiscn we used the learning reported (1, 3, and 10) and follow Shi et al. [2022) for the remaining hyperparameters for Fluen:Prompt. For these experiments, we prepend our T0 token prompt to each input text. We employ early stopping for all methods using a hold-out set of 5000 examples for each dataset, evaluating every 100 steps.
Table 4 shows that we are comparable to other methods in sentiment analysis and outperform the other methods on AGNEWS by about 2%. Examining the prompts, we find prompts are not coherent English for any of the methods. However, does produce relevant tokens and phrases. For example, our method for SST-2 with the fluency constraint produced "negative vibeThis immatureollywodd MunderinollywoodThis energetic screenplay." " This suggests the optimization process is finding relevant words to the task but lacks the ability to create full sentences.
Table 4: Validation accuracy for 10 discrete tokens trained prepended at the beginning of the input text. Best accuracy across three learning with standard error reported over speeds.
Method
SST-2
AGNEWS
Amazon
AutoPromptsan
87.56
74.3640.47
87.75
FluentPrompt
88.33_0.35
74.6240.22
87.4220.18
Ouršno Fleency
88. 1.
77.06
87.70
OurSytuency
88.05+0.50
76.94
87.78+0.119
Soft Prompt
93.35
92.76.0.01
94..65.0.01
Prompt Transferability. Table for each method the five prompts trained on GPT-2 Large transferred to other L.MMs. Interestingly, simply scaling model-with no additional training-does not guarantee that the model will scale perform according on SST-2 We see that all gradient-based methods are able to transfer compared to evaluating just the template, finding that our prompts trained with the fluency constraint transfer better than the other prompts. Additionally, we can see the largest boost from OPT-6. 7B with our fluent method with about 14% increase over just the template baseline. Additionally, we see our AGNEWS prompts are able to transfer from GPT-2 Large to GPT-2 XL in Table 5
Table 5: Shows the validation accuracy with standard deviation from transferring hard prompts learned on GPT-2 Large to GPT-2 XL.
Method
GPT-2 Large (755M)
GPT-2 XL, (I.3B)
Emptynempiate
58.34
52.42
AutoPrompt
74.36=0.47
63.7943.61
FluentPrompt
74..2210.44
61.57
Ours Fluency
77.06
59.45
OurSFlueacy
76.94
67.59
'Although we initialize the tokens with the label tokens, when examining the prompt over the optimization process, all tokens moved away from the initial tokens. This suggests that the process able to relearn the class label.
" quick experiment with and without the template on GPT-2 Large and XL showed that the template boosts performance differently for different models.
17

---

Table 6: Average validation accuracy with standard error on AGNEWS with & examples/shots per class using early stopping (including soft prompt) for all methods across 100 seeds for three tokens append to the end of the text similar to the original template ("It was about"). We set À = 0.03 for these experiments. "Empty" is the template with no additional prompt.
Method
k=2
4=4
Empty Tempptte
58.34
58.34
OurSNo Floency
70.07
73.99
OurSPtuency
70.9310.60
74.1520.45
Soft Prompt
74.92:10.58
79.93
Prompt Discovery. shows that even with just a few shots we can achieve high validation accuracy compared to prepended counterparts. It is worth noting that each few-shot takes about 5min. We ran 100 seeds where the training set contains A samples each class and did a quick examination of the top prompts, and although many of the prompts were gibberish, many of them were coherent. For example, even for k = 2, some of the prompts included news sources like "BBC", while other prompts found new approaches the classification task considering the text coming from a blog: "Brian blog," or "Blog Revolution analyze." Due to the efficiency of these gradient-basee methods, these methods can allow new ways for prompt engineers discover novel prompts.
18


Reviews:
[{"review_id": "QDUJZXUcDk", "reviewer": "Reviewer_Lu7o", "summary": "This work proposes a new paradigm which optimizes discrete prompts simply by gradient projection to bridge the gap between relatively “easy-to-optimize” soft prompts and their hard counterparts. Their methodology is straightforward and effective in downstream text-to-image task requirements. And more specifically, comparing to existing gradient-based approaches, such as AutoPrompt, their approach is less hyper-parameter sensitive equipping with their gradient projection. All these advantages offer a new lens on how to optimize discrete prompts for text-to-image generation models, and LLMs likewise.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This framework is easy to implement, by simply gradient projection in the soft embedding space, and the performance seems promising compared to popular caption-based \"CLIP interrogator\". It does contribute another approach for such optimization problem.\n2. The experiments do show the advantages of such framework, such as \"the insensitiveness to hyper-parameters\". \n3. The paper organization is good, and the writing is well-polished.", "weaknesses": "1. I think adding more baselines in your main table would definitely be a plus! Currently, you mainly compare this to a heuristic-based yet popular method called \"CLIP interrogator\". Can you compare your approach with more diversified baselines, e.g., \"AutoPrompt with different LRs\", \"RLPrompt\", etc.. These can let readers gain a basic intuitive understandings of your method's performance against others. \n2. In your main table, the improvements of yours compared to CLIP interrogator seems to be marginal, where your performance is almost the same level of performance with CLIP interrogator, even though you only include fewer tokens. \n3. Besides, can you show some quantitative and qualitative examples on directly showing on the same set of images, what are the optimized prompts by CLIP interrogator, and what are the optimized prompts by yours (and perhaps, their word overlapping by distillation), such that we can enhance our understandings that your approach identify a \"secret\" language, like initial work [1] (in addition to possible application contributions, the secret language could also contribute to scientific research if your pipeline is good at identifying such phenomenon). Otherwise, to one of your contributions that your identified prompt is shorter, we may speculate that the prompts in CLIP interrogator can also be distilled into the same short length, and even with the same key tokens as your pipeline. Therefore, the secret language observation would be weaken. Besides, the CLIP interrogator even does not require any training with similar level of performance and even similar level of short lengths through distillation, why not we use their easier pipeline? --- Currently, figure 6 further enhances my confidence on such point as well, as your prompts include a lot of natural tokens in accordance with our common-sense understandings, such as cat images should use \"cat\" tokens. \n4. Additionally, practitioners would be interested in your optimization efficiency as well for such algorithm to identify well-crafted prompts, comparing to many other soft/hard prompt optimization work. In terms of this, if you have better efficiency, that is also another good point, where I am also interested. \n\n[1] Discovering the hidden vocabulary of DALLE-2, arxiv 2022 \n\nSo right now, I am more viewing this work's contribution as a different perspective/approach to prompt optimization for text-to-image generation models, and even more models. Of course, that is interesting, and useful/insightful to the community. However, a key limitation would be such utilities are not quite clear alongside all current prompt optimization approaches. And the performance improvement seems to somewhat limited compared to CLIP interrogator. \n\nHopefully, my suggestions would be useful for you to improve your content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address each of your points below:\n\n>1. I think adding more baselines in your main table would definitely be a plus! Currently, you mainly compare this to a heuristic-based yet popular method called \"CLIP interrogator\". Can you compare your approach with more diversified baselines, e.g., \"AutoPrompt with different LRs\", \"RLPrompt\", etc.\n\nWe include AutoPrompt (SGD) with different learning rates in Figure 5(a). Additionally, we believe RLPrompt is infeasible for optimizing images as each experiment (a single image in our case) takes between 1.5 and 4 hours according to [1]. This makes RLPrompt infeasible for this particular set of experiments. Also, please see Table 2 in the Supplementary Material for more baselines.\n\n[1] Deng, Mingkai, et al. \"Rlprompt: Optimizing discrete text prompts with reinforcement learning.\" arXiv preprint arXiv:2205.12548 (2022).\n\n>2. In your main table, the improvements of yours compared to CLIP interrogator seems to be marginal, where your performance is almost the same level of performance with CLIP interrogator, even though you only include fewer tokens.\n\nCLIP Interrogator is a purpose-built tool that has a much stronger prior than PEZ - that we are even able to match it with a general optimization scheme, is to us an indication that our much more general-purpose optimizer works quite well. \n\nSpecifically, CLIP-Interrogator uses the BLIP image-to-text model and a word bank of predefined artistic phrases to search over. Problems with using this tool in any other context than generating captions for art are highlighted in the difference we see in the Celeb-A dataset, where PEZ outperforms CLIP Interrogator as these images are \"OOD\" for both BLIP and the small word bank. Additionally, due to the compressibility of using PEZ, we can combine different prompts together as highlighted in Sections 4.3 and 4.4. However, under the current functionality of the CLIP Interrogator, this is impossible. In short, PEZ outperforms in recreating images that fall outside the CLIP Interragator's prior and has additional functionality, while matching this purpose-built tool on its exact task. \n\n>3. Besides, can you show some quantitative and qualitative examples of directly showing on the same set of images, what the optimized prompts by the CLIP interrogator, and what the optimized prompts by yours (and perhaps, their word overlapping by distillation), such that we can enhance our understandings that your approach identifies a \"secret\" language, like initial work [1] (in addition to possible application contributions, the secret language could also contribute to scientific research if your pipeline is good at identifying such phenomenon). Otherwise, to one of your contributions that your identified prompt is shorter, we may speculate that the prompts in the CLIP interrogator can also be distilled into the same short length, and even with the same key tokens as your pipeline. Therefore, the secret language observation would be weakened. Besides, the CLIP interrogator even does not require any training with a similar level of performance and even a similar level of short lengths through distillation, why not use their easier pipeline? --- Currently, figure 6 further enhances my confidence on such a point as well, as your prompts include a lot of natural tokens in accordance with our common-sense understandings, such as cat images should use \"cat\" tokens.\n\nDo you mind clarifying the latter part of this statement? We are a little confused by what exactly you are suggesting. Thank you!\n\nRegarding the other questions, note the differences between CLIP interrogator that we mention above. We do evaluate CLIP Interrogator for shorter prompt lengths, see Table 1.\n\n> 4. Additionally, practitioners would be interested in your optimization efficiency as well for such an algorithm to identify well-crafted prompts, compared to many other soft/hard prompt optimization work. In terms of this, if you have better efficiency, that is also another good point, where I am also interested.\n\nThank you for the suggestion. Prompted by your feedback, we have now conducted a series of experiments comparing the efficiency between PEZ and AutoPrompt$_\\text{SGD}$ using varying numbers of steps, as outlined in the table below. Our findings indicate that PEZ achieves faster convergence compared to AutoPrompt.\n\n|     Method     | 100 Steps |  200  |  400  |  600  |  800  |  1000 |\n|:--------------:|:---------:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| AutoPrompt     |   0.642   | 0.654 | 0.656 | 0.672 | 0.677 | 0.684 |\n|       PEZ      |   0.668   | 0.674 | 0.675 | 0.684 | 0.690 | 0.695 |\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback including experiments. We have updated our draft accordingly and will include these updates in our camera-ready version, and we would appreciate it if you would consider raising your score in light of our response. Do you have any additional questions we can address?"}, {"review_id": "DY4EGbZt3i", "reviewer": "Reviewer_LSBN", "summary": "This paper works on hard prompt optimization with gradient methods, especially a discrete text prompt is discovered using CLIP, and optimized to prompt stable diffusion.", "questions": "1. Ablation study shows performance saturation with middle number of prompt length. More analysis is needed to explain this result.\n2. Prompt distillation seems interesting to explain the prompt length issue, however, how the distillation ratio is correlated with the quality of the generated samples should also be explained further.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Without hand-crafted design of hard prompt, the proposed solution directly discover and optimize prompt with gradient descent, leading to very efficient prompt engineering.\n2. The learned prompt works great on stable diffusion and achieve effective style transfer, further explains the superiority of the proposed solution. ", "weaknesses": "The proposed solution works on discrete text prompt optimization without constraints on meaningfulness of the text, making it hard to directly understand to the effectiveness of prompt optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address each of your points below:\n\n#### Weaknesses\n\n> The proposed solution works on discrete text prompt optimization without constraints on meaningfulness of the text, making it hard to directly understand to the effectiveness of prompt optimization.\n\nWhile we do experiment with fluency constraints, showing that it is possible to add such constraints (see experiments in the Supplementary Material), this is not actually our primary goal. We are not concerned with the fluency of the generated prompts. In all applications, a user would be able to provide any series of tokens to a generation API, and there is no need for the optimized prompt to be fluent. We agree that this is a departure from some of the literature on prompt optimization, which is concerned with finding interpretable prompts, but for the use cases we consider, interpretability is not required. \n\nFor the CLIP experiments, our study also provides interesting prompts that hint at a secret language within the diffusion models. We further highlight the \"secret language\" in Section 5 (Safety Concerns), where are able to bypass the Midjourney content filters. In safety questions, the ability to bypass a word filter with a non-interpretable prompt is a core necessity of the attack.\n\n#### Questions:\n\nRegarding Q1, our suspicion is rooted in overfitting, evident from the decreasing loss accompanied by a lower CLIP Score. Additionally, upon extending the prompt length, we observed an increase in the occurrence of model-specific unicode token usage within the optimized prompts. Consequently, this led to diminished transferability in comparison to the usage of shorter prompts.\n\nWe believe that introducing a constraint, such as enforcing optimized tokens to adhere to the English language, has the potential to alleviate the overfitting issue. To validate this notion, we have now conducted a preliminary assessment. When utilizing the keyword bank as a constraint during optimization, we noted a notable improvement in mitigating the overfitting concern, as illustrated in the table below.\n\n|                    | 4 tokens |   8   |   16  |   32  |   64  |\n|:------------------:|:--------:|:-----:|:-----:|:-----:|:-----:|\n|         PEZ        |   0.686  | 0.697 | 0.699 | 0.689 | 0.670 |\n| PEZ + Keyword Bank |   0.684  | 0.699 | 0.712 | 0.716 | 0.708 |\n\nFor Q2, we explore this question in Figure 8 in Supplementary Material. We find here that this is a step down in performance from a ratio of 0.3 to 0.1.\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback including experiments, and we would appreciate it if you would consider raising your score in light of our response. Do you have any additional questions we can address?"}, {"review_id": "TbQu3EdJCm", "reviewer": "Reviewer_DXf6", "summary": "The paper presents an easy-to-use approach to automatically obtain hard prompts for images. The work introduces PEZ, a gradient-based approach to obtain hard prompts for images. The experiments comparing a popular baseline show improved CLIP score. Finally, the qualitative results show that the method can distill prompts into shorter sequences and concatenates ", "questions": "Please address the questions in the weakness section. \n\nMinor: since you are using cosine distance as a loss function, does it run into instability issues? Is there a particular reason you are using cosine distance as a loss function? \n", "limitations": "The method requires the model weights to produce hard prompts. Given that recent methods use APIs, it would be useful to highlight this as a potential limitation.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is easy to follow. The algorithm is well explained and the results are quite promising. The results for prompt inversion appear to be fair and outperform the baseline. The analysis is quite clear and thorough. Overall, the work presents a straightforward method to get prompt tokens for images. ", "weaknesses": "The contribution of the work is limited: the work is mostly an application of existing work that discretizes soft prompts to tokens [a,b]. There is no significant difference between the proposed work and existing related work. The authors should clarify, in-depth, the main differences between their work and other work. Further, like AutoPrompt, the work suffers from gibberish prompts according to the authors (see Figure 2, “uuuu romantic canvas impressionist …”). This limitation makes the hard prompts harder to interpret. The explanation regarding the emoji appearing in the hard prompt (Figure 2 and line 174) is not entirely convincing. It would be really helpful to also know how often the model produces gibberish. \n\nLimited Evaluation: the experiments do not overwhelming support that PEZ is better than CLIP interrogator. On two out of the four datasets under consideration, CLIP interrogator achieves a higher CLIPScore. It is also unclear why AutoPromptSGD is not included in Table 1. Wouldn’t this be a good baseline?\n\nNegative Qualitative Examples: The paper could benefit from showing negative results where the method doesn’t perform as well. \n\nNit: \nIt would be helpful to look at the results along with Tables/Figures on the same page. At the moment, style transfer, prompt concatenation, and prompt distillation do not appear on the same page.\n\n[a] Gradient-Based Constrained Sampling from Language Models. EMNLP 2022.\n\n[b] Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? ArXiv 2022. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address each of your points below:\n\n> 1. The contribution of the work is limited.\n\nThe changes between **PEZ** and $FluentPrompt$ may appear subtle, but they are important for performance, and the comparison can be found in the appendix. **PEZ's** main difference is how the rounding is done and can be thought of as the combination of optimizing soft prompts and $AutoPrompt_\\text{SGD}$. $AutoPrompt_\\text{SGD}$ updates the set of tokens at every step using these same tokens to get the gradient information for the update. Soft prompts optimize the same continuous set of embeddings $P$ which is used for every forward and backward pass and the update step $P=P-\\eta\\nabla_{\\mathbf{P}}\\mathcal{L}$. FluentPrompt adds a decaying noise term during the optimization process. \n\nIn contrast, **PEZ** is like optimizing a soft prompt, but for each gradient update the information is going to come from the nearest-neighbor rounding prompt $P'=\\text{Proj}(P)$ instead of soft prompt, $P$. Then, the update is on $P$ not $P'$, $P=P-\\eta\\nabla_{\\mathbf{P'}}\\mathcal{L}$, where $P'=\\text{Proj}(P)$. Thus, the process is storing a continuous set of embedding that is required for the forward pass. This means that although it may suffer from the stochastic rounding issues in the earlier steps, where the tokens project to the same set of tokens, eventually **PEZ** will find a new set of tokens since $P$ in a sense is summing the gradient information at every step. The problem with FluentPrompt and $AutoPrompt_\\text{SGD}$ is sensitive to learning rates, but PEZ is quite robust under different learning rates as shown in Figure 5a. Conceptually, this tries to solve the problem that certain models require some interesting (and non-obvious) learning rates that can greatly vary from model to model.\n\nAdditionally, we have now conducted a series of experiments comparing the efficiency between PEZ and AutoPrompt$_\\text{SGD}$ using varying numbers of steps, as outlined in the table below. Our findings indicate that PEZ achieves faster convergence compared to AutoPrompt.\n\n|     Method     | 100 Steps |  200  |  400  |  600  |  800  |  1000 |\n|:--------------:|:---------:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| AutoPrompt     |   0.642   | 0.654 | 0.656 | 0.672 | 0.677 | 0.684 |\n|       PEZ      |   0.668   | 0.674 | 0.675 | 0.684 | 0.690 | 0.695 |\n\nWe believe that both properties of PEZ are very important to the prompt optimization applications mentioned in the paper.\n\n>Limited Evaluation: the experiments do not overwhelmingly support that PEZ is better than CLIP interrogator.\n\nWe have updated our local draft by noting that the CLIP interrogator is a purpose-built tool, built to generate fitting prompts for digital art images, from a word bank of possible artistic styles and expressions. That the proposed generic algorithm is able to match the performance of this tool is a strong statement of its general capability.  We will include these updates in our camera ready version.\n\nThe issues with using the CLIP Interrogator in any other context than generating captions for art is highlighted in the difference we see in the Celeb-A dataset, where PEZ outperforms CLIP Interrogator as these images is \"OOD\" for both the BLIP model and the small word bank. Additionally, due to the compressibility of using PEZ, we can combine different prompts together as highlighted in Sections 4.3 and 4.4. However, under the current functionality of the CLIP Interrogator, this is not a possibility. In short, PEZ outperforms in recreating images that fall outside the CLIP Interrogator's prior and has additional functionality, while matching this purpose-built tool on its exact task. \n\nA full table of these comparisons can be found in Table 2 in the Supplementary Material.\n\n\n> Negative Qualitative Examples.\n\nWe also think it's helpful to show some failure examples. We have added some failure generations in the rebuttal PDF. As we explained in the experiment part, the results on the Celeb-A dataset aren't as good as on other datasets. This might be because learning to make human portraits is tough, especially for small details in the face. As we can see in Figure 1 in the rebuttal PDF, the prompts from PEZ can still make pictures of human faces that look similar to the target images, but not exactly the same.\n\nMeanwhile, we provide other two typical failure cases in Figure 1: 1) instances where the generated images exhibit distinct styles from the target image (the second-to-last example); 2) cases where the prompt captures abstract concepts rather than specific objects (the final example). It's important to note, however, that these two scenarios can be mitigated by optimizing with various initializations.\n\nWe will include more failure examples and a comprehensive discussion and investigation of these cases in our future version.\n\n> Since you are using cosine distance as a loss function, does it run into instability issues?\n\nCLIP used cosine similarity to train the model, so we actually did not suffer from these instability issues as we used the same objective function.\n\n> The method requires the model weights to produce hard prompts. Given that recent methods use APIs, it would be useful to highlight this as a potential limitation.\n\nThis is a great point and we have extended our discussion of this limitation. However, many API models are based on publically-available CLIP checkpoints, for example, Midjourney. A great example of this process can be found in Section 5, where we show that in fact, we can bypass the content filters of the Midjourney API using optimized prompts created with open-source CLIP weights.\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback including multiple paper edits, and we would appreciate it if you would consider raising your score in light of our response. Do you have any additional questions we can address?"}, {"review_id": "YkoetiBld6", "reviewer": "Reviewer_7Vg6", "summary": "The paper presents an approach to optimize hard text prompts for generative models through efficient gradient-based optimization. The method automatically generates hard text-based prompts for both text-to-image and text-to-text applications, allowing users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. The paper highlights the advantages of hard prompts over soft prompts and demonstrates the effectiveness of the approach in tuning language models for classification.\n\n", "questions": "You mentioned that longer prompts do not necessarily produce better results and can lead to overfitting. Could you elaborate on the mechanisms behind this phenomenon and suggest potential strategies to mitigate this issue while maintaining the quality of the generated prompts?", "limitations": "As the author has discussed, the generated prompts may still contain several un-interpretable tokens. Also, it may lack the ability to create full sentences.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper's approach is efficient and can be optimized on smaller language models and then transferred to other, much larger models.\n\n2. The method performs consistently across all four datasets and outperforms other gradient-based optimization baselines. It can achieve similar performance to CLIP Interrogator. However, the proposed method only uses the CLIP model for prompt discovery and 8 tokens in total demonstrating its simultaneous simplicity and strength.\n\n3. The proposed approach can also be easily adapted to style transfer. Given several examples that share the same style, the method can extract their shared style characteristics into a single hard prompt and use this prompt to apply the style to new objects or scenes (Page 5).\n", "weaknesses": "1. While the method can generate relevant tokens and phrases, the prompts are not always coherent English. This suggests that while the optimization process is finding relevant words to the task, it lacks the ability to create full sentences. \n\n2. Longer prompts do not necessarily produce better results when generating with Stable Diffusion, even though they strictly reduce loss on the CLIP image encoder. Long prompts thus overfit and are less transferable. Thus, it may require hyperparameter tuning, which may heavy and tedious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Below, we address specific points you raised:\n\n> You mentioned that longer prompts do not necessarily produce better results and can lead to overfitting. Could you elaborate on the mechanisms behind this phenomenon and suggest potential strategies to mitigate this issue while maintaining the quality of the generated prompts?\n\nUpon extending the prompt length, we observed an increase in the occurrence of gibberish and Unicode tokens within the optimized prompts. Consequently, this led to diminished transferability in comparison to the usage of shorter prompts. There is a transfer happening here, as we optimize on CLIP, but test on text-to-image diffusion models. If we overoptimize a prompt against CLIP, it cannot transfer to the diffusion model.\n\nWe believe that introducing a constraint, such as enforcing optimized tokens to adhere to the English language, has the potential to alleviate the overfitting issue. To validate this notion, we conducted a preliminary assessment. When utilizing the keyword bank as a constraint during optimization, we noted a notable improvement in mitigating the overfitting concern, as illustrated in the table below. We will include more experiments with different constraints in the future version.\n\n|                    | 4 tokens |   8   |   16  |   32  |   64  |\n|:------------------:|:--------:|:-----:|:-----:|:-----:|:-----:|\n|         PEZ        |   0.686  | 0.697 | 0.699 | 0.689 | 0.670 |\n| PEZ + Keyword Bank |   0.684  | 0.699 | 0.712 | 0.716 | 0.708 |\n\nThank you for your feedback on this submission. Hope our additional experiment resolves your question. Please let us know if you have other questions and comments that we can address."}]

Author rebuttal:
We thank all the reviewers for their time and thoughtful feedback. We have attached a PDF containing additional figures, showing explicit failure cases of the approach as requested by one reviewer.

We would like to emphasize the significance of this work. Our proposed, general-purpose, prompt optimization method is able to achieve comparable results to the purpose-built art prompt tool, CLIP interrogator, without any prior domain-specific knowledge, image-to-caption models, or keyword banks. Additionally, we find that PEZ demonstrates that it is possible to break content filters in deployed generative AI models, such as Midjourney - a new vulnerability that we uncover immediately, without even tuning the optimization scheme.

Meanwhile, the differences between PEZ and other baselines like AutoPrompt may appear subtle, but they are crucial for performance. As shown in Figure 5 (a), PEZ is much more robust under different learning rates. Also, our additional experiments, presented in the table below, suggest that PEZ converges much faster than the baseline under the same learning conditions. We believe that the flexibility and efficiency of PEZ are very important for applications like prompt optimization, really making "hard prompts easy" to use. This has been reflected in the usage and implementation of this method in a number of applications, as well as in GUI tools for hobby users of diffusion models.

|     Method     | 100 Steps |  200  |  400  |  600  |  800  |  1000 |
|:--------------:|:---------:|:-----:|:-----:|:-----:|:-----:|:-----:|
| AutoPrompt     |   0.642   | 0.654 | 0.656 | 0.672 | 0.677 | 0.684 |
|       PEZ      |   0.668   | 0.674 | 0.675 | 0.684 | 0.690 | 0.695 |

Meta-review: The paper presents an approach to optimize hard text prompts for diffusion models through efficient gradient-based optimization. The authors have done a nice job during rebuttal. After rebuttal, it received scores of 5567. The scores look nice, though 3 of the 4 reviewers only gave a confidence level of 3. 

On the positive side, the proposed method is easy to implement, by simply gradient projection in the soft embedding space, and the performance seems promising compared to popular caption-based "CLIP interrogator". Also, the paper is well written, and the analysis is clear and thorough. On the other hand, during the rebuttal, the authors have provided additional results, and reviewers commented that some points needed to be further adjusted to be a nice work, e.g., the advantage in terms of efficiency, compared to other methods. Otherwise, the paper shows limited technical contribution, such as novelty and performance compared to other simpler baselines. Furthermore, the discussion on "hidden vocabulary"/"secret language" can be further polished. 

Overall, the AC thinks that the merits of this paper outweigh its flaws, and would like to recommend acceptance by the end. The authors are highly encouraged to add the new results into the main paper, and revise the paper accordingly based on reviewers' detailed comments.
Decision: Accept (poster)