Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation
Authors: Wanrong Zhu
Xinyi Wang
Yujie Lu
Tsu-Jui Fu
Xin Eric Wang
Miguel Eckstein
William Yang Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Text-to-image
LLM
Efficiency

Abstract: The field of text-to-image (T2I) generation has garnered significant attention both within the research community and among everyday users. Despite the advancements of T2I models, a common issue encountered by users is the need for repetitive editing of input prompts in order to receive a satisfactory image, which is time-consuming and labor-intensive. Given the demonstrated text generation power of large-scale language models, such as GPT-k, we investigate the potential of utilizing such models to improve the prompt editing process for T2I generation. We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process. We found that GPT-k models focus more on inserting modifiers while humans tend to replace words and phrases, which includes changes to the subject matter. Experimental results show that GPT-k are more effective in adjusting modifiers rather than predicting spontaneous changes in the primary subject matters. Adopting the edit suggested by GPT-k models may reduce the percentage of remaining edits by 20-30%.

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-lmage Generation
Wanrong Zhu Xinyi Wang', Yujie Lu?, Tsu-Jui Fu!, Xin Eric Wang.. Miguel Eckstein?, William Yang Wang! 'UC Santa Barbara, SCC Santa Cruz
|wanrongzhu, xinyi_wand wang, yujichu, isn-juifu, migueleckstein, william | @uesb.edu, xwang366@@use.cdu
Abstract
Percenlagel
Count
80
The field of text-to-image (T2T) generation has garnered significant attention both within the re- search community and among everyday users. Despite the advancements of T21 models, a common issue encountered by users the need for repetitive editing of input prompts in or- der receive a satisfactory image, which is time-consuming and inteesii Given the demonstrated text generation power of large- scale language models, such as GPT-k, we in- vestigate the potential of utilizing such mod- els to improve the prompt editing process for T2I generation. We conduct series of experi- ments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine fac- tors that may influence this process. We found that GPT-A models focus more on inserting modifiers while humans tend to replace words and phrases, which includes changes to the subject matter. Experimental results show that GPT-k are more effective in adjusting modifiers rather than predicting spontaneous changes in the primary subject matters. Adopting the edit suggested by GPT-k models may reduce the percentage remaining edits by 20-30%.
60
#Edit
Figure The histogram plots #edit t per trace in Diffu- sionDB (Wang et al., 2022) while the lineplot shows the cumulative percentage of traces less than certain #edi t. The y-axis on the left is on log-scale. On average, there 6.9 edits being made in each user editing trace.
"prompt-cngineering" which is writing accurate prompts to describe the desired image in this sce- nario still remains significant challenge. Users often need to edit the prompt for several rounds be- fore arriving a satisfactory image, which makes the process of T2I generation timecconumiing and laborious (and expensive for commercialized mod- cls). Figure shows the #edits distribution the editing traces made by StableDiffusion Discord server users (Wang et al., 2022). This phenomenon highlights the need to improve the efficiency and effectiivness of prompting T2I models.
Recently, large language models (LLMs) such as GPT-k (Radford al., 2019; Brown et al., 2020; Ouyang et al., 2022) have demonstrated impressive abilities in text generation. This leads to the natural question of whether these models can be utilized to improve the T2I prompting process (Chakrabarty et al., 2023). However, LLMs and T2I models have different architectures and are often trained on different modalities, which makes it challenging to integrate them seamlessly.
1 Introduction
The task of text-to-image (T2I) generation, which involves generating images from natural language descriptions, holds significant potential to create new avenues and job opportunities for content cre- ators while also providing insights into the ground- ing of natural language in the visual world through the application of generative AI. A number of mod- els have demonstrated exceptional performance in terms of image quality, such StableDiffu- sion (Rombach al., 2021), Midjourney (Mid- journey, 2022), Imagen (Saharia al., 2022), and DALLE-2 (Ramesh al., 2022). Despite the popu- larity of T2I generation and the ability of these mod- els to generate impressive images, the difficulty of
In this paper, we conduct a series of experiments and analyses on user editing traces on StableDiffu- sion,' and attempt to modify the T2I prompts with eight GPT-k models. The primary objective of our research is to examine the potential of modifying T2I prompts with GPT-k models. More specifically,
'Our experiments are conducted upon StableDiffusion since it is a wide-adopted open-source large text-to-image generative model with SoTA performance.

---

we aim to investigate the common edits made by humans and by GPT-k, as well as the performance of prompting T21 generation with GPT-k models. Additionally, we aim to identify and investigate pos- sible factors that might influence the performance of GPT-k in T2I generation tasks.
GPT-A
Model Name
#Parameters
gpt2-base
117M
apt2-mediun
345M
GPT-2 (Radford al., 2019)
apt2-large
774M
gpt2-xl
1.2B
text-ada-801
350M
GPT-3 (Brown et al., 2020)
text-babbage-081
1.3B
text-curie-001
6.7B
Through our experiments, observe that:
GPT-3.5 (Ouyang et al., 2022)
text-davinci-e03
175B
While GPT-k models tend to focus more on inserting modifiers, human users have greater tendency towards replacing words and phrases, which may include spontaneous changes the subject matter of the prompt. Modifying the T2I prompt with GPT-k models may not necessarily result in a direct increase in similarity to the final target image in the editing trace. Instead, the edit suggested by GPT-k may be closely related to intermediate editing steps, and the percentage of remain- ing edits may decrease by 20-30% if the edit suggested by GPT-k is adopted.
Table I: The names and corresponding parameter sizes of the GPT-k models covered in our study.
each trace is determined by the timestamps. We receive 100k traces of edits, the mean #edit per trace is 6.9, with a standard deviation of 16.1. Fig- ure plots the #edit in the clustered traces, which shows a long-tail distribution with about 5% of the traces having more than 20 edits. Thus, in the following experiments, the evaluation results were reported on traces with at most 20 edits.
GPT-k & Text-to-Image Models Table lists the names and parameter sizes of the eight GPT-k models we covered this study. We conduct T21 experiments with the open-source StableDiffusion v1-4 (Rombach et al., 2021), which is used to ren- der images in DiffusionDB (Wang et al., 2022). Annotations For each trace of length we denote the prompts as (t1, t2, .... t,), and the generated images (61, i2. .... 1,). We refer to the GPT- k-modified prompt as t' and refer to the image rendered from the modified prompt i'. Metrics We use the cosine similarity of CLIP- ViT/B-32 (Radford et al., 2021) visual features to evaluate the similarity between images. Let é de- note the image in the original edit trace that is most similar i' define the Relative Number of Edits (RNE) n- The RNE metric reflects the X-1-
These findings suggest that instead of attempl- ing to predict spontaneous changes made by human users on the primary subject matter, GPT-k models are more effective in improv- ing prompts by rewriting and performing edits related to modifiers adjustment.
2 Research Questions and Settings
To investigate the potential of modifying T21 prompts with GPT-k models, we conduct series experiments and analysis, aiming to answer the following questions:
1. To what extent can GPT-k models help prompt text-to-image generation?
2. What are the common types of edits made by humans and by GPT-k models?
relative position of the edit in the original trace that most similar to the edit suggested by GPT-k and also represents the percentage of remaining edits after the edit suggested by GPT-k is performed.
3. What are the factors that may influence GPT-k prompting text-to to to-image generation?
We describe the dataset, models and evaluation metrics used in this study below.
3 Prompting T2I w/ GPT-k
Dataset DiffusionDB-22N (Wang et al., 2022) scrapes 2M groups of user prompts, hyperpa- rameters and images generated by StableDiffu- sion (Rombach et al., 2021) from the official Stable Diffusion Discord server. We group the prompts by anonymized user_id, then cluster the user prompts into traces of edits based on the prompt contents. More specifically, we encode prompts with Universal Sentence Encoder (Cer et al., 2018), then cluster upon the prompt embeddings with DB- SCAN (Ester et al., 1996). The order edits within
In the following experiments, we only reveal the initial prompts in the user editing traces to GPT-k, and ask the models to perform one round of edit. Procedure We split the 100k trace of edits into two parts, with 30k traces used for evaluations and the remaining 70k serving as heldout set. For each of the listed GPT-k models, we provide the first prompt t1 in each evaluation trace to the model, and let GPT-k generate the modified prompt t'. GPT-2 models are finetuned with the prompts in
2

---

Model
i i--1-
F-h.
RNE(%)
Effeetiveness
Likelihood
Ept2-base
69.58
80.16
75.28
Win
Tie(%)
Loset())
Win(%)
Tie(1))
Lose(())
gpt2-mediun
69.63
80.39
78.28
BPt2-x1
38.57
29.77
31,66
38.99
22.01
38.99
gpt2-large
69,70
80,50
78.88
spti-curie
40.89
21.33
37.78
39.33
21.56
39.11
gpt3. S-davinii
39.58
21.67
38.75
38.13
25.00
36.87
gpt2-x1
69.57
80.37
75.25
71.72
74.82
got3-ada
66,95
76,37
69.43
Table 3: Human evaluation results of head-to-head com-
spt3-babbage
68,79
78.81
72.33
parison between edits suggested by GPT-k and human- made edits. We evaluate the effectiveness of the edit and the likelihood of this edit being adopted by humans. "Win" and "Tie" indicate that SEPT---suggested edits are better than or comparable to human edits.
8pt3-curie
68.45
78.28
71.41
gpt3.5-davinci
68.79
78.09
69.22
Table 2:
The CLIP cosine similarity scores between images and the relative number of edits (RNE). Here, é1. in-1. in denotes the first, last-but-one, and last image in the trace of edits; i' is the image generated from the modified prompt, and ass is the image that is most similar to i' with regard to CLIP cosine similarity. RNE scores suggest a 20-30% decrease in the percentage of remaining edits adopting edits suggested by GPT-k.
Insert
Delete
Swap
Replace
Human
29.9%
21.1%
2.0%
47.0%
gpt2-base
95,4%
0.0%
4.6%
apt2-redion
95.6%
0.1%
0.0%
4.4%
gpt2-lerge
95.0%
0.1%
0.0%
4.9%
gpt2-x1
95.9%
0.0%
0.0%
4.1%
GPT-k
gpt3-ada
36.5%
14.9%
2.0%
46.6%
the heldout traces for two epochs with a learning rate of 5e 5 and batch size of 128. For GPT- 3/3.5 models, an in-context learning approach is adopted. Following previous studies (Yang al.. 2021; Alayrac al., 2022), supporting examples for in-context learning are selected by comparing the similarity of the prompt text features and re- trieving (f1. tx pairs from the top-m most similar traces. Modified prompts are generated through 16-shot in-context learning with m=16. Appendix Table 8 shows how we prompt GPT-3/3.5.
gpt3-babbage
39.6%
18.8%
3.3%
38.3%
gpt3-curie
42.9%
17.7%
4.1%
35.3%
gpt3. 5-davinc1
68.5%
3.9%
2.3%
25.3%
Table 4: The distribution of the common types of edits made by human and by GPT-K models.
that the percentage of remaining edits decreases by 20-30% if the edit suggested by GPT-k is adopted. Human Evaluation For each editing trace, we present MTurk annotators with the initial prompt and image (t1, i1), and two candidate edits: (1) the GPT-k-modified prompt, t': (2) the human edit, tMs. from the original editing trace. Here, tys is the corresponding prompt to iMs. which has the highest CLIP cosine similarity with i. The annotators are then asked to decide which edit was more effective and more likely to be adopted by human editors, t' or tass We evaluate each listed GPT-k model with 200 traces. For each trace, three annotators are invited to provide their judgments.
After receiving the GPT-k-modified prompt t', wc provide it to StableDiffusion to generate image i. The effectiveness of GPT-k in prompting T2I generation is evaluated by comparing the similarity of the generated image i' to images in the original trace. The results reported in the following sections are the mean of three repeated runs.
Automatic Evaluation The CLIP cosine similar- ity scores, as listed in Table 2, are used to evaluate image similarity. Two model-agnostic baselines are established for comparison: the similarity between the first and last images (,)- and the similarity between the last-but-one and last images (in-1-i,))
As shown in Table 3, the three GPT-k models all show tight wins against the human edits regarding both effectiveness and likelihood of being adopted. This verifies that the edits made by GPT-k models are similar or comparable to the intermediate steps in the human editing trace.
We denote Ms as the image in the original trace that is most similar to the generated image i' (has the highest CLIP cosine similarity score). Exam- ining results in Table 2, we notice that the image i' generated from the modified prompt may not be directly similar to the final target én, as (i'-i,)) is lower than the baselines. However, it appears that i' may be related to the intermediate steps in the editing trace, evidenced by the significantly higher similarity between i' and is compared to the baselines. RNE scores show that, i' is most sim- ilar to images in the first one- third of the trace, and
4 Human's Common Edits vs. GPT-k's
Upon examination of the user editing traces, we identify four types of edits commonly made by hu- mans: (1) Insert: adding descriptive terms such modifiers to the prompt to specify the style, artistic technique, camera view, lighting, etc: (2) Delete:
removing certain terms; (3) Swap: changing the
order of certain terms in the prompt; (4) Replace: changing the modifiers or the main subject of the
3

---

Al-mixed
CLIP CLI cosing
fess fa-2 VS. Reggt 1o MS. Fy
Insert
Delete
Swap
0
10801
Replace
Figure 2: The CLIP cosine similarity scores between the image generated from the GPT-3.5-modified prompt and the last image. Results are reported on Seval (All-
RONE 0
/Eale/Trace #Ealt/ Trace
14
16
20
mixed), Smment Stdeee, Swaap and Speplace-
Figure 3: The CLIP cosine similarity (upper) and RNE (lower) on traces with #edit ranging from 2 to 20. Prompts for T2I generation are modified with GPT-3.5. Here, é,, $1 11==2p 8 1- 2n denotes the first, middle, last-
prompt. Appendix Table 5 shows edit examples. To count the frequency of edits, we first re- move punctuation marks and stopwords using NLTK (Bird et al., 2009). We then utilize the Se- quenceMatcher' to compare the adjacent prompts the trace and detect the edit type. Table 4 lists the frequency of common edits made by humans and by GPT-k models. We notice a discrepancy between the distribution of human edits and the ones made by GPT-k. Nearly half of human ed- its pertain to replace, followed by insert and delete. GPT-2 models, due to their autoregressive training nature, have tendency towards continual generation, resulting in majority of edits being insert. While GPT-3/3.5 also undergoes autore- gressive training, its extensive training data and en- larged model size empower it for emergent ability, Unlike GPT-2, which solely receives the specific prompt requiring editing during testing, GPT-3/3.5 is also provided with additional editing instances supportive exemplars for in-context learning. Consequently, while GPT-2 indeed adheres to its autoregressive inference manner, the emergent ca- pability of GPT-3/3.5 enables it to attempt other edit types that simulate human-like editing behav- ior. For GPT-3, as the model size increases, we see an increase in the frequency of insert and a decrease in replace. For GPT-3.5, the most fre- quent edit is insert, followed by replace; while delete and swap are relatively rare.
but-one, and last image in the trace of edits; i' is the image generated from the modified prompt, and ias is the image that most similar to
compared the baseline results obtained from the complete Seval that mixes all types of edits.
Figure illustrates the impact of different edit types on image similarity. The CLIP cosine similar- ities of traces that solely consist of insert, delete, and swap edits are higher or comparable to the all-mixed baseline. This suggests that GPT-A per- forms better at adding, removing, and reordering modifiers. Conversely, we observe that replace edits lead to lower image similarities. This is likely due to the fact that the replace edit sometimes results in a change of the subject matter, which can drastically alter the painting. It is worth not- ing that shifting the primary subject matter of the painting is a relatively spontancous action made by humans. Appendix Table 6 shows a trace with multiple replace operation. The vast number of potential replacements makes it particularly diffi- cult for GPT-k to accurately select the desired edit. Effects of #Edits Figure 3 depicts how the CLIP cosine similarity changes with the #edit in the trace. As #edit t increases, the similarity between the last image in and those at the beginning or middle of the trace (i1 or ) decreases. This trend may be attributed the higher likelihood of changing primary subject matters in longer traces. Meanwhile, the similarity between the last-but-one and the last images (in vs. i,)) remains consis- tent, suggesting that the majority of edits made towards the end of the prompt editing process are minor in nature. As the trace of edits gets longer, the similarity between the image i' generated from the modified prompt and the most similar image iss the trace remains constant, while the RNE metric gradually increases to approximately 80%. This indicates that the modified prompt is related
5 Ablations & Analyses
Effects of Edit Types To investigate the effects of each individual edit type e;, we create four subsets Sinserts Stelete. Swap and Speplace from the evaluation set Seval- Each subset Se., comprises of traces that meet the criteria of "if and only if edit is applied on the first prompt can we receive the last prompt." For each edit type e;, the image similarity between the image generated from the modified prompt and the last image for traces in Se, calculated and
4

---

the early edits in the trace. This aligns with our previous findings, which suggest that GPT-k pro- ficient in rewriting prompts and adjusting modifiers, but may struggle to predict spontaneous changes in the main subject matter of the painting.
content. In light of this, we utilize GPT-k models solely for assisting humans in the prompt editing process, and will continue to keep humans the loop for the content creation process.
Acknowledgments
6 Conclusion
The UCSB authors were sponsored by the Robert N. Noyce Trust. We thank the Robert N. Noyce Trust for their generous gift to the University of California via the Noyce initiative. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsor.
Through our experiments and analyses, we hope to gain a deeper understanding of the capabilities and limitations of using GPT-k to edit prompts for text- to-image generation, and provide valuable insights for future research in this field.
Limitations
In this work, sion model for rext-to-image generation. However, we have not examined user traces from other online text-to-image generation servers such as Midjour- ney or DALLE-2, as these models have not been publicly released and therefore cannot be replicated locally for experimentation.
focusing the StableDiffu-
References
Jean-Ban -ann-Bapisste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- Binkowski. Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198.
Additionally, the current experimental setup only uses GPT-k for a single round of editing, and future work should include human studies that use GPT-k to suggest edits for multiple turns, and compare these to user traces without the use of GPT-k. Furthermore, the current implementation only utilizes the initial prompt as input to GPT-k. In future studies, it would be beneficial to provide additional information to the GPT-k models, in- cluding additional steps of editing the generated image, in order to receive more specific feedback on the editing suggestions.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media, Inc.".
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv. 0lss2205514166.
Ethics Statement
The field of text-to-image generation is of great interest to the broad public, and every content cre- ator should have the opportunity explore its po- tential. However, the current prompt engineering process can be time-consuming and expensive for content creators, and the repetitive editing and re- generation process is computationally intensive and not envirommentally friendly. The goal of our work is to improve the efficiency of prompting text-to- image models, benefiting both users and service providers.
Daniel Matthew Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Con- stant, Mario Suajardo-Cessedae Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Univer- sal sentence encoder for english. In Conference on Empirical Methods in Natural Language Processing.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn. Artemis Panagopoulou, Yue Yang, Marianna Apid- ianaki, and Smaranda Muresan. 2023. I spy metaphor: Large language models and diffusion mod- els co-create visual metaphors. In Findings of the As- sociation for Computational Linguisties: ACL 2023, pages 7370-7388, Toronto, Canada. Association for Computational Linguistics.
We recognize that large language models (LLMs) such as GPT-k may have the possibility of generating biased content, as they may prefer the content they have seen during training, resulting in a smaller chance of showing other possibly related
5

---

Chao Dong, Chen Change Loy, Kaiming He, and Xi- aoou Tang. 2014. Image super-resolution using deep convolutional networks. IEFF Transactions Pat- tern Analysis and Machine Intelligence, 38:295-307.
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson W. H. Lau. 2019. Spatial atten- tive single-image deraining with high quality real rain dataset. 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12262-12271.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xi- aowei Xu. 1996. A density-based algorithm for dis- covering clusters in large spatial databases with noise. In Knowledge Discovery and Data Mining.
Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Ecro P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. JEEE Transactions on Image Processing, 13:600-612.
Wei Han, Shiyu Chang, Ding Liu, Mo Yu, M. Witbrock, and Thomas S. Huang, 2018. Image resouuieso via dual-state recurrent networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1654-1663.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2022. Diffusiondh: A large-scale prompt gallery dataset for text-to-image generative models.
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018. Recurrent squeeze-and- excitation context aggregation net for single image deraining. In European Conference on Computer Vision.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2021. An empirical study of gpt-3 for few-shot knowledge- based vqa. In AAAI Conference on Artificial Intelli- gence.
Lab Midjoumny.. 2022. [link].
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano. Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
Zeng. Zhe L. Lin, and Vishal M. Patel. 2021. Sketchedit: Mask-free local image manipulation with partial sketches 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 5941-5951.
A
Appendix
A.1 DiffusionDB
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In International Conference on Machine Learning.
DiffusionDB-21 (Wang et al., 2022) is on MIT Li- cense. It contains 2M user prompts with an average #token of 30.7 + 21.2. Figure 4 shows the distri- bution of #token per prompt.
100000
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Count
60000
20000
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.06125.
20
40
60
80
100
120
140
A Tokens
Figure 4: The distribution of the number of tokens per prompt in DiffusionDB (Wang et al., 2022).
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High- resolution image synthesis with latent diffusion mod- cls. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 10674- 10685.
A.2 Hyperparameters
For the DBSCAN clustering, we set eps= 0.5, min_samples= (note that we will filter out the clusters with only 1 sample, since it does not form trace of edits), and metric= For the GPT2 model, our implementation is based on GPT2LMHeadModel on HuggingFacce During inference, we set do_sample=True, num_ num_beams=5, max_new_tokens=50,, and early_stopping-True.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes. Tim Salimans, Jonathan Ho, David J. Fleet, and Moham- mad Norouzi. 2022. Photorcalistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.1 1487.
6

---

For the GPT3 models, we set temperature=0.7 and max_tokens=256. The other parameters just follows the default setting the OpenAl API.
Uver Input Prooptt
Grnented Image
side profile centered painted por trait, apidergven, naite painting concagt ars, are nonoeas, teaitirully ly backins, merrly vibrant color lires, fontastically goady, westutic octare rurder, 8 hid resclution, by 11yn knvathnns and rustart krentr and gilleard sowes
For the Stable Diffusion model, our im- plementation is based on Compvis/stable- di liffusion-v1-4 on Huggingface. We set the random seed to 41,42, 42, 43 for the three repeated runs. The heigth, width, guidance_scale and num_ wum_infereece_steee are kept the same as the original user specification collected in DiffusionDB for each trace of edits.
side profille contered painted portrast, esgan fox spickeramen, salwe painting art nouermn beuttsullly backist, sdrry vibrant fantestioally sauly, asstlatic resouttion, by tlyn cushort krontz gilleard ganes
lines,
profilee
apiderman, patating coneept art nouveau, beautifully backlii, swirly vidmant waiar liws, fantastically aeady anstrettr octone render, B neeo ton, by 1550 kuvshinev and custart lirenzz and gilleard jemss
A.3 Showcases
Table 5 presents an excerpt from an editing trace and provides examples of the four types of com- mon edits, including insert, delete, swap, and replace. Table 6 illustrates user editing trace where the edit of replace occurs in every step, and the primary subject matter of the prompt is constantly changing. Table displays two sets of prompts that have been modified by the eight covered GPT-k models. Figures 6 and 7 provide ex- amples of the edits suggested by GPT-k and those made by humans for head-to-head comparisons in terms of effectiveness and likelihood to be adopted by humans.
stde proftle centered poirtted portrast, boba fets, matie painting concapt ar art nowwees, beautifully backlst, worll vibront color Itres, fontastscally gousy. aestietic octine rerder, 5 há reselution, ilya kuvshine and sushart krentz and gilleaed gonee
stde proftle centered patitted portrans, darth beautsfully aestretic octone render, resolution, by ilyw kusshinev and sustar kreniz and ge llased yemes.
hetsatically araty.
sisk profile centered painted por trait, the puntaher, matte painting concept wet, art nouvess, teauttrully backist, sctrly vibrant color lires, fantestically gaady, aesthetic octane rarder, 5 hi resclution, by tlyn lavvshnno and custart krente and gilleard James
User Input Prompt
Generated Image
side proft in centered painted porteant, walverine, sotte pointing concept art, noureaa, besuifully backlit, swarly vibrant en lines, fantastioally gaaty. aesthettc octone rerder, K resolution, kusshinee and gillesed james.
circular ornated ceiling highly detailed
photo of ornated circular ceiling, full of paintings of angels, centered symntrrical, highly detailed
Table A user trace where the edit of replace con- stantly occurs. The primary subject matters of the prompts are in bold, and we can see that the main subject changes from step step in this editing trace.
SWAP: "circular emated" >> "omated circular"
INSERTION: "fall of painting of angels". "centered symmetrical"
ornate narble gold wall, full of paintings angels, highly detailed
than 99% and the completion of more than 100 approved HITs. The instructions provided to the MTurk annotators are displayed in Figures 6 and 7. Each HIT assignment was compensated with pay- ment of $0.30, and the average completion time for each assignment was approximately 2 minutes. This equates to an average hourly pay of $9-10.
REPLACE: "ceiling" >> "marble and gold wall" DELETION: removed "centered symmetrical"
Table 5: Common types of edits.
A.4 Human Evaluation
A.5 Other Image Similarity Metrics
In our study, we recruited Amazon Mechanical Turk annotators to conduct human evaluations. To ensure the quality of the evaluations, we established two additional qualifications for the annotators, which included a HIT Approval Rate of higher
In line with previous studies on image edit- ing/generation (Dong et al., 2014; Han et al., 2018: Li et al., 2018; Wang al., 2019; Zeng et al., 2021), we also employed the following metrics to evaluate the similarity between images: (1) SSIM (Wang
7

---

SSIM
PSNR
Model
fn-1-fn
RNB
KNB
spt2-base b-as
17.60
52.92
3pt2-redium
17.54
22.96
52.60
9.36
10.45
53.30
spt2-1arge
17.52
22.94
$2.80
9.36
10.45
$2.57
gpt2-xl
12:51
22.86
52.98
9.35
10.44
53.29
18.93
20.39
9.16
9.57
spt3-ada
17.11
22.28
53.71
9.33
10.37
$2.67
upt3 bebbbbe
16.97
22.12
$2.85
9.26
10.30
$2.40
3pt.3-curie
17.41
22.80
53.51
9.29
10.36
52.25
gpt3. Spt33n-dai S-davinci
16.91
22.00
51.92
9.22
10.27
52.32
Table The SSIM and PSNR scores to evaluate image similarity. Here, 21, 4 in denotes first, last-but-one, and last image in the trace of edits; e' is the image generated from the modified prompt, and i is the image that is most similar to i' with regard to current similarity metric.
Prompt
The user wants to draw picture and is deciding upon what key elenents should be included. You add, swap, or renove the modifiers, or even replace the main character. Please refer to the examples of edits and predict the elements in the final painting.
Input: (first promot the k-nost related trace) Output: (last pronot in the le-ost related trace)
Input: (first pronot in the (k-1)-nost related trace]
Shot
Output: (last prampt (l-t))-ost related trace)
Input: (first pronot in the most related trace)
Output: (last pronot the related trace)
Input: (first pronpt in current trace)
Output:
Table 8: The prompts for GPT-3 and GPT-3.5 models.
et al., 2004) which assesses pixel-wise errors from the perspective of luminance, contrast, and struc- ture; (2) PSNR, which compares pixels using the mean squared error.
mar All-mixed
Insert
Delete
mace
Swap
Replace
70
GB
Table 7 presents the SSIM and PSNR scores used to evaluate image similarity. For each metric, we established the similarity score between the first and last images (11-t,) and the last-but-one and last images (in-1-i,) as baselines. The similarity between the images generated from the modified prompts and the target image of the current trace (i'-i,,) is comparable the baselines per PSNR, however, is lower as per SSIM. However, also observed that the similarity between i'- is significantly higher than the basclincs, as per both listed metrics. This suggests that even though the image generated from the modified prompt may not be directly similar to the final target, it may be related to the intermediate steps in the editing trace. The RNE results, as per SSIM and PSNR, indicate that E' is most similar to images in the middle of the trace in terms of pixel-wise comparison.
16
SSIM
8.8
PSNA
CLIP COSTe
Figure The SSIM, PSNR and CLIP cosine similarity scores between the image generated from the GPT-3.5- modified prompt and the last image. Results are reported on Seval (All-mixed), Smseet Saclete Sswap and Steplace.
tion 5, SSIM and PSNR have similar performances on the ablated evaluation traces metric scores on traces that only involve insert, delete and swap are higher than or comparable to correspond- ing baselines. Meanwhile, we notice that replace leads lower image similarities, which applies all three image similarity metrics.
Figure plots the effect of the four common types of edits. Similar to what we found Sec-
8

---

Tracek1
Iettal Prompt User lddiiin Thoce
and mers on crimon and Nock hockgronnd inroan Uluoraaton
Last Pronue in User Ediing Tracs
und csscs uural cciino and Bock backgronl, inziore illustatios highly oceate, eupislee deall, its,
gptc-bose
and noes on crimon and Nack bociground iorcare liusration, highly decaled, simple, no jagged lines, smooth ansuatin, cenrered anwerk ohey yoire fazlers, wlup and arge calkmakii and unketo shinkai and studo
und nocs ciinnu Hack lockground intaatee Ulustaaioon
greli tymt animation
Moditted Prompe
apt2-lorge
back bakaroudd
Nack bacegound.
iluurraionn denalled aword wiining. treeding anstation, R.
2pt3-bsbtagg aptimcurte Le
girs
bacegeonnd by deceech, ingiieed by
puss noes ciaroo Nack badkpound, intieute Illsttaionn davil hellyer
gpt3. devinii
pura and neck un crime back hadeemounl intraaie ilkudnsim, oolon, ounzapt arstation
Tracok2
Isitial
bouls iiled
Phorge
apt
botis fileal decato, desiee.
suscapt slaap
2p1?
hight digial painting
ontzape shop
by angoen grop stphenee
aptz-large
Moditiel Poosps
4pt2-e3
butil
decuro, by
Wayar Buzkowe high deail, cincuatic, appccisty
1pt add
dream filkdd heatl:
3pt3-bablogg
transporem planti bonle filld with dreamss, clese up, mm less phonogaph
apt2mour Lo
botile filed will, dicure, desunlike baskgroual
spt3. 3 drdeiinci
bettiz of cotrefail crcems, vitee orkor, droentiee clemants arrralion, nagical realion, abstrcci painting
Table 9: Two sets of examples that compares the modified prompt generated by each GPT-k model.
Instructions
Shortzute
Whithh with offecive?
Select option
Edit more effective
The goal this stucy examine the we can better prompt text-lo-mmage ganeration models (such Midooumey, Stable Diffusion. DALLE-2, etc).
2 Tie
3 Edit more effective
presanted an initial prompt the corraspondigg image generated Stable Diffusion. Your task consider yourself artist and decide how you would modily the promot to improve the image. You will be provided with two candidate edits of the inttial prompt, along with images generated from the edited prompts.
an annotator, you
Please select the edit that you belleve is more effective in Improving the Image. To be cleer, an effective would be that producs an image that more visually appealing or closer desired outcome.
Prompt for Texa-to-Image
medieval city map, lankhmar, waterdeep, dungeons and di dregons, black and white, engraving, city walls, towers, port, twisting streets, allays, market square
Image generated from the Initial prompt:
Edit
medioval lankhhaar waterdoop, dungsons and dragons, black and white, engreving, city walls, towers, port, twisting streets, alleys, market square, highly realistic Illustrttonn.
medieval city mao, lankhmar, waterdeepp. dungeors and dragons, black and white, Illustationn, city walls, towers, port, twisting streets, alleys, square, fantasy carlography, serial view. tourist map
Image generated Edit
Image generated
Figure 6: This screenshot illustrates the interface used in the MTurk study for head-to-head comparison of the effectiveness the edits suggested by the GPT-k model and those made by humans. In this specific example, "Edit I" was suggested by GPT-2-x1, while "Edit 2" represents the most similar human edit in the original editing trace.
9

---

Instructione
Shortzuts
Witch edi would you acopt to impiove the met-to-imapp prompe?
Select an option
lively Edit
The goel this study is to examine the how we can betler prompt the texi-to-image generation models (such es Midjoumey, Stable Difusioon DALLE-2, 2, etc).
Tie
3 I'm more likely to adopt Edit
As an annotatior you will be presented with an intaal prompt and the corresponding image generated by Siable Diffusion, Your tack is to conslder youreelf as an artist and decide how you would modily the promot improve the image. You provided along with the images generated from the editdd prompts.
candidate edits the initial prompt,
more likely adooted by you. words, please select the edit that tnink would result in better image.
Please select
believe
Prompt Text-to-lmage
painting nice summer day, flowers and hills, on camvves
Image generated ennrrtee from the initial prompt:
Edit
peaceful day, roiling hills, flowers, and small stream, on carvas.
painting
day, and hills oll canvas
generated Edit
Image generated from Edit 1:
Figure 7: This screenshot illustrates the interface used in the MTurk study for head-to-head comparison of the edits suggested by the GPT-k model and those made by humans. The purpose of this evaluation was to assess the likelihood of an edit being adopted by humans. this specific example, "Edit 1" was suggested by GPT-3.5- davinci, while "Exlit 2" represents the most similar human edit in the original editing trace. Notice that in this example, the user actually made typo (misspelled "canvas" into "camvas" in both the initial prompt and the prompt in Edit 2), which confused the Stable Diffusion model and thus led to unrelated renderings.
10


Reviews:
[{"review_id": "XaFIfc96tf", "reviewer": "Reviewer_AHM3", "summary": "", "questions": "**Question A**: Would you mind providing some more details on how you used the GPT-3 models in the evaluation, especially what parameters were provided to the API and also how expensive was it to execute the experiments described in the paper?\n\n**Question B**: Is there any specific reason why the `gpt-3.5-turbo` model wasn't used in the experiments as opposed to `text-davinci-003`, which is both more expensive and which the OpenAI's docs now consider legacy?\n\n**Question C**:  Would you be able to provide any intuition as to what extent are the GPT-3 results robust to changes of the prompt shown in Table 8?\n\n**Question D**: Similarly to the previous question, how did you arrive at choosing 16 shots for the in-context learning and to what extent would varying this number change the results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful and insightful review. We appreciate your feedback and will take them into account when revising our paper.  Here are our responses to your questions: \n\n> **Question A: Detailed parameter setup for GPT models and Stable Diffusion**\n\nFor the GPT2 model, our implementation is based on `GPT2LMHeadModel` on HuggingFace. During inference, we set `do_sample=True`, `num_beams=5`, `max_new_tokens=80`, and `early_stopping=True`.\n\nFor the GPT3 models, we set `temperature=0.7` and `max_tokens=256`. The other parameters just follows the default setting the OpenAI API. \n\nFor the Stable Diffusion model, our implementation is based on `CompVis/stable-diffusion-v1-4` on Huggingface. We set the random seed to `41,42,43` for the three repeated runs. The `heigth`, `width`, `guidance_scale` and `num_inference_step` are kept the same as the original user specification collected in DiffusionDB for each trace of edits.\n\n> **Train/validation split**\n\nThanks for pointing out! The train/validation split is 70k vs. 30k. We apologize for causing the confusion and will definitely fix this in the revision.\n\n> **How to classify the edit category if multiple types of edits appear?**\n\nWe will count the appearance of each edit. As an illustrative example, if we compare the following two prompts:\n\n* prompt1: *'The quick brown fox jumps over the lazy dog'*\n\n* prompt2: *'Look! The brown quick fox jumps over the dog!'*\n\nThen there are three types of edits from prompt1 to prompt2:\n* insert x1: (prompt2 adds “Look”)\n\n* swap x1: (prompt1 “quick brown” -> prompt2 “brown quick”)\n\n* delete x1: (prompt2 removes “lazy”)\n\n\n\n> **Question B: Why wasn’t gpt-3.5-turbo used?**\n\nThanks for asking! The main experiments of this submission were conducted during January 2023, by which time gpt-3.5-turbo has not been released yet (release date: March 1st, 2023). \n\n\n> **Question C: Would you be able to provide any intuition as to what extent are the GPT-3 results robust to changes of the prompt shown in Table 8?**\n\nThanks for asking! In our pilot study, we experimented with several different framings of the instructional prompts for GPT-3, such as:\n\n* framing the task as summarization: *“Summarize the following prompt to around {num_kept_tokens} tokens.”*\n\n* framing the task as rewriting: *“Rewrite the following prompt to list out the key elements in a painting, no numbering, connect with comma.”*\n\n* describing the scenario but not specifying the type of edits: *“The user wants to draw a picture, but is not satisfied with the current drawing and keeps modifying the key elements in it. The trace of edits is provided. Please predict the elements in the final painting.”*\n\nWe notice that when using the framing of summarization or rewriting, GPT-3 tends to summarize/rearrange the existing keywords that appeared in the original prompts, which is quite different from human edits. \n\nWith the third framing, the GPT-3 predictions are more flexible, but we notice that the edit of “insertion” occurs more often than other types of edits. As a result, we further add a line of instruction specifying the possible types of edits: *“You may add, swap, or remove the modifiers, or even replace the main character”*, which makes up the instruction used in the submission, as listed out in Table 8. \n\n\n> **Question D: How did you arrive at choosing 16 shots for the in-context learning and to what extent would varying this number change the results?**\n\nWe conduct pilot studies varying the number of prompting examples for GPT3 models, and choosing k=16 is based on empirical evidence. \n\nThe following table shows prompting GPT-3.5-davinci with k=2, 4, 8, 16, 32 examples, as an extension to the results in Table 2 in the submission. We can see that the change of k does not have a significant impact on the final results. \n\n| k  | $i’$ - $i_n$ | $i’$ - $i_{MS}$ | RNE (%) |\n|----|--------------|-----------------|---------|\n| 2  | 68.31        | 77.33           | 68.97   |\n| 4  | 68.66        | 77.85           | 69.47   |\n| 8  | 68.84        | 78.07           | 69.53   |\n| 16 | 68.79        | 78.09           | 69.22   |\n| 32 | 68.90        | 78.02           | 69.17   |\n\n\n> **Typos**\n\nThanks for pointing out! We apologize for the oversight and will fix the writing issues in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An analysis of the prompt edits made by users at the Stable Diffusion Discord server\n- Exploration of the difference between the images generated by user-edited vs. GPT-generated prompts\n- Evaluation of various GPT-based models on this task, with the GPT-2 model family being finetuned and GPT-3 model family being prompted\n- Human evaluation of the prompts generated/edited by the GPT-based models, showing them to be better or comparable to user edited prompts\n- An ablation study on the edit types (insert/delete/swap/replace) on the generated output image", "reasons_to_reject": "The paper is well written and hence there are only a few reasons why this paper might be rejected.\n\nPerhaps the strongest one is the fact that while the GPT-based models are a key part of the analysis, except for the prompt the authors used the paper doesn't contain any further details on how it was used as a metric. Furthermore, the OpenAI's APIs provide various parameters such as `temperature`, `top_p`, `presence_penalty` and `frequency_penalty`, which are not specified in the paper, further complicating potential future replication efforts. This should also be applicable to GPT-2 models that were publicly released and are not available only via an API (e.g. the HuggingFace Transformers provide APIs for specifying similar parameters).\n\nA similar issue can be found with the Stable Diffusion model that was used for image generation. It is not obvious what parameters, except for the prompts, were varied when the images were generated, and to what extent would changing these have an impact on the presented results.\n\nThere were also two relatively minor inconsistencies we found in the paper which unfortunately detract from the quality of the paper. \n\nThe first can be found on line 151 when they authors mention \"We split the 100k trace of edits into two halves,\" but the next part of the sentence says \"with 30k traces used for evaluations and the remaining 70k serving as holdout set\". When splitting something into two halves, it would be reasonable to expect a 50:50 split but that is not what we observe here. It is therefore unclear which of the potential interpretations the authors meant in this sentence.\n\nThe second can be found at the discussion of edit types which starts on line 217. When using SequenceMatcher, it is unclear how are the edits classified. If for instance one generated prompt both swaps the order of words, inserts a few new words towards the beginning and removes a few words from the end of the prompt, which category would such a change fall into? Would it be all of them or just one of them or neither?", "questions_to_authors": "**Question A**: Would you mind providing some more details on how you used the GPT-3 models in the evaluation, especially what parameters were provided to the API and also how expensive was it to execute the experiments described in the paper?\n\n**Question B**: Is there any specific reason why the `gpt-3.5-turbo` model wasn't used in the experiments as opposed to `text-davinci-003`, which is both more expensive and which the OpenAI's docs now consider legacy?\n\n**Question C**:  Would you be able to provide any intuition as to what extent are the GPT-3 results robust to changes of the prompt shown in Table 8?\n\n**Question D**: Similarly to the previous question, how did you arrive at choosing 16 shots for the in-context learning and to what extent would varying this number change the results?", "paper_topic_and_main_contributions": "The paper investigates the concept of \"prompt editing\" associated with text-to-image generation, in which the users apply successive edits to their prompts in order to generate their desired image. It does so by analyzing a dataset of user prompts scraped from the official Stable Diffusion Discord server and employing various GPT-based models to improve the prompt editing process. In their experiments the authors observe that while the GPT-based models seem to mostly insert modifiers, human users tend replace words and phrases, yielding a significantly different image in the end. Furthermore, it also seems that editing the prompt via GPT-based models results in images more similar to intermediate edits made by the users rather than the final image, suggesting that the GPT-based models may be more effectively used to reduce the number of edits necessary to arrive at the final image.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 051: \"issue\" -- it would be best to replace this word with say \"phenomenon\" or \"figure\"\n- line 060: \"LLMs and T2I models are trained on different modalities and architectures\" -- although the models do use different architectures, it is not clear what training on architectures would refer to\n- line 138: \"denotes\" -> \"denote\""}}, {"review_id": "ksL6z0NIr1", "reviewer": "Reviewer_bKnd", "summary": "", "questions": "A: What are hyperparameters for clustering prompts into traces of edits and how were they chosen? I was surprised to see that you have clusters with such a huge number of edits, did you inspect them to see that the clusters make sense? How many clusters in absolute numbers do you have after selecting only those that have at most 20 edits?\n\nB: 131-135: How is the ordering for the traces determined?\n\nC: 150-152 this does not make sense to me. Before you mentioned you only focused on 95% of the traces. How come now it is the full 100K?\n\nD: 225-227 How exactly did you identify edit type using sequenceMatcher?\n\nE: 233: gpt-3 are also autoregressive but you say their distribution is more similar. so the claim \"GPT-2 models, due to their autoregressive training nature, have a tendency towards continual generation, resulting in a majority of edits being insert\" does not hold.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful reading and constructive comments! We will incorporate them in the revision. Here are our responses to your questions: \n\n> **Evaluation or verification of edit clusters**\n\nWe manually checked 177 editing traces written by 50 users. Common types of clustering errors include:\n\n* \"over-segmentation\" where DBSCAN splits an editing trace into several sub-traces (12/177); \n\n* \"under-segmentation\" where one predicted editing trace covers several unrelated traces (4/177). \n\nOverall, more than 90% of the checked clustering results are valid and correct. Among the two types of clustering errors, we would like to note that \"over-segmentation\" is less problematic, since the sub-traces still center around the same topic. Consequently, the feasibility of deriving meaningful edits endures in this scenario.\n\nWe’ll incorporate such results in the revision.\n\n> **Why only using the n-th edit of the trace for fine-tuning and few-shot prompts?**\n\nThanks for raising the question on the experimental design. We are more than delighted to provide a deeper insight into our rationale. \n\nIn the submission, we directly employ the last prompt in the editing trace for tuning and in-context learning. This decision emanates from our primary objective of conducting more *efficient* prompt editing. In line with this goal, our intention is to expedite the evolution of the prompt toward its final form, seeking to minimize the number of necessary edits.\n\nIt is indeed acknowledged that this approach might bypass several intermediate editing steps. However, we wish to emphasize that this particular choice is entirely intentional. By employing this strategy, we are intentionally encouraging the language model to proficiently skip or amalgamate redundant editing stages, in turn striving for an elevated efficiency in the prompt editing process.\n\n\n> **How did you select models for human evaluation?**\n\nThank you for inquiring about our model selection process for the human evaluation phase. Specifically, we made a deliberate decision to select one model from each of the model categories encompassing GPT-2, GPT-3, and GPT-3.5, as outlined in Table 1. Within each distinct model category, our focus gravitated towards the selection of the most extensive model available. Thus, we choose the largest model with the most parameters (`gpt2-xl` for GPT-2, `text-curie-001` for GPT-3, and `text-davinci-003` for GPT-3.5).\n\n> **Framing of the human evaluation**\n\nWe invite the annotators to conduct head-to-head comparisons, and provide three options of “win/lose/tie”. If the annotator finds it hard to decide which edit is better (e.g., Figure 6), then the annotator could choose “tie”. \n\n\n\n> **Compare the modified prompt to the first image on which the edit type was applied**\n\nIn the submission, we compare the modified prompt to the last prompt in the editing trace. This is because we attempt to conduct more efficient prompt editing, which means we want the modified prompt to be as similar to the final prompt as possible. \n\nWe are happy to provide more results comparing the modified prompts of each edit type to the original prompt. In the following table, we list out the CLIP similarity between the modified prompt $i’$ and the first prompt in the trace of edits $i_1$, as an extension to the results in Figure 2.\n\n| Edit Type | $i’$ vs. $i_1$ | $i’$ vs. $i_n$ (results in Figure 2) |\n|-----------|----------------|--------------------------------------|\n| All-mixed | 75.36          | 68.81                                |\n| Insert    | 76.08          | 70.54                                |\n| Delete    | 76.08          | 69.61                                |\n| Swap      | 75.98          | 69.08                                |\n| Replace   | 74.78          | 66.22                                |\n\nWe can see that the edit types of `insert`, `delete` and `swap` have comparable impacts on raising the CLIP image similarity, while the edit of `replace` lowers the CLIP image similarity. The trend aligns with the results comparing the modified prompt with the last prompt as discussed in the submission (Figure 2, line 243-270).\n\n> **Question A: Hyperparameters for clustering prompts into trace of edits**\n\nFor the DBSCAN clustering, we set:\n\n*  `eps = 0.5`. The `eps` is the maximum distance between two samples for one to be considered as in the neighborhood of the other. The value of eps of 0.5 is the default value as suggested by the package readme. After varying the eps value and empirically checking the clustering results, we decided to continue with the default eps value of 0.5.\n\n* `min_samples=1`. The `min_samples` is the number of samples (or total weight) in a neighborhood for a point to be considered as a core point. Note that we will filter out the clusters with only 1 sample, since it does not form a trace of edits.\n\n* `metric ='cosine'`, which is the metric to use when calculating the distance between instances in a feature array.\n\n\n\n> **Question A & C: How many clusters in absolute numbers do you have after selecting only those that have at most 20 edits?**\n\nThanks for asking! We received 100433 traces of edits after clustering. All of the 100k traces were used in our study. The validation set (30k) was randomly sampled from the traces with less than 20 edits. \n\nWhat we meant by *“in the following experiments, we consider traces with at most 20 edits”* (line 123-124) is that the evaluation results were reported on traces with at most 20 edits. \n\nWe sincerely apologize for causing the confusion, and would definitely clarify this in the revision.\n\n\n> **Question B: How is the ordering for the traces determined?**\n\nThe DiffusionDB dataset collects the timestamp of each user prompt. We order the prompts by corresponding timestamps in each trace of edits. We will clarify this in the revision.\n\n\n> **Question D: How exactly did you identify edit type using SequenceMatcher?**\n\nThanks for asking! We will clean and release the code used in this study upon acceptance. Since we are not allowed to put external links and considering the fact that there is no word limit in the rebuttal, we decided to directly paste the script we used here: \n\n\n```python\nimport string\nfrom difflib import SequenceMatcher\nfrom nltk.tokenize import word_tokenize\nfrom nltk.corpus import stopwords\nfrom collections import defaultdict\n\nstopword_list = set(stopwords.words('english'))\n\ndef _preprocess(sent):\n   sent = sent.lower().translate(str.maketrans('', '', string.punctuation))\n   word_tokens = word_tokenize(sent)\n   filtered_tokens = [w for w in word_tokens if not w in stopword_list]\n   return filtered_tokens\n\n\ndef find_edit_type(a, b):\n   edit_type = defaultdict(lambda:0)\n   a = _preprocess(a)\n   b = _preprocess(b)\n   s = SequenceMatcher(None, a, b)\n   opcode_list = list(s.get_opcodes())\n\n   # check if there is any equal sub-sequence\n   # if there is no equal, then not valid\n   no_equal = True\n   for tag, _, _, _, _ in opcode_list:\n       if tag == 'equal':\n           no_equal = False\n           break\n   if no_equal:  # not valid, early return with all zeros\n       return edit_type\n\n   # check the last op\n   tag, i1, i2, j1, j2 = opcode_list[-1]\n   if tag == 'replace':\n       if i1 == i2 and j1 < j2:\n           edit_type['insert'] += 1\n       elif i1 < i2 and j1 == j2:\n           edit_type['delete'] += 1\n       else:\n           edit_type['replace'] += 1\n   else:\n       edit_type[tag] += 1\n  \n   # check the previous ops\n   a_str_list = []\n   b_str_list = []\n   for idx, (tag, i1, i2, j1, j2) in enumerate(opcode_list):\n       if idx < len(opcode_list)-1:\n           edit_type[tag] += 1\n\n       # check if there is any swapping\n       if tag != 'equal':\n           a_str = ' '.join(a[i1:i2])\n           b_str = ' '.join(b[j1:j2])\n           if a_str != '':\n               for _str in b_str_list:\n                   if _str.find(a_str) > -1:\n                       edit_type['swap'] += 1\n                       edit_type[tag] -= 1\n                       break\n           if b_str != '':\n               for _str in a_str_list:\n                   if _str.find(b_str) > -1:\n                       edit_type['swap'] += 1\n                       edit_type[tag] -= 1\n                       break\n\n           if a_str != '':\n               a_str_list.append(a_str)\n           if b_str != '':\n               b_str_list.append(b_str)\n\n   return edit_type\n\n\ndef compare_prompts_and_find_edits(prompt_list):\n   all_edit_types = defaultdict(lambda:0)\n\n   for a, b in zip(prompt_list[:-1], prompt_list[1:]):\n       edit_count = find_edit_type(a, b)\n       for edit_type, count in edit_count.items():\n           all_edit_types[edit_type] += count\n\n   for edit_type in ['insert', 'swap', 'delete', 'replace']:\n       print(f'{edit_type}: {all_edit_types[edit_type]}')\n\n   return all_edit_types\n\n\nif __name__ == '__main__':\n   edit_trace = [\n       'The quick brown fox jumps over the lazy dog',\n       'Look! The brown quick fox jumps over the dog!',\n   ]\n   compare_prompts_and_find_edits(edit_trace)\n\n   \"\"\"\n   Output:\n       insert: 1\n       swap: 1\n       delete: 1\n       replace: 0\n   \"\"\"\n```\n\n> **Question E: Why GPT-2 and GPT-3 have different patterns of edits?**\n\nThanks for pointing out the loose statement! We will provide a more rigorous analysis in the revision. \n\nWe agree that both GPT-2 and GPT-3 models undergo training in an autoregressive fashion. However, a significant difference between the two types of models is that GPT-3 demonstrates emergent ability, which enables it to engage in in-context learning. Unlike GPT-2, which solely receives the specific prompt requiring editing during testing, GPT-3 is also provided with additional editing instances as supportive exemplars for in-context learning. This emergent ability within GPT-3 is a pivotal factor that diverges from mere plain continuation. Consequently, while GPT-2 indeed adheres to its autoregressive inference manner, the emergent capability of GPT-3 enables it to attempt other edit types that simulate human-like editing behavior, as illustrated by the in-context examples.\n\n\n> **Missing reference**\n\nThanks for pointing us to this publication on related topics! We will definitely include this in the revision.\n\n> **Typos**\n\nThanks for pointing out! We apologize for the oversight and will fix the writing issues in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides interesting and detailed experiments on the benefits and drawbacks of incorporating GPT-k models into the text-to-image pipeline. Some recent work was focusing on incorporating LLMs and diffusion models (see missing references) but this study provides more formal empirical investigation of this phenomenon. \n\n2. The experiments are well-defined and executed. While additional models and settings are possible, we can already draw some preliminary conclusions from this study about the usefulness of GPT models to improve t2i generation.", "reasons_to_reject": "1. Authors create a dataset of t2i prompt edits by clustering a dataset of t2i prompts into 100K clusters using sentence embeddings. However, no evaluation or verification of edit clusters is provided. Thus, it is hard to judge if the clusters make sense or not or if they even refer to the same image. Taking the clusters with more than just a few instructions (authors select clusters of up to 20 instructions) seems like an unrealistic setting to me and potentially increasing the chance that some of the clusters will have unrelated prompts. It would be interesting to see how smaller clusters of e.g. under 5 instructions would affect the results.\n\n2. Each edit cluster has 1 to n edits. I find problematic how only the n-th edit of the trace was used for fine-tuning and few-shot prompts. First, as there was no guarantee that the edits were about the same image, but also the first and last edits will be very different and skipping multiple steps in the edit trace. This potentially teaches the model to hallucinate edits as it has no way to understand the intermediary steps. For example, steps in the middle might have have had different edit types completely altering the final prompt. There needs to be control for semantic/lexical similarity of first and last prompt. The experiment could also include models fine-tuned on a single edit step for a single or multiple edit types. Another way would be to vary n-th edit e.g. from 1 k=n to see the impact on i'-i_MS.\n\n3. The human evaluation is not very clear to me. First, gpt-3-curie was chosen even though babbage has better CLIP scores, RNE (Table 2), and closer distribution to human edits (Table 4). Second, the framing of the task as choosing between prompts is very vague. For example in Figure 6 it is very hard to choose what edit would be best. It would be more clear to frame it explicitly as an image selection task (which I suspect crowdworkers did anyway).\n\n4. In the effects of edit types experiment, the CLIP score reported is only between generated image and the last image in the edit cluster. Since each edit cluster can contain multiple types of edits, it would be more appropriate to compare to the first image on which the edit type was applied.", "questions_to_authors": "A: What are hyperparameters for clustering prompts into traces of edits and how were they chosen? I was surprised to see that you have clusters with such a huge number of edits, did you inspect them to see that the clusters make sense? How many clusters in absolute numbers do you have after selecting only those that have at most 20 edits?\n\nB: 131-135: How is the ordering for the traces determined?\n\nC: 150-152 this does not make sense to me. Before you mentioned you only focused on 95% of the traces. How come now it is the full 100K?\n\nD: 225-227 How exactly did you identify edit type using sequenceMatcher?\n\nE: 233: gpt-3 are also autoregressive but you say their distribution is more similar. so the claim \"GPT-2 models, due to their autoregressive training nature, have a tendency towards continual generation, resulting in a majority of edits being insert\" does not hold.", "paper_topic_and_main_contributions": "The paper investigates the benefits of incorporating GPT-k into the text2image pipeline by using GPT-k to edit the initial t2i prompt. They then investigate how these edits perform compared to human edits. They also investigate what common edit types are made by humans compared with GPT-k and how edit types and # of edits affect the results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, and Smaranda Muresan. 2023. I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7370–7388, Toronto, Canada. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "023-024 \"predicting spontaneous changes in the primary subject matters\"  not very clear what are primary subject matters\n\n049-053 Figure 1 and description is very unclear. What is a trace and how did you define an edit? the division symbol is also confusing as it seems # of edits /trace is simply the size of a cluster of prompts, probably just keeping # edits is simpler\n\n059-062 \"LLMs and T2I models are trained on different modalities\" that's not entirely true as many multimodal models are being released recently (including announced GPT-4)\n\n094 adjusting -> adjustment\n\n152 holdout -> heldout"}}, {"review_id": "8cVjdhciaA", "reviewer": "Reviewer_zSe4", "summary": "", "questions": "In Figure 7 in the appendix, it was quite surprising to see that Edit2 and the initial prompt are the same but ended up with both failure generations, though totally different, have you analyzed factors of such failure cases/prompts/edits? Or if any filtering has been done from the traces before performing in-context learning/prompting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thought-provoking feedback! Here are our responses to your questions: \n\n> **Analyze the failure case in Figure 7 in the appendix**\n\nThanks for raising the question! For this specific example in Figure 7, the initial prompt and the prompt in Edit2 are all user-written prompts from the DiffusionDB dataset. Notice that in this example, the user actually made a typo (misspelled “canvas” into “camvas”), which confused the Stable Diffusion model and thus led to unrelated renderings. We will note this down in the revision.\n\n\n> **Is there any filtering on the traces before performing in-context learning/prompting?**\n\nTo provide clarity, we want to emphasize that we refrained from implementing any filters on the clustered edit traces. It was a deliberate choice not to address potential errors (e.g., typos,  misspelled artist names) within the user prompts. Our rationale behind this decision is rooted in the fact that both the introduction of such errors and the potential subsequent rectification process are integral components of the broader \"prompt editing\" procedure, a focal point within the scope of our study.\n\nIllustratively, let's consider Figure 7. When the user mistakenly typed \"camvas\" instead of \"canvas,\" leading to an unsatisfactory and unrelated image rendering, a pivotal question arises: how might the user modify the prompt to enhance Stable Diffusion's rendering outcome? Would they incorporate additional modifiers or alter the keywords? From our vantage point, we possess insight into the root cause behind the failure in Stable Diffusion's rendering, which is the typo within the prompt itself. Given the presence of this typo, any subsequent edits would likely still result in unsuccessful and unrelated image generation. Consequently, a more efficient and effective approach to editing involves addressing the typo initially. Notably, this is precisely the strategy employed by GPT-3.5-davinci in Edit1, underscoring how leveraging LLMs effectively enhances the efficiency of prompt editing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well written with clear motivation, solid experiments, and presented with concrete examples. It is interesting to see such analysis and how GPT models can contribute to the prompting process of text-to-image generation.", "reasons_to_reject": "Some of the qualitative examples are a bit confusing without detailed analysis, please see the questions.  ", "questions_to_authors": "In Figure 7 in the appendix, it was quite surprising to see that Edit2 and the initial prompt are the same but ended up with both failure generations, though totally different, have you analyzed factors of such failure cases/prompts/edits? Or if any filtering has been done from the traces before performing in-context learning/prompting?", "paper_topic_and_main_contributions": "In this paper, GPT models are used to modify prompts for text-to-image generation. The authors show that, in the generation process, GPT modified prompts can reduce the number of edits by around 20%. A human study is conducted to compare the type of GPT generated edits and human edits, as well as the practicality of the usefulness of the generated prompts.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the capabilities and limitations of using GPT-k to edit prompts for text-to-image generation. It focuses on understanding the capability of the GPT-k models for text-to-image generation rather than trying to improve it. Overall this is a solid paper. Reviewers found this paper well written and easy to follow. 2 reviewers found the experiments well-defined and executed with concrete examples and good details. One reviewer commented on the lacking of qualitative evaluation and verification of the edit clusters. The authors were able to provide the manual verification results and the cluster error breakdown in the response. This is an important aspect for supporting the finding presented in this paper. The authors should consider add these details in the later version.
Decision: Accept-Main