Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images
Authors: Ali Naseh
Katherine Thai
Mohit Iyyer
Amir Houmansadr
Venue: COLM (2024)
Keywords: Trustworthy ML, Text-to-Image models
TL;DR: This paper presents a novel method using multi-modal AI models to efficiently replicate prompts from popular text-to-image APIs, highlighting potential new threats to AI-generated and natural image markets.

Abstract: With the digital imagery landscape rapidly evolving, image stocks and AI-generated image marketplaces have become central to visual media. Traditional stock images now exist alongside innovative platforms that trade in prompts for AI-generated visuals, driven by sophisticated APIs like DALL-E 3 and Midjourney.  This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of these platforms, introducing an original attack strategy. Our method leverages fine-tuned CLIP models, a multi-label classifier, and the descriptive capabilities of GPT-4V to create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at a markedly lower expense. In presenting this strategy, we aim to spotlight a new class of economic and security considerations within the realm of digital imagery. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices (\$0.23 - \$0.27 per image), emphasizing the need for awareness and strategic discussions about the integrity of digital media in an increasingly AI-integrated landscape. Additionally, this approach holds promise as a tool for data augmentation, potentially enhancing machine learning models by providing varied and cost-effective training data. Our work also contributes to the field by assembling a dataset consisting of approximately 19 million prompt-image pairs generated by the popular Midjourney platform, which we plan to release publicly.

Published as a conference paper at COLM 2024
Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images
Ali Naseh, Katherine Thai, Mohit lyyer & Amir Houmansadr University of Massachusetts Amherst {anaseh, kbthai, Sithai,myyn miyyer, amir 0cs. umass. edu
Abstract
With the digital imagery landscape rapidly evolving, image stocks and Al-generated image marketplaces have become central to visual media. Traditional stock images now exist alongside innovative platforms that trade in prompts for Af-generated visuals, driven by sophisticated APIs like 3 and Midjourney. This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of these platforms, introducing original attack strategy. Our method leverages fine-tuned CLIP models, multi-label classifier, and the descriptive capabilities of GPT-4V to create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at markedly lower expense. In presenting this strategy, we aim to spotlight new class of economic and security considerations within the realm of digital imagery. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices ($0.23 $0.27 per image), emphasizing need for awareness and strategic discussions about the integrity of digital media in an increasingly Al-integrated landscape. Additionally, this approach holds promise as a tool for data augmentation, potentially enhancing machine learning models by providing varied and cost-effective training data. Our work also contributes to the field by assembling dataset consisting of approximately 19 million prompt-image pairs generated by the popular Midjourney platform, which we plan release publicly.
Introduction
In recent years, image stocks and marketplaces have become increasingly important in the commercial and business sectors. Alongside traditional stock images, known for their high quality and compositions by expert photographers, new trend has emerged in the form of marketplaces for AT-generated images, such as PromptBase, PromptSea, and Neutron Field. & Unlike traditional stocks where the images themselves are traded, these innovative platforms trade in the prompts that lead to the creation of Al-generated images. Advanced text-to-image APIs like DALL-E 3 (Betker et al., 2023) and Midjourney, with their extraordinary ability to generate stunning visuals, are at the forefront of this trend. However, identifying the right prompts to produce such images is not straightforward task (Cao et al., 2023; Oppenlaender, 2023), leading to the development of marketplaces where users can exchange their crafted prompts.
With the growing demand for purchasing prompts from Al-generated sources, and the continued interest in traditional stock images, pivotal question emerges: Can an adversary
Pntps://prompttbse..cmm
https://wwww.promptseaaio
"https://neuttootiilll.com
"htps:/wwww.middummey.yc

---

Published as a conference paper at COLM 2024
Image Embedding
Closest Image and Text Embeddings
Targeted
Keyword Extractor
Prompt
Fine -une-tuned CLIP model
Image Embadding
Mutt-label Classifier
Extracted Keywords
Targeted Image
Modiflers
+8
Generated Prompt
Instructions
Text-to-lmage
API
GPT-4V
Figure 1: Overview of our attack.
find a prompt and utilize current state-of-the-nte text-to-image models to generate similar images at lower cost? This question becomes particularly significant when considering that some of the natural images in traditional stocks are very expensive. This paper investigates this query, demonstrating how the latest multi-modal models with visual understanding capabilities can be harnessed for such attacks against these marketplaces. Our study exposes potential vulnerabilities in the digital imagery landscape and emphasizes the necessity for strategic countermeasures in response to these evolving challenges.
In this paper, we first demonstrate that if adversary is given images generated by one of the text-to-image APIs featured in Al-generated image marketplaces, they can, by accessing one of these APIs, find prompt to generate similar images. Additionally, we show that the same attack methodology can be applied to generate images that closely resemble natural images offered in traditional stock markets. This dual capability of the attack strategy underscores its potential impact on both Al-generated and natural image domains.
Can we simply utilize off-the-shelf image captioning models to recover the prompts used to generate an imput image? Current image captioning models (Li et al., 2023; 2022a; Wang et al., 2022; Chen et al., 2022) often produce general descriptions that capture the broad aspects of an image but typically lack the specificity required for text-to-image API prompts like those used in Midjourney. The textual input prompts for these APIs need to be more specific and follow particular format, usually by incorporating keywords and key phrases that significantly influence the generation (Oppenlaender, 2023). Directly using captions from standard image captioning models may not yield effective results since they often omit critical details and stylistic elements. While tools like CLIP Interrogator" can suggest corresponding text and keywords, they may not provide enough detail and are especially limited when describing natural images. Similarly, although multimodal models like Gemini and GPT-4V provide more detailed descriptions, they might still overlook essential keywords or named entities.
To bridge the gaps presented by these tools, we introduce ftree-component attack strategy: fine-tuned CLIP model (Radford al., 2021) on large dataset of Midjourney generations, multi-label classifier for related keywords and named entities, and GPT-4V for its ability to generate comprehensive prompts based on our instructions and information from the CLIP model and the classifier. We then implement a cyclic approach to refine these prompts, comparing the generated images with the original (ground-truth) image(s). The overview of our attack strategy is illustrated in Figure 1.
parent
7 htttp://hhuggingaae.o//ssssssss Ctt.ai////II----IIrggaaa hnttps:// httpss///eeeppnnn.ooogo///totoor htt.://mmnniitiittroouccioooo
2

---

Published as a conference paper at COLM 2024
Besides highlighting the potential threats to digital imagery, our approach also serves effective tool for data synthetic and augmentation. By generating varied visual content using different random seeds with the final prompts created by our pipeline, can produce diverse image sets. This capability is particularly valuable considering that traditional data collection is often time-consuming and expensive. These generated image sets can enhance the training of machine learning models across various applications, providing cost-effective alternative to real data acquisition (Tian al., 2024).
As significant contribution of our work, we have collected a large-scale dataset consisting of 19,137,11 7,40 generations from Midjourney's Discord server. This dataset, which pairs each prompt with its corresponding image or gridded image, aids in fine-tuning the CLIP model and training the multi-label classifier to identify related keywords and named entities. Our approach is validated through both automatic metrics and human evaluation, confirming that our attack outperforms existing baselines. Additionally, we provide a cost analysis to justify the feasibility of the attack. Our cost estimation indicates that an attacker can generate a reasonably similar image to the targeted one at cost of only $0.23 to $0.27.
To summarize our contributions:
1. To the best of our knowledge, this work presents the first systematic investigation of potential attacks against images generated by two of the most popular text-to-image APIs, as well as against natural images from commercial stock collections.
2. We demonstrate that attacker can generate images similar to those produced by AI, typically priced between $3 and $7, and natural stock images valued between $70 and $500 per image, all at mere fraction of the cost just-just just a few cents.
3. A significant portion of our effort was dedicated to collecting and preprocessing substantial dataset of Midjourney's generations. This dataset is not only pivotal to our project but also stands as a valuable resource for future research in this field.
2 Related Work
2.1 Text-to-Image Generation Models
The journey of text-to-image generation began with methods based Generative Adver- sarial Networks (GANs) (Goodfellow et al., 2014). These GAN-based models paved the way for the field, focusing synthesizing visual content from textual descriptions (Reed et al., 2016; Mansimov et al., 2015; Zhang et al., 2017; Xu et al., 2018; Zhu et al., 2019). In recent years, the emergence of diffusion models (Sohl-Dickstein et al., 2015) and large pre-trained autoregressive models has led to the introduction of many new text-to-image models (Ramesh et al., 2021; Ding et al., 2021; Cho al., 2020). These developments in- troduce multimodal transformer language models, which are proficient at learning the distribution of sequences of discrete image codes from given text inputs. Parti (Yu al., 2022) introduces sequence-to-sequeee thesis as a translation task, converting text descriptions into visuals. Imagen (Saharía et al., 2022) employs large language model for quality text features and introduces an Efficient U-Net for diffusion models. Latent Diffusion Models (LDM) (Rombach et al., 2022), such as Stable Diffusion (Rombach et al., 2021), employ diffusion processes within the latent space. This approach facilitates efficient model training on systems with constrained computational resources, without compromising the fidelity and quality.
autoregressive model treating text-to-image syn-
2.2 Prompt Stealing Attack
Prompt stealing attacks represent a domain closely related to our work. In such attacks, the objective of the attacker is typically to reconstruct the original prompt used in generating an image. However, our goal is to replicate Al-generated and natural images, where the resulting prompt does not necessarily need to match the original one precisely, word-by- word. Although this area has seen limited exploration, there have been a few metods in the context of both Large Language Models (L..Ms) (Zhang & Ippolito, 2023; Sha & Zhang,
3

---

Published as a conference paper at COLM 2024
2024) and -ax-t models (Shen et al., 2023). Concurrently, Liu et al. (2024) attempts to reverse-engineer the textual prompt to generate a specific image for later customization, highlighting another approach within this emerging field.
3 Threat Model
3.1 Attacker's Capabilities.
In our proposed attack, we consider an attack scenario that adheres to a realistic paradigm where the attacker is granted black-box access to all text-to-image APls. Specifically, the attacker lacks access to the underlying model and its training data; their interaction is confined to providing input and receiving the corresponding output. A key assumption in our model is that the attacker possesses inference-tinee time access to the API and also access to the generations of one of these APIs, namely Midjourney's Discord server. This is realistic assumption, as we have access to it. Furthermore, we extend our consideration to more complicated scenario where the attacker gains access to an alternate API, distinct from the primary target, to facilitate the attack. Additionally, in the case of natural images, the attacker only has access the targeted image(s) and nothing more.
3.2 Attacker's Objective.
The attacker's goal in this scenario is to identify a prompt that can generate images similar to given image set of images, whether produced by text-to-image API or featured in one of the commercial stock image collections. This objective necessitates that the attacker has the capability to send queries to the API to execute the attack. As previously noted, our investigation also includes scenarios where the attacker utilizes an alternative API, distinct from the primary target, to carry out the attack.
3.3 Attacker's Target.
In our attack scenario, the main targets are two well-known text-to-image APls: Midjourney and DALL-E 3. We aim to target the images generated by these APIs in order to find prompts that produce similar images. Additionally, for natural images, we focus on images from the Getty Images website," which represents one of the popular image stock sources.
4 Methodology
Our approach to executing the attack centers around three primary components: keyword extraction, modifier extraction, and prompt generation. Each plays a crucial role the attack strategy's 's overall effectiveness, as detailed in subsequent subsections.
Keyword Extraction. As previously noted, image captioning models often lack the speci- ficity required for text-to-image API prompts, typically omitting crucial details and stylistic elements needed for APIs like Midjourney. Adding to this challenge, key limitation of these models is their inconsistent accuracy in extracting specific keywords, special names, or named entities depicted in the images. Although they capture some details, they often lack the precision needed for completely accurate prompt replication. This combination of generalization and lack of precision in keyword and entity extraction significantly hin- ders their effectiveness in accurately generating prompts for text-to-image generation. An example of BLIP2's failure to detect the keywords is shown in Figure 2.
To address this challenge, we employ substantial dataset obtained from Midjourney consisting of subset of 2 million pairs of prompts and corresponding upscaled images generated by Midjourney. We then fine-tune a CLIP model on this dataset. The CLIP model, initially pre-trained on 2 billion pairs of captions and images, many of which are
4

---

Published as a conference paper at COLM 2024
palitting sitting In riels ot #Lovers
in flner gardes, Elustration, Ricoletta ceccoli
BLIP2
Our Keyword
french buildoo
and Named
Entiyy
close mountain with sun in the secograudd paper cuttuts of platn colors,
Lopo, design, risea betweer the sontalins text, lestaring, istters, tnfographte
ttret-diinnsion Layered style
Interrogator
Our Modifler
Extractor
design
Figure 2: Examples illustrating scenarios where the baseline models, BLIP2 and CLIP Interrogator, fail to accurately extract relevant keywords and modifiers from given prompts.
not precisely aligned, requires fine-tuning the Midjourney subset to better align with the nature of the prompts and images in our study.
The fine-tuned CLIP model is then utilized to generate set of 5 million text and correspond- ing image embeddings. During the inference phase, we use the fine-tuned CLIP model to obtain image embeddings of the targeted image(s) and identify the closest text and image embeddings. From these, we extract a list of named entities and keywords from the most closely related prompts and the corresponding prompts of the nearest images. Finally, this information, along with other relevant data, is fed into GPT-4V.
Modifier Extraction. Besides keywords and named entities, incorporating adjectives, adverbs, and specific styles, referred to as modifiers, can significantly enhance the quality of generations (Oppenlaender, 2023). However, image captioning models typically are unable to detect these modifiers, failing to recognize and extract them accurately. Additionally, the CLIP Interrogator also does not consistently perform well in detecting these modifiers. An example of the CLIP Interrogator's failure to detect modifiers is illustrated in Figure To overcome this, we train Multi-Layer Perceptron (MLP) as multi-label classifier image embeddings. After identifying 1800 frequent modifiers from our Midjourney dataset, train the multi-label classifier selected subset of around 300, 000 samples. During inference, we determine the top modifiers with the highest probability scores. These modifiers, along with other pertinent information, are then provided to GPT-4V.
Prompt Generation. The final step involves using model with strong visual understand- ing capabilities for detailed image description. For this, we select GPT-4V, multi-modal model known for its exceptional visual understanding. However, GPT-4V alone may not always suffice. To bridge this gap, we complement with detailed instructions and informa- tion derived from the other two components. Our comprehensive instructions to GPT-4V, called initial instructions, include detailed task description, relevant modifiers, general keywords, named entities, and an example prompt from Midjourney samples. While the fine-tuned CLIP model and the multi-label classifier provide relevant information, GPT-4V acts as an additional classifier, selecting the most appropriate modifiers, keywords, and named entities. This leads to two-level extraction process. All the related information, along with the targeted images, is fed into GPT-4V to generate the prompt.
Refinement of the Initial Prompt. The initial prompt generated by GPT-4V might not always capture every detail in the image. To enhance the prompt quality, we implement a cyclical refinement process. This involves devising a new set of instructions for GPT-4V, including elements like detailed task description, relevant modifiers, keywords, named entities, an example from Midjourney, the targeted image(s), the prompt used in the previous round of the cycle, and the corresponding generated image(s). The task description also
5

---

Published as a conference paper at COLM 2024
Model
Top-1 Accuracy
Top-5 Accuracy
Top-10 Accuracy
Original CLIP model
0.802
0.915
0.9395
Fine-tuned on 200K samples
0.8065
0.9413
0.967
Fine-tuned on 500K samples
0,9003
0.9653
0.9783
Fine-tuned on 2M samples
0.9167
0.9762
0.9857
Table 1: Comparative metrics of the original CLIP and fine-tuned models on varying dataset sizes, highlighting improvements in Top-1, Top-5, and Top-10 accuracy.
emphasizes comparing the generated image(s) with the targeted ones, refining the prompt based differences in styles, themes, or elements. This iterative process continues over multiple rounds to progressively refine the prompt's accuracy and relevance. The initial and refining prompts are displayed in Table 6 and Table 7, respectively. The overview of the attack is presented in Figure
Experimental Settings and Results
5.1 Midjourney Dataset
As previously mentioned, Midjourney stands as a frontier text-to-image API, known for generating high-quality and realistic images. Users interact with Midjourney's bot their official Discord server to input prompts and generate images, making these generations accessible to anyone who joins the server. Acknowledging the importance of this API, we collect millions of samples from Midjourney's Discord channels. This substantial dataset is crucial for our attack, providing wide array of prompts and corresponding images. Details about the dataset and its pre-processing can be found in the Appendix A.
5.2 Fine-Tuning the CLIP Model
Model Configuration and Dataset Details. For the task of fine-tuning, we select the CLIP ViT-G/14 model (Ilharco et al., 2021), which is pre-trained on a vast collection of 2 billion samples from the LAION dataset (Schuhmann et al., 2022). We pick this model variant as shows better performance in ImageNet zero-shot classification tasks." 10 Through iterative experimentation, we optimize the hyperparameters, setting learning rate (Ir) of x 10-5, batch size of 32, and gradient accumulation steps to Resource constraints lead us to fine-tune the CLIP model on 2 million sample subset of the Midjourney dataset. The outcomes of fine-tuning with various dataset sizes are shown in Table
Metric and Evaluation Results. To eval-
uate the fine-tuned CLIP models, we test set of 10,000 samples, employing CLIP's image and text encoders generate embed- dings. We calculate cosine similarity for each test image identify the closest cor- responding text, measuring the model's ss ac- curacy by whether the ground-truth text is within the Top-1, Top-5, or Top-10 closest texts. For example, Top-5 accuracy 90% indicates that the ground-truth text ranks among the top-5 closest texts for 90% of the samples. Table 1 presents the evaluation metrics for both the original and the fine- tuned CLIP models across different dataset sizes. The fine-tuned model significantly outperforms the original for this data type, with
Metric Value
3.4
5.2
Predision
Recall
51
Value
Figure 3: Overview of the multi-label classi- fier's 's performance for different values of k.
10https://github http:///s....ttaa...ooooo///ot// clip

---

Published as a conference paper at COLM 2024
BL.P2
CLIP Interrogator
Our Attack
Setting
CLIP-S
CLIP-S
HE
CLIPSS
HE
Midjourney
0.8477
0.8874
3.88
0.8946
4.36
(multiple images)
Midjourney
0.7483
3.6
0.8437
4.04
0.8844
4.52
(single image)
DALI--
(multiple images)
0.8534
4.16
0.8569
3.68
0.8883
4.56
3ALL-E3 (single
0.7932
2.8
0.8007
2.84
0.896
4.08
image)
Cross-setting
0.798
3.8
0.8169
4.04
0.859
4.4
(multiple images)
Cross-setting (single image)
0.7911
3.04
0.8215
3.28
0.886
4.08
Natural images
0.7516
3.8
0.748
3.92
0.8031
4.36
(single image)
Table Comparison of different methods using CLIP-S (CLIP-score) and HE (Human Evaluation) metrics across various settings.
the model fine-tuned million samples showing the best performance, though the improvement over other dataset sizes is marginal.
5.3 Training Multi-Label Classifier
Model Configuration and Dataset Details. We utilize a Multi-Layer Perceptron (MLP) architecture with three hidden layers for this task. The ReLU activation function is employed, along with learning rate of 0.001. To mitigate overfitting, we incorporate dropout rate of 0.3. Upon analyzing the Midjourney dataset, modifiers. From this, we select around 360, 000 samples, labeling them based on the presence of these modifiers. We allocate 10% of this dataset as validation set to monitor and adjust model performance. Detailed information about the process of extracting these modifiers can be found in the Appendix A.5.
identify approximately 1800 distinct
Metrics and Evaluation Results. For this task, we utilize precision and recall as the primary metrics. During inference, the top k modifiers are selected based on their scores from the classifier, where precision indicates the accuracy of selected modifiers and recall assesses the proportion of correctly predicted ground-trus truth modifiers. Figure displays precision and recall values for different k thresholds, highlighting B trade-off: increasing k lowers precision, reducing the proportion of correctly selected modifiers, but enhances recall, capturing more correct modifiers. Though higher recall is preferable for comprehensive modifier coverage, avoiding excessive inclusion is essential. To balance these metrics, we choose k 20 for our overall attack evaluation.
5.4 Overall Attack Evaluation
Experimental Settings. We evaluate two major text-to-image APIs, Midjourney and DALL- 3, for Al-generated images and Getty Images website for natural images. Our scenarios include both multiple images from single prompt and single image cases for Midjourney and DALL-E 3. We also explore an adversary using a different API, with Midjourney as the target and DALL-E
the alternate, across multiple and single image variations. This results in seven distinct settings, with 150 samples for setting 1, 50 samples each for settings 2-4, and 30 samples for settings 5-7.
Model Configuration and Dataset Details. In scenarios 1, 2, and 7, we utilize Midjourney as the text-to-image model for generating images. In scenarios 3-6, DALL-E is employed for image generation, with the output size set to 1024 x 1024 and standard quality. Across all scenarios, GPT-4V serves as the multimodal model for visual understanding and prompt generation. For scenarios involving Midjourney-geneee
images, small subset of the
7

---

Published as a conference paper at COLM 2024
CLIP
Setting
Original Image
Our Attack
BLIP2
Interrogator
Midjourney (Setting 2)
DALL-E3 (Setting 4)
Cross-setting (Setting 6)
Figure 4: Comparison of the original image with those generated by our method, BLIP2, and CLIP Interrogator for three different settings.
Midjourney dataset, not used in CLIP model fine-tuning or multi-label classifier training, is selected. In scenarios where DALL-E generates the targeted images, we utilize subset of Midjourney prompts to generate the corresponding images, which then form the basis of our evaluation dataset. For scenarios involving natural images, we carefully select diverse range of samples from the Getty Images website.
Metric. To assess the similarity between the images generated by our approach and the ground-truth images, we employ both automated metrics and human evaluation. The automated metric, termed Clip-score, involves calculating cosine similarity be- tween the image embeddings from the orig- inal CLIP model provided by OpenAL. Ad- ditionally, for human evaluation, we select five random samples from each setting (35 samples in total) and ask five annotators to rate these images. They similarity. More details about the human evaluation process are presented in Appendix B.
CLIP Inter-
Text-to- Image API
Our
BLIP2
Attack
rogator
DALL-E3 Midjourney
0.7509
0.7044
0.8284
0.7516
0.7480
0.8132
Table Image similarity scores for our at- tack on natural images using different text-to- image APTs, compared with baseline models.
5-point Likert scale to score the images based perceived
Baselines. Our evaluation includes BLIP2, recent image captioning model, and CLIP Interrogator 2, as baselines. For scenarios involving a single image, we use the textual output provided by these baselines directly. In settings with multiple images, to ensure fair comparison, we first process each image through the baselines and then select the one with the highest similarity score.
Evaluation Results. The results for all seven settings are presented in Table 2. Across these settings, our approach consistently outperforms the baselines. The margin of superiority is substantial in most settings, except for the first, where CLIP Interrogator shows close performance based on CLIP-score. It's observed that the effectiveness of all methods diminishes in scenarios 5 and 6, likely due to the use of different API by the attacker. Furthermore, the results show improved performance images compared to those with single image, likely because multiple images provide more comprehensive information for analysis. Some examples of generated images by our approach and other baselines are presented in Figure 4.
scenarios involving multiple
8

---

Published as a conference paper at COLM 2024
5.5 Natural Images: DALL-E3 3 vs Midjourney
In scenarios where the target imagery com- prises natural scenes from commercial stock providers, our investigation encompassed the use of both DALL-E 3 and Midjourney as the underlying text-to-image APTs. As delineated in Table 4, DALL-E 3 and Mid- journey exhibit comparable performance in terms of image similarity metrics. How- ever, a qualitative assessment reveals that Midjourney's s outputs are often more life- like and convincing. This observation is consistent with the platform's 's design philosophy, which emphasizes artistic expression and photorealism. An illustrative example contrasting the outputs from both APIs, with respect to a natural image from commercial stock collection, is provided in Figure 6.
CLIP
Text-to- Image API
Our
BLIP2
Inter-
Attack
rogator
DALL-E3 Midjourney
0.7509
0.7044
0.8284
0.7516
0.7480
0.8132
Table 4:
Image similarity scores for our at- tack on natural images using different text-to- image APIs, compared with baseline models.
Limitations
Like any other method, our approach may encounter failure cases due the complexity of its multi-component pipeline. Potential reasons for these failures include:
1. The fine-tuned CLIP model might not retrieve related keywords if the words are rare, or the model itself may fail to identify the closest texts to the images.
2. The multi-label classifier might not select appropriate modifiers.
3. GPT-4V might not accurately extract related keywords, named entities, and modi- fiers, or it might fail to generate appropriate prompt.
4. The text-to-image models might not produce images that align well with the text.
The inherent uncertainty in both multimodal LLMs and text-to-image models complicates the task. We include examples from setting 1, which yielded the lowest image similarity scores, to explore the factors leading suboptimal results. These examples and their analyses are available in Appendix C. The multi-component nature of our attack pipeline presents numerous opportunities for refinement. Each component, from the fine-tuning of the CLIP model and the training regimen of the multi-label classifier to the optimization of modifiers extracted from data, holds the potential to elevate the attack's ' effectiveness.
Moreover, the art of prompting GPT-4V is not monolithic; alternative prompt constructions could yield improved results within our cyclic approach. While this paper establishes a solid foundation, the practical enhancements of each component present as promising directions for future work, offering prospects for even more complex attacks.
7 Cost Estimation
7.1 Financial Cost
To justify the practicality of our attack, we provide an estimation of its associated costs. These are mainly divided into two main components: the cost of using GPT-4V and the cost associated with the Text-to-lmage API. We detail each as follows:
According to OpenAl's pricing, there is a charge per million tokens for both input and output. To estimate the cost for using GPT,-V, we consider the number input and output tokens required per round of our attack. Based on our analysis, the average number of input tokens is 900 for the initial prompt and 1165 for each subsequent round. The average number of rounds to achieve maximum similarity is approximately 2.7, leading to a total of 4395 input tokens per test sample. It's noteworthy that the examples from natural images shown in Figure 7 demonstrate that even with 1-2 rounds, we can achieve
GPT-4V Cost.

---

Published as a conference paper at COLM 2024
approximation of the targeted image. For output tokens, the average is 381 per round. Given the current rates on OpenAI's website, the total cost for using GPT-4V, considering both input and output, is approximately $0.09. Additionally, small fee of approximately $0.03 applies for including images in queries, based on the average number of rounds.
Text-to-lmage API Cost. Our experiments utilize either Midjourney or DALL-E 3, so we calculate the cost for each separately. Based on the rates from Midjourney's s and OpenAl's websites, the cost per image generation is about S0.03 for Midjourney and $0.04 for DALL-E3 (for 1024*1024 resolution images). Consequently, the total generation cost using Midjourney is approximately $0.12, and for DALL-E 3, it's around $0.16.
Overall Cost. The total cost per sample depends on the text- text-to-image API used. If the attacker utilizes Midjourney, the cost is approximately $0.235 per sample, and for DALL-E 3, it is around $0.275. These costs are substantially lower than the typical prices Al- generated image marketplaces (ranging from $3 to $7) and significantly below the cost of stocks of real images (which can be $50 to $500).
7.2 Computational Cost
Fine-tuning the CLIP model with 1.37 billion parameters and training the classifier are two critical components of our approach. Fine-tuning the CLIP model for 10 epochs 2 million samples require the use of four A100 GPUs and takes six days to complete. Additionally, training the Multi-Layer Perceptron (MLP) classifier on approximately 360,000 samples for 100 epochs is executed in just 25 minutes on single A100 GPU.
8 Ablation Study
To highlight the significance of the additional information provided by the CLIP model and the classifier, conduct an ablation study within one of our experimental settings (setting 1). In this experiment, we use the same GPT-4V prompts as described in Tables 6 and 7, but exclude information from the CLIP model and the classifier. We also include specific tips and instructions to demonstrate how prompt engineering operates in Midjourney, incorporating an example of prompt-image pair from Midjourney. We assess both variants-win CLIP scores of 0.8415 and 0.8395, respectively. For
and without the example-achievin comparison, the CLIP scores for BLIP2, CLIP Interrogator, and our approach in setting are 0.8477, 0.8874, and 0.8946, respectively. These results underscore the critical role of the information extraction component enhancing the effectiveness of our strategy.
Conclusion
This research has unveiled novel attack strategy in the realm of digital imagery, specifically targeting Al-generated image marketplaces and premium stock image providers. By effec- tively employing state-of-the-art multi-modal models, our method has demonstrated the ability to generate visually comparable content at a significantly reduced cost, challenging the current economic dynamics of the digital imagery landscape. Moreover, the compilation of a large-scale dataset from Midjourney is pivotal contribution to this field. This dataset not only aids our research but also serves as a valuable resource for future studies, offering insights into the capabilities and challenges of text-to-ine text-to-image technologies. Additionally, our approach can serve as a powerful tool for data augmentation, generating varied vi- sual content to enhance machine learning model training, which is often hampered by the expensive and time-consuming nature of traditional data collection. In conclusion, our study highlights new threats against digital imagery, stressing the need for urgent action in identifying and countering these risks. Future research should aim at developing protective measures to preserve digital image integrity amidst advancing AI technologies.
10

---

Published as a conference paper at COLM 2024
Ethical Considerations
This paper presents an attack designed to generate images similar to specific, selected stock images at a fraction of their actual cost. The primary aim of this research is to highlight potential new avenues for misuse of multimodal LLMs against the business models of stock image services and content creators. We believe it is vital to understand such threat models to develop appropriate defensive mechanisms effectively. While watermarking techniques (Wen et al., 2023; Zhao et al., 2023; Zhang et al., 2024) effectively protect content, assessing visual LLMs' resilience against these measures is vital. Therefore, exploring robust countermeasures is essential for future research.
In recognition of the potential risks posed to image creators by such attacks, we take several measures to minimize the possibility of reproduction. We will not freely publicize the code, the fine-tuned CLIP model, or the classifier. Additionally, the data used to fine-tune the CLIP model will be excluded from the Midjourney dataset that we intend to release publicly, further mitigating the risk of misuse.
Acknowledgements
This work supported by the NSF grant 2131910 and by the Young Faculty Award program of the Defense Advanced Research Projects Agency (DARPA) under the grant DARTA-RA22--0-00-YYA99P-000
References
James Betker, Gabriel Goh, Li Jing, lim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. httpss//cdnn. openai. com/papers/dall-e3. pdf, 2(3):8, 2023.
Tingfeng Cao, Chengyu Wang, Bingyan Liu, Ziheng Wu, Jinhui Zhu, and Jun Huang. Beautifulprompt Towards automatic prompt engineering for text-to-image synthesis. arxiv preprint ar Xxx::2311.06752 2023.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Gondman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXim:2209.0679. 2022.
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, and Aniruddha Kembhavi. X-Ixmert: Paint, caption and answer questions with multi-modal transformers. Xiv preprint ar Xix::2000.1111 11278, 2020.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances Neural Information Processing Systems, 34:19822-19835, 2021.
lan Goodfellow, Jean Pouget-Abadias Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Gabriel Illarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL inttps://dol org/10. 5281 22281/2e0 /ww.//eeooooooo 5143773. If you use this software, please cite it as below.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference machine learning, pp. 12888-12900 PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMI.R 2023.
11

---

Published as a conference paper at COLM 2024
Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. StyletZi: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18197-18207, 2022b.
Shihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak, and Deva Ramanan. Language models as black-box optimizers for vision-language models. In Proceddings of the iEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12687-12697, 2024.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint ar arXiv:1511.02799, 2015.
Jonas Oppenlaender. A taxonomy of prompt modifiers for text-to-image generation. Be- haviour & Information Technology, pp. 1-14, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp. 8748-8766 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, Pp. 8821-8831. Pmlr, 2021.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Ad- vances in neural information processing systems, 35:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15225278-25294 2022.
Zeyang Sha and Yang Zhang. Prompt stealing attacks against large language models. arXiv preprint ar Xiv::2402.12959, 2024.
Xinyue Shen, Yiting Qu, Michael Backes, and Yang Zhang, Prompt stealing attacks against text-to-image generation models. ar Xiv preprint ar Xxx::23022.0992,, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXi preprint arXi::2205.1111 14100, 2022.
12

---

Published as a conference paper at COLM 2024
Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring water- marks: Fingerprints for diffusion images that are invisible and robust. aiv preprint (7Xi::2305.20050, 2023.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference compuler vision and pattern recognition,, pp. 1516-1324, 2018.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint ar arx::22206.011 2(3)5, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Xuanyu Zhang, Runyi Li, Jiwen Yu, Youmin Xu, Weiqi Li, and Jian Zhang. Editguard: Versa- tile image watermarking for tamper localization and copyright protection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11964-11974, 2024.
Yiming Zhang and Daphne Ippolito. Effective prompt extraction from language models. 2023. URL https: //api. semanticscholar org Ctrp//orrssSD.... 259847681.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303.10137, 2023.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for coxt-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5802-5810, 2019.
A Data Collection
In this section, we discuss our approach to gathering data from Midjourney. First, we provide brief overview of Midjourney and its functionality. Next, we delve into the
Distribution of Token Counts
120000
Mean: 31.62
100000
specific steps of our data collection process.
Frequency
80000
Finally, we describe the characteristics of the data we collected from Midjourney. This offers readers clear picture of the founda- tion of our research.
60000
20000
A.1 Introduction to Midjourney
0
20
40
60
Number of Tokens
Midjourney is powerful black-box text- to-image API that generates images from natural language descriptions, commonly referred to as "prompts." It operates in a manner analogous to other AT tools like OpenAl's DALL-E and Stability AI's Stable Diffusion. Midjourney has consistently ad- vanced its algorithms, launching new model versions periodically. The algorithm's Version 2 was introduced in April 2022, followed by Version 3 in July 2022. Alpha iterations of Versions and were released in November 2022 and March 2023, respectively. The en- hancements in Version 5.1 leaned towards distinct image stylization, whereas its RAW variant optimized interpretations for clearer prompts. The subsequent Version 5.2 further refined the output image quality.
Figure 5: The distribution of the number of tokens in prompts from Midjourney samples.
13

---

Published as a conference paper at COLM 2024
Midjourney operates exclusively via a Discord bot present on their official server. Users can engage with this bot directly or have it on a different server. To produce images, the / /imagine command is employed along with desired prompt, resulting in set of four generated images. Users then have the option to upscale their preferred images.
A.2
Data Crawling
The Midjourney server offers a variety of channels designed for collaborative work, in- cluding dedicated rooms for beginners, themed generations, and general discussion areas. Both prompts and the corresponding generated images are posted within these channels. We target the general channels, which contain broader range of generations compared to theme-specific channels. In our approach, we target 10 such general channels. has to be noted that for our data collection, we utilize DiscordChatExporter: extracting chat histories from Discord channels. This tool facilitates the extraction of chat records from Discord, enabling us to handle direct and group messages, export from various channel types, and retain rich media and specific Discord formatting nuances.
tool adept at
A.3 Data Characteristics
In our data collection process, we successfully crawled over 20 million samples. These en- compass duplicated captions associated with distinct generated images. Among these, more than 12 million samples have unique captions. The collected samples can be categorized into two main groups. The first group consists of samples displaying grid of 4 generated images. The second group presents samples with a single upscaled image.
The collected raw data include several features: AuthorID, Author, Date, Content, Attachment, and Reactions. Both AuthorlD and Author remain consistent for all samples, pointing to Midjourney's BOT. The Date field contains timestamps in the Unix timestamp (or Epoch time) format, which counts the number of seconds that have elapsed since January 1, 1970 (UTC). An example of such timestamp is 1 688516818. Attachment denotes the URL of images corresponding to each caption. Lastly, Reactions pertains to the emoji reactions associated with the data.
However, the Content feature requires a more detailed description. The Content comprises the following items:
1. URL (Optional):
Users may embed the URL of an image in their prompt, particu- larly they wish modify or obtain different variations of an image.
2. Main Prompt: The primary textual prompt provided by the user.
3. User's ID: The identifier of the user who submitted the prompt.
4. Type: Specifies whether the image is a high-resolution upscaled version or grid containing lower resolution variations.
5. Parameters:
Parameters are options added to a prompt that change how image is generated. They can alter aspects such as the image's aspect ratios, switch between Midjourney model versions, and much more. Parameters are always appended to the end of a prompt, following double dash ("-). Multiple parameters can be included each prompt. Detailed documentation of each parameter available on Midjourney's website.
A.4 Data Pre-processing
We implement various data processing steps on this expansive dataset. Some key steps include:
1. Extracting the original prompt from the Content feature.
2. Removing empty rows or rows with NaN values.
14

---

Published as a conference paper at COLM 2024
3. Categorizing samples based whether their prompts contain image URL. For our method, we exclude these samples, ensuring none are included in our training data.
4. Converting the Unix timestamp in the Date feature into more standard date format.
5. Determining the version of the model using the Date feature and the known release dates of each version.
6. Excluding rows where prompts contain emojis non-English words.
Lastly, from our processed dataset, select approximately 4.8 million samples with upscaled images. We particularly choose prompts with 77 tokens or fewer, considering the constraint of the CLIP model which only accepts text inputs of up to 77 tokens. The distribution of token counts in these prompts illustrated in Figure
A.5 Collecting Modifiers
To extract modifiers from our large-scale dataset, we analyze the common format Midjour- ney's prompts. Upon reviewing numerous samples, observe that prompts often contain a general description followed by modifiers, typically separated by commas. While this pattern is not always consistent, is prevalent enough in large dataset to yield significant set of frequent modifiers. After examining millions of samples, we identify approximately 1800 modifiers, each occurring more than 1000 times in the dataset. Table presents some of the most frequent modifiers along with their corresponding frequencies.
B Human Evaluation
For the human evaluation component of our study, we adopt a methodology similar to that done by Li et al. (2022b). As outlined in the experimental section, we randomly select five samples from each of the seven settings, amounting to total of 35 samples. These are then presented five annotators, who are instructed to score each image us- ing 5-point Likert scale. The user interface for this task is a Google Form consisting of 35 multiple-choice grid questions, one for each sample.
Modifier
Frequency
8k octane render
0.0882
0.0453
photorealistic
0.0442
cinematic
0.044
realistic 4k highly detailed
0.0329
0.0324
0.0297
cinematic lighting
0.0266
hyper realistic unreal engine
0.0257
0.0249
In each question, annotators are presented with four images: the original image at the top, and three other images below it, gener- ated by attack and the baselines, shuf- fled in no specific order. The task requires the annotators to assign similarity score ranging from (not similar at all) to 5 (very similar) for each of the three shuffled images, based its resemblance to the original image. Detailed instructions and scoring guidelines are provided to the annotators to ensure consistent and accurate evaluation process. Here are the instructions and scoring guide:
Table 5;
The 10 most frequent modifiers tracted from the Midjourney dataset, with fre- quencies indicating the percentage of prompts containing each modifier relative to the total number of samples.
Instructions to Annotators:
"In this task, you will be shown set of images. For each set, the top image is the original, which may a grid four images showing different variants of single generation. Below the original are three generated images. Your role is to assess how similar each of the three generated images is to the original image (or its variants) and give similarity score from 1 to 5. Score for 'Completely Different' if there are no discernible similarities, and score 5 for 'Very Similar' if the generated image nearly identically matches
15

---

Published as a conference paper at COLM 2024
the original in content, style, and details. Use the intermediate scores to indicate varying degrees of similarity."
Scoring Guide:
1. Completely Different: No discernible similarities exist between the generated image and the original, presenting entirely distinct content and style.
2. Barely Similar: The original and generated images may share some elements or themes, but they significantly differ in both content and style.
3. Somewhat Similar: The generated image bears resemblance to the original in terms of content or subject matter, despite noticeable differences in style.
4. Closely Similar: Minor variations are present, but the generated image generally mirrors the original in content and subject matter.
5. Very Similar: The generated image closely resembles the original in content, style, and fine details, indicating strong likeness.
C Failure Cases
In Figure 8, we present three examples from setting that receive the lowest image similarity scores, highlighting specific instances where our method does not perform expected.
In the first failure case, our method successfully identifies several elements of the image, including the Asian woman, the expression, and the water pouring over her. However, the images are not deemed highly similar. A potential reason for this discrepancy is the word " "Awkwwainaa.. Although our named entity extractor feeds this keyword alongside other related keywords to GITT-V, the model fails to select it as a related element of the prompt, leading less accurate representation in the generated image.
The second failure case also exhibits mismatch. While our method extracts many elements and features from the image, the degree of similarity is not substantial. The original prompt contains special names that our method does not extract. Furthermore, Midjourney fails to accurately render images of these characters, instead generating unrelated figures. This outcome underscores dual failure: both our named entity extractor and the text-to-image model do not fulfill their intended functions effectively.
In the final example, the simplicity of the original prompt grants Midjourney significant freedom in varying most aspects of the image, complicating our method's ability to generate singular prompt that accurately represents all images simultaneously. This issue arises because we use a completely random sample from Midjourne''s ' outputs to construct our test dataset, resulting in some instances with less effective prompts.
These cases underline the complex challenges in prompt-based image generation, where both the extraction of relevant information and the generative capabilities of the model play critical roles in achieving accurate results.
16

---

Published as a conference paper at COLM 2024
Text-to-lmage
CLIP
Original Image
BLIP2
Our Attack
API
Interrogator
DALL-E3
Midjourney
DALL-E3
Midjourney
DALL-E3
Midjourney
Figure 6: Comparison of natural images from two different Text-to-lmage APIs with those generated by our method, BLIP2, and CLIP Interrogator.
Original Image
Initial Prompt
Round
Round 2
Round
Figure 7: Examples of natural images where our attack method achieves an acceptable approximation of the original model with only 1-2 rounds of refinement.
17

---

Published as a conference paper at COLM 2024
Original Prompt
Original Image
Generated Prompt An 8k high-definition, realistic, detailed photo-rendered portrait of Constance Wu from the waist up, caught off guard and visibly reacting to being suddenly soaked in
Generated Image
Gallons of water are pouring down on Awkwafina's head, soaking wet, full body -v 5 -q 2
torrential downpour
during daytime...
Early 1970s Kodak
A group photo of Alex Silva, Jesse
Portrait style portrait of
a group of nerdy young
Chevan, Justin Gonçalves, and Charlie Hack; 30 years old, portrait, editorial photograph, 8k, weird and odd -s 750 -v 5
men with curly hair and
vintage round glasses, dressed in muted, natural-coloned academic attire, photographed against simple backdrop with
even,
Premium 8k resolution portrait conveying nuanced, refined family dynamic, with subtle punk fashion influences and minimal tattoos. The atmosphere should be moody yet sophisticated, with attire that is stylish and understated,...
adult kids with black hairs talking real image
Figure 8: Three examples of failed cases with the lowest image similarity from setting 1, showcasing the original and generated prompts alongside their corresponding images.
18

---

Published as a conference paper at COLM 2024
Prompt template to start the cycle
You are tasked with analyzing grid of four images generated by a text-to-image model, keeping in mind that they represent varied interpretations from single prompt, rather than a series of similar objects or scenes. The objective is to deduce the original prompt used to create these images. Your analysis should focus on identifying common themes, elements, or characteristics present across all four images. The challenge is to construct a descriptive prompt that encapsulates these observations. Here are some instructions to generate the prompt:
Task:
Be within 15-50 tokens in length, as per the CLIP tokenizer.
Encompass all common features visible in the images.
Examine the provided list of modifiers. Select and include those you deem relevant and which meaningfully contribute to capturing the essence of the images: ...list of modifiers here...
Prioritize the integration of names from the provided list into your analysis of the image grid, especially when they align with the context or themes depicted. When creating the prompt, you recognize a famous individual in the image and their name is included in the special names list I've provided, please use that name directly in the prompt. Avoid using phrases like 'a character resembling someone' and instead use the actual name from the list, provided it aligns with the person identified in the image: ...list of named entities here...
Choose general words from the list that are relevant to the images, reflecting their themes and atmosphere. After ensuring relevance, use these words to enhance the prompt's descriptiveness: list of general words here...
Avoid phrases like 'Create a series of' or using plural terms that might lead the image generation API to produce multiple versions in each variation.
Typically, follow a structure starting with a general description, followed by spe- cific features and modifiers, separated by commas: "Dark street Tokyo environment Traditional Japanese illustration of Funky Musician. With bird mask traditional Japanese elements and neo electro string instrument big japanese graffiti, Crisp contemporary illus- trations, bold lines, vibrant, metallic, moving, geometric, expressive"
Table 6: The initial prompt template provided to GPT-4V to begin the iterative refinement cycle. The sections highlighted in red represent the information obtained from the fine-tuned CLIP model and the multi-label classifier.
19

---

Published as a conference paper at COLM 2024
Prompt template for refining the prompts in the cycle
Task: You are tasked with analyzing grid of four images generated by a text-to-image model (The first grid). Keeping in mind that they represent varied interpretations from single prompt, rather than a series of similar objects or scenes. The objective is to deduce the original prompt used to create these images. Your analysis should focus on identifying common themes, elements, or characteristics present across all four images. The challenge is to construct descriptive prompt that encapsulates these observations. Instructions: Here are some instructions to generate the prompt:
Be within 15-50 tokens in length, as per the CLIP tokenizer.
Encompass all common features visible in the images.
Examine the provided list of modifiers. Select and include those you deem relevant and which meaningfully contribute to capturing the essence of the images: ...list of modifiers here...
Prioritize the integration of names from the provided list into your analysis of the image grid, especially when they align with the context or themes depicted. When creating the prompt, if you recognize famous individual in the image and their name is included in the special names list I've provided, please use that name directly in the prompt. Avoid using phrases like 'a character resembling someone' and instead use the actual name from the list, provided it aligns with the person identified in the image: ...list of named entities here...
Choose general words from the list that are relevant to the images, reflecting their themes and atmosphere. After ensuring relevance, use these words to enhance the prompt's ss descriptiveness: .i.sist of general words here...
Avoid phrases like 'Create series of' or using plural terms that might lead the image generation API to produce multiple versions in each variation.
Typically, follow structure starting with general description, followed by spe- cific features and modifiers, separated by commas: "Dark street Tokyo environment Traditional Japanese illustration of Funky Musician. With bird mask traditional Japanese elements and neo electro string instrument big japanese graffiti, Crisp contemporary illus- trations, bold lines, vibrant, metallic, moving, geometric, expressive"
Your generated prompt given the information above: ...geerttted prompt from the previous round...
Consider the second grid of images that were generated using the prompt I obtained from you above. Compare these images with the original image I first provided (the first grid of images). Your task now is to refine your prompt to achieve closer resemblance to the original image. Identify areas where the generated images differ from the original, focusing on style, subject matter, themes, and other key elements. Utilize all provided information above which are related to the images, including different artistic styles, names of entities, or other relevant elements, to make the necessary adjustments. The objective is to modify your previous prompt in such way that it results in images more accurately reflecting the original in appearance, atmosphere, and concept. Suggest specific changes, whether it involves a different artistic style, more accurate incorporation of certain entities or names, or thematic adjustments. The goal is for the improved prompt to bridge the gap between the generated images and the original image.
Table 7: The prompt provided to GPT-4V to refine the prompt at each round based on the generated image in the previous round.
20


Reviews:
[{"review_id": "8VaIqnXquu", "reviewer": "Reviewer_BmpJ", "summary": "The paper introduces an original attack strategy for generating images that look like traditional stock images  or images generated with APIs like DALL-E 3 or Midjourney. The proposed strategy uses a fine-tuned multi-modal CLIP model to identify keywords and named entities representative for an image, and also a multi-label classifier to identify relevant modifiers. Such information is provided as instruction to GPT-4V, which finally produces a prompt that can be used to generate the desired images. The authors also perform a cost analysis and show that the proposed attach strategy can be used to generate images at at fraction of the current cost.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to Reviewer for their thorough and insightful review.\n\nThe reviewer expressed concern that our approach leverages existing tools in a straightforward way. We agree with the reviewer and wish to further emphasize the potential misuse of state-of-the-art APIs to target AI-generated image marketplaces and natural image stocks. Our goal is to highlight these emerging threats and provide a foundational approach that future work can build upon to develop more complex strategies.\n\nThe reviewer inquired whether it is allowed under Midjourney's policies to publicly release and share our dataset. We commit to publicly releasing our dataset, as there is nothing in the Midjourney Terms of Service (https://docs.midjourney.com/docs/terms-of-service) that prohibits this action. In fact, these terms explicitly state that anyone is free to use and \"remix\" images and prompts that are posted on their public Discord.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a novel attack strategy and show that they can generate images that look like desirable real stock images or images generated by APIs such as DALL-E 3 or Midjourney at a fraction of the cost. \n\nThe authors perform both quantitate and qualitative evaluation of the proposed approach. \n\nThe authors clearly point out the limitations of the work.", "reasons_to_reject": "While the proposed approach can be seen as a novel attack strategy, it leverages existing tools in a pretty straightforward way. \n\nThe success of the approach depends on the collection of a large set of prompt/image pairs from Midjourney to fine-tune CLIP.", "questions_to_authors": "The authors mention that the dataset assembled from Midjourney will be released publicly. Is that allowed by Midjourney? Can that data be shared?", "extra_scores": {}}, {"review_id": "W6SLzHwcMe", "reviewer": "Reviewer_w2oy", "summary": "This research work includes design attacks for generating images in a refined matter (iterative) that are significantly cheaper than online digital stock images using text-to-image APIs and LLMs. \n\nParticipants are shown 4 images and an original image and are asked to tell how similar each image is to the original image using a Likert scale for similarity. From these 4 images, one is generated by the author’s attack, while the others are the baseline-generated images (CLIP interrogator 2 and BLIP2).\n\nAuthors conclude that overall, while both midjourney and DALLE-3 (text-to-image) are high-performing, midjourny produces more realistic photos. \n\nThe metrics used in the study is CLIP score alongside human evaluation.\n\n\n\n\nMain contributions:\n\n- A large-scale dataset of images from Midjourney collected from their official Discord channel that can enhance the text-to-image line of research.\n\n- Created an attack to create images that have a high similarity scores to the digital images from Web via 1-2 rounds of refining. GPT-4V and midjourney/DALLE-3 are used for the experiments.\n\n- Their work showcases that attention should be paid towards maintaining the integrity of digital images.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you to Reviewer for their positive feedback and strong support of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n- Great comparison of results both visually and quantitatively.\n\n- Cost analysis of the attack shows that with negligible money, images that are very close to the original digital image can be created.\n\n- Authors have provided enough details that makes this research paper reproducible for the most part.\n\n- Providing a set of failure cases that depicts what are some main flaws with the process. Further, expanding on limitations of their current work sheds light on what could be picked up and improve by future researchers.", "reasons_to_reject": "I am pro accepting this paper. It is a novel concept and beneficial to the community.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGTenmK9Ig", "reviewer": "Reviewer_imxU", "summary": "This paper explores a novel and intriguing question in the realm of text-to-image system safety: Is it possible to recover the prompt used to generate an image using a text-to-image system? Given the potential value of high-quality prompts in the AI-generated content (AIGC) domain, this question is worth investigating.\n\nThe paper assumes that the attacker has black-box access to all text-to-image APIs without prior knowledge of the underlying models and data, and that the generated images are accessible. This setup seems to be very reasonable. \n\nThe proposed method involves fine-tuning CLIP on a dataset of 4.8 million prompt-image pairs to generate text and image embeddings. Similar images to the target images are retrieved, and the corresponding text in the database is used to extract key entities. A multi-label classifier is trained based on the image embeddings to extract image modifiers from a set of 1,800 high-frequency modifiers. GPT-4V is then employed with task demonstration, instructions, the target image, and extracted information to generate the prompt. This generation process can be refined iteratively to improve accuracy.\n\nEvaluation results demonstrate that the proposed framework is more effective in terms of the quality of generated images, with Figure 4 providing clear illustrations of the cases. Furthermore, the method can be transferred from DALL-E 3 (source) to Midjourney (target), and the overall cost (excluding training) is low, at approximately $0.24 per sample.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to Reviewer for their insightful review and suggestions.\n\nThe reviewer asked if we could estimate the computational cost for training the CLIP model and the classifier. Fine-tuning the CLIP model for 10 epochs on 2 million samples using four A100 GPUs took six days. Additionally, training the MLP classifier on ~300K samples for 100 epochs took only 25 minutes using an A100 GPU. We will include these details in the final version of the paper.\n\nWe agree with the reviewer that adding an oracle analysis to compare the generated prompts with the real prompts would provide interesting insights regarding the evaluation. As an initial observation, upon comparing the generated prompts with the original ones, we found that the generated prompts are typically longer and more detailed. It's important to note that different prompts can lead to similar images, so the final prompt might not exactly match the original one. Please note that natural images do not have original prompts. We will include a more detailed analysis in the final version of the paper.\n\nThe typos, including the citation format and the issue in Figure 2, will be corrected in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall this paper is well-written, and the reading is enjoyable.\n\n- I like the question explored which might be valuable for future text2image safety research.\n\n- The framework is well-motivated and effective given the current results and the cases in Figure 4.", "reasons_to_reject": "I did not find anything significant to reject this paper.", "questions_to_authors": "Minor Suggestions:\n\n- When estimating the cost, it would be beneficial to consider the computational cost for training CLIP and the classifier. Although these costs are one-time and can be amortized later, it would still be informative to know the training cost.\n- Regarding the evaluation, adding an oracle analysis by comparing the generated prompts with the real prompts used to create the target images would provide interesting insights.\n\n\nTypos:\n\n- DALL-E 3 Betker et al. (2023) -> DALL-E 3 (Betker et al., 2023): Use \\citep instead of \\cite for these citations.\n- Figure 2: There is an empty period on the new line.", "extra_scores": {}}, {"review_id": "rU5OTTGr2Q", "reviewer": "Reviewer_1ZQM", "summary": "This paper proposed a method to recover the prompt used to generate the input image and use the prompt the reproduce the image. The method is basically a captioning system consisting of (1) a fine-tuned CLIP model for constructing a pool of image and text embeddings, which will be used to retrieve relevant information for the input image, (2) a multilabel classifier to predict the modifiers given the image, and (3) use GPT-4V to generate and refine the final prompt given the information from (1) and (2) with the image to generate the prompt. The proposed method performs better than two off-the-shelf captioning models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We appreciate the detailed review and thank you for highlighting these concerns.\n\nThe reviewer expressed concerns that this paper only chooses two off-the-shelf captioning models as baselines. To the best of our knowledge, there are two suitable open-source tools for this task: CLIP Interrogator and captioning models, for which we selected BLIP2 due to its recent advancements and effectiveness in image captioning. Besides, there are no existing baselines with large multimodal LLMs.\n\nThe reviewer suggested fine-tuning the baselines on the same dataset we used. However, this approach presents some difficulties. The CLIP Interrogator, which merges CLIP with BLIP for optimized text prompts, complicates practical fine-tuning on new datasets due to its integrated nature. Moreover, the smallest BLIP2 version holds 3.87 billion parameters, significantly larger than the 1.37 billion parameters of our CLIP model. Our prior experience required 2 million samples and six days on four A100 GPUs to fine-tune the CLIP model, indicating that fine-tuning BLIP2 is not feasible with our resources.\n\nThe reviewer expressed some concern regarding the comparability of the sizes of the baselines and our model. It is difficult to find models with exactly matching sizes for specific tasks. However, the parameter counts of our models and the baselines are generally comparable. The baseline BLIP2 model contains 3.87 billion parameters, whereas the CLIP Interrogator, combining CLIP and BLIP, totals about 898 million parameters. The specific CLIP model used in our approach has 1.37 billion parameters.\n\nThe reviewer asked for an ablation study that shows the importance of each component. We conducted an experiment by removing the information from the CLIP model and the classifier in one of our experimental settings (Setting 1). We used the same GPT4-V prompt, excluding these components' data. Additionally, we incorporated specific tips and instructions to demonstrate how prompt engineering operates in Midjourney, including an example of a prompt-image pair from Midjourney. We tested the settings with and without the example, obtaining CLIP scores of 0.8415 and 0.8395, respectively (The CLIP scores for BLIP2, CLIP Interrogator, and our approach are 0.8477, 0.8874, and 0.8946, respectively in setting 1). This comparison underscores the essential role of the information extraction component in our strategy's success. These findings will be detailed in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is technically sound and could have broad applications.", "reasons_to_reject": "1. **Unfair baseline comparison.**\n\nThis is the main problem for this paper, and important baselines are missing for a fair comparison to evaluate the effectiveness of the proposed method. This paper only chooses two off-the-shelf captioning models as baselines, while the proposed model is trained on **additional data** (2M samples for fine-tuning CLIP and 300k examples for modifier classification). The **model size** is also not comparable. The proposed model uses the backbone ViT- G/14 pretrained on LAION-2b. Do the BLIP2 and CLIP Interrogator use the same backbone and pretrained on a similar amount of data? Moreover, the proposed system uses GPT-4V, which is a big advantage.\n\nTo make a fair comparison, the following baselines are suggested:\n1) fine-tune the baselines with the same amount of data as the proposed method uses.\n2) make sure the model size is comparable.\n3) use GPT-4V as the captioning model.\n\n2. **Lack of ablation studies**\n\nThis is also important to evaluate the effectiveness of the model design. The proposed pipeline involves multiple steps: fine-tuning, multi-label classifier, and GPT-4V to summarize based on the entities, keywords, etc. Are all those steps necessary? Which step is more important to the final performance? This version of the paper did not carefully ablate those components. The performance gain over the baselines could be just because of using more data and a bigger model, as mentioned before.\n\nOverall, this paper has problems in baseline comparison and lacks ablations, making it difficult to evaluate the method's effectiveness.", "questions_to_authors": "", "extra_scores": {"ethics_concerns_details": "The method in this paper can be used to plagiarize copy-righted artworks. The authors should discuss the potential impact of using this method."}}]

Author rebuttal:

Meta-review: All reviewers appreciated the contribution and especially mentioned its cost efficiency. One reviewer mentions it has "a novel concept and beneficial to the community." Rebuttal period was lively and productive. Based on this, I am recommending acceptance. 

For the next version of your paper: I would recommend the authors include the cost analysis mentioned by imxU. We would also like to see the ablations etc. recommended by 1ZQM incorporated into the paper.

On the ethics review: it is expected that the authors will take into account this reviewer's suggestions for minor reframe, and the suggested update to limitations.

[ethics comments from PCs] This paper has been flagged by reviewers for potential concern that the attack can be used to obtain trademark images for free/cheaply and wider discussion has to be made. The motivation of this work can be also be presented in a positive light (e.g., improved image search but by generation) rather than framing it as an attack on stock images. **We strongly recommend addressing these issues and adding/revising an ethics statement as needed.**
Decision: Accept