SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prompt inversion aims to address this gap, involving the process of taking a single image and predicting the corresponding prompts associated with it. We anticipate that the proposed dataset would further facilitate the development of prompt inversion through the in-depth analysis of the prompts.
To evaluate the effectiveness of prompt inversion, we extend the metrics commonly utilized in image captioning, including Bleu, METEOR, ROUGE, and CIDEr. Additionally, we adopt the approach employed in a related Kaggle competition [46] to calculate the Cosine Similarity of the sentence-transformers features [47].
Furthermore, in the supplementary materials, we propose a Question Answering Score (QAS) for evaluating the prompt inversion results. In this paper, we establish a benchmark for the zero-shot prompt inversion task by leveraging state-of-the-art multi-modal models, namely BLIP-2 OPT2.7B [4], BLIP-2 FlanT5XL[4], Flamingo9B[3], MiniGPT-4 [38], and Uni-Perceiver v2 [30]. To ensure optimal performance in this novel task, we customize different prompts for each model.
We evaluate these models on the test set of our dataset, and the results are presented in Table 3. During the experiment, we observed that the existing models struggle to capture the intricate details and style-related information of the input image, leading to lower performance compared to conventional datasets.
To verify the effectiveness of our dataset, we fine-tuned Uni-Perceiver v2 for 20 epochs and noted a significant improvement in the prompt inversion task. It is important to note that we followed the training methodology outlined in [30] without tuning the hyperparameters or utilizing data augmentations. This demonstrates that our JourneyDB can complement existing image-text datasets for training prompt inversion models. Nevertheless, it is evident that there is still a substantial way to go in developing a robust and effective prompt inversion model.
4.2 Image Caption
Image captioning tasks require multi-modal models to generate textual descriptions for the visual content of an image. In comparison to existing image captioning benchmarks such as COCO Caption [10], JourneyDB encompasses both detailed descriptions and high-level summarizations of images, thereby assessing the model's proficiency in fine-grained recognition and holistic understanding.
Figure 3: Samples from the validation set of JourneyDB captioning. The examples show that existing multi-modal models failed to recognize some key concepts from the AI-generated images.
Table 5: Style Retrieval Results. The metric used there is the Recall.
Method	Validation	Test
Over-All	Per-Category	Over-All	Per-Category
CLIP-ViT-L/14 [2]	0.65	41.72	0.47	41.33
We evaluate various existing multi-modal models on the image captioning sub-task of JourneyDB. The results are presented in Table 4, indicating the challenges faced by multi-modal models trained on natural images in providing accurate descriptions for AI-generated content. The quantitative performance is notably poorer (significantly worse than COCO Caption results) due to two primary factors: GPT-3.5 tends to generate verbose descriptions for the images in JourneyDB, resulting in lengthy ground-truth captions. This discrepancy between lengthy ground-truth captions and shorter predicted captions undermines the quantitative performance. When describing AI-generated images, the focus may differ in terms of concepts such as emotions, human/object attributes, etc., compared to natural images. However, existing image captioning models have not adequately addressed these concepts.
We provide qualitative examples in Fig 3. Existing multi-modal approaches fail to describe key concepts present in the AI-generated content (e.g., Fig 3(b) depicts kids in astronaut suits, Fig 3(d) shows a sad potato). Moreover, some models may hallucinate contents that do not exist in the images (e.g., Open-Flamingo hallucinates objects and text in Fig 3(a, c)).
4.3 Style Retrieval
We inhabit a captivating world enveloped in a multitude of vibrant colours and ever-shifting illumination. Artists, in turn, develop their distinct styles to convey their unique perspectives of the world. Elements such as weather conditions, moods, and atmospheres all contribute to the style portrayed in an image, resulting in a complex "style system." As detailed in Section 3.2, we have compiled a comprehensive collection of over 150,000 style words to describe the style-related attributes of images.
Table 6: Evaluation results of the content-relevant and style-relevant zero-shot Multiple-Choice Visual Question Answering on JourneyDB. The evaluation metric here is accuracy.
Method	Validation	Test
Content	Style	Content	Style
Flamingo9B [3]	32.1%	31.9%	35.6%	41.4%
MiniGPT-4 [38]	28.2%	26.6%	31.1%	29.3%
BLIP-2 FlanT5 [4]	65.8%	54.9%	69.7%	57.4%
Style-Relative Questions
Blip-2: A
A: Just the couple,
Blip-2: A
Blip2: A
Figure 4: Failure cases of BLIP-2 [4] for Multiple-Choice Visual Question Answering.
Blip2: A
Blip2: B
Blip2: B
Blip2: D
Ground Truth: C
Given the vast expanse of the style space, identifying the style of a given image poses a significant challenge, even for human observers. Consequently, there is a need for style retrieval techniques to aid in comprehending the style exhibited in an image.
Directly retrieving a style prompt from an extensive pool of candidates is a complex and time-consuming task. Therefore, we employ clustering techniques to group the style prompts into 344 categories, including camera parameters, lighting, artist style, colour schemes, and more, as outlined in Section 3.2. By doing so, we effectively narrow down the search space for style prompt retrieval within each category. To establish a benchmark, we employ CLIP [2] for zero-shot style retrieval evaluation. We extract the features of both the images and the style prompts, subsequently calculating the inner product between the image feature and all candidate style prompts. The results are presented in Table 5. Notably, we observe that conducting retrieval in the overall style prompt space yields significantly low recall. Conversely, the model performs substantially better when performing retrieval in the sub-space of each category.
4.4 Visual Question Answering (VQA)
JourneyDB comprises a collection of images encompassing abundant and diverse prompts. These prompts not only encompass stylistic attributes but also describe the visual contents of the generated images. To assess the model's proficiency in comprehending both style and content of generative data, we establish two tasks: multiple-choice visual question answering (MC-VQA)[48, 49, 12]. In the MC-VQA tasks, we utilize GPT-3.5 to generate "Style-relevant questions" and "Content-relevant questions" along with three distracting options for each question, in addition to the correct answer. The evaluation metric employed is accuracy, where the model selects its answer from the four options based on the given question and the image. A-OKVQA[12] highlights that MC-VQA overcomes several inherent challenges of direct answer evaluation, such as ambiguity, which is prevalent in open-ended VQA [16, 17]. The versatility of language expression implies that MC-VQA, by directly matching the selected option with the correct answer, provides a lucid and objective evaluation approach. This characteristic proves advantageous, especially considering the extensive spectrum of answers in our benchmark, encompassing a wide range of descriptions for diverse styles and contents.
To assess the performance of current multimodal models in the zero-shot visual question answering task within our benchmark, we adopt a methodology inspired by recent studies [50, 51, 52, 53]. In this approach, we provide the model with a question and its corresponding candidate answers enumerated with symbols ("A", "B", "C", "D"). By assigning the highest probability to a predicted token ("A", "B", etc.), the model selects the associated answer choice as its response.
The evaluation outcomes for the zero-shot multiple-choice visual question answering tasks, specifically the content-relevant and style-relevant tasks, are presented in Table 6. It is evident that the performance of existing multimodal models falls significantly short of satisfactory levels in both the content-relevant and style-relevant MC-VQA tasks. BLIP-2 [4] outperforms Flamingo9B [3] and MiniGPT-4 [38], yet its accuracy remains below  $70\%$ . These results highlight the substantial challenges that generative data poses to existing models in comprehending the visual contents and stylistic attributes. Generative data often represents scenes and object compositions that are absent in reality, thereby posing difficulties for multimodal models pre-trained on real images to interpret the visual elements of generated images when answering content-relevant questions. For instance, as illustrated in the second row and fourth column of Figure 4, the model fails to recognize the relationship between the piano and the tree in the image and predicts the option "D: the tree and piano are not related." This failure arises due to the rarity of scenes depicting a tree growing out of a piano in the real world. In comparison to answering content-relevant questions, the performance of all models generally deteriorates when addressing style-relevant questions. The generation of multiple-choice questions from text prompts encompassing diverse stylistic aspects, such as camera angle, lighting, and artistic styles, enables a comprehensive evaluation of a model's capacity to identify the stylistic attributes of generative data within JourneyDB. However, previous multimodal models are pre-trained using descriptions derived from captioning real images, thereby lacking exposure to the broad range of stylistic variations prevalent in generative data. Consequently, these models encounter difficulties in recognizing and distinguishing the various styles manifested in generative data. As illustrated in Figure 4, BLIP-2 provides incorrect answers to the style-relevant questions in the first row pertaining to camera angle, visual elements, lighting type, and animation style depicted in the images of JourneyDB.
We introduce JourneyDB, an extensive benchmark comprising four distinct downstream tasks, aiming to foster advancements in the comprehension of generative content. By providing a platform that facilitates the development of visual understanding in relation to generated images, researchers and practitioners are empowered to drive progress in this field.
Thanks Mengwei R. for her insightful feedback. Thanks Mingjie Z. for his assistance with data curation. This project is funded in part by National Key R&D Program of China Project 2022ZD0161100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is a PI of CPII under the InnoHK.
[57] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
[59] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890-16902, 2022.
[60] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332, 2022.
[61] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021.
[62] Epicdiffusion. https://civitai.com/models/3855/epic-diffusion.
[63] Dalle mini. https://github.com/borisdayma/dalle-mini.
[64] Dreamlike photoreal 2.0. https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0.
[65] Deliberate. https://civitai.com/models/4823/deliberate.
[66] Y Zhou, R Zhang, C Chen, C Li, C Tensmeyer, T Yu, J Gu, J Xu, and T Sun. Lafite: Towards language-free training for text-to-image generation. arxiv 2021. arXiv preprint arXiv:2111.13792, 2021.
[67] Realistic vision. https://civitai.com/models/4201/realistic-vision-v20.
[68] Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization. arXiv preprint arXiv:2112.01573, 2021.
[69] Majicmix realistic. https://huggingface.co/sinkinai/majicmix-realistic-v5.
[70] Chilloutmix. https://huggingface.co/swl-models/chilloutmix.
[71] Openjourney. https://openjourney.art.
A Data samples
In this section, we randomly sample instances from the dataset to provide a visualization for users. Shown in the Figure 5 are the uncurated samples of JourneyDB. The 5 columns from left to right are images generated by Midjourney, the corresponding prompts, the captions, style-relative and content-relative VQA annotation. We mark the content-relative words with green color, and the style-relative words with orange color.
B Details of Data Annotation
In this section, we introduce the details of data annotation, including image-prompt consistency filtering, downstream visual understanding annotations, and style clustering.
B.1 Image-Prompt consistency filtering
To build a clean test set, we hire a team of 40 professional annotators to mark the prompt word not presented in the corresponding image. Specifically, Given a picture and a corresponding piece of text, the annotators are required to find words or phrases in the text that do not match the picture. Definition of "mismatch": 1) Content that exists in the text but is missing (not reflected) in the picture. For example, if the text is "apples and bananas are placed on a table", and only apples are placed on the table in the picture, then mark "and bananas". 2) It exists in the text, but the content in the picture is inconsistent with the text. For example, if the text is "a red apple is placed on a table", but the picture is a green apple, then mark "red". We show the examples we use in the annotation document in Table 7.
Table 7: Image-Prompt Consistency Filtering Examples.
Num	Images	Text	Analysis
1	face portraits of two mechanical cy-borg warriors facing off, in the style of a UFC poster, cinematic blue or-ange lighting, volumetric lighting, smoke, hyper detailed armor, hard surface, neon light tubes, realistic, intricate details, symmetrical	All elements are fully represented in the picture. Thus this pair passes review without annotation.
2	a cute 1950s style alien with glow-ing green eyes waving	All elements are well represented in the picture. Thus this pair passes review without annotation.
3	Pokémon picnic, detailed, realistic	“Pokémon” are not clearly repre-sented in the picture, so mark it. “picnic”, “detailed”, and “realistic” are all reflected and not marked.
4	sketch, a blue glowing gem with two pieces of rope tied on each side, fan-tasy, Lord of The Rings, watercolor	“with two pieces of rope tied on each side” it does not conform to the ef-fect shown in the picture, mark it. “Lord of The Rings” does not mani-fest. “sketch”, “a blue glowing gem and fantasy, watercolor” are all re-flected and not marked.
Figure 5: Randomly sampled instances.
B.2 Visual Understanding Annotation
We employ GPT-3.5 to generate the answer to downstream tasks according to the given prompt. We follow this format to query GPT-3.5:
You are a visual art designer. Here is a prompt for a text-to-image generation model: [PROMPT]. You will be required to do the following 3 tasks based on the prompt you receive. Please be faithful to the prompt given to you and do not hallucinate. Directly answer the questions without any redundant sentences in the format of Python dict. The first task is to separate the prompt into 'Style', 'Content', 'Atmosphere', and 'Other' categories. 'Style' words describe the whole image style. 'Content' words describe the image content. 'Atmosphere' words describe the emotional and psychological elements associated with the image, including the mood and feeling conveyed by the scene. If some words are hard to be sorted into 'Style', 'Content', or 'Atmosphere', put them in the 'Other' category. You should try to limit putting words into the 'Other' category as much as possible. The second task is to provide the caption according to the content of the prompt. Only consider the 'Content' words separated in the first task and ignore the 'Style' and 'Atmosphere' words. Be faithful to the 'Content' prompt and do not add any redundant information. The caption should be in a tone that a visual AI assistant is describing the image. The caption should be a single complete sentence. The Third task is to design a set of multiple-choice questions based on the style and content that are separated in the first task. The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers and also provide wrong options. Only include questions that have definite answers that satisfy the following conditions: 1) one can see the content in the image that the question asks about and can answer confidently, 2) one can determine confidently from the image that wrong options are not in the image. Do not ask any questions that cannot be answered confidently. The answer should not be 'Unknown'. Please include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss events happening in the image, etc. Never ask about uncertain details. Never ask questions you cannot determine from the given prompt. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary. Ask at least one question for each category. For each question, there should be 4 options. The options should be relevant but only one is correct. Return a single json file with the following format: [FORMAT]. Strictly follow the provided format please. Directly return the python dict ONLY! Do not say the polite words. Make sure your answer can be parsed by json directly.
In this way, we encourage the output of the GPT-3.5 can be loaded by the "json" package directly.
B.3 Style Clustering
Similarly, we ask the GPT-3.5 to cluster the style prompts into several categories. The prompt we use to query GPT-3.5 is:
Here are some style-relative prompts: [PROMPTS]. Please help me build a hierarchical tree, to summarise these prompts into several categories like photography styles, camera parameters, colour grading, lighting, film looks, mood, artist style, etc. And each category may have fine-grained sub-categories. Please return a Python dict in the format like: [FORMAT]. You should design the keyword to make sure it summarizes its following list. One prompt can belong to more than one category. Directly return the python dict ONLY! Do not say the polite words. Make sure your answer can be parsed by json directly.
C.1 Question Answering Score (QAS)
The grammar structure of the prompts is quite different from the caption. The image caption is usually a complete sentence, while some prompts might simply be some broken phases instead. Prior metrics treat the prompts as complete sentences, which do not always hold and bring noise into the evaluation. Therefore, we propose a Question Answering Score (QAS) for the evaluation of the prompt inversion task, which is computed by feeding the predicted prompt to a large language model (LLM) and calculating the accuracy of the LLM answers to the relevant questions provided in the annotation.
Specifically, we make use of Vicuna [54] as the LLM  $L$ . Given a generated image  $I$ , a prompt inversion model predicts the prompt  $\hat{p}$ . And as the annotation, there are  $N$  style-relevant questions  $q_{\mathrm{s}}$  and answers  $a_{\mathrm{s}}$  concerning the style elements in the ground-truth prompts, as well as  $M$  content-relevant questions  $q_{\mathrm{c}}$  and answers  $a_{\mathrm{c}}$  concerning the content. In the following, we do not distinguish between symbols for style and content, but in implementation, we treat them separately to calculate  $\mathrm{QAS}_{\mathrm{s}}$  and  $\mathrm{QAS}_{\mathrm{c}}$ . We construct a prompt  $P$  with  $\hat{p}, q,$  and  $a$  in the following format:
Here is a prompt:  $[\hat{p}]$ . Assume you see the image generated from the given prompt. Answer the question: [q]. Choose from: [a]. Directly answer A or B or C or D without any further illustration. Your answer should be one single letter.
By feeding the constructed prompt  $P$  into the LLM  $L$  we obtain the predicted result:
$$
\hat {a} = L (P). \tag {1}
$$
We calculate the average accuracy separately among the  $N$  style questions and  $M$  content question for this image, and obtain the final QAS by computing the average for all the  $K$  images:
$$
\mathrm {Q A S} _ {s} = \frac {1}{K} \sum_ {K} \left(\frac {1}{N} \sum_ {N} \mathbb {I} (\hat {a} = a)\right) \tag {2}
$$
$$
\mathrm {Q A S} _ {c} = \frac {1}{K} \sum_ {K} \left(\frac {1}{N} \sum_ {M} \mathbb {I} (\hat {a} = a)\right) \tag {3}
$$
In this way, we convert the "prompt similarity" problem to compute the accuracy of the question-answering task, which is more interpretable. We show QAS results in the last two columns in Table 8.
Table 8: Evaluation results of Prompt Inversion on JourneyDB. We list results on the validation set in the upper half, results on the test set in the lower. For all metrics, the higher, the better.
Mode	Models	Test
BLEU-4	METEOR	ROUGE-L	CIDEr	Similarity	QASs	QASc
ZeroShot	BLIP-2 OPT [4]	0.29	2.85	7.06	6.46	0.36	12.42%	18.55%
BLIP-2 FlanT5 [4]	0.40	2.95	7.69	8.86	0.37	13.79%	18.58%
MiniGPT-4 [38]	1.71	6.51	13.13	11.40	0.43	17.12%	26.79%
Uni-Perceiver v2 [30]	0.37	2.73	9.88	15.45	0.34	12.43%	18.49%
Finetune	Uni-Perceiver v2 [30]	4.68	8.56	16.98	34.01	0.51	19.71%	24.84%
Images generated by SD
HPS:0.2303
HPS:0.1974
HPS:0.2008
HPS:0.2000
Images from JourneyDB
HPS:0.2489
Figure 6: Comparison between images generated by Stable Diffusion v1.4 and images in JourneyDB regarding HPS. Images in the same row are generated with the same prompt.
Prompt: A camping tent in the middle of a dark forest, three somalian girls with their black tumbaredforest.  
A camp with two tumbareds in the distance middle of a tall, dark forest.
HPS:0.2396
Prompt: post-apocalyptic, giant mutnat rabbits, serious, scary
HPS:0.2217
Prompt: ijsihdgfh, whuhidhda, fhrjjr, a horse fantasy character
HPS:0.2283
Prompt: water, ice, winter, 8k, photorealistic, cold, dark blue, starry sky, mirror reflection
C.2 Analysis of Image Quality
As shown in Fig. 6, Images in JourneyDB (credit to Midjourney [55]) exhibit better visual quality than those generated by Stable Diffusion, quantified by Human Preference Score (HPS) [56].