JourneyDB: A Benchmark for Generative Image Understanding
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Midjourney
Dataset
Generative Image Understanding
TL;DR: We present JourneyDB, a large-scale dataset with 4 million generated image-prompt pairs, 1 million caption annotations, and 8 million VQA annotations, for multi-modal visual understanding in the realm of generative image.

Abstract: While recent advancements in vision-language models have had a transformative impact on multi-modal comprehension, the extent to which these models possess the ability to comprehend generated images remains uncertain. Synthetic images, in comparison to real data, encompass a higher level of diversity in terms of both content and style, thereby presenting significant challenges for the models to fully grasp. In light of this challenge, we introduce a comprehensive dataset, referred to as JourneyDB, that caters to the domain of generative images within the context of multi-modal visual understanding. Our meticulously curated dataset comprises 4 million distinct and high-quality generated images, each paired with the corresponding text prompts that were employed in their creation. Furthermore, we additionally introduce an external subset with results of another 22 text-to-image generative models, which makes JourneyDB a comprehensive benchmark for evaluating the comprehension of generated images. On our dataset, we have devised four benchmarks to assess the performance of generated image comprehension in relation to both content and style interpretation. These benchmarks encompass prompt inversion, style retrieval, image captioning, and visual question answering. Lastly, we evaluate the performance of state-of-the-art multi-modal models when applied to the JourneyDB dataset, providing a comprehensive analysis of their strengths and limitations in comprehending generated content. We anticipate that the proposed dataset and benchmarks will facilitate further research in the field of generative content understanding. The dataset is publicly available at https://journeydb.github.io.

JourneyDB: A Benchmark for Generative Image Understanding
Keqiang Sun';
Junting Pan Yuying Ge Hao Li',
Haodong Duan',
Xiaoshi Wu', Renrui Zhang', Aojun Zhou', Zipeng Qin', Yi Wang', , Jifeng Dai Yu Qiao', Limin Wang Hongsheng Li1.344
Multimedia I Laboratory, The Chinese University of Hong Kong The University of Hong Kong SShanghai Artificial Intelligence Laboratory Centre for Perceptual and Interactive Intelligence
Abstract
While recent advancements in vision-language models have had transformative impact on
comprehension, the extent which these models possess the ability to comprehend generated images remains uncertain. Synthetic images, in comparison to real data, encompass a higher level of diversity in terms both content and style, thereby presenting significant challenges for the models to fully grasp. In light of this challenge, we introduce a comprehensive dataset, referred as JourneyDB. that caters the domain of generative images within the context of multi modal visual understanding. Our meticulously curated dataset comprises 4 million distinct and high-quality generated images, each paired with the corresponding text prompts that were employed in their creation. Furthermore, we additionally introduce external subset with results of another 22 text-to-image generative models, which makes JourneyDB a comprehensive benchmark for evaluating the comprehension of generated images. On our dataset, have devised four benchmarks to assess the performance of generated image comprehension in relation both content and style interpretation. These benchmarks encompass prompt inversion, style retrieval, image captioning, and visual question answering. Lastly, we evaluate the performance of state-of-the--rt multi-modal models when applied to the JourneyDB dataset, providing a comprehensive analysis of their strengths and limitations in comprehending generated content. We anticipate that the proposed dataset and benchmarks will facilitate further research in the field of generative content understanding The dataset publicly available https://journeydb.iihhee gittbu...b
1 Introduction
In recent times, notable progress has been achieved in the domain of Artificial Intelligence Generative Content (AIGC), particularly in the advancement of diffusion models ( that have significantly enhanced the quality of generative content. As a consequence, AIGC platforms such as DALLE, Stability AI, Runway, and Midjourney have gained considerable popularity, enabling exceptionally high-quality images using text prompts composed in natural language. These text prompts encompass both content and style descriptions provided by users, playing pivotal role in image generation (see Figure for an illustrative prompt). Unlike descriptions acquired from captioning real images, text prompts for image generation tend exhibit a high level of detail and specificity, surpassing mere portrayal of salient content. The primary objective behind the creation
generate
"Equal Contribution
Project Lead
1 Corresponding Authors
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Prompts: ultra realistic, maltese poodle dogs, brown fur, singing Karaoke, spotlight
Content maltese poodle dogs. brown fur, sing Karaoke
Caption
QA
Q1: What are the dogs doing?
group of brown maltese poodle dogs are singing Karaoke.
Styles Spotlight, ultra realistic
D. Singing Karaoke
Q2: What type of light is used in this image? B. Spotlight
Figure 1: Data Collection Procedure. To collect enough generated images, investigate the Midjourney channel on Discord to collect the available pictures. Then we employ the GPT-3.5 to annotate the downstream tasks, including 1) separating the prompt into "Style" and "Content", 2) generating the caption according to the content words obtained from task 1, 3) generating "Style- relevant questions" and "Content-relevant questions", providing 4 options for each question, together with the answer. Please refer to Section 3 for more details.
of these prompts lies in visual generation, resulting in intricate descriptions that encompass diverse stylistic facets such as lighting, camera angle, artistic style, medium, and more. Moreover, the generated content originates from the users' imagination, often depicting scenes and compositions that are entirely fictional and devoid of real-world existence.
Considering the aforementioned characteristics, we contend that both the elaborate textual prompts and the generated images themselves serve as valuable sources of information that can be incorporated into existing visual understanding benchmarks. On one hand, the detailed text prompts offer more comprehensive interpretation of the visual scene, enabling us to perceive the scene and comprehend its underlying style. On the other hand, the abundance of novel object compositions in the generated images provides insights into a realm unrestricted by conventional sense biases, facilitating exploration beyond the constraints of traditional visual representations.
Foundation models have achieved unparalleled capabilities across various visual understanding tasks, owing large-scale pre-training datasets, such as CLIP (2) Flamingo [3, and BLIP-2 [4] However, is essential to acknowledge that current foundation models are primarily pre-trained on real data, giving rise to concerns regarding their generalization ability and effectiveness in handling the distinctive characteristics associated with generative content. These models may not fully capture the nuanced aspects of generative content and might encounter difficulties in comprehending and generating high-quality images based on complex text prompts.
In view of this challenge. our research initiative seeks address this gap by curating dataset comprising a substantial number of 4 million meticulously generated images accompanied by corresponding text prompts. This dataset serves as the fundamental basis for a benchmark consisting of four distinct tasks, which collectively facilitate a comprehensive evaluation of generative content understanding.
The initial task, referred to as prompt inversion, involves identifying the text prompts employed by the user to generate the given images. This task serves decipher the original prompt or description, assessing the model's ability to comprehend both the content and style of the generated images. The second task involves style retrieval, wherein the model is tasked with identifying and retrieving similar generative images based on their stylistic attributes. This task evaluates the model's 's proficiency in discerning subtle stylistic nuances within generative images. The third task centres around image captioning, requiring the model to generate descriptive captions that accurately represent the content of the generative image. This task evaluates the model's capability to effectively comprehend and express the visual elements of the generated content using natural language. The fourth and final task is visual question answering (VQA), in which the model is expected to provide accurate answers to questions related to the generative image. This task evaluates the model's 's ability to comprehend the visual and stylistic content and deliver relevant responses based the provided questions.
We collected a total of 4, 692, 751 pairs of image-text prompts, which were subsequently divided into a training set comprising 4, 453, 193 pairs, a validation set comprising 234, 156 pairs, and test

---

Table I: A comparison between JourneyDB and other commonly-used Text-Image multi-modal datasets. Among all the commonly-un multi-modal datasets, the proposed dataset is the most versatile, supporting four downstream tasks. H: Human, M: Models
Dataset
Total Image Num
Label Source
Image Caption
VQA Prompt Inversion Style Retrieval
Flickr Caption
9
32k
H
V
COCO Caption
10
164k
VQA v2 III
204k
A-OKVQA 12
24k
LAION-COCO
13
600M
DiffusionDB
14
14M
Ours
4M
H M
set comprising 5, 402 pairs. We also include 45, 803 images from 22 other text-to-image models provided by HPD v2 [51, including VQ-Diffusiin (6, DALL-E 2 (7), StableDiffusion-XX XL [8 etc., to build the external set for cross dataset evaluation. Given that the generative model not flawless, some discrepancies in the text prompts may be present. Consequently, for the test set, we carried out human verification, where annotators were tasked with removing word descriptions that do not align with the corresponding images. To create annotations for tasks 3, and utilized GPT-3.5 to convert text prompts into annotations specific to each task.
To comprehensively evaluate the performance of current tfate-of-the-arr art multi- models, conducted extensive assessments using our benchmark dataset. Furthermore, we performed in-depth analyses to gain insights into the strengths and limitations of these models when applied to generative content. Overall, we observed that the state-of-the-ar models do not perform as effectively as they do on real datasets, and fine-tuning the proposed dataset significantly enhances their performance.
In conclusion, our contribution encompasses three key aspects: 1) the best of our knowledge, we are the first to draw attention to the visual understanding of generated images. 2) We propose JourneyDB, a large-scale benchmark that serves as both a training and evaluation resource for this emerging field. 3) We conducted an extensive evaluation of state-of-the-art visual understanding models using the proposed dataset, revealing their relatively limited performance on generative content. We hope that endeavours will contribute to further advancements in the field of generative content understanding.
2
Related Works
2.1 Image-Ter Text Datasets
We present summary of existing image-text datasets in Table ! The Flickr Caption dataset [9] consists of 32, 000 images obtained from the Flickr [15) platform, accompanied by five reference sentences provided by human annotators. The COCO Caption dataset [10] comprises 164 thousand images, with five independent sunan-generated captions provided for each image for training and validation, resulting in over 1.5 million captions. These datasets play crucial role in fostering the development of the Image-Caption Task. The Visual Question Answering (VQA) v2.0 [I] dataset, which is the second version of the VQA dataset [16], contains open-ended questions about images that require an understanding of vision, language, and commonsense knowledge to answer.
A-OKVQA [12) an augmented successor of OK-VQA (17). encompasses diverse of 24 thousand questions that demand a broad foundation of common and world knowledge for accurate responses. These datasets involve human employees in the annotation process, ensuring consistently high-quality annotations. However, manual annotation by human annotators is time-consuming and costly endeavour, thereby limiting the scalability of the datasets. LAION-COCO dataset containing 600 million detailed captions. Although these datasets may contain noise due to the cleaning or generation process using pre-trained neural network models, they have demonstrated their utility in training multi-modal models. However, it is important to note that these datasets primarily focus on real images and cater to specific task. A comparable dataset to the present study is DiffusionDB (18). a large-scale text-to-image prompt dataset comprising 14 million images generated using Stable Diffusion. However, the image quality from Stable Diffusion is plausible, and no further annotations
M another large-scale pairs, where GPT3.5 is employed to generate more
3

---

are available. In this paper, we collect data from Midjourney and provide annotations generated by GPT3.5 support four downstream tasks.
2.2 Text-to-Image Generative Models
Text-to-image generative models 19 20
aim at generating images according to text
conditions, apart from traditional generative models [2 23 24 25], which map random noise to images. Text-to-image generative models have experienced rapid development in recent years, empowering users to create image content through natural language specifications. This field has seen significant progress since Mansimov al.demonstrated that Deep Recurrent Attention Writer (DRAW) can generate images conditioned on text [26] (7]. Since then, several generative architectures and modeling approaches have been applied for text-to-image generation, including autoregressive models 19]. GANs [20], and diffusion models [14 7 211. Among these, diffusion models have shown better computational efficiency and the ability to produce higher-quality samples compared to autoregressive models [7]. These diffusion models have reached a level of maturity where they can generate high-quality images suitable for industrial deployment. Notably, Midjourney provides state- of-the-art text-to- to-image generation service using diffusion models [3] A vast number of artificial images are generated each day at unprecedented speed. As perception and generation tasks are double sides of the same coin, the achievements in the generative models open new probability for the perception studies. In this context, our dataset aims to organize and consolidate recent progress in text-to- text-to-image generative models while laying the foundations for further research in perception studies.
2.3 Multi-modal Foundation Models and Datasets
Aided by data from diverse sources, multi-modal foundation models capable of understanding and connecting data across multiple modalities, such as image, text, audio and so on. As prioneering vision-language models, CLIP and ALIGN (28) adopt contrastive learning paradigms and are pre- trained by millions of web--collected image-text pairs, which showcases promising visual zero-shot capabilities. Flamingo 3 and BLIP-2 4 further align pre-trained vision backbones with language models with intermediate networks and billions of data pairs, exhibiting superior results on vision- language tasks. OFA (2)]. Uni-Perceivers 30 31 32], and Unified-IO 33 also introduce unified training architectures for different modalities with competitive performance to uni-modal methods. Recently, inspired by the powerful GPT-4 (34) many efforts have been devoted to multi-modal [35] 33). LLaVA (37) and MiniGPT-4 (38). Given the textual prompts with image conditions, these models fine-tune a frozen LLaMA 39 to respond to multi-modality instructions, the training data of which is either existing image-caption data [10 or GPT-annotated pairs 37.. Despite the popularity of multi-modal models, it is still rarely explored for their generalization capacity generated vision-language data, considering the difference between the world pre-training data and generative content. In this paper, we propose a large-scale synthetic dataset, JourneyDB, along with customized benchmarks to fully validate the extension efficacy current multi-modal models.
models, such as LLaMA-
2.4 Training with Generated Data
It worth noting that the annotations generated by GPT demonstrate a lower level of noise than expected, validating the effectiveness of these models. Notably, LLaVA 337 introduces a novel instruction-tuning dataset that leverages the capabilities of both GPT3.5 and GPT4. Their experiments reveal a remarkable relative score increase of 295.8% elevating the score from 21.5 to 85.1, thus emphasizing the effectiveness of their generated data. LaCLIP [40] integrates text augmentations by employing text rewriting techniques with GPT3.5 and Bard. By rewriting the textual descriptions within existing image caption datasets, they achieve notable improvement of 36.08%, raising the score from 15.8 to 21.5. StableRep [41] unveils the remarkable potential of using exclusively synthetic data generated from text-to-image models to train highly effective visual representations, surpassing the performance of models trained solely on real image datasets. In similar vein, VideoChat [42 constructs a dataset by sequentially feeding dense captions to GPT3.5 in temporal order. Despite the inherent challenges in comprehending individual frames with GPT3.5, their successful mastery of understanding the entire video demonstrates the effectiveness of their approach. The generated annotations not only validate the effectiveness of GPT models but also significantly contribute to

---

Table 2: Statistics of JourneyDB. We provide 4 million generated image-prompt pairs, million captions and over million VQA annotations.
Dataset
Image
Prompt
Labeled Image
Labeled Prompt
Style QA
Content QA
Training Set
4,453,193
1,643,375
4,189,737
1,385,317
7,056,394
8,775,971
Validation Set
234,156
82,093
234,156
82.093
311,569
374,310
Testing Set
5,402
5,171
5,402
5,171
10,040
11,369
External Set
45,803
45,365
45,803
45,365
74,407
81,565
Total
4,738,554
1,776,004
4,475,098
1,517,946
7,452,410
9,243,215
advancing the understanding of images. Therefore, based on our demonstrated results, we firmly believe that our JourneyDB can serve valuable tool to enhance numerous image-related tasks.
Dataset
In this section, present the methodology employed for dataset collection and annotation, along with relevant statistical insights to gain a deeper understanding of the dataset.
3.1 Data Collection
The data collection procedure is presented in Figure In order to obtain a sufficient number of generated images, we investigated the Midjourney channel 43 on the Discord platform [44 to access the available pictures. Within the public Discord channel named "Midjourney." " users submit text prompts to the channel, and the Midjourney bot responds with the corresponding generated images. Users then select the preferred image for upscaling, and Midjourney provides the corresponding upscaled images. The chat history contains numerous publicly accessible prompt-image pairs. To collect the data, we utilized DiscordChalExporter [45]. a widely used Discord crawler, to download the publicly available images along with their corresponding prompts. In this version of the dataset, we only retained images that were generated solely based on text prompts, filtering out any images conditioned given images. Additionally, removed setfjourney-speccin arguments, such as "-y 4", to enhance the generalizability of the prompts and ensure their comprehensibility for existing large language models.
Moreover, to improves the diversity of JournyDB,, we additionally introduce another 22 text-to- image generative models into JourneDB,, such VQ-Diffusion (6]. DALL-E 2 771. StableDiffusion XL 18), etc., which makes our data a comprehensive benchmark for evaluating the comprehension of generated images. For each generative model, originally generated 3, 200 images, and a group of 60 annotators helped clean up the pairs without consistency to obtain the final cross-model test set containing 45, 803 images in total. Please find more details of this part in the appendix
3.2 Data Annotation
We provide ample annotations for multiple visual understanding tasks. The dataset is compared with existing methods in Table demonstrating its versatility in supporting four downstream tasks.
Annotation for Visual Understanding. In this section, GPT-3.5 is employed to annotate the downstream tasks. Specifically, a set of Midjourney prompts and explicit instructions are provided to GPT-3.5. The objectives "Atmosphere", and "Others", 2) generating captions based on the content words identified in task 1, 3) generating "Style-relevant questions" and "Content-relevant questions," accompanied by four answer choices for each question. The detailed instructions provided to GPT-3.5 can be found the Supplementary Materials.
follows: 1) segmenting the prompt into "Style", "Content"
Clustering of Styles. Numerous prompts related to style are highly intricate for style retrieval. Taking inspiration from existing prompt engineering platforms we propose hierarchical clustering
"naps:/www..bprompp..om
5

---

grows
lowkey lighting
Outdoor
7000
Photography Styles
sweet light
Shimmering lights
moody lighting
cover photography
Phlo
Tnen heart
cinematic perfect light
gorgeous lighting
6000
Vertical frame photographed with Canon EOS
Carl Zeiss lens Depth of Field
general lighting
By Rossdraws
dappling light
Lighting
5000
underwater photography
Depth assily Kandinsky
Abstract by Keith Negley
1970s studio photography
4000
photo studi
Style H Geiger
Art by Brian Viveros
Akira Hiramoto
3000
ilip Hodas
Photographed by Peter Lindbergh
Kenard Pak
Photographyy Styles
Varameters
2000
Colour Gradi ig
by Robert Mapplethorpe
Johanna Barforrd
Howard Scho
1000
Artist Style
Film Lonks
Lighting
Camera
Fashio
Design
Textu
sired by Eddie Mendoza Painting by Georg Baselitz
Fine
Alma-Tadema style
lim
Artist Style
0
Figure 2: Distribution and samples of the style prompts.
approach for organizing styles, which simplifies style retrieval and facilitates user reference. Since traditional word embedding and clustering methods struggle to handle sophisticated style words, we leverage GPT-3.5 for this task. Specifically, we divide the prompts into smaller patches, each comprising 200 prompts, and instruct GPT-3.5 to cluster the words. Subsequently, manually merge the categories from different patches to obtain the final "style tree". The distribution of the clustered style space is visualized in Figure
Filtering for Image-Prompt Consistency. Due to the limitations of Text-to-Image generative models, inconsistencies may arise between the prompt and the generated image. To ensure the quality of the test set, we engaged 40 annotators to identify inconsistent prompt words in the test set. Specifically, given pair of text prompts and the corresponding generated image, the annotators are instructed to verify if each word is depicted in the image. Words annotated as "Not Appear" are removed to obtain the clean prompts.
3.3 Data Statistics
General Statistics In this iteration, total of 4, 692, 751 images were collected, all with resolution exceeding 1024 x 1024, accompanied by corresponding text prompts. Among them, 1, 730, 639 prompts were found be independent. Furthermore, 1,472, 581 instances were annotated using the GPT-3.5 model, following the procedure outlined in Figure [ Additionally, 5, 402 images were filtered out due inconsistencies between the images and prompts, determined by the Image-Prompt consistency check. Moreover, 45, 803 images from 22 other text-to-image models provided by HPD v2 are introduced build the external set for cross dataset evaluation. In addition, clustering process was conducted to summarize the 70, 521 fine-grained styles into 334 style categories, displaying a long-tail distribution pattern, illustrated in Figure
Dataset Split Detailed statistics for each subset of the dataset provided in Table The entire dataset was randomly divided, with approximately a 20 : ratio, to create the training and validation sets. The training set comprises 4, 189, 737 images and 1, 385, 317 prompts, while the validation set consists of 234, 156 images and 82, 093 prompts. Additionally, a separate testing set was sampled for manual filtering, consisting of 5, 402 images and 5, 171 prompts.
Benchmarks
4.1 Prompt Inversion
The prompt, which determines both the content and style of generated image, contains crucial and comprehensive information regarding the image. When presented with an appealing generated image, individuals are eager to discern the prompt employed for its creation. By accurately identifying the prompts, they can further enhance the corresponding image, such as modifying its content or generating images with a similar style.

---

Table 3: Evaluation results of Prompt Inversion on JourneyDB, We list results on the validation set in the upper half, results on the test set the lower. For all metries, the higher, the better.
Validation ROUGE-L
Test
Models
BLEU-4
METEOR
CIDEr
Similarity
BLEU-4
METEOR
ROUGE-L CIDEr Similarity
BLIP-2 OPT 4
0.18
2.39
6.75
5.42
0.36
0.29
2.85
7.06
6.46
0.36
BLIP-2 F5 4
0.27
2.46
7.19
6.88
0.38
0.40
2.95
7.69
8.86
0.37
MiniGPT-4 88
1.49
5.50
12.51
10.39
0.43
6.51
13.13
11.40
0.43
Miii-Perceiier v2 30
0.23
2.44
9.11
12.38
0.33
0.37
2.73
9.88
15:45
0.34
Uni-Percceive V2pT
20.6
16.9
29.1
123.2
0.59
4.68
8.56
16.98
34,01
0.51
Table 4: Evaluation results of Image Captioning on JourneyDB. We list the zero-shot results the upper half, and the fine-tuned results in the lower. For all metrics, the higher, the better. FT denotes "Fine-Tune".
Validation METEOR
Test
COCO Caption
Models
BLEU-4
ROUGE-L
CIDEr
BLEU-4
METEOR
ROUGE-L
CIDEr
CIDEr
BLIP-2 OPT
4
0.82
5.43
19.87
22.00
2.35
7.88
22.40
37.60
145.8 (FT)
BLIP-2 F5an15
0.54
5.02
19.94
22.18
2.07
7.62
23.12
39.62
144.5 (ITT
Flamingo9B 3
0.94
6.58
14.19
10.19
1.39
6.84
17.75
19.10
(PS)
MiniGPT-4
38
2.28
7.39
19.24
16.78
2.79
9.84
20.31
22.34
Uni- Sni-Perceeivr
0.41
4.50
18.72
21.88
0.94
5.21
16.71
20.13
122.5 (FT)
PaiiPeereiier v2FT
30
8.20
12.53
27.09
50.72
3.23
10.12
22.45
31.76
However, predicting the prompts of image is challenging task. Existing visual understanding models, such as image-caption models, often fall short in providing a detailed description of the image's main elements, such as the subject, while neglecting other indispensable details like the viewpoint, illumination, or art style.
Prompt inversion aims to address this gap. involving the process of taking single image and predicting the corresponding prompts associated with it. We anticipate that the proposed dataset would further facilitate the development of prompt inversion through the in-depth analysis of the prompts.
To evaluate the effectiveness prompt inversion, we extend the metrics commonly utilized in image captioning, including Bleu, METEOR, ROUGE, and CIDEr. Additionally, we adopt the approach employed in related Kaggle competition [46] to calculate the Cosine Similarity of the sentence-traasffommers features
Furthermore, in the supplementary materials, we propose Question Answering Score (QAS) for eval- uating the prompt inversion results. In this paper, establish a benchmark for the zero-shot prompt inversion task by leveraging state-of-the-ee multi-modal models, namely BLJIP-2 OPT2. [], BLIP-2 Flan FlanT5XT. [4), Flamingo9B 3. MiniGPT-4 38 and Uni-Perceiver v2 30 To ensure optimal performance in this novel task, we customize different prompts for each model.
We evaluate these models on the test set of our dataset, and the results are presented in Table During the experiment, we observed that the existing models struggle to capture the intricate details and style-related information of the input image, leading to lower performance compared to conventional datasets.
To verify the effectiveness of our dataset, we fine-tuned mni-Perreiier v2 for 20 epochs and noted significant improvement in the prompt inversion task. It is important to note that we followed the training methodology outlined in (30 without tuning the hyperparameters or utilizing data augmentations. This demonstrates that our JourneyDB can complement existing image-text datasets for training prompt inversion models. Nevertheless, it is evident that there is still a substantial way to go in developing robust and effective prompt inversion model.
4.2 Image Caption
Image captioning tasks require multi-modal models to generate textual descriptions for the visual con- tent of an image. In comparison to existing image captioning benchmarks such as COCO Caption [10] JourneyDB encompasses both detailed descriptions and high-level summarizations of images, thereby assessing the model's proficiency in fine-grained recognition and holistic understanding.
7

---

Blip2
[OPT]: walking background
spaceship
Blip2 [Fla faturistie space with neon lights
[Plan15] walking through moon in the background
Flaminge: image Heawei Ascend W1
MiniGPI4 IVicuna 1381: This futuristie
Flamingo:
astronauts
space, background
with ncon lights and a statrcase leading the next level.
MiniPP44 [Vicuna
astronauts walking on the surface of the moon
GroundTruthh: Inside mother ship from distunt future, big portal is activated, revealing intricate details and neon lights, all captured cinemutic style with Sk resulation und soft and bold lights.
(b) GroundTruth:: A group of kids lunding the
2 JOPT: II: a cartoon potato sitting at computer
Blip2 [OPT]: I]: the swiss flgg. with cross and an cagle on it
Blip2 Tian potato sitting froet of computer
[FlanT5]: with wings
switzerland
Flaminge: An image of Bittle robot sitting loptop
Swiss with words Flay" "Switterfand". Swinzerland is country in central Europe.
Flamsingo:
MiniGPT4 |Vicuna 13B): The image shows cartoon potato sitting
computer
MiniGPT [Vicuna 13B]; The flag of Switzerfand is white cross on red barkground with white cross red baclgground.
GroundTruth:
Ground Truth: A chibi potato looking sad work the office.
GroundTruth: nurse logo representing public le with silver, bluc, and red colors on pure white background.
Figure 3: Samples from the validation set of JourneyDB captioning. The examples show that existing multi-modal models failed to recognize some key concepts from the Al Al-generated images.
Table 5: Style Retrieval Results. The metric used there the Recall.
Validation
Test
Method
Over-All
Per-Category
Over-All
Per-Category
CLIP-ViT-L/14 2
0.65
41.72
0.47
41.33
We evaluate various existing multi-modal models on the image captioning sub-task of JourneyDB. The results are presented Table indicating the challenges faced by multi-modas models trained on natural images in providing accurate descriptions for Al-generated content. The quantitative performance notably poorer (significantly worse than COCO Caption results) due to two primary factors: GPT-3.5 tends to generate verbose descriptions for the images in JourneyDB, resulting in lengthy ground-truth captions. This discrepancy between lengthy ground-truth captions and shorter predicted captions undermines the quantitative performance. When describing Al-generated images, the focus may differ terms of concepts such as emotions, human/object attributes, etc., compared to natural images. However, existing image captioning models have not adequately addressed these concepts.
We provide qualitative examples in Fig 3 Existing multi-modal approaches fail to describe key concepts present in the Al-generated content (c.g., Fbg depicts kids astronaut suits, 3)) shows sad potato). Moreover, some models may hallucinate contents that do not exist in the images (e.g., Open- Flamingo hallucinates objects and text in Fig 3/a c)).
4.3 Style Retrieval
We inhabit captivating world enveloped multitude of vibrant colours and ever-shifting illumina- tion. Artists, in turn, develop their distinct styles to convey their unique perspectives of the world. Elements such as weather conditions, moods, and atmospheres in an image, resulting in complex "style system." As detailed in Section 3.2 we have compiled a comprehensive collection of over 150,000 style words describe the style-related attributes of images.
contribute to the style portrayed
8

---

Table 6: Evaluation results of the content-relevant and style-relevant zero-shot Multiple-Choice Visual Question Answering on JourneyDB. The evaluation metric here accuracy.
Validation
Test
Method
Content
Style
Content
Style
Flamingo9B
32.1%
31.9%
35.6%
41.4%
MiniGPT-4
38
28.2%
26.6%
31.1%
29.3%
BLIP-2 2 Flan
65.8%
54.9%
69.7%
57.4%
Style-Relathve Quesiios
Q1: What type camera angle image?
QZ: What the dominast visual image?
Q3: What of lighting is used in the
Q4: Which animation studio most sosociated
this
Clese-up, Medium,
Lines, Shupes,
A: Soft lighting, Directional lighting,
Disney, Dreamworks,
C: Top-down, Side-view.
Colors.
Textures.
C: Volumetric lighting, D:. Ambleat lighting.
C: Pisar, D: Scudio Ghibli.
Blip-2:
Blip2: Ground Truth:
Hlip2: Ground Truth: C
Blip2: Ground Truth:
Ground Truth: C
Content-Relativ Quertons
present
image?
subject
woddling?
image?
piano image?
At large spider with the head of IJy. B: giant scorpion with head of butterfly. maviive glant crah with the head of
At baby
A: The tree behind the plano,
A: Just the couple, B: large croad. Only family members,
Bi musical instrument.
B: The tree growing out of the piano,
C: of kittens, fire pit.
top piano,
coloseal
wasp.
related.
Blip-2: A Ground Truth:
Hlip2: A
Blip2: Ground Truthc:
Blip2: Ground Troth:
Ground Truth:
Figure 4: Failure cases of BLIP-2 4 for Multiple-Choice Visual Question Answering.
Given the vast expanse of the style space, identifying the style of given image poses significant challenge, even for human observers. Consequently, there is need for style retrieval techniques to aid in comprehending the style exhibited in an image.
Directly retrieving style prompt from extensive pool of candidates is complex and time- consuming task. Therefore, we employ clustering techniques to group the style prompts into 344 categories, including camera parameters, lighting, artist style, colour schemes, and more, as outlined in Section 3.2 By doing so, we effectively narrow down the search space for style prompt retrieval within each category. To establish a benchmark, we employ CLIP (2 for zero-shot style retrieval evaluation. We extract the features of both the images and the style prompts, subsequently calculating the inner product between the image feature and all candidate style prompts. The results are presented in Table Notably, we observe that conducting retrieval in the overall style prompt space yields significantly low recall, Conversely, the model performs substantially better when performing retrieval in the sub-space of each category.

---

4.4 Visual Question Answering (VQA)
JourneyDB comprises collection of images encompassing abundant and diverse prompts. These prompts not only encompass stylistic attributes but also describe the visual contents of the generated images. To assess the model's proficiency in comprehending both style and content of generative data, we establish two tasks: multiple-choice visual question answering (MC-VQA) 48 49 12 In the MC-VQA tasks, we utilize GPT-3.5 to generate "Style-relevant questions" and "Content-relevant questions" along with three distracting options for cach question, in addition to the correct answer. The evaluation metric employed is accuracy, where the model selects its answer from the four options based on the given question and the image. A-OKVQAII [2] highlights that MC-VQA overcomes several inherent challenges of direct answer evaluation, such as ambiguity, which is prevalent in open-ended VQA [16 [7]. The versatility of language expression implies that MC-VQA, by directly matching the selected option with the correct answer, provides lucid and objective evaluation approach. This characteristic proves advantageous, especially considering the extensive spectrum of answers our benchmark, encompassing a wide range of descriptions for diverse styles and contents.
To assess the performance of current multimodal models in the zero-shot visual question answering task within our benchmark, we adopt a methodology inspired by recent studies [50) 51 |2 (3). In this approach, we provide the model with a question and its corresponding candidate answers enumerated with symbols ("A", "B", "C", "D"). By assigning the highest probability to a predicted token ("A", "B", etc.), the model selects the associated answer choice as its response.
The evaluation outcomes for the zero-shot multiple-choic visual question answering tasks, specif- ically the content-relevant and style-relevant tasks, are presented in Table It is evident that the performance of existing multimodal models falls significantly short of satisfactory levels in both the content-relevant and style-relevant MC-VQA tasks. BLIP-2 [4] outperforms Flamingo9B ( and MiniGPT-4 38 yet its accuracy remains below 70%. These results highlight the substantial challenges that generative data poses to existing models in comprehending the visual contents and stylistic attributes. Generative data often represents scenes and object compositions that are absent in reality, thereby posing difficulties for multimodal models pre-trained on real images to interpret the visual elements of generated images when answering content-relevant questions. For instance, as illustrated in the second row and fourth column of Figure 4 the model fails to recognize the relationship between the piano and the tree in the image and predicts the option "D: the tree and piano are not related." " This failure arises due to the rarity of scenes depicting tree growing out of a piano in the real world. In comparison to answering content-relevan models generally deteriorates when addressing style-relevant questions. The generation of multiple- choice questions from text prompts encompassing diverse stylistic aspects, such as camera angle, lighting, and artistic styles, enables a comprehensive evaluation of model's capacity to identify the stylistic attributes of generative data within JourneyDB. However, previous multimodal models are pre-trained using descriptions derived from captioning real images, thereby lacking exposure to the broad range of stylistic variations prevalent in generative data. Consequently, these models encounter difficulties in recognizing and distinguishing the various styles manifested in generative data. As illustrated in Figure 4. BLIP-2 provides incorrect answers to the style-relevant questions in the first row pertaining camera angle, visual elements, lighting type, and animation style depieted in the images of JourneyDB.
questions, the performance of all
5 Conclusion
We introduce JourneyDB, an extensive benchmark comprising four distinct downstream tasks, aiming to foster advancements in the comprehension of generative content. By providing a platform that facilitates the development of visual understanding in relation to generated images, researchers and practitioners are empowered to drive progress in this field.
Acknowledgement
Thanks Mengwei R. for her insightful feedback. Thanks Mingjie Z. for his assistance with data cura- tion. This project is funded in part by National Key R&D Program of China Project 2022ZD0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technol- ogy Commission (TTC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is Pl of CPII under the InnoHK.
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image symthesis with latent diftusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[3] lean-Raptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Newral Information Processing Systems, 35:2371 16-23336, 2022.
[4] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language -egggge-mmage pre- wre-training with frozen image encoders and large language models. arXiv preprint arXiv:2331. 12597, 2023.
[5] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferenees text-to-image synthesis. arXiv preprint arXiv:2306.0 09341, 2023.
[6] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[7] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.061 2022.
[8] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0195 2023.
[9] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[10] Xinlei Chen, Hao Fang, Tsung-Yt Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504 00325, 2015.
[11] Yash Goyal. Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[12] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162. Springer, 2022.
[13] Christoph Schuhmann, Andreas Köpf, Richard Veneu, Theo Coomhes, and Romain Beaumont. Laion coco: 600m synthetic captions from laion2b-en, 2022.
[14] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021.
[15] Flickr. https//wwww.ticcrrcom.
[16] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[17] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[18] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang. Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models, arXiv:2210.144: 14896 [cs]. 2022.
11

---

[19] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation via transformers. Advances in Neurul Information Processing Systems, 34:19822-19835, 2021.
[20] Katherine Crowson, Stella Biderman, Daniel Koris, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision-ECCV ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 88 105. Springer, 2022.
[21] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedch Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorcalistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 1487, 2022.
[22] lan Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11 11399 144, 2020.
[23] Jingtan Piao, Keqiang Sun, Quan Wang, Kwan- Yee Lin, and Hongsheng Li. Inverting generative adversarial renderer for face reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15628, 2021.
[24] Keqiang Sun, Shangzhe Wu, Zhaoyang Huang, Ning Zhang, Quan Wang, and Hongsheng Li. Controllable 3d face synthesis with conditional generative occupancy fields. In Advances in Neural Information Processing Systems, 2022.
[25] Keqiang Sun, Shangzhe Wu, Ning Zhang, Zhaoyang Huang, Quan Wang, and Hongsheng Li. Cgof++: Con- trollable face synthesis with conditional generative occupancy fields. arXiv preprint arXiv:2211.13251, 2022.
[26] Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. CoRR, abs/151 1e//11111029999 .02793, 2015.
[27] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. ArXiv, abs/1502.0462. 2015.
[28] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Lec, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[29] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.
[30] Hao Li. Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li. Chun Yuan, Xiaohua Wang. Yu Qiao. Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 2691-2700, 2023.
[31] Xizhou Zhu, Jinguo Zhu, Hao Li. Xiaoshi Wu, Hongsheng Li. Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unitied architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804- 16815, 2022.
[32] Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-pereeiver-moe:: Learning sparse generalist models with conditional moes. In Advances in Neural Information Processing Systems, 2022.
[33] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddaa Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.088) 10911, 2022.
[34] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.087777 087744, 2023.
[35] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-effiee visual instruction model, arXiv preprint 150000 2023.
12

---

[36] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu. Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint urXiv:2303. 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint axXiv:330.088433 2023.
[38] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar0iv: 13971, 2023.
[40] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In NewrlPS, 2023.
[41] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners, arXiv preprint arXiv:2306.0090 2023.
[42] KunChang Li. Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 260505530,, 2023. 2023.
[43] Join the midjourney discord server! hutp://diioonleemm/miiiemmmmmen
[44] Discord your place to talk and hang out. htpps//diisorrd.comm/
[45] Discordchatexporter. https://githhb..omm//// http:///ww..motee/i/ioooooeeoeee
[46] Will Cukierski Ashley Chow, inversion. Stable diffusion image to prompts, 2023.
[47] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 11 2019.
[48] Yuke Zhu, Oliver Groth, Michacl Bersstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 4995-5004, 2016.
[49] Jiaying Lu, Xin Ye, Yi Ren, and Yezhou Yang. Good, better, best: Textual distractors generation for multiple-choice visual question answering via reinforcement learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4921-4930, 2022.
[50] Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. arXiv preprint ar Xiv:2210 12353, 2022.
[51] Jack W Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmannn Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 2021.
[52] Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv: 15556, 2022.
[53] Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. Can large language models reason about medical questions? arXiv preprint -xXx:22277.08143. 2022.
[54] Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[55] Midjourney,
[56] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- to-iate- models with human preference, 2023.
13

---

[57] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEE/CVF conference on computer vision and patiern recognition, pages 12873-12883, 2021.
[58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever. Mark Chen. Glide: Towards photorealistic image generation editing with text-guided diffusion models. arXiv preprint vrXiv:2112.10741 2021.
[59] Ming Ding, Wendi Zheng, Wenyi Hong, Jie Tang. Cogview2: Faster and better text-to- to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890 169002 2022.
[60] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332 2022.
[61] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block- wise quantiza- tion. arXiv preprini arXiv:2110.02811. 2021.
[62]
Epicdiffusion. http:///wwwaaa.cccmmmmmm///////a
[63] Dalle mini. https//ghhhl...co/uouisssaaaa/aa
[64] Dreamlike photoreal 2.0. http://wwwnooaecccccccemeleeeeer
[65]
Deliberate. https///ttttaacooommmmm////////t
[66] Y Zhou, R Zhang, C Chen, Li, C Tensmeyer, T Yu, J Gu, J Xu, and T Sun. Lafite: Towards language-free training for eext-to-image generation. arxiv 2021. arXiv preprint arXiv:2111.1399, 2021.
[67] Realistic vision. http://iii.....ommmmmm////////rr
[68] Xingchao Liu, Chengyue Gong. Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fusedream: Training-fiee free text-to-image generation with improved clip+ gan space optimization. arXiv preprint
arXiv:211201573. 2021.
[69] Majiemix realistic. http:///aeeiigge...cotiiiiiiiiir
[70]
Chilloutmix. http:://wuggiiggne...oosmmmmeeet
[71] Openjourney. https://openjourree.artt
14

---

Appendices
A
Data samples
In this section, randomly sample instances from the dataset provide visualization for users. Shown the Figure 5 are the uncurated samples of JournyDB.. The 5 columns from left to right images generated by Midjournyy, the corresponding prompts, the captions, style-relative and content-relative VQA annotation. We mark the content-relative words with green color, and the style-relative words with orange color.
B Details of Data Annotation
In this section, we introduce the details of data annotation, including image-prone consistency filtering, downstream visual understanding annotations, and style clustering.
B.1 Image-Prompt consistency filtering
To build a clean test set, we hire team of 40 professional annotators to mark the prompt word not presented in the corresponding image. Specifically, Given picture and corresponding piece of text, the annotators are required to find words or phrases in the text that do not match the picture. Definition of "mismatch":
Content that exists in the text but missing (not reflected) in the picture. For example. if the text is "apples and bananas are placed on table", and only apples are placed on the table in the picture, then mark "and bananas". 2) It exists in the text, but the content in the picture is inconsistent with the text. For example, if the text is "a red apple is placed on a table", but the picture is green apple, then mark "red". We show the examples we use the annotation document in Table 7
Table 7: Image-Prompt Consistency Filtering Examples.
Num
Images
Text
Analysis
face portraits of two mechanical cy- borg warriors facing off, in the style of a UFC poster, cinematic blue or- ange lighting, volumetric lighting, smoke, hyper detailed armor, hard surface, light tubes, realistic, intricate details, symetrical
All elements are fully represented in the picture. Thus this pair passes review without annotation.
All elements are well represented in the picture. Thus this pair passes review without annotation.
cute 1950s style alien with glow- ing green eyes waving
"Pokémon" are not clearly repre- sented the picture, so mark "picnic", "detailed", and "realistic" are all reflected and not marked.
Pokémon picnic, detailed, realistic
"with two pieces of rope tied each side" it does not conform to the ef- fect shown in the picture, mark it. "Lord of The Rings" does not mani- fest. "sketch". "a blue glowing gem and fantasy, watercolor" are all re- flected and marked.
sketch, blue glowing gem with pieces of rope tied on each side, fan- tasy, Lord of The Rings, watercolor
15

---

Images
Promuts
Caption
(Style)
(Content)
Whish this image?
vinual
depicted
hoastifal amondds scrfitid ana ferot wiich grece mons her drow foolng camera,, geteal siew, high detail,
Amsanaa Stythied partrngs farest watch in imses. drawed in gann betioe
Notalle Partnan,
A: Migaetaa Diteilind,
Laira Watnes,
Arannaa Seyfried.
fireet.
W heal the stline the feeet witch the madcaf??
Sike gures dircctly at the caneers caploring
Q2: What viee dass imnage praside?
eeter odkes moadl steosphere, style miggobar diterited
portralt view that accestentis
Landscape Bird' begg nlew,
kaves
Flamers,
captivatiag
Fauthers.
ope
many rabortul aninaais
colerfal antinal in wendertand drvane sighmaars, presested photer stylo.
Miliinalis,
Binds
it weedertand dream sighnnare 2t photercaliiiic
Recees,
Fich
C: Gattie Berequo
Lises.
Banns
buauty particles is state of cxperpecilie. their
What depioned Image?
What concept reprosented
Inside a quantara state
deticancte interiniaed tranccedeett dance.
functiono
Az Emprosiiaiion
Resloa,
Osentere
#lemmanagtet Supergedttie quqantur partides.
Nenton's lans mation.
Quanttm cntaggeemett
What retoorannt shana Imags?
What
A: Chinner and That feod,
plate Chinons fand, allea, arearevns, hreary orkutal rostaraatt
plsti Chincee clanaily
everyy eriental.
trattarla,
raqueraa,
Iodian earry.
Japasese
Talken grsts.
presented percilats adaraed with dragon
decorative dicucats peredala??
Q2: What the materta of the plate sned?
Parcetain,
Dragen eranucets.
Flower patterus.
Glass.
Makaninos
locl
Mark phareh pasther ged crysta perple insuncly intricate
A Aypernasinnater, elegeas, and omatt II
B: Mideratts
Mininnl
Sther.
GaM.
image katuring stansingle deteled depiction panther pad surroanded by perple fllgree.
C: Blah.
Parple.
typerecireellt, slegant, amats, realitte, detailed, ik
would you decrrioe
What anitnall depkcned Imxx??
Imsgc??
Az Misinnabik,
Lise,
Summal
Eaple
2D retra cumpater with an astresntiaaelts
An sumenaut's hand reaching our from ledis evenge
QU: What oygc Eluntration vode ls saed the Imsre?
Q1: W main eabjert lnaide the congrader?
opon,
art, thick liccs brnght
atole with vector kcsturing thick lino. bright calain, and
C: Yedtve
redertee
Flower.
cakers, single, btadi auttiee wa
Engressinaionn
Dr. Rocket.
steple dedign cottiied to
Film willl of rabile sitting counter ari- dees lnanpdaar, drtslaut whicks froe fambier glas in
QU: in which file's vtyle is this imsuee depicted?
QU: W bat the rabiit delaz in the image??
rabili conster
of are- lanagahar,
Playing plano.
whiskl.
shot, qualit plaos. sharp, book at that detall -qualiy - tot.
Encepiion (2019)
Wriling litter.
Figure Randomly sampled instances.
16

---

B.2 Visual Understanding Annotation
We employ GPT-3.5 to generate the answer to downstream tasks according the given prompt. We follow this format to query GPT-3.5:
You are a visual art designer. Here is prompt for a text-to-image generation model: [PROMPT). You will be required to do the following tasks based on the prompt you receive. Please be faithful to the prompi given to you and do not hallucinate. Directly answer the questions without any redundant sentences in the format of Python dict. The first task is to separate the prompt into 'Style'. 'Content', 'Atmosphere' and 'Other' categories. 'Style' words describe the whole image style. 'Content' words describe the image content. 'Ammosphrre' words describe the emotional and psychological elements associated with the image, including the mood and feeling conveyed by the scene. If some words are hard to be sorted into 'Style', "Content', or 'Ammosphre'' put them in the 'Other' category. You should try to limit putting words into the 'Other' category as much as possible. The second task is to provide the caption according to the content of the prompt. Only consider the 'Content" words separated in the first task and ignore the 'Style' and 'Atmosphere' words. Be faithful to the "Content' prompt and do not add any redundant information. The caplion should be in tone that a visual Al assistant is describing the image. The caption should be a single complete sentence. The Third task is to design set of multiple-choie questions based on the style and content that are separated in the first task. The answers should be in a tone that a visual Al assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers and also provide wrong options. Only include questions that have definite answers that satisfy the following conditions: 1) one can the content in the image that the question asks about and can answer confidently. 2) one can determine confidently from the image that wrong options are be answered confidently. The answer should not be 'Unknown'. Please include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss events happeming in the image, etc. Never ask about uncertain details. Never ask questions you cannot determine from the given prompt. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps make the content more convincing and well-organized. You can include multiple paragraphs if necessary. Ask at least one question for each category. For each question, there should be options. The options should be relevant but only one is correct. Return a single json file with the following format: [FORMAT), Strictly follow the provided format please. Directly return the python dict ONLY! Do not say the polite words. Make sure your answer can be parsed by json directly.
the image. Do not ask any questions that cannot
In this way, we encourage the output of the GPT-3.5 can be loaded by the "json" package directly.
B.3 Style Clustering
Similarly, we ask the GPT-3.5 cluster the style prompts into several categories. The prompt we use to query GPT-3.5 is:
Here are some style-relative prompts: [PROMPTS]. Please help me build hierarchal tree, to summarise these prompts into several categories like photography styles, camera parameters, colour grading, lighting, film looks, mood, artist style, etc. And each category may have fine -re-grained cabb-aareeoiss. Please return a Python dict in the format like: [FORMAT]. You should design the keyword to make sure it summarizes its following list. One prompt can belong to more than one category. Directly return the python diel ONLY! Do not say the polite words. Make sure your answer can be parsed by json directly.
17

---

Additional Experiments
C.1 Question Answering Score (QAS)
The grammar structure of the prompts is quite different from the caption. The image caption is usually a complete sentence, while some prompts might simply be some broken phases instead. Prior metrics treat the prompts as complete sentences, which do not always hold and bring noise into the evaluation. Therefore, we propose a Question Answering Score (QAS) for the evaluation of the prompt inversion task, which is computed by feeding the predicted prompt to large language model (LLM) and calculating the accuracy of the LLM answers to the relevant questions provided in the annotation.
Specifically, make of Vicuna [54] as the LLM L. Given generated image 1, prompt inversion model predicts the prompt p. And as the annotation, there are NN style-relevant questions ds and answers a, concerning the style elements in the ground-truth prompts, as well as M content- relevant questions and answers concerning the content. In the following, we do not distinguish between symbols for style and content, but implementation, QAS, and QAS.. We construct a prompt P with p. 9, and a in the following format:
treat them separately to calculate
Here is prompt: [p]. Assume you the image generated from the given prompt. Answer the question: [q]. Choose from: [a]. Directly answer or B or C or D without any further illustration. Your answer should be one single letter.
By feeding the constructed prompt P into the LLM obtain the predicted result:
à L(P).
We calculate the average accuracy separately among the N style questions and M content question for this image, and obtain the final QAS by computing the average for all the K images:
QAS,
I(a =a)
K
and
QAS,
Σ I(ā I(ā= a) M
(3)
In this way, convert the "prompt similarity" problem compute the accuracy of the question- answering task, which is more interpretable. We show QAS results in the last two columns in Table
Table Evaluation results of Prompt Inversion on JourneyDB. We list results on the validation set in the upper half, results on the test set in the lower. For all metrics, the higher, the better.
Test ROUGE-I. CIDEr Similarity
Mode
Models
BLEU-- METEOR
QAS,
QAS,
BLIP-2 OPT
0.29
2.85
0.36
12.42%
18.55%
BLIP-2 Flan T5
0.40
2.95
7.69
8.86
0.37
13.79%
18.58%
ZeroShot
MiniGPT-4
38
1.71
6.51
13.13
11.40
0.43
17.12%
26.79%
Uni- Peiceeeee v2
30
0.37
2.73
9.88
15.45
0.34
12.43%
18.49%
Finetune
Uni-Perceiver
30
4.68
8.56
34.01
0.51
19.71%
24.84%
18

---

HPS 0.23%
1952 0.285
Pronutt compling fewd middle with thair bocks cameadieent. sastnnce mildll of dank forost
Proseet part part-apeealgrii, gland
Nighllgghh Shayod fontang chanacter
Prowett watter, sibtte, Ba), mirror rellaction
scory
itarry
Figure 6: Comparison between images generated by Stable Diffusion v1.4 and images in JourneyDB regarding HPS. Images in the same row are generated with the same prompt.
C.2 Analysis of Image Quality
As shown in Fig. 6 Images in JourneyD (credit to Midjourney 55
exhibit better visual quality
than those generated by Stable Diffusion, quantified by Human Preference Score (HPS) [56].
19

---

Table 9: Statistics of JourneyDB. We provide 4 million generated image-prompt pairs, million captions and over million VQA annotations.
Dataset
Labeled Image
Laheled Prompt
Style QA
Content QA
VQ-Diffusion
1,888
1,869
3,067
3,346
VQGAN + CLIP
1,888
1,869
3,067
3,346
GLIDE [58]
1,898
1,878
3,101
3,349
Cog View2
59
1,914
1,896
3,135
3,387
Latent Diffusion
1,942
1,942
3.159
3,438
Versatile Diffusion
60
1,953
1,935
3,179
3,427
Stable Diffusion v1.4
2,028
2,010
3,301
3,621
DeepFloyd-XI. 61
2,052
2,028
3,334
3,655
Epic Diffusion
2,066
2,047
3,366
3,685
DALL-E mini 63
2,097
2,075
3,415
3,739
Dreamlike Photoreal 2.0
64
2.100
2,080
3,393
3,737
Stable Diffusion v2.0
2,104
2,084
3,405
3,756
Deliberate (65
2,122
2,101
3,447
3,781
LAFITE 66
2,124
2,105
3,439
3,804
Realistic Vision 67
2,144
2,119
3,475
3,832
FuseDream 6X
2,176
2,157
3,536
3,893
SDXL Refiner 0,9
8
2,184
2,161
3,540
3,915
MajicMix Realistic
2.189
2.167
3,568
3,910
ChilloutMix 77
2,207
2.185
3,571
3,923
DALL-E 7
2,220
2.197
3,593
3,967
Openjourncy 71
2,237
2,214
3,630
3,992
SDXL Base 0.9
g
2,270
2,246
3,686
4,062
Total
45,803
45,365
74,407
81,565
Table Evaluation results of Prompt Inversion and Image Captioning on the extension test set of JourneyDB.
Prompt Inversion METEOR ROUGE-L
Image Caption
VQA
Models
BLEU-
CIDEr
BLEU-4
METEOR ROUGE-L CIDEr
Style
Content
BLIP-2 OPT
4
3,46
8.06
24.82
51.81
3.99
9.00
26.25
55.64
BLIP-2 Flan
5.15
9.41
26.06
54.57
4.56
9.77
27.57
59.28
68.38%
66.57%
D Cross-model Test Set
As listed Table 9. we additionally introduce another 22 text-to-image generative models into JourneyDB, such as VQ-Diffusion (6] DALL-E (7]. StableDifflsion-XI. etc., which significantly improves the diversity of JourneyDB, making comprehension of generated images. For each generative model, we originally generated 3, 200 images, and a group of 60 annotators helped clean up the pairs without consistency to obtain the final cross-model test set containing 45, 803 images in total.
comprehensive benchmark for evaluating the
We evaluate the BLIP models on the new dataset on the image caption and prompt inversion tasks. Results are shown in Table 10 This additional text-image dataset, with manually cleaned text prompt, image captions, and VQA annotations, serves divergent and comprehensive benchmark for the evaluation of the visual understanding model for generated images.
20


Reviews:
[{"review_id": "WpKrhQ7pgB", "reviewer": "Reviewer_r9bw", "summary": "", "questions": "", "limitations": "The authors admit themselves that \"we did not check all images to make sure the content is free of offensive or violent content.\" Unfortunately, while I appreciate the authors honesty, this issue might be particularly strong, and admitting this issue might not be enough to release the data. ", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. the scale of the dataset is impressive. \n2. it's interesting to see the creation of such dataset under different styles. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cKSTfYyGuQ", "reviewer": "Reviewer_Wqw6", "summary": "", "questions": "", "limitations": "The work could be improved by\n- comparing real image evaluation with the generated image ones\n", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Strengths include\n- the largest released dataset for generated image / text\n- augmentations of prompts to try and provide more information per image\n- evaluation of the datasets using 4 tasks and multiple models", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3tZSN5F8bp", "reviewer": "Reviewer_J527", "summary": "", "questions": "", "limitations": "The authors have not addressed their work's limitations and potential negative impact. There exists a debate on whether certain opaquely trained generative models constitute copyright infringement, and scraping one prominent example of such algorithms certainly deserves such discussions in the manuscript.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Clearly explained data collection process and extensive benchmark\n- Dataset is comprehensive", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TRyepB4C00", "reviewer": "Reviewer_Q5nG", "summary": "", "questions": "", "limitations": "The authors only mention the risk of having objectionable content in the images, which is an important point when releasing an image dataset.\n\nThough, the main issue lies in the fact that the dataset was scraped, potentially violating the rules set by MidJourney and Discord. This has not been discussed in the paper.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper proposes relevant benchmarks in the context of image generation and image understanding.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Key contribution: 
This paper introduced a large dataset: 4M generated images with prompts scrapped from midjourney discord
an evaluation of the capacity of models to analyze such generated images: prompt inversion, style retrieval, imagecaptioning, visual question answering.  GPT was used to convert prompts into more specific information (content, style, caption, qa) and humans in the loop were utilized to filter the GPT generated prompts. The dataset was used to evaluate image captioning by model construction and test using their dataset and comparison with models trained on other datasets. The capacity of pretrained CLIP to do style retrieval on their dataset along with the capacity of 3 models to do zero shot visual question answering on their dataset was illustrated.
The central claim is that the results obtained on these evaluations show that the understanding of generated images by pretrainedmodels is poor.

Pros: the largest released dataset for generated image / text, human filtered augmentations of prompts to try and provide more information per image, evaluation of the datasets using 4 tasks and multiple models.  

Cons:   A number of concerns were raised by the reviewers including:  the nature of generative AI and the dynamics of generative AI models and what the meaning of this benchmark will be from a scientific perspective. Moreover, the ethical aspects of scraping data from discord were raised as potential issues. 

The authors have addressed a lot of the concerns of the reviewers and the ratings were raised.  With the exception of one of the reviewer who gives a high rating (8), two other reviewers have after the dialogue and revision increased their ratings to 5 and 6. A final reviewer who had ethical concerns communicated after the close of the review process that he also increases the rating to a 5 based on the rebuttal and discussions (this unfortunately is not reflected in the rating since he could not update the review in time and private communication was used to ascertain his final opinion).  The overall average rating thus is a 6 (marginally above acceptance threshold).

Significance and Recommendation:   The extent of discussion based on the revision based on the rebuttal has clarified some of the elements raised including ethics, correctness verification of data, etc.  I find that the size of the dataset and the nature of question raised is interesting as generative AI is a rapidly evolving space and it is important to have benchmarks. However, I have reservations on the impact of this dataset and benchmark. As nicely articulated by one of the reviewers the central question we need to ask is what specifically the benchmark’s purpose is from a scientific point of view.  The efforts that went into curation and preparing such a large scale benchmark is to be lauded, but I am afraid that the impact is short-term. Thus, I find that the paper is marginally above acceptance threshold as a Poster.
Decision: Accept (Poster)