SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Keqiang Sun $^{1*}$ , Junting Pan $^{1,3*}$ , Yuying Ge $^{2}$ , Hao Li $^{1}$ , Haodong Duan $^{1}$
Xiaoshi Wu $^{1}$ , Renrui Zhang $^{1}$ , Aojun Zhou $^{1}$ , Zipeng Qin $^{1}$
Yi Wang $^{3}$ , Jifeng Dai $^{3}$ , Yu Qiao $^{3}$ , Limin Wang $^{3}$ ; Hongsheng Li $^{1,3,4\dagger}$
<sup>1</sup>Multimedia Laboratory, The Chinese University of Hong Kong
$^{2}$ The University of Hong Kong  $^{3}$ Shanghai Artificial Intelligence Laboratory
<sup>4</sup>Centre for Perceptual and Interactive Intelligence
While recent advancements in vision-language models have had a transformative impact on multi-modal comprehension, the extent to which these models possess the ability to comprehend generated images remains uncertain. Synthetic images, in comparison to real data, encompass a higher level of diversity in terms of both content and style, thereby presenting significant challenges for the models to fully grasp. In light of this challenge, we introduce a comprehensive dataset, referred to as JourneyDB, that caters to the domain of generative images within the context of multi-modal visual understanding. Our meticulously curated dataset comprises 4 million distinct and high-quality generated images, each paired with the corresponding text prompts that were employed in their creation. Furthermore, we additionally introduce an external subset with results of another 22 text-to-image generative models, which makes JourneyDB a comprehensive benchmark for evaluating the comprehension of generated images. On our dataset, we have devised four benchmarks to assess the performance of generated image comprehension in relation to both content and style interpretation. These benchmarks encompass prompt inversion, style retrieval, image captioning, and visual question answering. Lastly, we evaluate the performance of state-of-the-art multi-modal models when applied to the JourneyDB dataset, providing a comprehensive analysis of their strengths and limitations in comprehending generated content. We anticipate that the proposed dataset and benchmarks will facilitate further research in the field of generative content understanding. The dataset is publicly available at https://journeydb.github.io.
In recent times, notable progress has been achieved in the domain of Artificial Intelligence Generative Content (AIGC), particularly in the advancement of diffusion models [1] that have significantly enhanced the quality of generative content. As a consequence, AIGC platforms such as DALLE, Stability AI, Runway, and Midjourney have gained considerable popularity, enabling users to generate exceptionally high-quality images using text prompts composed in natural language. These text prompts encompass both content and style descriptions provided by users, playing a pivotal role in image generation (see Figure 1 for an illustrative prompt). Unlike descriptions acquired from captioning real images, text prompts for image generation tend to exhibit a high level of detail and specificity, surpassing mere portrayal of salient content. The primary objective behind the creation
†Project Lead
‡ Corresponding Authors
Figure 1: Data Collection Procedure. To collect enough generated images, we investigate the Midjourney channel on Discord to collect the available pictures. Then we employ the GPT-3.5 to annotate the downstream tasks, including 1) separating the prompt into "Style" and "Content", 2) generating the caption according to the content words obtained from task 1, 3) generating "Style-relevant questions" and "Content-relevant questions", providing 4 options for each question, together with the answer. Please refer to Section 3 for more details.
of these prompts lies in visual generation, resulting in intricate descriptions that encompass diverse stylistic facets such as lighting, camera angle, artistic style, medium, and more. Moreover, the generated content originates from the users' imagination, often depicting scenes and compositions that are entirely fictional and devoid of real-world existence.
Considering the aforementioned characteristics, we contend that both the elaborate textual prompts and the generated images themselves serve as valuable sources of information that can be incorporated into existing visual understanding benchmarks. On one hand, the detailed text prompts offer a more comprehensive interpretation of the visual scene, enabling us to perceive the scene and comprehend its underlying style. On the other hand, the abundance of novel object compositions in the generated images provides insights into a realm unrestricted by conventional sense biases, facilitating exploration beyond the constraints of traditional visual representations.
Foundation models have achieved unparalleled capabilities across various visual understanding tasks, owing to large-scale pre-training on datasets, such as CLIP [2], Flamingo [3], and BLIP-2 [4]. However, it is essential to acknowledge that current foundation models are primarily pre-trained on real data, giving rise to concerns regarding their generalization ability and effectiveness in handling the distinctive characteristics associated with generative content. These models may not fully capture the nuanced aspects of generative content and might encounter difficulties in comprehending and generating high-quality images based on complex text prompts.
In view of this challenge, our research initiative seeks to address this gap by curating a dataset comprising a substantial number of 4 million meticulously generated images accompanied by corresponding text prompts. This dataset serves as the fundamental basis for a benchmark consisting of four distinct tasks, which collectively facilitate a comprehensive evaluation of generative content understanding.
The initial task, referred to as prompt inversion, involves identifying the text prompts employed by the user to generate the given images. This task serves to decipher the original prompt or description, assessing the model's ability to comprehend both the content and style of the generated images. The second task involves style retrieval, wherein the model is tasked with identifying and retrieving similar generative images based on their stylistic attributes. This task evaluates the model's proficiency in discerning subtle stylistic nuances within generative images. The third task centres around image captioning, requiring the model to generate descriptive captions that accurately represent the content of the generative image. This task evaluates the model's capability to effectively comprehend and express the visual elements of the generated content using natural language. The fourth and final task is visual question answering (VQA), in which the model is expected to provide accurate answers to questions related to the generative image. This task evaluates the model's ability to comprehend the visual and stylistic content and deliver relevant responses based on the provided questions.
We collected a total of 4,692,751 pairs of image-text prompts, which were subsequently divided into a training set comprising 4,453,193 pairs, a validation set comprising 234,156 pairs, and a test
Table 1: A comparison between JourneyDB and other commonly-used Text-Image multi-modal datasets. Among all the commonly-used multi-modal datasets, the proposed dataset is the most versatile, supporting four downstream tasks. H: Human, M: Models
Dataset	Total Image Num	Label Source	Image Caption	VQA	Prompt Inversion	Style Retrieval
Flickr Caption [9]	32k	H	✓
COCO Caption [10]	164k	H	✓
VQA v2 [11]	204k	H	✓
A-OKVQA [12]	24k	H	✓
LAION-COCO [13]	600M	M	✓
DiffusionDB [14]	14M	M	✓
Ours	4M	H + M	✓	✓	✓	✓
set comprising 5,402 pairs. We also include 45,803 images from 22 other text-to-image models provided by HPD v2 [5], including VQ-Diffusion [6], DALL·E 2 [7], StableDiffusion-XL [8], etc., to build the external set for cross dataset evaluation. Given that the generative model is not flawless, some discrepancies in the text prompts may be present. Consequently, for the test set, we carried out human verification, where annotators were tasked with removing word descriptions that do not align with the corresponding images. To create annotations for tasks 2, 3, and 4, we utilized GPT-3.5 to convert text prompts into annotations specific to each task.
To comprehensively evaluate the performance of current state-of-the-art multi-modal models, we conducted extensive assessments using our benchmark dataset. Furthermore, we performed in-depth analyses to gain insights into the strengths and limitations of these models when applied to generative content. Overall, we observed that the state-of-the-art models do not perform as effectively as they do on real datasets, and fine-tuning on the proposed dataset significantly enhances their performance.
In conclusion, our contribution encompasses three key aspects: 1) To the best of our knowledge, we are the first to draw attention to the visual understanding of generated images. 2) We propose JourneyDB, a large-scale benchmark that serves as both a training and evaluation resource for this emerging field. 3) We conducted an extensive evaluation of state-of-the-art visual understanding models using the proposed dataset, revealing their relatively limited performance on generative content. We hope that our endeavours will contribute to further advancements in the field of generative content understanding.
2.1 Image-Text Datasets
We present a summary of existing image-text datasets in Table 1. The Flickr Caption dataset [9] consists of 32,000 images obtained from the Flickr [15] platform, accompanied by five reference sentences provided by human annotators. The COCO Caption dataset [10] comprises 164 thousand images, with five independent human-generated captions provided for each image for training and validation, resulting in over 1.5 million captions. These datasets play a crucial role in fostering the development of the Image-Caption Task. The Visual Question Answering (VQA) v2.0 [11] dataset, which is the second version of the VQA dataset [16], contains open-ended questions about images that require an understanding of vision, language, and commonsense knowledge to answer. A-OKVQA [12], an augmented successor of OK-VQA [17], encompasses a diverse set of 24 thousand questions that demand a broad foundation of common and world knowledge for accurate responses. These datasets involve human employees in the annotation process, ensuring consistently high-quality annotations. However, manual annotation by human annotators is a time-consuming and costly endeavour, thereby limiting the scalability of the datasets. LAION-COCO [13] is another large-scale dataset containing 600 million image-caption pairs, where GPT3.5 is employed to generate more detailed captions. Although these datasets may contain noise due to the cleaning or generation process using pre-trained neural network models, they have demonstrated their utility in training multi-modal models. However, it is important to note that these datasets primarily focus on real images and cater to a specific task. A comparable dataset to the present study is DiffusionDB [18], a large-scale text-to-image prompt dataset comprising 14 million images generated using Stable Diffusion. However, the image quality from Stable Diffusion is plausible, and no further annotations
are available. In this paper, we collect data from Midjourney and provide annotations generated by GPT3.5 to support four downstream tasks.
2.2 Text-to-Image Generative Models
Text-to-image generative models [19, 20, 14, 7, 21] aim at generating images according to text conditions, apart from traditional generative models [22, 23, 24, 25], which map random noise to images. Text-to-image generative models have experienced rapid development in recent years, empowering users to create image content through natural language specifications. This field has seen significant progress since Mansimov et al.demonstrated that Deep Recurrent Attention Writer (DRAW) can generate images conditioned on text [26, 27]. Since then, several generative architectures and modeling approaches have been applied for text-to-image generation, including autoregressive models [19], GANs [20], and diffusion models [14, 7, 21]. Among these, diffusion models have shown better computational efficiency and the ability to produce higher-quality samples compared to autoregressive models [7]. These diffusion models have reached a level of maturity where they can generate high-quality images suitable for industrial deployment. Notably, Midjourney provides state-of-the-art text-to-image generation service using diffusion models [5]. A vast number of artificial images are generated each day at unprecedented speed. As perception and generation tasks are double sides of the same coin, the achievements in the generative models open new probability for the perception studies. In this context, our dataset aims to organize and consolidate recent progress in text-to-image generative models while laying the foundations for further research in perception studies.
2.3 Multi-modal Foundation Models and Datasets
Aided by data from diverse sources, multi-modal foundation models are capable of understanding and connecting data across multiple modalities, such as image, text, audio and so on. As prioneering vision-language models, CLIP [2] and ALIGN [28] adopt contrastive learning paradigms and are pretrained by millions of web-collected image-text pairs, which showcases promising visual zero-shot capabilities. Flamingo [3] and BLIP-2 [4] further align pre-trained vision backbones with language models with intermediate networks and billions of data pairs, exhibiting superior results on vision-language tasks. OFA [29], Uni-Perceivers [30, 31, 32], and Unified-IO [33] also introduce unified training architectures for different modalities with competitive performance to uni-modal methods. Recently, inspired by the powerful GPT-4 [34], many efforts have been devoted to multi-modal instruction-following models, such as LLaMA-Adapter [35, 36], LLaVA [37] and MiniGPT-4 [38]. Given the textual prompts with image conditions, these models fine-tune a frozen LLaMA [39] to respond to multi-modality instructions, the training data of which is either existing image-caption data [10] or GPT-annotated pairs [37]. Despite the popularity of multi-modal models, it is still rarely explored for their generalization capacity on generated vision-language data, considering the difference between the real-world pre-training data and generative content. In this paper, we propose a large-scale synthetic dataset, JourneyDB, along with customized benchmarks to fully validate the extension efficacy current multi-modal models.
2.4 Training with Generated Data
It is worth noting that the annotations generated by GPT demonstrate a lower level of noise than expected, validating the effectiveness of these models. Notably, LLaVA [37] introduces a novel instruction-tuning dataset that leverages the capabilities of both GPT3.5 and GPT4. Their experiments reveal a remarkable relative score increase of  $295.8\%$ , elevating the score from 21.5 to 85.1, thus emphasizing the effectiveness of their generated data. LaCLIP [40] integrates text augmentations by employing text rewriting techniques with GPT3.5 and Bard. By rewriting the textual descriptions within existing image caption datasets, they achieve a notable improvement of  $36.08\%$ , raising the score from 15.8 to 21.5. StableRep [41] unveils the remarkable potential of using exclusively synthetic data generated from text-to-image models to train highly effective visual representations, surpassing the performance of models trained solely on real image datasets. In a similar vein, VideoChat [42] constructs a dataset by sequentially feeding dense captions to GPT3.5 in temporal order. Despite the inherent challenges in comprehending individual frames with GPT3.5, their successful mastery of understanding the entire video demonstrates the effectiveness of their approach. The generated annotations not only validate the effectiveness of GPT models but also significantly contribute to
Table 2: Statistics of JourneyDB. We provide 4 million generated image-prompt pairs, 1 million captions and over 8 million VQA annotations.
Dataset	Image	Prompt	Labeled Image	Labeled Prompt	Style QA	Content QA
Training Set	4,453,193	1,643,375	4,189,737	1,385,317	7,056,394	8,775,971
Validation Set	234,156	82,093	234,156	82,093	311,569	374,310
Testing Set	5,402	5,171	5,402	5,171	10,040	11,369
External Set	45,803	45,365	45,803	45,365	74,407	81,565
Total	4,738,554	1,776,004	4,475,098	1,517,946	7,452,410	9,243,215
advancing the understanding of images. Therefore, based on our demonstrated results, we firmly believe that our JourneyDB can serve as a valuable tool to enhance numerous image-related tasks.
In this section, we present the methodology employed for dataset collection and annotation, along with relevant statistical insights to gain a deeper understanding of the dataset.
The data collection procedure is presented in Figure 1. In order to obtain a sufficient number of generated images, we investigated the Midjourney channel [43] on the Discord platform [44] to access the available pictures. Within the public Discord channel named "Midjourney," users submit text prompts to the channel, and the Midjourney bot responds with the corresponding generated images. Users then select the preferred image for upscaling, and Midjourney provides the corresponding upscaled images. The chat history contains numerous publicly accessible prompt-image pairs. To collect the data, we utilized DiscordChatExporter [45], a widely used Discord crawler, to download the publicly available images along with their corresponding prompts. In this version of the dataset, we only retained images that were generated solely based on text prompts, filtering out any images conditioned on given images. Additionally, we removed Midjourney-specific arguments, such as "-v 4", to enhance the generalizability of the prompts and ensure their comprehensibility for existing large language models.
Moreover, to improve the diversity of JourneyDB, we additionally introduce another 22 text-to-image generative models into JourneyDB, such as VQ-Diffusion [6], DALL-E 2 [7], StableDiffusion-XL [8], etc., which makes our data a comprehensive benchmark for evaluating the comprehension of generated images. For each generative model, we originally generated 3,200 images, and a group of 60 annotators helped clean up the pairs without consistency to obtain the final cross-model test set containing 45,803 images in total. Please find more details of this part in the appendix D.
We provide ample annotations for multiple visual understanding tasks. The dataset is compared with existing methods in Table 1, demonstrating its versatility in supporting four downstream tasks.
Annotation for Visual Understanding. In this section, GPT-3.5 is employed to annotate the downstream tasks. Specifically, a set of Midjourney prompts and explicit instructions are provided to GPT-3.5. The objectives are as follows: 1) segmenting the prompt into "Style", "Content", "Atmosphere", and "Others", 2) generating captions based on the content words identified in task 1, 3) generating "Style-relevant questions" and "Content-relevant questions," accompanied by four answer choices for each question. The detailed instructions provided to GPT-3.5 can be found in the Supplementary Materials.
Clustering of Styles. Numerous prompts related to style are highly intricate for style retrieval. Taking inspiration from existing prompt engineering platforms<sup>4</sup>, we propose a hierarchical clustering
<sup>4</sup>https://www.mbprompt.com/
Figure 2: Distribution and samples of the style prompts.
approach for organizing styles, which simplifies style retrieval and facilitates user reference. Since traditional word embedding and clustering methods struggle to handle sophisticated style words, we leverage GPT-3.5 for this task. Specifically, we divide the prompts into smaller patches, each comprising 200 prompts, and instruct GPT-3.5 to cluster the words. Subsequently, we manually merge the categories from different patches to obtain the final "style tree". The distribution of the clustered style space is visualized in Figure 2.
Filtering for Image-Prompt Consistency. Due to the limitations of Text-to-Image generative models, inconsistencies may arise between the prompt and the generated image. To ensure the quality of the test set, we engaged 40 annotators to identify inconsistent prompt words in the test set. Specifically, given a pair of text prompts and the corresponding generated image, the annotators are instructed to verify if each word is depicted in the image. Words annotated as "Not Appear" are removed to obtain the clean prompts.
General Statistics In this iteration, a total of 4,692,751 images were collected, all with a resolution exceeding  $1024 \times 1024$ , accompanied by corresponding text prompts. Among them, 1,730,639 prompts were found to be independent. Furthermore, 1,472,581 instances were annotated using the GPT-3.5 model, following the procedure outlined in Figure 1. Additionally, 5,402 images were filtered out due to inconsistencies between the images and prompts, as determined by the Image-Prompt consistency check. Moreover, 45,803 images from 22 other text-to-image models provided by HPD v2 [5] are introduced to build the external set for cross dataset evaluation. In addition, a clustering process was conducted to summarize the 70,521 fine-grained styles into 334 style categories, displaying a long-tail distribution pattern, as illustrated in Figure 2.
Dataset Split Detailed statistics for each subset of the dataset are provided in Table 2. The entire dataset was randomly divided, with approximately a  $20:1$  ratio, to create the training and validation sets. The training set comprises 4,189,737 images and 1,385,317 prompts, while the validation set consists of 234,156 images and 82,093 prompts. Additionally, a separate testing set was sampled for manual filtering, consisting of 5,402 images and 5,171 prompts.
4.1 Prompt Inversion
The prompt, which determines both the content and style of a generated image, contains crucial and comprehensive information regarding the image. When presented with an appealing generated image, individuals are eager to discern the prompt employed for its creation. By accurately identifying the prompts, they can further enhance the corresponding image, such as modifying its content or generating images with a similar style.
Table 3: Evaluation results of Prompt Inversion on JourneyDB. We list results on the validation set in the upper half, results on the test set in the lower. For all metrics, the higher, the better.
Models	Validation	Test
BLEU-4	METEOR	ROUGE-L	CIDEr	Similarity	BLEU-4	METEOR	ROUGE-L	CIDEr	Similarity
BLIP-2 OPT [4]	0.18	2.39	6.75	5.42	0.36	0.29	2.85	7.06	6.46	0.36
BLIP-2 FlanT5 [4]	0.27	2.46	7.19	6.88	0.38	0.40	2.95	7.69	8.86	0.37
MiniGPT-4 [38]	1.49	5.50	12.51	10.39	0.43	1.71	6.51	13.13	11.40	0.43
Uni-Perceiver v2 [30]	0.23	2.44	9.11	12.38	0.33	0.37	2.73	9.88	15.45	0.34
Uni-Perceiver v2FT [30]	20.6	16.9	29.1	123.2	0.59	4.68	8.56	16.98	34.01	0.51
Table 4: Evaluation results of Image Captioning on JourneyDB. We list the zero-shot results in the upper half, and the fine-tuned results in the lower. For all metrics, the higher, the better. FT denotes "Fine-Tune".
Models	Validation	Test	COCO Caption
BLEU-4	METEOR	ROUGE-L	CIDEr	BLEU-4	METEOR	ROUGE-L	CIDEr	CIDEr
BLIP-2 OPT [4]	0.82	5.43	19.87	22.00	2.35	7.88	22.40	37.60	145.8 (FT)
BLIP-2 FlanT5 [4]	0.54	5.02	19.94	22.18	2.07	7.62	23.12	39.62	144.5 (FT)
Flamingo9B [3]	0.94	6.58	14.19	10.19	1.39	6.84	17.75	19.10	79.4 (ZS)
MiniGPT-4 [38]	2.28	7.39	19.24	16.78	2.79	9.84	20.31	22.34	-
Uni-Perceiver v2 [30]	0.41	4.50	18.72	21.88	0.94	5.21	16.71	20.13	122.5 (FT)
Uni-Perceiver v2FT [30]	8.20	12.53	27.09	50.72	3.23	10.12	22.45	31.76	-
However, predicting the prompts of an image is a challenging task. Existing visual understanding models, such as image-caption models, often fall short in providing a detailed description of the image's main elements, such as the subject, while neglecting other indispensable details like the viewpoint, illumination, or art style.