SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiqun T. Chen
Department of Biomedical Data Science
yiqunc@stanford.edu
Departments of Biomedical Data Science;
Electrical Engineering; and Computer Science
Recent progress in generative artificial intelligence (gen-AI) has enabled the generation of photo-realistic and artistically-inspiring photos at a single click, catering to millions of users online. To explore how people use gen-AI models such as DALLE and StableDiffusion, it is critical to understand the themes, contents, and variations present in the AI-generated photos. In this work, we introduce TWIGMA (TWItter Generative-ai images with MetadatA), a comprehensive dataset encompassing over 800,000 gen-AI images collected from Jan 2021 to March 2023 on Twitter, with associated metadata (e.g., tweet text, creation date, number of likes), available at https://zenodo.org/records/8031785. Through a comparative analysis of TWIGMA with natural images and human artwork, we find that gen-AI images possess distinctive characteristics and exhibit, on average, lower variability when compared to their non-gen-AI counterparts. Additionally, we find that the similarity between a gen-AI image and natural images is inversely correlated with the number of likes. Finally, we observe a longitudinal shift in the themes of AI-generated images on Twitter, with users increasingly sharing artistically sophisticated content such as intricate human portraits, whereas their interest in simple subjects such as natural scenes and animals has decreased. Our findings underscore the significance of TWIGMA as a unique data resource for studying AI-generated images.
Recent advancements in text-to-image generation models, such as DALLE [33] and StableDiffusion [37], have revolutionized the creation of realistic and visually captivating images. The ability to generate artistically inspiring images at the click of a button has attracted millions of daily active users. However, the surge in popularity has also given rise to intriguing questions about the boundaries of human and model creativity, as well as the diversity in topics and styles of the generated images. For instance, let's consider the StableDiffusion model: it passes the user-specified text input (known as prompts) into a text encoder CLIP [32] to obtain an encoding that captures the essence of the prompt. Next, the model utilizes a latent diffusion model that takes the text embedding as input and stochastically generates an image that is guided by the text encoding. Therefore, the output image is influenced by both the user's input prompt and the complex diffusion process. Consequently, the stochasticity and black-box nature of the diffusion process led to extensive practice and research in the field of prompt engineering, where content creators iteratively refine their text prompts to generate images that align closely with their desired targets [22, 28]. Similar to other emerging technologies with significant capabilities, generative image models have given rise to a plethora of intricate legal and ethical challenges. These include concerns such as the proliferation of AI-generated NSFW
(Not-Safe-For-Work, which broadly includes offensive, violent, and pornographic topics) images, the dissemination of fake news, and controversies surrounding copyright.
Given the ever-increasing popularity and controversy surrounding AI-generated images, understanding their themes, content, and variations is increasingly crucial. However, existing datasets available for research purposes do not sufficiently address these specific inquiries, as many of them were created for specialized investigations (e.g., fake art detection [44, 51]) and image quality evaluation [40]), resulting in limited content diversity. Some recent exceptions include the Kaggle dataset on Midjourney prompts [3] and DiffusionDB [52], two large-scale datasets compiled from Discord for Midjourney and StableDiffusion models, respectively. However, these pioneering datasets are limited in terms of model variations, user distribution, and relatively short data collection periods.
To address these limitations, we present TWIGMA (Twitter Generative-AI Images with MetadatA) — a large-scale dataset encompassing 800,000 AI-generated images from diverse models. Spanning January 2021 to March 2023, TWIGMA covered an extended timeframe and included valuable metadata, such as inferred image subjects and number of likes. To the best of our knowledge, TWIGMA is the first AI-generated image dataset with a substantial time span and rich metadata, enabling analysis of temporal trends in human-AI-generated image content. Moreover, we leverage unsupervised learning techniques and inferred image captions to understand themes of AI-generated images. This complements earlier research, which primarily concentrated on logged prompts' subjects and content [52, 53]. We demonstrate the value and the type of insights TWIGMA enables by using it to characterize the underlying themes and novel aspects of AI-generated images. In addition to offering opportunities for fine-tuned adaptations to generate images with high popularity on social platforms like Twitter, our dataset and accompanying analyses contribute substantial insights into the ways in which humans engage with generative models, a topic of much interest to a broader scientific framework: as the prevalence of generative content continues to grow, our study could serve as a pilot for future endeavors aimed at comprehending and enhancing the sociological [38] and human-machine interaction [9, 27] dimensions inherent to these data-driven gen-AI models.
Figure 1: Creation process and content overview of TWIGMA. (a): Curation process of the TWIGMA dataset, resulting in approximately 800,000 images posted from 2021 to March 2023. (b): Steady growth in the count of tweets featuring generative AI images over time. (c): Wordcloud showcasing the prevalent keywords extracted from the tweets; popular words include new, AI, art, and prompts. (d): Schematic depiction of the data and metadata incorporated in our analyses.
The rest of the paper is organized as follows. We review image datasets and evaluation metrics for model novelty and variation in Section 2. The data collection and analysis processes are outlined in Section 3. Section 4 presents empirical results addressing our research questions. Finally, Section 5 discusses limitations, safety and ethics concerns, and future research directions.
Table 1: Comparing TWIGMA with other image datasets analyzed in the paper.
Dataset	Source	# Images	# Prompts/Captions	Additional Features
TWIGMA (This paper)	Twitter	800,000	Inferred BLIP captions	Metadata (date, likes, clustering). Images come from multiple gen-AI models.
Jan. 2020–Mar. 2023
DiffusionDB [52]	Stable Diffusion Discord August 2022	18 Million	1.8 unique prompts	Creation date, user hash
ArtBench1201	WikiArt, Ukiyo-e.org 1400–2000	60,000	Painting names	Artist names, creation time, art genre.
LAION-5B [42]	Common Crawl Filtered for image-text similarity	5.85 Billion	Corresponding text	NSFW and watermark detection
AI-generated image dataset: The creation of large-scale image-text datasets has rapidly evolved, transitioning from carefully annotated datasets relying on human labels [16, 21] to vast collections of image-text pairs gathered from the web [32, 37, 47]. As text-to-image models continue to demonstrate unprecedented capabilities in generating images based on user prompts, researchers have started curating similar datasets featuring images generated by these models. For example, DiffusionDB and Midjourney Kaggle provide large-scale datasets (14 million and 250,000, respectively) with prompts and images generated by StableDiffusion and Midjourney, respectively. However, these pioneering, general-purpose datasets are limited in terms of style (as they originate from a single model variation), user distribution (restricted to Discord users of specific channels), and relatively short data collection periods (one month in 2022 for both datasets). Researchers have also constructed datasets of AI-generated images for specialized use, such as detecting generated art images [46, 51], evaluating qualities of specific contents such as human portraits [6, 18], investigating safety filters and hidden vocabularies [26, 34], and evaluating potential biases [24]. Given the problem-oriented nature, these datasets are often limited in size and skewed in themes. Recognizing the gap between existing datasets and our research questions on novelty, themes, and variation of AI-generated images, we curated TWIGMA to demonstrate its potential in this paper.
Novelty and variation of AI-generated images: Many text-to-image models rely on continuous training with a substantial amount of human artistic artifacts sourced from datasets such as LAION [42], which includes images from platforms such as Wikiart and Pinterest, and potentially copyrighted or proprietary contents [31, 34, 44]. Users have also fine-tuned the open-sourced StableDiffusion model on additional samples from specific artists or art genres, such as anime, to generate AI-generated art imitating those styles [5, 8, 23]. While prior research has examined the reproduction of training data in AI-generated art settings [46, 51] and from the perspective of adversarial attack [7, 26], large-scale empirical results comparing AI- and non-AI-generated images are relatively limited. This prompts our first research question (RQ) to investigate the novelty of AI-generated images:
RQ1: How distinct are the distributions of AI-generated images from those of non-AI images?
Furthermore, the stochastic nature of many text-to-image models has sparked inquiries about the variability of their generated image outputs. In a notable court case, a judge determined that AI-generated images do not qualify for copyright protection, citing the significant disparity between the user's intended prompts for Midjourney and the resulting visual material it produced. Recent research has explored the variation within the prompt space [22, 52, 53], and our second RQ builds upon these findings and zooms in a particular dimension of RQ1, the variation in the image space:
RQ2: How do AI-generated image variations compare to non-AI-generated counterparts?
Lastly, we leverage the extended time span offered by TWIGMA to explore our final RQ:
RQ3: How do the content and theme of AI-generated images change over time?
By answering RQ1-3, our paper makes the following contributions to the literature: Firstly, we utilize metrics proposed in generative model evaluation to demonstrate the distinctiveness of AI-generated images compared to real images, providing evidence of the novelty embedded in the underlying models. Moreover, we find that the distance between AI-generated and human images (i) correlates with the number of likes received; and (ii) can be leveraged to identify human images that served as inspiration for AI-generated creations. Secondly, we introduce quantitative measures to establish
that (i) images from generative models exhibit less diversity than real images; and (ii) a substantial portion of variations in the image space can be attributed to the variation in user prompts. Finally, we observed a shift in the preferences for generative models and image topics among Twitter users: they are increasingly sharing artistically sophisticated or distinct content, such as intricate human portraits and anime, while interest in simpler subjects like natural scenes and animals has declined.
Curating the TWIGMA dataset: To curate the TWIGMA dataset, we began with the initial filtering of available tweets using hashtags commonly associated with AI-generated images, such as #dalle, #stablediffusion, and #aiart. We then proceeded to iteratively refine the set of hashtags by incorporating new hashtags that co-occur with the existing ones and contain highly relevant tweets with AI-generated images. We evaluated the quality of these new hashtags based on the first 20 tweets returned by the hashtag search. This iterative process allowed us to create a final set of 19 hashtags, encompassing both generic community descriptions (e.g., #aiart, #generativeart) and specific models (e.g., #midjourney, #dalle2); please refer to Figure 1(a) for the complete list of hashtags.
Next, we utilized the official Twitter API to scrape tweets containing at least one of the identified hashtags. This process encompassed the time frame from January 1, 2021 to March 31, 2023, resulting in approximately 2.2 million tweets. Out of these potentially duplicated tweets, around 1.3 million contained downloadable photos as of April 2023. To ensure data quality, we conducted a two-step dedduplication process. Firstly, we utilized the media ID in Twitter to retain only one image in cases where the same tweet was associated with multiple hashtags. Secondly, we computed CLIP-ViT-L-14 embeddings [32] to remove images with identical embeddings. This dedduplication step resulted in a dataset of 623,707 tweets and 805,650 images in TWIGMA.
Furthermore, we extracted comprehensive metadata for each image in the TWIGMA dataset, including original tweet texts, engagement metrics such as likes, as well as user-related information such as follower counts. Recognizing that tweet texts may not always accurately describe the image content, we also generated captions for each image in TWIGMA using the BLIP model [19]. We release TWIGMA at https://zenodo.org/records/8031785 and provide a tutorial at https://yiqunchen.github.io/TWIGMA/. This dataset includes our curated metadata pertaining to the image creation date, number of likes, assigned cluster membership obtained through k-means clustering, and the inferred BLIP captions. Additionally, we provide a list of Twitter IDs and the necessary code to retrieve images and metadata using the Public Twitter API.
This paper presents a series of analyses to showcase the fascinating insights that TWIGMA can offer into human-AI-generated images. Our approaches represent an initial exploration effort, and numerous other intriguing questions can be addressed using TWIGMA.
Measuring distinctiveness of AI-generated images: To enable efficient computational evaluation without relying on human ratings, we operationalized the concept of "distinct" as the difference between the distribution  $Q$  of a generative model (e.g., images generated by StableDiffusion in TWIGMA) and the distribution  $P$  of real data (e.g., real images in LAION [42]); we then leverage recent advances in quantifying the difference between two continuous distributions. Our analysis incorporated a diverse range of image datasets, including AI-generated images from TWIGMA and DiffusionDB, as well as human images from LAION [42] and ArtBench [20] (see details in Table 1). ArtBench features 60,000 high-quality, genre-annotated images of artwork spanning ten different art genres from the 14th century to the 21st century; and LAION encompasses CLIP-similarity-filtered image-text pairs sourced from the Common Crawl.
To investigate the novelty of AI-generated images, we first employed two-dimensional UMAP [25] in the CLIP embedding space to visualize the differences in distributions between AI-generated and non-AI-generated images. Additionally, we used  $k$ -means clustering (with the efficient implementation in FAISS [13]) and computed Kullback-Leibler (KL) divergence among pairs of image distributions via the quantization approach outlined in Pillutla et al. [30]; our results are robust to the choices of divergence measures and specific estimators thereof.
Measuring the variation of AI-generated images: Quantifying the variation of a continuous distribution is a well-studied topic across many disciplines. In this paper, we primarily drew upon existing literature that assesses diversity in natural images and generated texts: (i) pairwise distance,
computed as  $\sum_{i\neq j}\| x_i - x_j\| _2 / N(N - 1)$ , where  $x_{i},x_{j}$  are embeddings of randomly sampled images; this metric is equivalent to cosine distance when  $x_{i}$  and  $x_{j}$  have unit  $\ell_2$  norms; (ii) inverse of explained variance (IEV) by the largest singular value [54],  $\sum_{i = 1}^{d}\sigma_{d} / \sigma_{1}$ , where  $\sigma_{i}$  denotes the ith-largest singular value of the embedding matrix; (iii) product of marginal variances (PMV) [17],  $\left(\prod_{i = 1}^{d}\hat{s}_{i}\right)^{1 / d}$ , where  $\hat{s}_i$  denotes the sample standard deviation of the ith feature; and (iv) entropy estimated using a quantization approach [35]. We adjusted the original definitions slightly for some measures to ensure that a higher value always indicates greater variability in the data.
Characterizing the themes of AI-generated images: To characterize the themes of AI-generated images posted on Twitter, we first applied  $k$ -means clustering to group images in TWIGMA. Subsequently, we employed the BLIP [19] model to caption the images in TWIGMA, which facilitated the interpretation of the obtained image clusters. To assess the quality of the theme revealed by the BLIP captions, we conducted an additional experiment with four volunteer annotators (two males, and two females; two of them have used generative text-to-image tools). The volunteers were tasked to: (1) annotate randomly-sampled images (10 from each identified cluster) with three relevant keywords; and (2) devise groupings for the images without knowledge of the clustering we had established.
4.1 Quantified novelty and variations of AI-generated images
In this section, we present our analysis of the novelty and variations of AI-generated images. Figure 2 displays different approaches we used to compare AI-generated and non-AI-generated images. The UMAP density plot in Figure 2(a) reveals minimal overlap between sampled LAION, TWIGMA, and ArtBench images, whereas the overlap between TWIGMA and DiffusionDB is substantial. We note a distinct density peak of TWIGMA images absent in DiffusionDB, primarily representing a subset of images generated in specific styles (such as anime) and often of NSFW nature (see Figure 4 for more details). This distinction is further confirmed by estimated  $k$ -means clusters and KL divergence between image distributions, as shown in panels (b) and (c) of Figure 2. Specifically, cluster 1 primarily consists of LAION images, while AI-generated images are present in clusters 2-4. Additionally, the estimated KL divergence between AI-generated and non-AI-generated data pairs tends to be much larger compared to the divergence between DiffusionDB and TWIGMA.
While panels (a)-(c) in Figure 2 focused on examining whether AI-generated and non-AI-generated images are distinct, we additionally investigated whether image distances contain useful information about an image's likability. Figure 2(d) presents the analysis results for TWIGMA data (limited to the subset for which we were able to obtain likes data using Twitter API): On the left side, we present the average similarities of the five nearest neighbors in LAION and ArtBench, determined using cosine similarities, for the images in TWIGMA. On the right side, we plot the average likes for each quintile of similarities, where quintiles 1 and 5 represent the least and most similar images to LAION or ArtBench, respectively. There is some evidence suggesting that AI-generated images that are less similar to their non-AI-generated counterparts receive more likes. For instance, quintile 1 images for LAION and Artbench received, on average, 30.1 (95% CI: [28.8, 31.5]) and 31.9 (95% CI: [29.7, 34.1]) likes, respectively, compared to 23.8 (95% CI: [21.9, 25.7]) and 24.1 (95% CI: [21.5, 26.7]) likes in quintile 5. However, it's important to note that the number of likes for a tweet is influenced by various factors, such as a user's social networks [11, 49]. To account for these factors, we used the following Poisson regression that incorporates the number of followers and similarity quintiles:
$$
\log (\mathbb {E} (\text {L i k e s} | X)) = \alpha + \beta_ {1} 1 (5 0 0 - 5, 0 0 0 \text {f o l l o w e r s}) + \beta_ {2} 1 (> 5, 0 0 0 \text {f o l l o w e r s}) + \sum_ {i = 1} ^ {4} \gamma_ {i} 1 (\text {Q u i n t i l e} i). \tag {1}
$$
The regression model in (1) shows that within the same follower count category (i.e.,  $< 500,500 - 5,000$ , or  $> 5,000$  followers), compared to TWIGMA images most similar to their ArtBench neighbors (quintile 5), images in quintile 1 receive, on average,  $19\%$  more likes (95% CI: [18%, 19%],  $p < 0.0001$ ); similar trends are observed for images in quintile 2 (12% more likes; 95% CI: [11%, 12%];  $p < 0.0001$ ), while the differences are less pronounced for more similar quintiles (0.8% fewer likes in quintile 3 and 2% more likes in quintile 4). Comparable results are found for LAION neighbors as well, where images in quintiles 1 and 2 receive, on average,  $10\%$  (95% CI: [9%, 10%],  $p < 0.0001$ ) and  $2\%$  more (95% CI: [2%, 3%],  $p < 0.0001$ ) likes, respectively, compared to quintile 5 images posted by accounts with the same follower categories. Upon further investigation, we observe
Figure 2: Comparing AI-generated and non-AI-generated images. (a): Kernel density plot of the 2D-UMAP embedding of randomly-sampled images from TWIGMA, LAION, DiffusionDB, and ArtBench. (b): K-means clustering (with  $k = 4$ ) separates AI-generated and non-AI-generated images. (c): Estimated KL divergence of pairs of image distributions using the quantization approach (with 10 clusters) outlined in Section 3. Each entry in the table represents the KL divergence between the row and column distributions, e.g., the 5.42 entry is the KL divergence between LAION and ArtBench, defined as  $\sum_{x} p_{\text{LAION}}(x) \log \left( \frac{p_{\text{LAION}}(x)}{p_{\text{ArtBench}}} \right)$ . (d): (Left) Average cosine similarity of the five nearest neighbors in LAION/ArtBench for TWIGMA images. (Right) Average number of likes per similarity quintile, with error bars representing 95% confidence intervals. On average, TWIGMA images least similar to LAION/ArtBench receive the most likes.
that many of the highly-liked images that stand out from the non-AI-generated images are often anime-style NSFW photos. This observation highlights the need for caution when interpreting the number of likes as an additional indicator of aesthetic and creative value for an image.
Next, we explore image variations across distributions (see Figure 3(a)-(b)). On average, LAION images exhibit the highest variability, followed closely by ArtBench, DiffusionDB, and TWIGMA, which show similar variation distributions based on pairwise Euclidean distance. Notably, ArtBench shows reduced variation when conditioned on image pairs from the same artist. Similarly, conditioning on prompts in DiffusionDB reduces around  $50\%$  of the variation in generated images, measured by the average distance between two image embeddings; that is, the average distance between images in DiffusionDB with the same prompt is roughly half that between randomly selected images with different prompts. Additional variation metrics in Figure 3(b) align with pairwise distance. The lower estimated entropy for ArtBench may be due to its smaller sample size compared to the other datasets.