SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abeba Birhane*
Mozilla Foundation and
School of Computer Science and Statistics
Trinity College Dublin, Ireland
Vinay Prabhu*
Independent Researcher
Sang Han
Vishnu Naresh Boddeti
Computer Science and Engineering Michigan State University
Alexandra Sasha Luccioni
'Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning to scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision-language datasets such as LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common CWEl, which is known to have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. Our results show that hate content increased by nearly  $12\%$  with dataset scale, measured both qualitatively and quantitatively using a metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, we found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with a reflection and a discussion of the significance of our results for dataset curation and usage in the AI community. Code and the meta-data assets curated in this paper are publicly available at https://github.com/vinayprabhu/hate_scaling.
Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.
Generative AI models have come to captivate diverse stakeholders, spanning from researchers [7, 42] to media institutions [28, 31] and even large-scale investment firms [18, 34]. The release of numerous generative models introduced generative AI to the general public. The emergence of the Dall-E 2 [57], a text-to-image vision-language model released in April 2022, which purportedly attracted over a million users within the first three months of its launch and was celebrated with claims like: "the first AI technology that has caught fire with regular people" [28]. Subsequently, models such as Stable Diffusion [58] and Midjourney emerged, followed by proprietary projects from large technology companies
such as Imagen [59], Parti [76], and BASIC [52], access to which was never given to the general public. While Stable Diffusion and its variants have been trained on open-sourced datasets from the LAION family, little is known about the datasets that are used to train models such as OpenAI's Dall-E [57], Google's Parti [76], and Imagen [59]. Fundamental to this multimodal model development is large-scale vision-language datasets containing image-text pairs, which form the main focal point of this paper.
Broadly speaking, these datasets are of two types: those that are open-source, "freely available," and mainly scraped from the Common Crawl (such as LAION-400M [63] and LAION-5B [62]), and those that are closed datasets, curated internally by big corporate labs (such as Google's ALIGN 1.7B/ALIGN 6.6B, JFT-5B [52], and OpenAI's WebImageText-WIT [54]). Although their products increasingly permeate society (and rarely remain within the bounds of corporate labs), information on training datasets remains outside the reach of independent audits and evaluations, where only some of the models trained on such datasets are accessible to the general public via commercial APIs (such as Dall-E 2 or Runway ML). Canonical trend-setting papers introducing state-of-the-art (SOTA) models such as ALIGN [33],Imagen [59] and Parti [76] systematically conceal critical details on training data (See Appendix D for more). These models are also adopted in various commercial tools and applications such as stock photo generation [41], further accelerating their adoption and usage.
The open-source variants of these datasets are getting bigger and now breaching the billion-samples mark for two reasons: firstly, the unquestioned subservience to the scale is all you need mandate handed down from models underlying the field exemplified by works such as [33]. Secondly, there is an increased alliance between dataset creation and venture capital, resulting in the financial backing of dataset creation efforts, which was hitherto missing, i.e., that companies are sponsoring dataset creation efforts. The LAION-5B [62] dataset, for example, was sponsored by Hugging Face, Doodlebot, and Stability.ai, as per their blog post announcement.
The scale is all you need mandate, in turn, emerges from reasoning that treats dataset creation, curation, and management in a lackadaisical manner in published literature accompanying datasets, often resting on vaguely-defined 'dataset scaling laws' (see Appendix C for further elaboration). This is further exacerbated by the vague and often un-reproducible empirical results buried in subsections of some of the canonical papers which (barely) describe the closed datasets used for training models such as ALIGN [33], Imagen [59] and Parti [76] (See Appendix D for more). We also note that these high-profile and trend-setting disseminations are increasingly turning into a flag-posting exercise that involves tactfully concealing critical details on how the dataset was curated, where the data came from, and what it contains (See Appendix E for a deeper exploration).
All of this has resulted in a *scrape-first-ask-later* data creation and curation culture generating gargantuan datasets, in turn eliciting several copyright lawsuits [40], en masse fetishization of women's bodies in an emergent synthetic digital culture [48], outright bans of model outputs from art forums [17], and marquee datasets filled with hundreds of millions of duplicates [75]. These dataset drawbacks and limitations, in turn, result in downstream negative impacts, often against marginalized identities, for example, worsening of negative stereotypes and biases [42, 21, 2, 3], discriminatory and harmful representations [66, 67, 61] and disparate performance and treatment based on gender, race, and other dimensions [64, 4, 25].
Hateful speech, which is commonly understood as speech that attacks individuals or groups based on attributes such as race, religion, sexual orientation, or gender identity [44, 74], has become a growing threat in online media [10, 19, 45, 2]. The United Nations Secretary-General Antonio Guterres recently appealed to the global community to "tackle the hate that spreads like wildfire across the internet" [47]. Owing to this growing threat of hate speech, toxicity classification and hate-speech detection have emerged as important challenges in the machine learning community (See [9, 22, 32, 53, 73] for systematic surveys).
Hateful, abusive, racist, aggressive, and targeted speech are overlapping phenomena where each can be characterized along dimensions such as directed, generalized, explicit, and implicit abuse or hate [74]. Often, the targets of hateful speech are minoritized groups. Based on analysis of generated data to improve hate detection [70] highlights that Black people, women, Muslims, and trans people constitute groups that are most often the targets of hate. Hateful, abusive, and aggressive speech is a systemic problem. For example, [2] studied the outputs generated from GPT-3 when the word "Muslim" is included in the prompt. They found that 66 out of the 100 completions were violent. Findings from studies of hate speech and abusive language detection datasets show systemic bias. For example, [12] examined hate speech and abusive language detection datasets and found systematic racial bias in
all datasets. Consequently, their findings show that classifiers trained on these datasets predict tweets written in African-American English as abusive at a substantially higher rate. Auditing vision-language datasets for hateful content is, thus, a critical first step to ameliorate the downstream effects of hateful content from models trained on such data. The extent of hate speech in a large-scale vision-language dataset, such as the ones we study, has yet to be established.
This paper sheds new light on the presence of hateful content and the impact of scale through a comparative audit of two open-sourced vision-language datasets, LAION-400M and LAION-2B-en. More specifically, we examine the impact of dataset size on the presence of hateful, targeted and aggressive content and what portion of this content is filtered out using existing filtering approaches.
The rest of the paper is organized as follows. Section 2 provides some context to the current AI landscape that treats scale as a solution to many problems and critical scholarship that illustrates shortcomings with such narratives. In Section 3, we detail our audit methodology; in Section 4, we present our findings. Our qualitative and quantitative analysis using HCR show that problematic content increases with dataset size. Our NFSW label analysis also reveals some correlation between 'hateful' and 'targeted' speech and NSFW values. We reflect upon these results and propose several recommendations for a more rigorous, transparent, and equitable dataset creation and curation practice in Section 5. We conclude the paper in Section 6.
2 Scaling Datasets: An Overview
Current thinking around scale can be broadly categorized under two differing approaches: that which views scale as a solution to a multitude of concerns such as model performance, bias, and generalization and that which emphasizes numerous concerns that arise with an unwavering commitment to scale and large-scale dataset scraping. We present both below.
Scale as a solution: Heralded by emblematic large-scale datasets such as ImageNet [14] and C4 [55] over the past decade, the field of machine learning (ML) has come to treat large scale as a mark of success, progress, and improvement as well as a solution. This can be observed in the popularity of the field-wide concept of scaling laws, where large scale is often juxtaposed with better model performance [35]. Following analysis of the top 100 most influential ML papers from the past decade published in two of the most prestigious ML conferences (NeurIPS and ICML) [6], for example, found that "scaling up" is one of the top sought out values in ML research. Similar sentiment about scale also prevails about datasets. For instance, model performance, according to Schumann et al. [63], can be improved simply by making datasets bigger.
Scale is furthermore presented as a shortcut that can circumvent various dataset-related problems. These include the presence of problematic content in datasets, resource-intensive dataset curation, and costly annotation processes, where a larger scale is seen as a substitute for quality data. According to Jia et al., for example, "heavy work on data curation and annotation" can be avoided by scaling up image-text datasets [33]. This "scale beats noise" narrative has tactfully re-framed the thoughtful handheld dataset curation process as a costly burden that can be "solved" by larger scale. Scale, subsequently, is introduced as liberating panacea that not only frees the downstream ML pipeline from the burdens of expensive filtering or post-processing steps but also something that makes up for "noisy" data, as if captioning errors in multimodal datasets of image and alt-text pairs can somehow be "averaged out" through the correct captioning elsewhere in the dataset. Similarly, in the 'data curation' section of FLOrenceDataset900M [77], image-alt-text data scraped from the World Wide Web is presented as 'Diverse' and 'Noisy free-form'. Such lines of thinking are not unique to this specific context but form a widespread belief that drives initiatives such as the LAION datasets and permeates the entire field of multimodal pursuit.
The cost of scale thinking: The primary motivation behind the LAION-400M undertaking was to produce open-source variants of the opaque Web-Image-Text (WIT) dataset, and the CLIP [54] and DALL.E [57] models. Such data open-sourcing initiatives are important first steps towards accountability and building rigorous and equitable AI, given that open access is a crucial prerequisite for independent audit and evaluation. Nonetheless, numerous concerns arise with web-sourced data. Subsequently, continual rigorous audits and assessments of these datasets are imperative for well-functioning, equitable, and healthy open-sourcing practices.
For instance, Science and Technology Studies (STS) scholars and critical data and AI studies have repeatedly emphasized that "scale thinking" stands in stark opposition to values such as societal equity
and effective systemic change [26, 36]. In fact, unwavering commitment to scalability is instrumental to the realization of central objectives driving big technology corporations, such as profit maximization, market monopoly, and the centralization of power in a handful few, all too often at the expense of prioritization of informed consent, justice, and consideration for societal impacts of models [6, 4].
3 Dataset Audit: LAION-400M and LAION-2B-en
The last few years have seen more targeted efforts within the ML community where the value and importance of work on datasets has become apparent. Artifacts such as data sheets proposed to accompany and document ML datasets [23] and the emergence of a robust body of work around dataset auditing, data exploration, and documentation [15, 50, 49, 60] has emphasized the need to consider such work an integral part of model development.
This has been accompanied by scholarly work looking at datasets such as the Common Crawl [43], C4 [16], and LAION-400M [7], which have found a multitude of problematic and harmful content and stressed the importance of data documentation and auditing. With this awareness, there has also been increased attention towards the need to evaluate and audit models trained on these datasets as an important intervention and accountability mechanism [56, 46, 68]. There have also been several efforts to carefully create and curate large-scale datasets to reduce the inclusion of problematic content and emphasize multilingualism (e.g., the Pile [20], the ROOTS corpus [38]) which have since been used and disseminated within the community.
Previous audits of multimodal datasets thus far have investigated the images contained within them, using techniques ranging from image content analysis [65, 7] to image source analysis (by analyzing the URL field), as well cross-modal analyses between images and text (i.e. looking for discordance between an image and its alt-text description). Findings from these analyses indicate that these datasets contain a multitude of issues. For instance, poor data quality is a significant concern. Nearly  $30\%$  (700 million image-text pairs) in the LAION-2B-en dataset, for example, are duplicates [75]. This, as addressed in [65] and [75], can manifest as Digital Forgery, or exact memorization of training examples present multiple times in training data, which was shown to be possible in recent work by Carlini et al. [8] – a phenomenon that has stark ramifications for the field of image generation at large. Other work has shown the presence of explicit images, pornography, malign stereotypes, and other extremely problematic content in the LAION-400M dataset [7].
3.1 Audit methodology
Multimodal datasets can present harm and biases in either modality. The focus of the current work is the alt-text accompanying the images in two LAION datasets. We particularly examine 1) the relationship between the presence of harmful content and scale and 2) whether harmful textual content correlates with the Not Safe For Work (NSFW) image labeling already carried out on the dataset. To this end, we use a SoTA open-source NLP toolkit known as *pysentimiento* [51]<sup>2</sup>.
We use the 'hate speech' analyzer of the framework, which is trained to detect misogyny and racism. In response to an input sentence, it outputs a  $3 \times 1$  vector containing probability scores across the three categories of hateful speech, targeted speech, and aggressive speech. While the 'hateful' score indicates if hate speech is present, 'targeted' indicates whether the target of the text is generic or a specific group of individuals (e.g., women or immigrants), and 'aggressiveness' indicates whether the content also includes aggression or violence.
For example, the following alt-text 'Biden's Spending Will Go To Illegal Immigrants While Tax Hikes Will Destroy American Jobs' results in psentimiento returning [hateful: 0.902, targeted: 0.024, aggressive: 0.449], whereas this one: 'slave punished by angry blond mistress' results in [hateful: 0.967, targeted: 0.891, aggressive: 0.919] (both examples are randomly sampled from alt-text descriptions found in LAION-400M dataset).
To support our initial study, we looked for a state-of-the-art open-source multilingual Python toolkit that supported GPU inference and one that was peer-reviewed, well cited, well documented, actively under development with responsive maintainers (See https://github.com/pysentimiento/pysentimiento/issues/39) and Pysentimiento satisfied all of the above criteria. The project has  $80+$  citations and  $400+$  Github stars. It supports Spanish, Portuguese, and Italian, which constitute approximately 1/6th of the Laion2B-multi dataset that is the next candidate for the type of analysis presented here. We outline the limitation of this tool in Section 5.
3.2 Experiment design
To evaluate the impact of scaling a dataset from 400 million to 2 billion samples on text quality pertaining to hate speech, targeted speech, and aggressive speech, we perform the following audits: we first sub-sample image rows from both datasets and extract the alt-text descriptions associated with the sampled image rows from the TEXT field. In all of our experiments, since we are limited by the amount of computational resources we have access to, we sub-sample 100,000 rows from each of the 160 constituent parquet files spanning the two datasets (the number of files the dataset is separated into). This yields  $N_{\text{samples},400M} = 3.2$  million samples for the LAION-400M dataset and  $N_{\text{samples},2B-en} = 12.8$  million samples for the LAION-2B-en dataset.
The extracted alt-text descriptions are then passed through  $\text{pysentimiento}$  to extract the  $N_{\text{samples}} \times 3$  text-quality score matrices for each of the two datasets with the three columns mapping to probability scores of hateful speech, targeted speech and aggressive speech respectively. We then perform a quality control step where we check if any of the three probability score values associated with an input alt-text description exceeds a specific pre-set threshold score  $P_{\text{threshold}}$ , in which case the input text is deemed to have failed the quality check at that threshold.
We define the metric, Hate Content Rate (HCR)  $3$ :  $\psi_{type}(P_{threshold})$  to be,
$$
\psi_ {t y p e} \left(P _ {\text {t h r e s h o l d}}\right) = 1 0 0 \times \frac {\sum_ {i = 1} ^ {N _ {\text {s a m p l e s}}} \mathbb {1} \left(\tilde {p} _ {\text {t y p e , i}} > P _ {\text {t h r e s h o l d}}\right)}{N _ {\text {s a m p l e s}}} \tag {1}
$$
where  $\mathbb{1}(\cdot)$  is the indicator function,  $\tilde{p}_{type,i}$  is the probability score assigned by the Pysentimiento model for the text associated with the  $i^{th}$  sample and type  $\in \{\text{'hateful', 'targeted', 'aggressive'}\}$ .
This captures the ratio of samples (in percent) that resulted in the *pysentimiento* model assigning the associated hate/targeted/aggressive speech probability score to be greater than  $P_{threshold}$ .
We also introduce the Any-of-the-three metric, which maps to the case where the input text fails the quality test if any of  $\tilde{p}_{\text{hateful}}, \tilde{p}_{\text{aggressive}}$  or  $\tilde{p}_{\text{targeted}}$  happens to be greater than  $P_{\text{threshold}}$ . The associated 'Any-of-the-three'-HCR,  $\bar{\psi}(P_{\text{threshold}})$  would be:
$$
\bar {\psi} \left(P _ {t h r e s h o l d}\right) = 1 0 0 \times \frac {\sum_ {i = 1} ^ {N _ {s a m p l e s}} \mathbb {1} \left(\max  \left\{\tilde {p} _ {h a t e f u l , i} , \tilde {p} _ {t a r g e t e d , i} , \tilde {p} _ {a g g r e s s i v e , i} \right\} > P _ {t h r e s h o l d}\right)}{N _ {s a m p l e s}} \tag {2}
$$
We use both metrics to compare the statistics associated with the text-quality score matrices to understand the nature of the text that was scooped in when the dataset expanded from 400 million samples to 2 billion samples, as well as to examine the relationship between harmful content detected in images and alt-text.
In the forthcoming section, we use both HCR and, more specifically, 'Any-of-the-three'-HCR,  $\bar{\psi} (P_{threshold} = 0.5)$  as the default metric of comparison to characterize the amount of hateful content in both LAION-400M and LAION-2B-en datasets (Section 4.1). We also carry out a file-wise comparison of specific shards of both datasets (Section 4.2) and analyze the correlation between toxic alt-text and NSFW images in Section 4.3.
4.1 Scaling is not benign: comparing LAION 400M and LAION 2B-en
For all the sentiment types (hate, targeted, and aggressive speech) detected by *psentimiento*, the curve corresponding to the 2B-en dataset lies strictly above the curve for the 400M dataset. This signifies that irrespective of what  $P_{threshold}$  is being chosen, the quality failure rate shows that the prevalence of hateful content is higher with the 2B-en dataset than its 400M counterpart. As shown in Figure 1,
3We use the metric Hate Content Rate (HCR) as a shorthand for not just hateful content but all three categories: hateful, targeted, and aggressive.
as the  $P_{threshold}$  is increased, the HCR curves monotonically decrease, indicating that fewer textual samples meet the more stringent constraint placed by a higher  $P_{threshold}$ . Out of the three sentiments – hateful, targeted, and aggressive – hateful content was the most prevalent in both datasets. The 2B dataset showed an HCR of up to 0.7, and the 400M dataset showed an HCR of 0.6, followed by targeted speech, with an HCR up to 0.25/0.2, and finally aggressive speech, with an HCR of 0.04/0.03.
Figure 1: HCR curves for the LAION400M and LAION-2B-en datasets using psentimiento outputs showing that Hate Content Rate increased with dataset size.
Based on the 'Any-of-the-three' curve (leftmost on Figure 1), we can see that the HCR curve for the 2B-en dataset is above the curves of the 400M dataset. Given that both LAION-400M and LAION-2B-en are extracted from the Common Crawl dataset, we hypothesize that during the race to expand the dataset to 5 billion samples, the dataset scraping module might have sampled from the lower-quality sub-parts of the Common Crawl at a rate worse than that during the LAION-400M dataset creation process. We also note that the CLIP-filtering threshold has been relaxed from 0.3 (for LAION-400M) to 0.28 (for LAION-2B-en), which could be another explanatory factor.
To further investigate the relationship between scale and quality, we conduct a binomial proportion confidence interval analysis to establish lower and upper confidence levels of HCR for both datasets at a given reasonable  $P_{threshold}$  of 0.5. For this, we use the Wilson Score interval method with coverage at 0.95 (or  $\alpha = 0.05$ ) that resulted in:
$$
\bar {\psi} (0. 5) \in \left[ \bar {\psi} _ {l b, d a t a s e t} ^ {(\alpha = 0. 0 5)} (0. 5), \bar {\psi} _ {u b, d a t a s e t} ^ {(\alpha = 0. 0 5)} (0. 5) \right]
$$
$$
\bar {\psi} _ {4 0 0 M} (0. 5) = 0. 2 9 8 \in [ 0. 2 9 2, 0. 3 0 4 ] \tag {3}
$$
$$
\bar {\psi} _ {2 B - e n} (0. 5) = 0. 3 4 4 \bar {\in} [ 0. 3 4 1, 0. 3 4 7 ]
$$