Into the LAION’s Den: Investigating Hate in Multimodal Datasets
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: multimodal datasets
dataset audit
hate speech detection
toxic content detection

Abstract: `Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning to scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision-language datasets such as LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common Crawl, which is known to have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. Our results show that hate content increased by nearly **12%** with dataset scale, measured both qualitatively and quantitatively using a metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, we found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with a reflection and a discussion of the significance of our results for dataset curation and usage in the AI community.
Code and the meta-data assets curated in this paper are publicly available at https://github.com/vinayprabhu/hate_scaling.
 Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.

Into the LAION's Den: Investigating Hate in Multimodal Datasets
Abeba Birhane"
Vinay Prabhu Independent Researcher
Mozilla Foundation and School of Computer Science and Statistics Trinity College Dublin, Ireland
Sang Han Independent Researcher
Vishnu Naresh Boddeti
Alexandra Sasha Luccioni Hugging Face Montreal, Canada
Computer Science and Engineering Michigan State University
Abstract
'Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision- language datasets such LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common Crawl, which is known have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-444 -400M and LAION-2B. Our results show that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively using metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with reflection and discussion of the significance of results for dataset curation and usage in the Al community. Code and the meta-data assets curated in this paper are publicly available at wtlhttps//ggitie thubbbb..b..... com/ v www.yypraahuua//a........aaaaaaa haatuu//tte..e/ http/haaatalanngggaeaaaaaaaaaaal Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.
Introduction
Generative Al models have come captivate diverse stakeholders, spanning from researchers 42
to media institutions [2 31 and even large-scale investment firms [18 34 The release of numerous
generative models introduced generative AI the general public. The emergence the Dall-E [57]. a text-to-image vision- language model released in April 2022, which purportedly attracted over million users within the first three months of its launch and was celebrated with claims like: "the first Al (28). Subsequently. models such as Stable Diffu- sion 588 and Midjourney emerged, followed by proprietary projects from large technology companies
technology that has caught fire with regular people"
"Equal contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

such as Imagen 1591 Parti 176,, and BASIC [52,, access to which was never given to the general public. While Stable Diffusion and its variants have been trained on open- sourced datasets from the LAION family, little is known about the datasets that are used to train models such as OpenAl's s -all-E [57]. Google's 's Parti [76,, and Imagen [59]. Fundamental to this multimodal model development is large-scale vision-language datasets containing image-text pairs, which form the main focal point of this paper.
Broadly speaking, these datasets are of two types: those that are open-source, "freely available." and mainly scraped from the Common Crawl (such as LAION-400M [63] and LAION-5B (2]). and those that are closed datasets, curated internally by big corporate labs (such as Google's ALIGN 1.7B/ALIGN 6.6B, JFT-5B [52], and OpenAl's AI's Weblmage TeT [54)) ). Although their products increasingly permeate society (and rarely remain within the bounds of corporate labs), information on training datasets remains outside the reach of independent audits and evaluations, where only some of the models trained on such datasets are accessible to the general public via commercial APIs (such as Dall-E 2 or Runway ML). Canonical trend-setting papers introducing state-of- tfae-offthhe-art (SOTA) models such as ALIGN 133. Imagen [59] and Parti [76 systematically conceal critical details on training data (See Appendix D for more). These models are also adopted in various commercial tools and applications such as stock photo generation [41, further accelerating their adoption and usage.
The open-source variants of these datasets are getting bigger and now breaching the billion-samples mark for two reasons: firstly, the unquestioned subservience to the scale is all you need mandate handed down from models underlying the field exemplified by works such as [33]. Secondly, there is an increased alliance between dataset creation and venture capital, resulting in the financial backing of dataset creation efforts, which was hitherto missing, i.e., that companies are sponsoring dataset creation efforts. The LAION-5B [62] dataset, for example, was sponsored by Hugging Face, Doodlebot, and aii, as per their blog post announcement
The scale is all you need mandate, in turn, emerges from reasoning that treats dataset creation, curation, and management in a lackadaisical manner in published literature accompanying datasets, often resting on vaguely-defined 'dataset scaling laws' (see Appendix [ further elaboration). This is further exacerbated by the vague and often un-reprodueible empirical results buried in subsections of some of the canonical papers which (barely) describe the closed datasets used for training models such as ALIGN [33. Imagen [59] and Parti [76] (See Appendix D for more). We also note that these high-profile and trend-setting disseminations are increasingly turning into a flag-posting exercise that involves tactfully concealing critical details on how the dataset was curated, where the data came from, and what contains (See Appendix for a deeper exploration).
All of this has resulted in a scrape-first-ask-lutee gargantuan datasets, in turn eliciting several copyright lawsuits (40), en masse fetishization of women's is bodies in an emergent synthetic digital culture [48), outright bans of model outputs from art forums [17, and marquee datasets filled with hundreds of millions of duplicates 775.. These dataset drawbacks and limitations. in turn, result in downstream negative impacts, often against marginalized identities, for example, worsening of negative stereotypes and biases 442 21 3. discriminatory
data creation and curation culture generating
and harmful representations (66, 67 61 and disparate performance and treatment based on gender,
race, and other dimensions 66
Hateful speech, which is commonly understood as speech that attacks individuals or groups based on attributes such as race, religion, sexual orientation, or gender identity 774,, has become growing threat in online media 19 45|2 2). The United Nations Secretary- -eneral Antonio Guterres recently appealed to the global community to "tackle the hate that spreads like wildfire across the internet" [4]. Owing to this growing threat of hate speech, toxicity classification and hate-speech detection
have emerged as important challenges in the machine learning community (See 22 32 53 73
for systematic surveys).
Hateful, abusive, racist, aggressive, and targeted speech are overlapping phenomena where each can be characterized along dimensions such as directed, generalized, explicit, and implicit abuse or hate 774. Often, the targets of hateful speech are minoritized groups. Based on analysis of generated data to improve hate detection 770 highlights that Black people, women, Muslims, and trans people constitute groups that are most often the targets of hate. Hateful, abusive, and aggressive speech is systemic problem. For example, (2) studied the outputs generated from GPT-3 when the word "Muslim" is included in the prompt. They found that 66 out of the 100 completions were violent. Findings from studies of hate speech and abusive language detection datasets show systemic bias. For example, [12] examined hate speech and abusive language detection datasets and found systematic racial bias in

---

all datasets. Consequently, their findings show that classifiers trained on these datasets predict tweets written in datasets for hateful content is, thus, critical first step to ameliorate the downstream effects of hateful content from models trained on such data. The extent of hate speech in a large-scale vision-langue -laggugg dataset, such as the ones we study, has yet to be established.
English as abusive at substantially higher rate. Auditing lasion-language
This paper sheds new light on the presence of hateful content and the impact of scale through a comparative audit of two open-sourced vision-language datasets, LAION-400M and LAION-2B-en. More specifically, we examine the impact of dataset size on the presence of hateful, targeted and aggressive content and what portion of this content is filtered out using existing filtering approaches.
The rest of the paper is organized follows. Section 2 provides some context to the current Al landscape that treats scale as a solution to many problems and critical scholarship that illustrates shortcomings with such narratives. In Section 3] we detail our audit methodology: in Section 4 we present our findings. Our qualitative and quantitative analysis using HCR show that problematic content increases with dataset size. Our NFSW label analysis also reveals some correlation between 'hateful' and 'targeted' speech and NSFW values. We reflect upon these results and propose several recommendations for more rigorous, transparent, and equitable dataset creation and curation practice in Section 5
We conclude the paper in Section
Scaling Datasets: An Overview
Current thinking around scale can be broadly categorized under two differing approaches: that which views scale as a solution to a multitude of concerns such as model performance, bias, and generalization and that which emphasizes numerous concerns that arise with an unwavering commitment to scale and large-scale dataset scraping. We present both below.
Scale as solution: Heralded by emblematic large-scale datasets such as ImageNet [44 and C4 (55 over the past decade, the field of machine learning (ML) has come to treat large scale as a mark of success, progress, and improvement as well as a solution. This can be observed in the popularity of the field-wide concept of scaling laws, where large scale is often juxtaposed with better model performance (35). Following analysis of the top 100 most influential ML papers from the past decade published in two of the most prestigious ML conferences (NcurIPS and ICML) (6. for example, found that "Scaling up" is one of the top sought out values in ML research. Similar sentiment about scale also prevails about datasets. For instance, model performance, according to Schumann et al. [63]
can be improved simply by making datasets bigger.
Scale is furthermore presented as a shorteut that can circumvent various dataset-related problems. These include the presence of problematic content in datasets, resource-intensive dataset curation, and costly annotation processes, where a larger scale is seen as a substitute for quality data, According to Jia et al., for example, "heavy work on data curation and annotation" can be avoided by scaling up image-text datasets (33. This "scale beats noise" narrative has tactfully re- re-framed the thoughtful handheld dataset curation process costly burden that can be "solved" by larger scale. Scale, subsequently, is intro- duced as liberating panacea that not only frees the downstream ML Mipeliiee from the burdens of expen- sive filtering or post-processing steps but also something that makes up for "noisy" data, as if captioning errors in multimodal datasets of image and alt-text pairs can somehow be "averaged out" through the correct captioning elsewhere in the dataset. Similarly, in the 'data curation' section of DatsenceDataset- 900M 77, image-alt-texx data scraped from the World Wide Web is presented as 'Diverse' and 'Noisy free-form', Such lines of thinking are not unique to this specific context but form widespread belief that drives initiatives such as the LAION datasets and permeates the entire field of multimodal pursuit.
The cost of scale thinking: The primary motivation behind the LAION-400M undertaking was to produce open- variants of the opaque Web-Image-Texx (WIT) dataset, and the CLIP [54] and DALL.E [57] models. Such data open-sourcing initiatives accountability and building rigorous and equitable AI, given that open access is crucial prerequisite for independent audit and evaluation. Nonetheless, numerous concerns arise with web-soureed data. Subsequently, continual rigorous audits and assessments of these datasets are imperative for well-functioning. equitable, and healthy open-sourcing practices.
important first steps towards
For instance, Science and Technology Studies (STS) scholars and critical data and AI studies have repeatedly emphasized that "scale thinking" stands in stark opposition to values such as societal equity
3

---

and effective systemic change [26 36). In fact, unwavering commitment to scalability is instrumental to the realization of central objectives driving big technology corporations, such as profit maximization. market monopoly, and the centralization of power in a handful few, all too often at the expense of prioritization of informed consent, justice, and consideration for societal impacts of models 66 4
3 Dataset Audit: LAION-400M and LAION-2B-en
The last few years have seen more targeted efforts within the ML community where the value and importance of work datasets has become apparent. Artifacts such as data sheets proposed to accompany and document ML datasets [33 and the emergence of a robust body of work around dataset
auditing, data exploration, and documentation [15] 50 49 60] has emphasized the need to consider such work an integral part of model development.
This has been accompanied by scholarly work looking at datasets such as the Common Crawl 4 C4 [T6], and LA0ON-400M (7). which have found multitude of problematic and harmful content and stressed the importance of data documentation and auditing. With this awareness, there has also been increased attention towards the need to evaluate and audit models trained on these datasets as an important intervention and accountability mechanism [56] 46 68). There have also been several efforts to carefully create and curate large-scale datasets to reduce the inclusion of problematic content and emphasize multilingualism (e.g., the Pile [20), the ROOTS corpus (38) which have since been
used and disseminated within the community.
Previous audits of multimodal datasets thus far have investigated the images contained within them, using techniques ranging from image content analysis 65 to image source analysis (by analyzing the URL field), as well cross-modal analyses between images and text (i.e. looking for discordance between an image and its alt-text description). Findings from these analyses indicate that these datasets contain multitude of issues. For instance, poor data quality is significant concern. Nearly 30% (700 million image-text pairs) in the LAION-2B-en dataset, for example, are duplicates (75). This, as addressed in [65 and r5 can manifest Digital Forgery, exact memorization of training examples present multiple times in training data, which was shown to be possible in recent work by Carlini et al. [ a phenomenon that has stark ramifications for the field of image generation at large. Other work has shown the presence of explicit images, pornography, malign stereotypes, and other extremely problematic content in the LAION-400M dataset (7).
3.1 Audit methodology
Multimodal datasets can present harm and biases in either modality. The focus of the current work the alt-text accompanying the images in two LAION datasets. We particularly examine 1) the relationship between the presence of harmful content and scale and 2) whether harmful textual content correlates with the Not Safe For Work (NSFW) image labeling already carried out on the dataset. To this end, we use a SoTA open-source NL. toolkit known as pysentimiento [51
We use the 'hate speech' analyzer of the framework, which trained detect misogyny and racism. In response to an input sentence, it outputs a 3 x 1 vector containing probability scores across the three categories of hateful speech, targeted speech, and aggressive speech. While the 'hateful' score indicates if hate speech is present, 'targeted' indicates whether the target of the text is generic or specific group of individuals (e.g., women or immigrants). and 'aggressiveness' indicates whether the content also includes aggression or violence.
For example, the following alt-text 'Biden's 's Spending Will Go To Illegal Immigrants While Tax Hikes Will Destroy American Jobs' results in pysentimiento returning [hateful: 0.902, targeted:
0. 020, aggressive: 0. 0494,, whereas this one: 'slave punished by angry blond mistress'
results in [hateful: 0. 0.9 targeted: 0. 0.891, aggressive: 0.919] (both examples are
randomly sampled from alt-text descriptions found in LAION-400M dataset).
"To support our initial study, we looked for state-of-the-ar open-source multilingual Python toolkit that supported GPU inference and one that was peer-reviewed, well cited, well documented, actively under development with responsive maintainers (See httehtttp//gglle thgbbbb..ttubtt www.com/////p. cwmppysentiniiiiiic pysentimientoo www.ttimnnnnttnttttt.... wyppptttsiimeeeee /ttett/mmiitt//ssee/////3 39) and Pysentimiento satistied all of the above criteria. The project has 80+ citations and 400+ Github stars. It supports Spanish, Portuguese, and Italian, which constitute approximately 1/6th of the Laion2B- on2o-multtt ti dataset that is the next candidate for the type of analysis presented here. We outline the limitation of this tool in Section

---

3.2 Experiment design
To evaluate the impact of scaling dataset from 400 million to 2 billion samples on text quality pertaining to hate speech, targeted speech, and aggressive speech, we perform the following audits: we first sub-sample image rows from both datasets and extract the alt-text descriptions associated with the sampled image rows from the TEXT field. In all of our experiments, since we are limited by the amount of computational resources we have access to, we sub-sample 100,000 rows from each of the 160 constituent parquet files spanning the two datasets (the number of files the dataset separated into). This yields samples, 400M 3.2 million samples for the LAION-400M dataset and N samples 2B en 12.8 million samples for the LAION-2B-en dataset.
The extracted alt-text descriptions are then passed through pysentimiento to extract the N samplee text-quality score matrices for each of the two datasets with the three columns mapping to probability scores of hateful speech, targeted speech and aggressive speech respectively. We then perform quality control step where we check if any of the three probability score values associated with an input alt-text description exceeds specific pre-set threshold score Pthreshold: which case the input text is deemed to have failed the quality check at that threshold.
We define the metric, Hate Content Rate (HCR) Ptype (I (thrreshodd) to be,
Naumplex
1 (pegpe > Pthreshhoaa
Vtypel Pthreshhldd 100
N
Nsamples
where 1() is the indicator function, Ptype. is the probability score assigned by the Pysentimiento model for the text associated with the 1thh sample and type { ('hateful' /targeted'" 'aggressive' }.
This captures the ratio of samples (in percent) that resulted in the pysentimiento model assigning the associated nate/targeted/aggresstt speech probability score to be greater than Pihreshold-
We also introduce the any-of-the-thre metric, which maps to the case where the input text fails the quality test if any of Phatefals Paggressive or Ptargeted happens to be greater than Pahreshold- The associated 'Any-of Sche-three! -HCR, v6 (Pshreshota) would be:
N Nammpie ples
1 (mmax(Phote wl. Pooggeted.sPoggresssii Pihreshola)
/ P Pthreshold) = 100x X
N sammpess
We use both metrics to compare the statistics associated with the text-quality score matrices to understand the nature of the text that was scooped in when the dataset expanded from 400 million samples to billion samples, as well as to examine the relationship between harmful content detected in images and alt-text.
4 Results
In the forthcoming section, use both HCR and, more specifically. "Any-of-the-three--HCC (PPtsreehotu=:.3 0.5) as the default metric of comparison to characterize the amount of hateful content in both LAION-400M and LAION-2B-oa en datasets (Section 4.1p. We also carry out file-wise comparison of specific shards of both datasets (Section 4.2) and analyze the correlation between toxic alt-text and NSFW images in Section 4.3
4.1 Scaling is not benign: comparing LAION 400M and LAION 2B-en
For all the sentiment types (hate, targeted, and aggressive speech) detected by pysentimiento, the curve corresponding to the 2B-en dataset lies strictly above the curve for the 400M dataset. This signifies that irrespective of what Pthreshold is being chosen, the quality failure rate shows that the prevalence of hateful content is higher with the 2B-en dataset than its 400M counterpart. As shown in Figure []]
We use the metric Hate Content Rate (HCR) as shorthand for not just hateful content but all three categories: hateful, targeted, and aggressive.
5

---

as the Pibreshold is increased, the HCR curves monotonically decrease, indicating that fewer textual samples meet the more stringent constraint placed by a higher Pihreshold- Out of the three sentiments hateful, targeted, and aggressive hateful content was the most prevalent in both datasets. The 2B dataset showed an HCR of up to 0.7, and the 400M dataset showed an HCR 0.6, followed by targeted speech, with an HCR up to 0.25/0.2, and finally aggressive speech, with an HCR of 0.04/0.03.
Any-of-thh-3
hateful
largeted
aggressive
0.6
2B
400M
0.6
400M
0.2
28
0.04
2B
0.4
0.4
0.1
0.02
0.2
0.4
0.9
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Phvvstold
Pirecsaal
Figure 1: HCR curves for the LAION400M and LAION-2B-or datasets using pysentimiento outputs showing that Hate Content Rate increased with dataset size.
Based on the Any-of-the-thre curve (leftmost on Figure 1). the 2B-en dataset is above the curves of the 400M dataset. Given that both LAION-400M and LAION-2B-crt
see that the HCR curve for
extracted from the Common Crawl dataset, we hypothesize that during the race to expand the dataset to 5 billion samples, the dataset scraping module might have sampled from the lower-quality sub-parts of the Common Crawl at worse than that during the LAIÓN-400M dataset creation process. We also note that the CLIP-filtering threshold has been relaxed from 0.3 (for MATON-400M 0.28 (for LAION-2B-en), which could be another explanatory factor.
To further investigate the relationship between scale and quality, we conduct a binomial proportion confidence interval analysis to establish lower and upper confidence levels of HCR for both datasets at a given reasonable Pihreshold of 0.5. For this, we use the Wilson Score interval method with coverage at 0.95 (or a = 0.05) that resulted in:
(0.5) E ((==000)
Ib, datoser (0.5), ( 7(a=0.05)
dataset (0.5)
1100 (0.5) = 0 0.898 E (9222222.00))
2 en (0.5) 0.344 E (0.341,0.347]
where ((a=0.05)
datasst and / ub.dataset are the lower-bound and the upper-bound values of the confidence interval at a 0.05.
(la=0)05)
As seen above in equation the lower-bound HCR for the 2B-en dataset is markedly higher than the upper-bound estimate of HCR for the 400M dataset thus leading to change-of-HCR, òer = ((am0.06) 06)
1b.2B -n (0.5) - fff==0000 055
40n (0.5)
100 of 12.26%. Simply put, even under an optimistic setting where
(10-6.006 (0.5)
we compute the difference between the lower-bound estimate of HCR for the 2B-en dataset and the upper-bound estimate of HCR for the 400M dataset, we still see a 12.26% normalized increase in HCR.
We have so far established the risks of extending the LAION-400M dataset quality statistics to its bigger counterpart, that is, LAION-2B-en. This, as we hypothesize, may be a consequence of rich non-iid inter-sample correlations emerging from a graph structurd prior for CommonCrawi This begs the question, given that our dataset was uniformly sampled at the shard/file level, whether the HCR statistics computed at the shard/file level can meaningfully compare with the global dataset-level HCR statistic. We examine this in the sub-section below.
4.2 Intra-dataset filewise comparisons
Given that the two datasets, LAION-4000 4400 and LAION-2B-cn are split into, respectively, 32 and 128 purportedly uniformly sampled shards, we also examine the validity of the file-level HCR metrics to the global dataset- metrics. To this end, we use the three 100,000-sizee file-level text-quality score matrices obtained from the pysentimiento model and compute what fraction of these rows larger than Pihreshotd of 0.5 for all the three columns. This yields file-level HCRs (in %) for each dataset, with the three columns mapping hateful speech, targeted speech, and aggressive speech.
We found that the file-wise HCRs were tightly clustered around the mean levels for the individual datascts. Figure 2 shows the fused swarm-box-violin plot that captures the file-wise HCR metrics

---

0.40
400M 2B-En
0.35
hateful
targeted
aggressive
0.30
HCR (fllewise)
T
14.48
13.80
4.44
0.25
Degrees of freedom
53.32
54.91
47.96
0.15
P-value
2.02e-20
8,44c-20
2.60e-05
Cohen-d BF-10
2.64
2.47
0.87
3.785c+27
5.957c+25
2131.144
0.05
0.00
P_hateful
P.targeted
aggressive
Figure 2: Fused swarm-box -o----oo-vvoioooo that cap- tures the file-wise HCR metrics for all the 160 (=32+128) parquet files from LAION400M and LAION-2B-ca HCRs for LAION-2B-BR (the red swarms) are higher than the 32 file-level HCRs for the LAION400M (the blue swarms) for all three sub-categories hateful, targeted, and aggressive speech.
Table 1: Results from the two-sample t-test while cor-
recting for unequal variances (using the Welch separate variances T-test). 'BF10' indicates the Bayes Factor the alternative hypothesis. For all three categories of hateful, targeted, and aggressive speech, the file-wise HCR asso- ciated with the 2B-en dataset is higher than the file-wise HCR for the 400M dataset, showing dataset degradation with dataset scaling.
for all the 160 (=32+128) parquet files spanning the two datasets. For example, the 'hateful' related HCR for LAION-400M has mean value of 0.298, which increased to 0.344 for LAION-2B-en. All the 32 constituent file-with fiee HCRs for this dataset fall within 0.26 and 0.33.
Furthermore, 97% of all the files have their HCRs within standard deviations of the mean -CRR for the dataset. Similarly, the mean-HCR for the entire LAION-2B-en dataset is 0.344, and the range across all the 128 files is (0.297.0.382) which indicates that 95% of all the constituent files have their file-level HCRs to be within two standard deviations of the dataset level mean HCR.
We also observe that the 128 lilellevel HCRs for LAION-20 2--N-BB-en- (the red swarms in Figure 2 are higher than the 32 file-level HCRs for the LAION400M (the blue swarms) for all the three sub-categories of hate speech. To ascertain if this difference is statistically valid, perform two-sample t-test while cor- recting for unequal variances (using the Welch separate variances T-test) and explicitly setting the alter- native hypothesis to be greater (with respect the alternate hypothesis that the mean of the 2B-en HCRs is greater than the mean
HCRs). The results of this wwo-sample t-test are captured in Table []
As seen, for all the three categories of hateful, targeted, and aggressive speech, the strong T-values (14.48, 13.8 and 4.44) combined with high Cohen's-4 s-d 22.64,2.47,0.87 and low p-values (all < le-1) strongly support the hypothesis that the file-wise HCR associated with the 2B-en dataset is higher than the file-wise HCR for the 400M dataset, thus adding further evidence to our claim of dataset degradation upon dataset scaling.
4.3 Connecting toxic alt-text and NSFW labels
The creators of LAION provide 'Not Safe For Work' (NSFW) tag alongside each image to filter out undesirable content the user. The score was computed using ccustom-trained NSFW classifier built by the LAION team to detect harmful content in the dataset. As input, this classifier takes CLIP VIT L/14 embeddings, based on which it estimates a value between 0 (safe) and (unsafe). As reported by the LAION team, manually annotated test set of 3000 samples, the accuracy of the classifier was 0.961. For filtering purposes, they recommend a safety threshold of 0.5, which allows excluding the most explicit images.
To establish a connection in terms of harmful content contained in either of the two modalities of the dataset, we measured the extent to which the images the LAION NSFW classifier (which uses only the images as input) flags as harmful are accompanied by alt-text that was tagged as harmful by pysentimiento. While the NSFW values provided for the LAION 400M dataset were discrete ("UNLIKELY", "UNSURE" and "NSFW"), therefore precluding any values-based analysis, those provided for the LAION 2B-en dataset were continuous. We, therefore, focused on the LAION 2B
7

---

NSFW >= 0.5
NSFW <0.5
hateful
targeted
aggressive
hateful
targeted
aggressive
p M 0.25
91.44%
95.42%
99.23%
99.57%
99.9%
99.9%
0.25 <p 2 0.5
2.31% %
1.86%
0.54%
0.24%
0.06%
0.10%
0.5 p < 0.75
2.40%
1.31% %
0.18%
0.12%
0.02%
0.00%
0<0.75
3.85%
1.40%
0.04%
0.12%
0.01%
0.00%
Table 2: The percentage of values of hateful, aggressive, and targeted speech (as identified by pysen- timiento in our sample of LAION 2B-en, divided based on the suggested NSFW threshold of 0.5. can be seen that even the 'safe' subset (on the right) contains traces of toxic speech based on the alt-text.
dataset to carry out our amalysis. To do so, we compared the probability of a given image being labeled as NSFW by the LAION safety classifier the probabilities assigned for the three classes (hateful, targeted, and aggressive text) by pysentimiento for each image-text pair in our sample. We found a slight correlation between 'hateful' and 'targeted' values and NSFW ones, with Pearson correlation coefficient of 0.227 and 0.215, respectively. The correlation with "aggressive' values was much lower, at 0.076, with very low p-values (all < 0.005). This would indicate slight but not substantial correlation between harmful content in images and that contained in their accompanying alt-text.
After filtering out the LAION 2B-en sample according to the suggested threshold of 0.5, some of the harmful content identified by pysentimiento does get removed, as shown in Table However, there are still trace amounts of hateful, targeted, and aggressive content that remain, e.g., 0.24% of likely hateful content (with a probability larger than 0.5) and 0.03% of targeted content in the "safe for work" portion of the sample (right side of Table 2) This suggests that filtering out data only based on the NSFW image classifier alone does not filter out all hateful and targeted content. When carrying out qualitative observations of the portion of samples about and below the NSFW threshold, we also found examples of extremely toxic content in the alt-text (for instance, targeting women in terms of sexual violence), which was accompanied by inoffensive images. This suggests that using both text- and image-based safety filtering concurrently can help detect this kind of data and remove it before training.
We present further analyses regarding the relationship between the image and text safety filtering and provide more detailed comparisons between the datasets in Appendix B All of our results, as well as the meta-eataset created as a result of our audit, are available in repository We hope that the meta-datasets we have generated will continue to be used by the community to guide further work.
5 Discussion and Recommendations
In this paper, we have systematically examined two datasets (LAION 400M and LAION-2B-en and models trained on them. Contrary to current discourse in ML that treats scale as an unambiguously good attribute, our findings reveal that scale exacerbates hateful content. Datasets are only fundamental to equitable, just, robust, and well-performing models, but also rigorous evaluation, audit, curation, and management of datasets critical for advancing the field. Below, we present a set of observations, recommendations, and limitations of our study. We hope such discussions help the machine learning community, dataset creators/curators and other stakeholders towards advancing not only data creation and curation but also the field as a whole in manner that is transparent, rigorous, responsible, and accountable.
Computational constraints: As models and datasets get ever larger, machine learning becomes field that dominated by (and accessible to) a handful few within tech corporations and clite universities with unconstrained computational resources, crowding out those outside of it [I] The presence of big tech- affiliated influential papers in machine learning, for example, shows an increase from 13% in 2008/09 to 47% in 2018/19 (6. Assembling sarge-scale datasets requires relatively fewer resources, time, and effort than auditing, investigating, and "cleaning" them. Conversely, big tech corporations and large institutes with abundant computing power assemble these datasets. At the same time, most often, the thorough investigation and cleaning up is left to critical scholars with little resources. this study, we have inves- tigated as thoroughly as possible, given our relatively limited resources. We encountered various issues through manual investigation, such as poor data quality. For instance, an overwhelming number of im- ages were screenshots. We could not perform thorough analysis to determine a precise estimate of such
8

---

poor-quality data due to the cost of access to image APIs and the substantial computational resources required to download the datasets their entirety to sift through them. Even when large datasets such as those that we have audited are open-sourced, getting the computational resources and tooling necessary for rigorous audit is a challenge. For instance, downloading LAION 2B-en requires 6. 6.2TB of storage, with additional resources needed to run analyses such as running pysentimiento and the NSFW classifier. We encourage corporations and institutes to perform such audits. However, such self-audits will remain insufficient given the validity and credibility of self-audits are limited compared to those carried out by independent auditors [.. Subsequently, we hope nope-perhaps through a coalition of the larger commu- nity, regulatory, and funding bodies- for the cultivation (through incentives) and creation of ecology that allocates compute resources for independent auditors without access to institutional compute.
The risks of extrapolation: The dct of 12.26%, calculated in Section 4.1 above, has important consequences on estimating the number of low-quality samples that either ought to be filtered out or at least re- re-investtgated on account of having failed the text-quality mechanism that we have proposed. a fixed threehhodd of 0.5, the estimated confidence interval (CT) for the number of failed-quality text samples in -100N-400M can be closer to 1.21 to 1.26 million samples. Suppose use the upper bound of this CI as a benevolent estimate of HCR and extrapolate this to the LAION-2B-en dataset. It produces upper-bound estimate of 7.1 million text-image pairs of hatiefu//ggrresiiv//rrggg content. However, as we have seen with our HCR analysis with actual 2B-en samples above in equation the CI for this number is much higher, at 7.9 to 8.6 million samples. That is, our upper bound estimates based on the LAION-400M analysis turn out to be lower than the souur-bound estimate obtained by looking at LAION-2B -en samples by margin of 0.8655 million images (See Figure Al in the Appendix). This massive margin illustrates that extrapolation using confidence intervals, especially on datasets with underlying graph structure and rich inter-node correlations such as the Common Crawl where the sample-level i.i.d. (Independent and Identically Distributed) assumptions may be invalidated, is imprudent and may lead under-estimation of haatcfu//agrresivv//arrgee content.
The dangers of thresholding: As we show in Section even a conservative NSFW threshold of 0.5 still keeps certain amount of toxic and hateful content in alt-text, on top of the false negatives it may have for images. Given that for some LAION releases, discrete safety tags were provided based on pre- defined thresholds like this one. there will be consequences on downstream data quality (given that data users may filter the dataset based on the discrete values). Also, above and beyond false negatives, any predefined threshold assumes that a set amount of NSFW content is universally acceptable regardless of downstream data usage, which should be a decision for the data users. Therefore, we advocate that the raw values of multiple rypes of hateful content detection approaches (based on text and images) provided with datasets, alongside evaluations of what kind of content different thresholds filter out. The final thresholding values would then be established by dataset users, who can make informed decisions based on the provided information - for instance, they may filter out more NSFW and toxic content if training a text-to-image model that illustrates children's stories, as opposed to one that generates stock photos. This would require more fine- mme-grained evaluation than the one currently carried out for content fil- ters. which does not analyze filtered content and uses accuracy as the sole metric for measuring success.
Appropriate and consistent metrics: Even though many of these datasets are created to train semantic search systems and image generation models that supposedly "democratize art-creation" for the general public (where a significant proportion are people of diverse gender, ethnicity, and race) the metrics used to check if progress indeed being made by dataset scaling rarely reflects that diversity. While specific analyses are being made regarding the risk of biases and harm in ethics and safety subsections of reports and papers accompanying datasets, the metrics that supposedly measure these harms are rarely incorporated as part of the model check-pointing process. For instance, in the ALIGN paper [33,, the dataset scaling ablation study focused only on two metrics: the MS-COCO zero-shot retrieval accuracy rates (12T- T-R@1 1 and T2I-R@ 1) and the ImageNet K- K-Nearest-neight model paper ablation study [52]. the authors gauge the impact of increasing the dataset size from 1.7B to 6B by comparing the ImageNet-I k zero-shot top-1 1 accuracy. Finally, in the LAION-5B paper (62). the authors use the zero-shot top-1 classification accuracy metric, once again on the ImageNet family of datasets (with the distribution shift variants) and a bespoke VTAB+ benchmark spanning 35 classification tasks covering different computer vision datasets. This means that is difficult for users to meaningfully compare the metrics and performance on any of these datasets to each other without re- running analyses. Using standardized, meaningful metrics for measuring progress is vital to ensure that we are indeed measuring what we set out to measure and that results are comparable and reproducible.
(KNN) R@ I rates. In the BASIC

---

Exploiting multimodality in toxic content detection; While there has been some work in multimodal hate speech in specific contexts such as memes [39 69 as well as on topies such as byber-bullying B7], we were not able to find existing multimodal approaches that were generic enough to be usable for creating multimodal datasets such as LAION. Our study intended to connect the two modality-speciffe approaches described in Section 4.3 However, there is much work to be done in connecting the type of content that is often the focus of image-based filtering (for instance, violence and sexually explicit content) as well as the focus of text-based filtering (for example, hate speech and aggression). It is crucial to understand the links between images and text better and to use them both when filtering datasets. Finally, no kind of filtering is, in itself, bulletproof solution against harmful content. Without careful contextual analysis, filtering mechanisms can censor and crase marginalized languages, identities, and experiences [6
The analysis of hate content and scale in this paper was performed through the pysentimiento library. This library is known to have high predictive accuracy and works in multiple languages [51] However, as far as we know, there has been no research on the variability of pysentimiento regarding linguistic variants of English or specific topics. To continue rigorous audits and improve multimodal-toxicit detection models, we encourage future work to use these various NLP models to investigate the impact of scale on hateful content and confirm if our finding of hate-scaling holds upon using different NLP models. This, for example, includes further investigation to establish if the alternative hate detection options, including open-source models (e.g. [71 24 27 13
Limitations of pysentimiento:
produce
different or complementary results to ours.
The multimodal datasets we audited form crucial backhone for mod- ern machine learning systems, including generative models. These models are not purely intellectual exercises but are integrated into society, directly or indirectly impacting actual people. Subsequently. legal issues arise from multiple angles, including consent and rights of individuals in datasets, what should be in datasets and how they should be evaluated and maintained, and mechanisms for responsibil- ity and accountability for problematic content in the dataset as well as the downstream effect on models trained on it. Closing access to datasets used for popular and impactful models and active obfuscation of information around these datasets present a significant obstacle to developing appropriate regulatory guidelines and guardrails. This audit study showed extensive evidence of the exacerbation of hateful content correlated with scale. We hope this work serves as an initial document for legal and policy experts alike that both demystifies multimodal datasets and illustrates the negative implications of scale.
Legal and policy implications:
6 Conclusion
We have conducted a dataset audit of two vision-language multimodal datasets, LAION-400M and LAION 2B-en, and presented evidence of hateful, aggressive, and targeted content, which was exac- erbated by dataset size. This type of analysis was not previously carried out by the dataset creators, who focused on analyzing the characteristies of the images and not their accompanying alt-texts- our analysis has found that these two types of analyses are correlated but complementary. We conclude with some final remarks: we cannot stress enough the importance of open-source in audit endeavors such as ours since any quantitative and qualitative dataset exploration hinges upon access to the artifacts them- selves. Providing access to datasets and models is essential to healthy, thriving research community. However, we are saddened that an increasing number of machine learning organizations fail to do so.
Today's 's state-of-the-art vision-languan multimodal models are trained on datasets like the ones we examined in the present article. These models are currently being deployed in real-world recommendation systems, information--ettiivall systems, semantic search systems, and image captioning systems, despite them exhibiting biases and stereotypes [5] 42 that are neither well understood nor documented. Given that such failures can result in dire consequences on real people, often those at the margins of society, we implore the research community as well as those developing and deploying these systems to carry out due diligence with rigorous audits of these models and training datasets and take necessary actions, including refraining from use in high-stake scenarios.
Acknowledgements: Vishnu Naresh Boddeti was supported by the National Science Foundation under Grant No. (#2147116).
10

---

References
[1] ABDALLA, M., AND ABDALLA, M. The grey hoodie project: Big tobacco, big tech, and the threat on academic integrity. In Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics, and Society (2021), pp. 287-297.
[2] ABID, A., FAROOQL, M., AND Zou, Persistent anti- mati-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on Al. Ethics, and Society (2021), pp. 298-306.
[3] BARLAS, P., KYRIAKOU, K., GUEST, O., KLEANTHOUS, S., AND OTTERBACHER, J. To" see" is to stereotype: Image tagging algorithms, gender recognition, and the accuracy-fairness trade-off. Proceedings of the ACM on Human- Human-Compunn Computer Interaction 4, CSCW3 (2021), 1-31.
[4] BENDER, E. M.. GEBRU, T., McMILLAN -AMOR,, A., AND SHMITCHELL, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610-623.
[5] BIANCHI, F., KALLURI, P., DURMUS, E., LADHAK, F., CHENG, M., NO7ZA, D., HASHIMUTO, T., JURAFSKY, D.. Zou, J., AND CALISKAN, A. Easily accessible text-to-image generation amplifies demographic stereotypes large scale. arXiv preprint arXiv:2211.03759 (2022).
[6] BIRHANE, A., KALLURI, P., CARD, D., AGNEW, W., DOTAN, R., AND BAO, M. The values encoded in machine learning research. arXiv preprint arXiv: 106. 5500 (2021).
[7] BIRHANE, A., PRABHU, V. U., AND KAHEMBWE, E. Multimodal datasets: misogyny, pornog raphy, and malignant stereotypes. arXiv preprint arXiv:22 (2021).
[8] CARLINI, N., HAYES, J., NASR, M., JAGIELSKI, M.. SEHWAG, V., TRAMER. F., BALLE,
B., IPPOLITO, D., AND WALLACE, E. Extracting training data from diffusion models. arXiv preprint arXiv:2301. 13/88 (2023).
[9] CHHABRA, A., AND VISHWAKARMA, D. K. literature survey on multimodal and multilingual automatic hate speech identification. Multimedia Systems (2023), 1-28.
[10] CITRON, D. K. Hate crimes in cyberspace. Harvard University Press, 2014.
[11]
COSTANZA-CHOCK, S., RAII, D., AND BUOLAMWINI, J. Who audits the auditors? recom- mendations from field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (2022), pp. 1571-1583.
[12] DAVIDSON, T., BHATTACHARYA, D., AND WEBER, Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv: 12516 (2019).
[13] DAVIDSON, T., WARMSLEY, D., MACY, M., AND WEBER, I. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference web and social media (2017), vol. 1, pp. 512-515,
[14] DENG, J., DONG, W., SOCHER, R., LI, L.-J., LI, K., AND FEI-FEI, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition
(2009), leee, pp. 2555.
[15] DENTON, E., HANNA, A., AMIRONESEL, R., SMART, A., AND NICOLE, H. On the genealogy of machine learning datasets: A critical history of imagenet. Big Data & Soccity 8, 2 (2021), 10535117211335555
[16] DODGE, J., SAP, M., MARASOVIC, A., AGNEW, W., ILIARCC, G., GROENEVELD, D., AND GARDNER, M. Documenting the english colossal clean crawled corpus. arXiv preprint arXiv:2104.0875 (2021).
[17] EDWARDS. B. Flooded with ai-generated images, some communities ban them com- pletely ars technica. httpps://asttteciiicaaaaaa htt.ar/eeec.cc.c.......oooa.aata
some art htt.ae/ee...a...ttttoataaaaaaaaa Sep 2022. (Accessed on 04/22/2023).
[18] ERIC SHERIDAN, K. R. Are we on the cusp of generative ai revolution? https://wwx goldmansachs. www.cooooooii. wtmm//niiggts/pocsstt//////oe/ee
pwtposssss/eissdsse/////// html Feb 2023. (Accessed on 04/21/2023).
iiterddan-ranger
[19] FOXMAN, A. H., AND WOLF, C. Viral hate: Containing its spread on the Internet. Macmillan,
2013.
11

---

[20] GAO. L., BIDERMAN, S., BLACK, S., GOLDING, L., HOPPE, T., FOSTER, C., PHANG, J., HE, H., THITE, A., NABESHIMA, N., ET AL. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2 10000000 (2020).
[21] GARCIA, N., HIROTA, Y., WU, Y., AND NAKASHIMA, Y. Uncurated image-text datasets:
Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 6957-6966.
[22] GARG, T., MASUD, S., SURESH, T., AND CHAKRABORTY, T. Handling bias in toxic speech detection: A survey, ACM Computing Surveys (2022).
[23] GEBRU, T., MORGENSTERN, J., VECCHIONE, B., VAUGHAN, W., WALLACH, H.,, III, H. D.,
AND CRAWFORD, K. Datasheets for datasets. Communications the ACM 64, 12 (2021), 86-92.
[24] GEHMAN, S., GURURANGAN, S., SAP, CHOI,, Y., AND SMITH, N. A. Realtoxicityprompp::
Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. (2020).
[25] GICHOYA, J. W., BANERJEE, I., BIIIIIRDDY, A. R., BURNS, J. L., CELI, L. A., CHENN, L.-C., CORREA, R., DULLERUD, N., GHHASSEI,, M., HUANG, S.-C., ET AL. Ai recognition of patient race in medical imaging: modelling study. The Lancet Digital Health 4, 6 (2022),
[26]
HANNA, A., AND PARK, T. M. Against scale: Provocations and resistances to scale thinking. arXiv preprint arXiv:2010.000 (2020).
[27]
HANU, L.,, AND TEAM. Detoxify. Github. https:/ggihubb.commmniaaaaaaaale
2020.
[28]
HEAVEN, W. D. the hype is gone?
Generative ai is changing everything. but what's left when httpp::/www technologyrevien, com/2002/12/1//155000 Dec 2022. (Accessed on 04/21/2023).
[29] HESTNESS, J., NARANG, S., ARDALANI, N., DIAMOS, G., JUN, H., KIANINEJAD, H., PATWARY, M., ALI, M., YANG, Y., AND ZHOU, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 172..0049 (2017).
[30]
HOFFMANN, J., BORGEAUD, S., MENSCH, A., BUCHATSKAYA, E., CAI, T., RUTHERFORD, E., CASAS, D. D. L., HENDRICKS, L. A., WELBL, J., CLARK, A., ET AL. Training compute- optimal large language models. arXiv preprint arXiv:2203. 15556 (2022).
[31] HUANG, H.
The generative ai revolution has begun-how did we get
the-generaaive--e www.vvv--aiireloooooo-----eee http:://aastccccaaaacoooo////gpp (Accessed on 04/21/2023).
here?
https: https://arstechnica.. cwm ars technica. bttp:aa---e---bbb----------eeeee
/013//1
Jan 2023.
[32] JAHAN, M. S., AND OUSSALAH, M. A systematic review of hate speech automatic detection using natural language processing. arXiv preprint arXiv:2106.00 00742 (2021).
[33] JIA. C., YANG, Y., XIA, Y., CHEN, Y.-T., PAREKH, Z., PHAM, H., LE, Q. V., SUNG, Y., LI,
Z., AND DUERIG, T. Scaling up visual and vision- Languae representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 (2021).
[34] JOSHUA LU, R. G. The generative ai revolution will enable anyone create games andreessen horowitz. https: //aw/2.///..... cttp://cccmm22222/ wwtp//////////////////// http:////..//../teeeeeeeeeaeaaaa 2023. (Accessed on 04/21/2023).
Mar
[35] KAPLAN, J., MCCANDLISH, S., HENIGHAN, T., BROWN, T. B., CHESS, B., CHILD, R., GRAY, S., RADFORD, A., J., AND AMODEI, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08836 (2020).
[36] KOCH, B., DENTON, E., HANNA, A., AND FOSTER, G. Reduced, reused and recycled: The life of a dataset in machine learning research. arXiv preprint 1rXxi::2/220177 (2021).
[37] KUMAR, A., AND SACHDEVA, N. Multimodal cyberbullying detection using capsule network with dynamic routing and deep convolutional neural network. Multimedia Systems (2021). 1-10.
[38] LAURENÇON, H., SAULNIER, L., WANG, T., AKTKI, C., VILLANOVA DEL MORAL, A., LE SCAO, T., VON WERRA, L.., MOU, C., GONZÁLEZ PONFERRADA, E., NGUYEN, H., RT AL.. The bigscience roots corpus: 1.6 th composite multilingual dataset, Advances in Neural Information Processing Systems 35 (2022), 31809-31826.
12

---

[39] LEE, R. K.- CAO, R., FAN, Z., JIANG, J., AND CHONG, W.-H. Disentangling hate in online memes. In Proceedings tfe 29th ACM International Conference on Multimedia (2021),
pp. 10.3118-147,
[40] LEE. T. B. Stable diffusion copyright lawsuits could be a legal earthquake for ai ars technica.
https://asstechhicc..com ottp://cccccccc.ccm/mcceeeoaaaao copyrighoyrrooeee wwt.:aawsssttccoodddbbeeeeee http::/:ww....oaaaaooooooooooaar
Apr 2023. (Accessed on 04/22/2023).
[41] LOMAS, N. Shutterstock to integrate OpenAI's DALL-E2 and launch fund for contributor artists, 2022.
[42] LUCCIONI, A. S., AKIKI, C., MITCHELL, M., AND JERNITE, Y. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303. 14408 (2023).
[43]
LUCCIONI, A. S., AND VIVIANO, D. What's in the box? analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:
(2021).
[44]
MACAVANEY, S., YAO, H.-R., YANG, E., RUSSELL, K., GOHARIAN, N., AND FRIEDER, O. Hate speech detection: Challenges and solutions. PloS one 14, 8 (2019), e0221152.
[45]
MANTILLA, K. Gendertrolling: How misogyny went viral: How misogyny went viral. ABC-
CLIO, 2015.
[46]
METAXA, D., PARK, J. S., ROBERTSON,,, R. E., KARAHIALIOS, K., WILSON, C., HANCOCK, J. SANDVIG, C., ET AL. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends@ in Human- Computerr Interaction 14, (2021), 272-344.
[47] NATIONS, U Understanding hate speech. https: /wwww//wwww.o un org/ en/ understanding www.hapeechheeeeeeeee sptech is- http://taatttttetpppppeeaeee
(Accessed on 16 October 2023).
[48] ORSELLI, B. Stable diffusion ai has mastered the female form niche gamer. https
//nichegamer. com/ hommstable-ddae www.ediiffffooooooooomoooooooo aa--aaiiaaaaseeeeeeeeeseee the fetale-emme---rrm 2022. (Accessed on (4/22/2023).
Oct
[49] PAULLADA, A., RAJI, I. D., BENDER, E. M., DENTON, E., AND HANNA, A. Data and its (dis) contents: A survey of dataset development and use machine learning research. Patterns 11 (2021), 100336.
[50] PENG, K., MATHUR, A., AND NARAYANAN, A. Mitigating dataset harms requires stewardship: Lessons from 1000 papers. arXiv arXiv:2108. 02922 (2021).
[51] PÉREZ, J. M., GIUDICI, J. AND LUQUE, pysentimiento: A python toolkit for sentiment analysis and socialnlp tasks. arXiv preprint arXiv:2106.09 094.6262660 (2021).
[52] PHAM, H., DAI, Z.,, GHIASI, G., KAWAGUCHI, K., LIU, H., Yu, A. W., YU, CHEN Y.-T., LUONG, M.-T., WU, Y., ET AL. Combined scaling for open-vocabulary image classification.
arXiv (2021), arXiv-2111.
[53] POLETTO, F., BASILE, V., SANGUINETTI, M., Bosco, C., AND PATTI, V. Resources and benchmark corpora for hate speech detection: systematic review. Language Resources and Evaluation 55 (2021), 477-523.
[54] RADFORD, A., J. W., HALLACY, C., RAMESH, A., GoH, G., AGARWAL, S., SASTRY, G., ASKELL, A., MISHKIN, P., CLARK, J., ET AL. Learning transferable visual models from natural language supervision. arXiv preprint 1rXiv:2103.00022 (2021).
[55] RAFFEL, C., SHAZEER, N., ROBERTS, A., LEE, K., NARANG, S., MATENA, M., ZHOU, Y., LI, W., AND LIU, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research 21, (2020). 5485-5551.
[56] RAJI, I. AND BUOLAMWINI,) J. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 429-435.
[57] RAMESH, A., PAVLOV, M., GOH, G., GRAY, S., Voss, C., RADFORD, A., CHEN, M., AND SUTSKEVER, I. Zero-shot text-to-image generation. arXiv preprint 2rX (2021).
[58] ROMBACH, R., BLATTMANN, A., LORENZ, D., ESSER, P., AND OMMER, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 10684-10695.
13

---

[59] SAHARIA, C., CHAN, W., SAXENA. S., LI, L., WHANG, J., DENTON. E. L., GHASEMIPOUR, K., GONTIJO LOPES. R., KARAGOL AYAN, B., SALIMANS, T., ET ET AL. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35 (2022), 36479-36499.
[60] SAMBASIVAN, N., KAPANIA, S., HIGHFILI,, H., AKRONG, D., PARITOSH, P., AND AROYO, L. M. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021),
pp. 1-15.
[61] SCHEUERMAN, M. K., WADE, K., LUSTIG, C.,, AND BRUBAKER, J. R. How we ve taught algorithms identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-compel
Interaction 4, CSCW1 (2020), 1-35.
[62] SCHUHMANN, C., BEAUMONT, R., VENCU, R., GORDON, C., WIGHTMAN, R., CHERTI, M., CоOMBES, T., KATTA, A., MULLIS, C., WORTSMAN, M., ET AL. Laion-5b: An open large- scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 (2022).
[63] SCHUHMANN, C., VENCU, R., BEAUMONT, R., KACZMARCZYK, R., MULLIS, C., KATTA,
A., COOMBES, T., JITSEV, J., AND KOMATSUZAKI, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint wwt.:iivv:21ii11. 11.02114 (2021).
[64] SOLAIMAN, I., TALAT, Z., AGNEW, W., AIMMAD, L., BAKER, D., BLODGETT, S. L., DAUMÉ III, H., DODGE, J., EVANS, E., HOOKER, S., ET Evaluating the social impact of generative ai systems in systems and society. arXiv preprint
(2023).
[65] SOMEPALLI, G., SINGLA, V., GOLDBLUM, M., GEIPING, J., AND GOLDSTEIN, T. Diffu- sion art digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.08:: 12.08600 (2022).
[66] TOMASEV, N., MAYNARD, J. L., AND GABRIEL, I. Manifestations of xenophobia in ai systems, 2022.
[67] VAN MILTENBURG, E. Stereotyping and bias in the flickr30k dataset. arXiv preprint 1rXiv:1605.06083 (2016).
[68] VECCHIONE, B., LEVY, K., AND BAROCAS, S. Algorithmic auditing and social justice: Lessons from the history of audit studies. In Equity and Access in Algorithms, Mechanisms, and Optimization. 2021, pp. 1-9.
[69] VELIOGLU, R., AND ROSE, J. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution hateful memes challenge. arXiv preprint arXiv:2012.1297. (2020).
[70] VIDGEN, B.. THRUSH, T., WASEEM, Z., AND KIELA, D. Learning from the worst: Dynamically generated datasets to improve online hate detection. arXiv preprint arXiv:2012.1574 (2020).
[71] VIDGEN, B., THRUSH, T., WASEEM, Z.., AND KIELA, D. Learning from the worst: Dynamically generated datasets to improve online hate detection. In ACI. (2021).
[72] VILLALOBOS, Scaling laws literature review, 2023. Accessed: 2023-4-22.
[73] WANG, K., YANG, AND WU, H. survey of toxic comment classification methods. arXiv preprint
(2021).
[74] WASEEM, Z., DAVIDSON, T., WARMSLEY, D., AND WEBER, I. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprini urXiv:/705.0989) (2017).
[75] WEBSTER, R., RABIN, J., SIMON, L., AND JURIE, F. On the de-duplication of laion-2b. arXiv preprint arxiv: (2023).
[76] Yu, J., Xu, Y., KOH, J. Y., LUONG, T., BAID, G., WANG, Z., VASUDEVAN, V., KU, A.. YANG, Y., AYAN, B. K. ET AL. Scaling autoregressive models for content-rich eoxt-to-image generation. arXiv preprini arXiv:2206. 10789 (2022).
[77] YUAN, L., CHEN, D., CHEN, Y.-L., CODELLA, N., DAI, X., GAO, J., HU, H., HUANG, X., LI, B., LI, C., ET AL. Florence: A new foundation model for computer vision. arXiv preprini 211.:2211. (2021).
14

---

Supplementary Materials
A
The risks of extrapolation
(a) CI comparison 400M & 28-en
(b) "Excess-images" estimate for LAION28-en
0.35
0.34
7.8 10a
óci = 12.26%
of 28
Cl. Claaee soaed Chwee
7.6 10
B6442220 images!
0.32
x.4
0.31
x 10f
0.29
400M
Nye 100M
Figure Al: Binomial proportion confidence interval (CI) analysis to establish the extent of HCR underestimation upon using LATON400M statisties.
B NSFW Analysis
NSFW >= 0.5
NSFW <0.5
hateful
targeted
aggressive
hateful
targeted
aggressive
0.749
0.356
0.104
0.021
0.013
0.011
min
0.501
0.008
0.013
0.00
0.013
0,005
max
0.988
0.961
0.894
1.00
0.499
0.513
Table Al: The mean, minimum, and maximum values for hateful, targeted, and aggressive content in our sample LAION 2B-En, are divided according the suggested NSFW threshold of 0.5.
C The origins of the dataset scaling laws: A cartoon sketch emerges
While attempting to unearth what this specific dataset scaling law was that the practitioners were so inspired by, we repeatedly encountered certain cartoon sketch 'power-law' plot referred to both personal exchanges as well as in surveys such as 772. As turns out, this cartoon sketch power-plot first appeared Figure 6 in "Deep learning scaling is predictable, empirically" [29), a work that emerged out of Baidu research in 2017. The authors that first presented this plot posit that the generalization error associated with ML model exhibits a three-phase behavior about its training dataset size. In the first phase, they state maps to the 'small data region,' where "models will struggle learn learn small number of training samples" resulting in high generalization errors. The second phase (or the middle portion of learning curves), they claim, is the 'power-law region,' where the generalization error monotonically decreases with training dataset size (linear with application-specifie slopes when plotted on a log-log scale). This phase stretches till we hit the point of the "glass-ceiling' "unbreachable error floor' on account of factors such model mismatch and mislabeled data (constituting the third phase). This, of course, has been further supplanted by the likes of the Chinchilla scaling laws (20 tokens per model parameter) 30 in the specialized context of LLMs.
D Blackbox non-reproducible empirical results
As for the black box non-reprodueible empirical results that validated the dataset-scaling mandate and championed the
narrative, refer to the ALIGN paper (33) that emerged in 202 In
15

---

the abstract section of this paper. we first encounter the following claim: "We show that the scale of our corpus can make up for its noise and leads to state-of-tiee oof-the-rr art representations even with such simple learning scheme". " The demonstration of this claim appears later in "Section 6. 2. Pre-training Datasets" where the authors state that "To understand better how data size scaling wins over the increased noise, we further randomly sample 3M, 6M, and 12M ALIGN training data and compare them with the cleaned CC-3M data on B7+BERT-bur data performs much worse than CC data with the same size (3M), the model quality trained 6M and 12M ALIGN data rapidly catches up. Despite being noisy, ALIGN data outperforms Conceptual Captions with only 4x size." We note that these experiments (or similar ones) have not been replicated elsewhere to check if these scaling ratios presented ipse dixit in these contexts indeed hold at all.
model. Table 10 (sic) shows that while the ALIGN
E
The tactical template: Fuzzy main section meets non-existent appendices
What unites the marquee projects of Dall-E, Parti, and Imagen is the near-same tactical template deployed when it comes to (non)declaring the training dataset information. The template runs something like this:
Step-1: Allocate a small nondescript subsection of the main section of the paper covering only the bare minimum details about the number of samples in the training dataset with cross-references to other similar black box datasets such as JFT. This coincidentally happens to be Section 4. in both Parti and Imagen papers (See A2).
Step-2: Declare that somewhere in the succeeding sections titled on the lines of broader impacts or societal impacts are details about the 'potentially problematic' aspects of the dataset and the downstream risks while patronizingly citing previously published audit papers (such as 7 that have done the grunt work exposing the gory details of such datasets. This happens to be Section 8 Broader impacts in Parti and Section 6 for the Imagen model.
Step-3: Setting the reader up for non-existent Appendix section that is not part of the main paper and docs not contain any details about how the dataset is constructed and where the data is sourced from while noting the fact that it's not mandatory for the reviewers to even glance at the Appendix section in peer-rev reviewed avenues of publishing.
is in this backdrop we observe that the authors of the BASIC model paper have not even addressed model safety and dataset auditing issues despite having trained their model on the largest image-text dataset ever assembled and presented full-length 47- 47-page paper with three revisions on ArXiv (See https://arxse wtgg/bss//e 10050).
16

---

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
4.1 Training Datasets
We train on a combination of image-text datasets for Parti models. The data includes the publicly available LAION-400M dataset [43]: FIT400M, a filtered subset of the full 1.8 billion examples used to train the ALIGN model [9]: JFT-4B dataset [44], which has images with text annotation labels. For textual descriptions JFT, we randomly switch between the original labels as text (concatenated if an image has multiple labels) or machine-generann the limitations of the data in Section 8, For all image inputs, we follow the DALL-E dVAE input processing (Section A.2. Training in [2]) for image tokenizer training and the DALL-E Transformer input processing (Section B.2. Training in [2] for encoder-decoder training.
captions from a SimVLM model [45]. We discuss
8 Broader Impacts
Bias and safety. Text-to-image generation models like GLIDE, DALL-E 2, Imagen, Make-a-Scene, Coe and Parti are all trained on large, often noisy, image-text datasets that are known contain biases regarding people of different backgrounds. This particutarly highlighted Birhane et al's [100] analysis of the LAION-400M dataset [43]: their study of the dataset surfaced many problems with respect to stereotyping, pornography, violence and more. Other biases include stereotypical representations of people described as lawyers, flight attendants, homemakers, and so on. Models trained on such data without mitigation strategies thus risk reflecting and scaling up the underlying problems. Our primary training data is selected and highly filtered to minimize the presence of NSFW content; however, we incorporated LAION-400M during finetuning with classifier-free guidance this improved model performance but also led to generation of NSFW images in some contexts. Other biases include those introduced by the use of examples that primarily have English texts and may be biased certain areas of the world. In informal testing, we have noticed, for example, that prompts mentioning wedding clothes seem to produce images biased towards stereotypically female and Western attire.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
4.1 Training details
We train on a combination of internal datasets, with s 460M image-text pairs, and the publicly available Laion dataset [61], with z 400M image-text pairs. There are limitations our training data, and we refer the reader to Section for details. See Appendix F for more implementation details.
6 Conclusions, Limitations and Societal Impact
Imagen's 's training data was drawn from several pre-existing datasets of image and English alt-text pairs.
A subset of this data was filtered to removed noise and undesirable content, such as pornographic imagery and toxic language. However, recent audit of one of our data sources, LAION-400M [61]] uncovered a wide range of inappropriate content including pornographic imagery, racist slurs, and harmful social stereotypes [4]. This finding informs our assessment that Imagen is not suitable for public use at this time and also demonstrates the value of rigorous dataset audits and comprehensive dataset documentation (e.g. [23, 45]) in informing consequent decisions about the model's appropriate and safe use. Imagen also relies on text encoders trained on uncurated web-scale data, and thus inherits the social biases and limitations of large language models [5, 3, 50].
Figure A2: The Google template used to (non)declare the training dataset information along with paper screenshots
17


Reviews:
[{"review_id": "jpWKPLQJCi", "reviewer": "Reviewer_ToJe", "summary": "", "questions": "", "limitations": "Yes, I cannot think of any other key limitations that are missed.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The topic is timely, the paper is extremely well-written, motivated, and explained. The analyses, are sound and they mostly show clear results. Large datasets are also processed in those analyses. \n\nFigure 1 is very impactful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "G2yKCtM3kA", "reviewer": "Reviewer_9brB", "summary": "", "questions": "", "limitations": "The authors do acknowledge that multi-modal models would need auditing on both linguistic and visual components of their dataset for proper auditing, something they do not do in the paper. The paper focuses on simply the linguistic portion. They also discuss the limitation of access to datasets of most major LLMs and how this can affect the ability to audit these models effectively. Unfortunately, they miss an important limitation of their analysis- their dependence on packages such as pysentimiento and proba that may incorrectly label certain things as hate speech or having a negative sentiment that may not be. One such example of this happening within the paper is the sentence the authors use in their auditing methodology paragraph: ‘slave punished by angry blond mistress’ that is rated with a .96 hate speech score- no doubt because of the inclusion of the word 'slave'. The problem is, what is actually hateful about that sentence? Was this sentence used when discussing an historical fact? If so, slaves WERE punished by their white slave owners that may have been blond and, yes, may have been angry. Is the problem the word 'mistress'? It is not clear what makes a sentence such as that hateful, and, as such, there can be other sentences labeled 'hateful' that are not hateful at all though they may make one uncomfortable to read. Slavery is a fact in this country and any model that tries to flag references to it may end up causing more harm than good. ", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The perspective of this paper is refreshing. There is an amazing need for explainability and auditing tools in the NLP/LLM space due to the massive explosion of Generative AI and the ever growing datasets that are used to build it. I also love that they want for the code resulting from this work to be open source. It is very important in these spaces to build upon prior work. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "iKNfuj1ZNl", "reviewer": "Reviewer_gEqv", "summary": "", "questions": "", "limitations": "1. The comparison does not seem quite valid to me as the data size for the two datasets is significantly different. A larger dataset is likely to contain more noisy data. Instead, the data is better sampled proportionally.\n2. Using only one measurement is not sufficient. The authors mention in section 5 that pysentimiento lacks research on its variability in terms of linguistic variants of English or specific topics. Hence it is questionable to interpret the reliability of results if no other models/metrics are used. The selection of pysentimiento is not very much justified. Is it better than other models/APIs like perspective? For choosing pysentimiento why did you mention misogyny and racism which is neither discussed in the rest of the paper nor used for the analysis of the LAION datasets?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper touches base on a hot topic - the impact/limitation of data scaling on model performance and society.\n2. It analyses multimodal datasets.", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Because reviews were mixed and only 3 of the ideal 6 were provided, I read the paper as well. I had a very similar impression as Reviewer ToJe, and have some (strong) disagreements with statements made by Reviewer gEqv. To summarize the views of the 3 reviewers + me:

Strengths:
- Very timely topic.
- In an important domain.
- At times writing is passionate, but written well. 🔥
- Sound analysis.

Opportunities For Improvement:
- More explanation of why hate speech is an issue in this domain.
- Using only one main system (pysentimiento) to derive general conclusions is not ideal. Either that choice should be better justified, or more than one system should be used.
- Explanation on what exactly the three measures capture: hateful, targeted, and aggressive.
- Make it more clear throughout that the packages are not completely reliable. In particular, I noticed several places where the authors write results about "hate"/"harm" where I think they mean *predicted* hate/harm. 

Limitations: Well written. Would have appreciated them being referred to earlier, such as noting that discussion on issues with pysentimiento are available in the Limitations section.

Correctness: Mostly correct. Not mentioned by other reviewers, but did you really intend for HCR to be multiplied by 100? The numbers provided appear to scale between 0 and 1, whereas I had thought you were scaling between 0 and 100(%). I had the sense that you were trying to convert probabilities to percentages, but then didn't end up doing it. You also describe it as a "rate" and "ratio" but I think I would call it an "average" (it's divided by the number of samples, giving you something like "average hate content"). I'd imagine a ratio to have the denominator as #samples - numerator.

Clarity: Good

Relation to Prior Work: Suggestions from reviewers of additional things to include.

Documentation: Reasonable

Additional comments:
- " dataset degradation " -- perhaps be more specific. More hateful?
- " precluding any kind of values-based analysis " -- Confusing. You mean any kind of *real* or *continuous*-valued analysis? The values given are ordinal. Not to mention the additional confusion with "value" often meaning peoples' values in this context.

Some typos (not all):
- 129: "such the ones"
- 139: "and scaleand to what harmful" (Also, I think by "what" you mean "whether"?)
- 168: "the amount of problematic"
- 169: "We also and"
- iid: You define in the second usage, not the first; flip.
Decision: Accept (Poster)