SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\bar{\psi}_{lb,dataset}^{(\alpha = 0.05)}$  and  $\bar{\psi}_{ub,dataset}^{(\alpha = 0.05)}$  are the lower-bound and the upper-bound values of the confidence interval at  $\alpha = 0.05$ .
As seen above in equation 3, the lower-bound HCR for the 2B-en dataset is markedly higher than the upper-bound estimate of HCR for the 400M dataset thus leading to change-of-HCR,  $\delta_{CI} = \left(\frac{\bar{\psi}_{lb,2B - en}^{(\alpha = 0.05)}(0.5) - \bar{\psi}_{ub,400M}^{(\alpha = 0.05)}(0.5)}{\bar{\psi}_{ub,400M}^{(\alpha = 0.05)}(0.5)}\right) \times 100$  of  $12.26\%$ . Simply put, even under an optimistic setting where we compute the difference between the lower-bound estimate of HCR for the 2B-en dataset and the upper-bound estimate of HCR for the 400M dataset, we still see a  $12.26\%$  normalized increase in HCR.
We have so far established the risks of extending the LAION-400M dataset quality statistics to its bigger counterpart, that is, LAION-2B-en. This, as we hypothesize, may be a consequence of rich non-iid inter-sample correlations emerging from a graph-structured prior for CommonCrawl. This begs the question, given that our dataset was uniformly sampled at the shard/file level, whether the HCR statistics computed at the shard/file level can meaningfully compare with the global dataset-level HCR statistic. We examine this in the sub-section below.
4.2 Intra-dataset filewise comparisons
Given that the two datasets, LAION-400M and LAION-2B-en, are split into, respectively, 32 and 128 purportedly uniformly sampled shards, we also examine the validity of the file-level HCR metrics to the global dataset-level metrics. To this end, we use the three 100,000-sized file-level text-quality score matrices obtained from the *pysentimiento* model and compute what fraction of these rows are larger than  $P_{threshold}$  of 0.5 for all the three columns. This yields file-level HCRs (in %) for each dataset, with the three columns mapping to hateful speech, targeted speech, and aggressive speech.
We found that the file-wise HCRs were all tightly clustered around the mean levels for the individual datasets. Figure 2 shows the fused swarm-box-violin plot that captures the file-wise HCR metrics
Figure 2: Fused swarm-box-violinplot that captures the file-wise HCR metrics for all the 160 (=32+128) parquet files from LAION400M and LAION-2B-en. HCRs for LAION-2B-en (the red swarms) are higher than the 32 file-level HCRs for the LAION400M (the blue swarms) for all three sub-categories – hateful, targeted, and aggressive speech.
hateful	targeted	aggressive
T	14.48	13.80	4.44
Degrees of freedom	53.32	54.91	47.96
P-value	2.02e-20	8.44e-20	2.60e-05
Cohen-d	2.64	2.47	0.87
BF-10	3.785e+27	5.957e+25	2131.144
Table 1: Results from the two-sample t-test while correcting for unequal variances (using the Welch separate variances T-test). 'BF10' indicates the Bayes Factor of the alternative hypothesis. For all three categories of hateful, targeted, and aggressive speech, the file-wise HCR associated with the 2B-en dataset is higher than the file-wise HCR for the 400M dataset, showing dataset degradation with dataset scaling.
for all the 160 (=32+128) parquet files spanning the two datasets. For example, the 'hateful' related HCR for LAION-400M has a mean value of 0.298, which increased to 0.344 for LAION-2B-en. All the 32 constituent file-wise HCRs for this dataset fall within 0.26 and 0.33.
Furthermore,  $97\%$  of all the files have their HCRs within two standard deviations of the mean-HCR for the dataset. Similarly, the mean-HCR for the entire LAION-2B-en dataset is 0.344, and the range across all the 128 files is [0.297,0.382], which indicates that  $\sim 95\%$  of all the constituent files to have their file-level HCRs to be within two standard deviations of the dataset level mean HCR.
We also observe that the 128 file-level HCRs for LAION-2B-en (the red swarms in Figure 2) are higher than the 32 file-level HCRs for the LAION400M (the blue swarms) for all the three sub-categories of hate speech. To ascertain if this difference is statistically valid, we perform a two-sample t-test while correcting for unequal variances (using the Welch separate variances T-test) and explicitly setting the alternative hypothesis to be greater (with respect to the alternate hypothesis that the mean of the 2B-en HCRs is greater than the mean of 400M-HCRs). The results of this two-sample t-test are captured in Table 1.
As seen, for all the three categories of hateful, targeted, and aggressive speech, the strong T-values (14.48, 13.8 and 4.44) combined with high Cohen's-d (2.64, 2.47, 0.87) and low p-values (all  $\ll 1e^{-4}$ ) strongly support the hypothesis that the file-wise HCR associated with the 2B-en dataset is higher than the file-wise HCR for the 400M dataset, thus adding further evidence to our claim of dataset degradation upon dataset scaling.
4.3 Connecting toxic alt-text and NSFW labels
The creators of LAION provide a 'Not Safe For Work' (NSFW) tag alongside each image to filter out undesirable content for the user. The score was computed using a custom-trained NSFW classifier built by the LAION team to detect harmful content in the dataset. As input, this classifier takes CLIP ViT L/14 embeddings, based on which it estimates a value between 0 (safe) and 1 (unsafe). As reported by the LAION team, on a manually annotated test set of 3000 samples, the accuracy of the classifier was 0.961. For filtering purposes, they recommend a safety threshold of 0.5, which allows excluding the most explicit images.
To establish a connection in terms of harmful content contained in either of the two modalities of the dataset, we measured the extent to which the images the LAION NSFW classifier (which uses only the images as input) flags as harmful are accompanied by alt-text that was tagged as harmful by psentimiento. While the NSFW values provided for the LAION 400M dataset were discrete ("UNLIKELY", "UNSURE" and "NSFW"), therefore precluding any values-based analysis, those provided for the LAION 2B-en dataset were continuous. We, therefore, focused on the LAION 2B-en
NSFW >= 0.5	NSFW <0.5
hateful	targeted	aggressive	hateful	targeted	aggressive
p ≤ 0.25	91.44%	95.42%	99.23%	99.57%	99.9%	99.9%
0.25 < p ≤ 0.5	2.31%	1.86%	0.54%	0.24%	0.06%	0.10%
0.5 < p ≤ 0.75	2.40%	1.31%	0.18%	0.12%	0.02%	0.00%
p < 0.75	3.85%	1.40%	0.04%	0.12%	0.01%	0.00%
Table 2: The percentage of values of hateful, aggressive, and targeted speech (as identified by psentimiento in our sample of LAION 2B-en, divided based on the suggested NSFW threshold of 0.5. It can be seen that even the 'safe' subset (on the right) contains traces of toxic speech based on the alt-text.
dataset to carry out our analysis. To do so, we compared the probability of a given image being labeled as NSFW by the LAION safety classifier to the probabilities assigned for the three classes (hateful, targeted, and aggressive text) by *pysentimiento* for each image-text pair in our sample. We found a slight correlation between 'hateful' and 'targeted' values and NSFW ones, with a Pearson correlation coefficient of 0.227 and 0.215, respectively. The correlation with 'aggressive' values was much lower, at 0.076, with very low p-values (all  $\ll 0.005$ ). This would indicate a slight but not substantial correlation between harmful content in images and that contained in their accompanying alt-text.
After filtering out the LAION 2B-en sample according to the suggested threshold of 0.5, some of the harmful content identified by *pysentimiento* does get removed, as shown in Table 2. However, there are still trace amounts of hateful, targeted, and aggressive content that remain, e.g.,  $0.24\%$  of likely hateful content (with a probability larger than 0.5) and  $0.03\%$  of targeted content in the "safe for work" portion of the sample (right side of Table 2). This suggests that filtering out data only based on the NSFW image classifier alone does not filter out all hateful and targeted content. When carrying out qualitative observations of the portion of samples about and below the NSFW threshold, we also found examples of extremely toxic content in the alt-text (for instance, targeting women in terms of sexual violence), which was accompanied by inoffensive images. This suggests that using both text- and image-based safety filtering concurrently can help detect this kind of data and remove it before training.
We present further analyses regarding the relationship between the image and text safety filtering and provide more detailed comparisons between the datasets in Appendix B. All of our results, as well as the meta-dataset created as a result of our audit, are available in our repository. We hope that the meta-datasets we have generated will continue to be used by the community to guide further work.
5 Discussion and Recommendations
In this paper, we have systematically examined two datasets (LAION 400M and LAION-2B-en) and models trained on them. Contrary to current discourse in ML that treats scale as an unambiguously good attribute, our findings reveal that scale exacerbates hateful content. Datasets are not only fundamental to equitable, just, robust, and well-performing models, but also rigorous evaluation, audit, curation, and management of datasets is critical for advancing the field. Below, we present a set of observations, recommendations, and limitations of our study. We hope such discussions help the machine learning community, dataset creators/curators, and other stakeholders towards advancing not only data creation and curation but also the field as a whole in a manner that is transparent, rigorous, responsible, and accountable.
Computational constraints: As models and datasets get ever larger, machine learning becomes a field that is dominated by (and accessible to) a handful few within tech corporations and elite universities with unconstrained computational resources, crowding out those outside of it [1]. The presence of big tech-affiliated influential papers in machine learning, for example, shows an increase from  $13\%$  in 2008/09 to  $47\%$  in 2018/19 [6]. Assembling large-scale datasets requires relatively fewer resources, time, and effort than auditing, investigating, and "cleaning" them. Conversely, big tech corporations and large institutes with abundant computing power assemble these datasets. At the same time, most often, the thorough investigation and cleaning up is left to critical scholars with little resources. In this study, we have investigated as thoroughly as possible, given our relatively limited resources. We encountered various issues through manual investigation, such as poor data quality. For instance, an overwhelming number of images were screenshots. We could not perform a thorough analysis to determine a precise estimate of such
poor-quality data due to the cost of access to image APIs and the substantial computational resources required to download the datasets in their entirety to sift through them. Even when large datasets such as those that we have audited are open-sourced, getting the computational resources and tooling necessary for rigorous audit is a challenge. For instance, downloading LAION 2B-en requires 6.2TB of storage, with additional resources needed to run analyses such as running *pysentimiento* and the NSFW classifier. We encourage corporations and institutes to perform such audits. However, such self-audits will remain insufficient given the validity and credibility of self-audits are limited compared to those carried out by independent auditors [11]. Subsequently, we hope – perhaps through a coalition of the larger community, regulatory, and funding bodies – for the cultivation (through incentives) and creation of an ecology that allocates compute resources for independent auditors without access to institutional compute.
The risks of extrapolation: The  $\delta_{CI}$  of  $12.26\%$ , calculated in Section 4.1 above, has important consequences on estimating the number of low-quality samples that either ought to be filtered out or at least re-investigated on account of having failed the text-quality mechanism that we have proposed. At a fixed  $P_{threshold}$  of 0.5, the estimated confidence interval (CI) for the number of failed-quality-text samples in LAION-400M can be closer to 1.21 to 1.26 million samples. Suppose we use the upper bound of this CI as a benevolent estimate of HCR and extrapolate this to the LAION-2B-en dataset. It produces an upper-bound estimate of 7.1 million text-image pairs of hateful/aggressive/targeted content. However, as we have seen with our HCR analysis with actual 2B-en samples above in equation 3, the CI for this number is much higher, at 7.9 to 8.6 million samples. That is, our upper bound estimates based on the LAION-400M analysis turn out to be lower than the lower-bound estimate obtained by looking at LAION-2B-en samples by a margin of 0.865 million images (See Figure A1 in the Appendix). This massive margin illustrates that extrapolation using confidence intervals, especially on datasets with underlying graph structure and rich inter-node correlations such as the Common Crawl where the sample-level i.i.d. (Independent and Identically Distributed) assumptions may be invalidated, is imprudent and may lead to under-estimation of hateful/aggressive/targeted content.
The dangers of thresholding: As we show in Section 4.3, even a conservative NSFW threshold of 0.5 still keeps a certain amount of toxic and hateful content in alt-text, on top of the false negatives it may have for images. Given that for some LAION releases, discrete safety tags were provided based on predefined thresholds like this one, there will be consequences on downstream data quality (given that data users may filter the dataset based on the discrete values). Also, above and beyond false negatives, any predefined threshold assumes that a set amount of NSFW content is universally acceptable regardless of downstream data usage, which should be a decision for the data users. Therefore, we advocate that the raw values of multiple types of hateful content detection approaches (based on text and images) are provided with datasets, alongside evaluations of what kind of content different thresholds filter out. The final thresholding values would then be established by dataset users, who can make informed decisions based on the provided information – for instance, they may filter out more NSFW and toxic content if training a text-to-image model that illustrates children's stories, as opposed to one that generates stock photos. This would require more fine-grained evaluation than the one currently carried out for content filters, which does not analyze filtered content and uses accuracy as the sole metric for measuring success.
Appropriate and consistent metrics: Even though many of these datasets are created to train semantic search systems and image generation models that supposedly "democratize art-creation" for the general public (where a significant proportion are people of diverse gender, ethnicity, and race) the metrics used to check if progress is indeed being made by dataset scaling rarely reflects that diversity. While specific analyses are being made regarding the risk of biases and harm in ethics and safety subsections of reports and papers accompanying datasets, the metrics that supposedly measure these harms are rarely incorporated as part of the model check-pointing process. For instance, in the ALIGN paper [33], the dataset scaling ablation study focused only on two metrics: the MS-COCO zero-shot retrieval accuracy rates (I2T-R@1 and T2I-R@1) and the ImageNet K-Nearest-neighbor (KNN)  $R@1$  rates. In the BASIC model paper ablation study [52], the authors gauge the impact of increasing the dataset size from 1.7B to 6B by comparing the ImageNet-1k zero-shot top-1 accuracy. Finally, in the LAION-5B paper [62], the authors use the zero-shot top-1 classification accuracy metric, once again on the ImageNet family of datasets (with the distribution shift variants) and a bespoke VTAB+ benchmark spanning 35 classification tasks covering different computer vision datasets. This means that it is difficult for users to meaningfully compare the metrics and performance on any of these datasets to each other without rerunning analyses. Using standardized, meaningful metrics for measuring progress is vital to ensure that we are indeed measuring what we set out to measure and that results are comparable and reproducible.
Exploiting multimodality in toxic content detection: While there has been some work in multimodal hate speech in specific contexts such as memes [39, 69] as well as on topics such as cyber-bullying [37], we were not able to find existing multimodal approaches that were generic enough to be usable for creating multimodal datasets such as LAION. Our study intended to connect the two modality-specific approaches described in Section 4.3. However, there is much work to be done in connecting the type of content that is often the focus of image-based filtering (for instance, violence and sexually explicit content) as well as the focus of text-based filtering (for example, hate speech and aggression). It is crucial to understand the links between images and text better and to use them both when filtering datasets. Finally, no kind of filtering is, in itself, a bulletproof solution against harmful content. Without careful contextual analysis, filtering mechanisms can censor and erase marginalized languages, identities, and experiences [16].
Limitations of psentimiento: The analysis of hate content and scale in this paper was performed through the psentimiento library. This library is known to have high predictive accuracy and works in multiple languages [51]. However, as far as we know, there has been no research on the variability of psentimiento regarding linguistic variants of English or specific topics. To continue rigorous audits and improve multimodal-toxicity detection models, we encourage future work to use these various NLP models to investigate the impact of scale on hateful content and confirm if our finding of hate-scaling holds upon using different NLP models. This, for example, includes further investigation to establish if the alternative hate detection options, including open-source models (e.g. [71, 24, 27, 13]) produce different or complementary results to ours.
Legal and policy implications: The multimodal datasets we audited form a crucial backbone for modern machine learning systems, including generative models. These models are not purely intellectual exercises but are integrated into society, directly or indirectly impacting actual people. Subsequently, legal issues arise from multiple angles, including consent and rights of individuals in datasets, what should be in datasets and how they should be evaluated and maintained, and mechanisms for responsibility and accountability for problematic content in the dataset as well as the downstream effect on models trained on it. Closing access to datasets used for popular and impactful models and active obfuscation of information around these datasets present a significant obstacle to developing appropriate regulatory guidelines and guardrails. This audit study showed extensive evidence of the exacerbation of hateful content correlated with scale. We hope this work serves as an initial document for legal and policy experts alike that both demystifies multimodal datasets and illustrates the negative implications of scale.
We have conducted a dataset audit of two vision-language multimodal datasets, LAION-400M and LAION 2B-en, and presented evidence of hateful, aggressive, and targeted content, which was exacerbated by dataset size. This type of analysis was not previously carried out by the dataset creators, who focused on analyzing the characteristics of the images and not their accompanying alt-texts – our analysis has found that these two types of analyses are correlated but complementary. We conclude with some final remarks: we cannot stress enough the importance of open-source in audit endeavors such as ours since any quantitative and qualitative dataset exploration hinges upon access to the artifacts themselves. Providing access to datasets and models is essential to a healthy, thriving research community. However, we are saddened that an increasing number of machine learning organizations fail to do so.
Today's state-of-the-art vision-language multimodal models are trained on datasets like the ones we examined in the present article. These models are currently being deployed in real-world recommendation systems, information-retrieval systems, semantic search systems, and image captioning systems, despite them exhibiting biases and stereotypes [5, 42] that are neither well understood nor documented. Given that such failures can result in dire consequences on real people, often those at the margins of society, we implore the research community as well as those developing and deploying these systems to carry out due diligence with rigorous audits of these models and training datasets and take necessary actions, including refraining from use in high-stake scenarios.
Acknowledgements: Vishnu Naresh Boddeti was supported by the National Science Foundation under Grant No. (#2147116).
A The risks of extrapolation
(a) CI comparison between 400M & 2B-en
(b) "Excess-images" estimate for LAION2B-en
Figure A1: Binomial proportion confidence interval (CI) analysis to establish the extent of HCR underestimation upon using LAION400M statistics.
B NSFW Analysis
NSFW >= 0.5	NSFW <0.5
hateful	targeted	aggressive	hateful	targeted	aggressive
mean	0.749	0.356	0.104	0.021	0.013	0.011
min	0.501	0.008	0.013	0.00	0.013	0.005
max	0.988	0.961	0.894	1.00	0.499	0.513
Table A1: The mean, minimum, and maximum values for hateful, targeted, and aggressive content in our sample of LAION 2B-En, are divided according to the suggested NSFW threshold of 0.5.
C The origins of the dataset scaling laws: A cartoon sketch emerges
While attempting to unearth what this specific dataset scaling law was that the practitioners were so inspired by, we repeatedly encountered a certain cartoon sketch 'power-law' plot referred to in both personal exchanges as well as in surveys such as [72]. As it turns out, this cartoon sketch power-plot first appeared as Figure 6 in "Deep learning scaling is predictable, empirically" [29], a work that emerged out of Baidu research in 2017. The authors that first presented this plot posit that the generalization error associated with an ML model exhibits a three-phase behavior about its training dataset size. In the first phase, they state maps to the 'small data region,' where "models will struggle to learn from a small number of training samples" resulting in high generalization errors. The second phase (or the middle portion of learning curves), they claim, is the 'power-law region,' where the generalization error monotonically decreases with training dataset size (linear with application-specific slopes when plotted on a log-log scale). This phase stretches till we hit the point of the 'glass-ceiling' or 'unbreachable error floor' on account of factors such as model mismatch and unlabeled data (constituting the third phase). This, of course, has been further supplanted by the likes of the Chinchilla scaling laws (20 tokens per model parameter) [30] in the specialized context of LLMs.
D Blackbox non-reproducible empirical results
As for the black box non-reproducible empirical results that validated the dataset-scaling mandate and championed the scale-beats-noise narrative, we refer to the ALIGN paper [33] that emerged in 2021. In