SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wanrong Zhu\* Jack Hessel\* Anas Awadalla Samir Yitzhak Gadre Jesse Dodge Alex Fang Youngjae Yu Ludwig Schmidt\* William Yang Wang Yejin Choi\* University of California, Santa Barbara Allen Institute for Artificial Intelligence Paul G. Allen School of Computer Science, University of Washington Columbia University Yonsei University LAION https://github.com/allenai/mmc4
In-context vision and language models like Flamingo [2] support arbitrarily interleaved sequences of images and text as input. This format not only enables few-shot learning via interleaving independent supervised (image, text) examples, but also, more complex prompts involving interaction between images, e.g., "What do image A and image B have in common?" To support this interface, pretraining occurs over web corpora that similarly contain interleaved images+text. To date, however, large-scale data of this form have not been publicly available.
We release Multimodal C4 (mmc4), an augmentation of the popular text-only c4 corpus<sup>2</sup> with images interleaved. We use a linear assignment algorithm to place images into longer bodies of text using CLIP features [24], a process that we show outperforms alternatives.mmc4 spans everyday topics like cooking, travel, technology, etc. A manual inspection of a random sample of documents shows that a vast majority  $(88\%)$  of images are topically relevant, and that linear assignment frequently selects individual sentences specifically well-aligned with each image  $(80\%)$ . After filtering NSFW images, ads, etc., the resultingmmc4 corpus consists of 101.2M documents with 571M images interleaved in 43B English tokens.
In-context learning [7] enables sequence models to adapt to new tasks without any parameter updates. By interleaving a few supervised examples in a prompt, few-shot learning can be formatted as a next-token prediction task, i.e.,  $x_{1}, y_{1}, x_{2}, y_{2}, \ldots, x_{n}$  is input to predict  $\hat{y}_{n}$ . Some image+text models also support in-context learning via interleaving of images/text jointly. Prior experiments [2] suggest that performant multimodal in-context learning is dependent upon pretraining on similarly interleaved sequences of images and text (rather than single image/caption pairs). However, such a large-scale corpus has not been made publicly available.
To address this, we introduce Multimodal C4 (mmc4), a public, billion-scale image-text dataset consisting of interleaved image/text sequences.  ${}^{3}$  mmc4 is constructed from public webpages contained in the cleaned English c4 corpus. In addition to standard preprocessing steps like dedduplication,
*equal contribution; work partly conducted while Wanrong Zhu was an intern at AI2.
$^{2}$ https://www.tensorflow.org/datasets/catalog/c4
3mmc4's datasheet [15] is available here.
Table 1: Comparison of mmc4 with other interleaved image/text pretraining corpora. In addition to the full version of the dataset, we also release: 1) fewer-faces subsets, which aim to remove all depicted human faces; and 2) "core" subsets, result from more stringent filtering.
# images	# docs	# tokens	Public?
M3W (Flamingo) [2]	185M	43M	-	×
Interleaved training data for CM3 [1]	25M	61M	223B	×
Interleaved training data for KOSMOS-1 [17]	≤ 355M	71M	-	×
Multimodal C4 (mmc4)	571M	101.2M	43B	✓
Multimodal C4 fewer-faces (mmc4-ff)	375M	77.7M	33B	✓
mmc4 core (mmc4-core)	29.9M	7.3M	2.4B	✓
mmc4 core fewer-faces (mmc4-core-ff)	22.4M	5.5M	1.8B	✓
NSFW removal, etc., we place images into sequences of sentences by treating each document as an instance of a bipartite linear assignment problem, with images being assigned to sentences (under the constraint that each sentence is assigned at most one image). We show that applying CLIP ViT-L/14 [24] to estimate bipartite weights in a zero-shot fashion results in state-of-the-art performance on intra-document alignment benchmarks, and then apply this process to  $100\mathrm{M}+$  documents to construct mmc4. Apart from the full corpus, we have created two additional subsets: mmc4-ff, which removes images with detected faces, and mmc4-core, a more strictly filtered and downsized version of the corpus, serving as an initial corpus for developers.
We explore mmc4, showing that: 1) the text and images in the corpus span expected everyday topics like cooking and travel; 2) filters like NSFW/ad removal work with high accuracy; and 3) the resulting images are relevant to the associated documents, and often, appropriately aligned to the most-relevant individual sentence. We conclude by discussing initial use-cases of mmc4, including OpenFlamingo [3],<sup>4</sup> an open source version of Flamingo [2]. Initial ablations show that training on the sequences of mmc4 enables few-shot, in-context adaptation to image captioning datasets.
2 Related Dataset Work
Most million/billion-scale, public multimodal pretraining datasets consist of images paired with their literal descriptions, e.g., LAION-2B [26], CC-12M [8], YFCC100M [32]. However, literal description is only one of many ways images can relate to text on the web [21]. mmc4 aims to capture a broader range of these relationship types. Some web datasets collect multiple images for one text snippet (e.g., the Google Local Restaurant Reviews Dataset [36] with 4.4M images), or situate images in longer bodies of text (e.g., the Wikipedia-based Image Text Dataset [30] with 11.5M images), but do not directly cover multi-image/multi-sentence interleaving. Table 1 provides summary statistics of other large-scale interleaved pretraining datasets. mmc4 contains more images than prior non-public datasets. [5] highlight risks associated with web-scale multimodal data.
In addition to the detailed curation steps described in § 3 and the considerations for data release outlined in § 3.1, we are hopeful that the availability of mmc4 can facilitate a more transparent and critical examination of interleaved corpora compared to previous privately held training sets. Models trained on mmc4 inherit its risks; we selected the widely-adopted c4 corpus as a starting point in part because there are existing auditing efforts on the text-only corpus, see § 3 and [23] for more discussion of transparency.
3 Data Curation Process
Initial data collection. Multimodal C4 is an expansion of the text-only c4 dataset [25], which was created by taking the April 2019 snapshot from Common Crawl and applying several filters with the intention of retaining high-quality, natural English text. Each document in c4 consists of the text
<sup>4</sup>https://github.com/mlfoundations/open_flamingo  
<sup>5</sup>https://commoncrawl.org/
Figure 1: A T-SNE [34] projection of LDA [6] topic clusters from a random sample of 22K documents from mmc4; mmc4 spans a variety of everyday topics, e.g., cooking, technology travel, etc. For 6 selected topics, we also show a sample of most-central images to the topic according to CLIP ViT-L/14 [24].
scraped from one URL. The full c4 dataset has 365M documents and 156B tokens, covering many domains [12]; it was first used to train T5 [25]. We built the mmc4 dataset on top of c4 because: 1) c4 is a web-scale dataset widely adopted as a pre-training corpus [25, 29, 9, 33, 31]; 2) c4 is constructed from web pages, which frequently contain multimedia content like images, which makes it a suitable basis for extending to a multimodal sequence version; and 3) c4-en, $^{6}$  the specific underlying subset from which we construct mmc4 has already been processed with several data-cleaning steps (including English-language identification by langdetect $^{7}$  with at least 0.99 confidence; text dedduplication removing duplicate three-sentence spans + placeholder text like “lorem ipsum”; and removal of any document containing any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”). $^{8}$  See [25] for more information about the text-only c4. Importantly, by building on the popular text-only c4, prior text-only documentation efforts [12] can provide insight about potential biases and risks that could arise when training on our multimodal extension. We use the NLTK [4] sentence tokenizer to chunk each c4 document into a list of sentences.
Gathering images. We first retrieve the original webpages for each document in the c4-en dataset from the Common crawl version 2019-18, which is the default version for c4. Next, we extract the URLs for downloadable images from the raw WAT files. We restrict the image extension to either png/jpeg/jpg, and exclude image URLs that contain the following tokens: {logo, button, icon, plugin, widget}. We attempt to download from these URLs, and resize images to a maximum dimension of  $800\mathrm{px}$ . We eliminate any c4 documents that do not contain valid, downloadable images at the time of collection (mid-to-late 2022). The starting point after this step is 115M documents and 1.37B images.
De-duplication+small resolution. We next run duplicate image detection using opennota's findimagedupes<sup>9</sup> which uses phash<sup>10</sup> to identify visually similar images.<sup>11</sup> We keep only one copy of an image if multiple versions are detected within the same document. We also remove images with more than 10 duplicates in a sample of 60K images. We discard images with a width or height smaller than 150px; this accounts for many small icons, e.g., navigation buttons. We discard images with an aspect ratio of greater than 2 or less than 0.5; this accounts for many banner-like ads. In
$^{6}$ https://www.tensorflow.org/datasets/catalog/c4#c4en_default_config
7https://pypi.org/project/langdetect/
$^{8}$ https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words
<sup>9</sup>https://gitlab.com/opennota/findimagedupes
10http://www.phash.org/
<sup>11</sup>We use a more aggressive de-duplication threshold of 5 compared to the default library setting of 0; this removes roughly 10M additional images. While some duplicates survive this process, we qualitatively found a threshold of 5 to be an appropriate balance of false positives/negatives.
Table 2: Performance on single document image-text benchmarks from [16] (higher=better in all cases). Applying CLIP ViT-L/14 in a zero-shot fashion [24] produces better within-document alignments compared to prior methods which rely on fine-tuning.
MSCOCO
AUC p@1	Story-DII
AUC p@1	Story-SIS
AUC p@1	DII-Stress
AUC p@1	RQA
AUC p@1	DIY
AUC p@1
Random	49.7	5.0	49.4	19.5	50.0	19.4	50.0	2.0	49.4	17.8	49.8	6.3
Hessel et al. (2019) [16]	98.7	91.0	82.6	70.5	68.5	50.5	95.3	65.5	69.3	47.3	61.8	22.5
Li et al. (2021) [20]	99.3	97.6	85.5	77.2	70.2	53.1	-	-	-	-	-	-
CLIP ViT-L/14 (Zero Shot)	99.4	95.7	92.8	93.9	79.1	73.3	98.7	93.0	80.7	70.7	74.0	57.6
a manual sample of 3.7K images that survive this (and the NSFW) filter, 91 images  $(2.5\%)$  were identified as ads potentially unrelated to document contents. $^{12}$
Discarding NSFW images. We employ strict NSFW image filtering, using DataComp's [14] dataset2metadata<sup>13</sup> NSFW binary image classifier. The model is a 4-layer MLP, trained on the NSFW dataset introduced in LAION-2B [26]. This MLP takes as input image features extracted from OpenAI's CLIP ViT-L/14 [24] and achieves  $97.4\%$  accuracy on the NSFW test set. We run this classifier on each image and discard cases with a model-predicted NSFW probability over 0.1, which removes approximately  $10\%$  of remaining images. Because the data distribution of the classifier and mmc4 may be slightly different, we also conduct a spot check on images that are marked safe for work. In a manual sample of 3.7K images, we discovered zero NSFW images.
Aligning images and sentences. After collecting a set of images for each document, we now describe our intra-document alignment process to interleave the collected images with the sentences. Given that the scope of the images and sentences may be different – the image set is collected from the whole webpage, while the sentence list is subject to preprocessing within the c4 dataset and thus may not represent the complete content of the webpage – we did not rely on Document Object Model placements in the raw HTML to establish the alignment between images and sentences in each document. Instead, to associate each image with a sentence, we consider each document as an instance of a bipartite assignment problem [19, 16], and use CLIP ViT-L/14 compute pairwise similarities between all sentences/images on a single page. Then, we discard images without at least a 0.15 CLIP cosine similarity to at least one sentence in the document. Finally, we use [18] to compute a bipartite assignment of images to sentences, under the constraint that each sentence can only be assigned a single image.[14] Table 2 shows that this zero-shot application of CLIP ViT-L/14 for within-document matching surpasses prior competitive, fine-tuned methods on image-text alignment benchmarks from [16] (we also distribute the raw intra-document similarity matrices with mmc4 so alternate assignment methods can be explored). Figure 2 illustrates two example documents with the images interleaved before or after the assigned sentences.
3.1 Considerations for data release
mmc4 contains all images that survive the previously described filters. In addition to the full version of the corpus, we construct two additional types of subsets.
3.1.1 Fewer Faces (mmc4-ff)
Like the text-only version of c4, mmc4 may contain webpages with personal information that individuals had not explicitly intended to make available for model training. For an initial public release, we make a version of mmc4 available, mmc4-ff (ff stands for "fewer faces"); similar to some prior image dataset curation efforts [13, 11], mmc4-ff aims to remove images containing detected faces.
<sup>12</sup>The delineation between an "irrelevant advertisement" and a "relevant image" is inexact: for example, we discovered images advertising specific, small events, e.g., ones hosted by a fishing club within a city (this type of image was not included in this count). We later assess advertisement-ess in the context of the text of documents, rather than assessing based on the image alone.
<sup>13</sup>https://github.com/mlfoundations/dataset2metadata
For documents with more images than sentences, after assigning an image to each sentence, we assign according to max similarity.
Example#2: Interleaving the image after each corresponding text
Figure 2: Two example image+text documents from mmc4. Following Flamingo [2], during training, images can be interleaved before or after their assigned sentences. More example documents are given in Appendix D.2.
Removing images with detected faces. To detect faces at billion-scale with the intent of removing them from the dataset, we first run RetinaFace[10] over a sample of 60K images with the default settings. This detector runs at a high resolution and would be computationally prohibitive to run in full precision for the whole corpus; it produces detailed localization information about the coordinates of each face in each image (which we discard). Using an 80/20 train/test split, we train a cross-validated logistic regression over CLIP ViT-L/14 features to predict whether or not RetinaFace detects a face: this classifier is several orders of magnitude faster compared to RetinaFace. This approximation performs well: we choose a confidence cutoff that achieves  $95\%$  recall for the label "RetinaFace detected any face" over the test set while preserving  $65\%$  of the original images.
Manual sample-based face image risk assessment. We performed a manual verification of face removal. In a random sample of 912 images that pass all filters including the "no faces" filter, 23  $(2.5\%)$  images arguably contain a mostly-un-obscured human face. In most cases (12/23), faces are very low resolution, e.g., a  $150 \times 150$ px image of a crowd of people from a distance, where each face accounts for  $3 \times 4$  pixels, or are motion shots where the face is blurred. In one case, the face is Marilyn Monroe's as depicted in art on a wall. In 6 cases, there is a plausibly identifiable face depicted: in 2 cases, these are models posing in ads; in 1 case, there is a low resolution image of politicians giving a speech; in 2 cases, the faces are obscured; in 1 case, a passerby was caught in the background of a city photograph and could feasibly be individually identified. Overall: the rate of unobscured, high-resolution, identifiable faces in mmc4-ff is low.
3.1.2 Core (mmc4-core)
Early conversations with some model developers revealed a desire to work with a smaller subset of the corpus as an initial step. We thus additionally release core versions of mmc4 (and mmc4-ff), which apply even more stringent filtration criteria. The aim of core is to identify a "higher-precision" subset of documents that: 1) have a minimum/maximum number of sentences/images per document; 2) pass an even stricter dedduplication step; and 3) have a higher image-text similarity. Hyperparameters<sup>17</sup> are selected heuristically and are balanced to downsize the original corpus by an order of magnitude.
Example#1: Interleaving the image before each corresponding text
<sup>15</sup>As implemented by [27, 28] available from https://github.com/serengil/retinaface.
RetinaFace is not perfectly accurate, so selecting a more aggressive threshold (e.g.,  $99.99\%$ ) would not necessarily result in significantly fewer face-containing images removed.
<sup>17</sup>Min/max number of sentences: 4/40; min/max number of images 2/15; findimageduples applied with a threshold of 10; documents are required to have at least  $75\%$  of image assignments have CLIP ViT-L/14 similarity of greater than 25.
(a) CLIP sim is similar between lin. assignment  $+$  max. In red: percent of images remaining at various CLIP thresholds.
(b) Lin. assignment results in a higher percentage of sentences being associated with an image.
Figure 4: Using linear assignment results in comparable image-text similarities to max assignment, but the former spreads images much more evenly, e.g., the per-document mean percent of sentences with an associated image increases from  $22\%$  to  $34\%$ .
Figure 5: Distribution of images and sentences per document; the median document has 2 images/13 sentences. Documents with more sentences tend to have more images, but the correlation is weak (Spearman  $\rho = 10.3$ ).
4 Exploringmmc4
Statistics. Table 1 gives basic summary statistics of mmc4 (and fewer-faces/core subsets) compared to some other interleaved image/text corpora. Overall, the full version of mmc4 is larger than prior non-public datasets across axes like number of images/number of documents. In addition, the various subsets of the corpus offer trade-offs between privacy, image/text similarity thresholds, etc. Figure 5 gives details about the mean/median number of images/sentences in each document (mean/median # sent.=2.0/5.7; # im = 13.0/24.3) based on a random sample of 22K documents.
Sources of documents & images. We trace back the top-level domains of documents (webpages) and images to better understand the origins of contents in mmc4. Figure 6 presents the top-20 top-level domains that host the highest number of documents and images in mmc4. The distribution of document sources in mmc4 reveals a relatively uniform pattern, with 101.2M documents distributed across 6.0M unique domains. On average, each domain contains approximately 16.9 documents, with a median value of 2.0. The top  $10\%$  most frequently appeared domains account for  $77\%$  of all documents in mmc4. The documents are most commonly hosted on news media outlets (e.g., BBC, NY Times, Daily Express, Daily Mail), academic publication sites (e.g., Springer), online encyclopedias (e.g., Wikipedia), and e-commerce sites (e.g., iTunes, Etsy). Conversely, the sources of images in mmc4 exhibit a higher level of clustering. The 571.4M images are hosted on 4.9M domains, with each domain having an average of 116.0 images and a median value of 7.0 images. The top  $10\%$  most frequent domains are responsible for hosting  $89\%$  of all images. Images are most commonly hosted on blogs (e.g., Blogspot, WordPress), shopping sites (e.g., Amazon), cloud storage sites (e.g., AWS S3, Google storage), or general image hosting sites (e.g., Flickr, Imgur). More detailed lists of top document/image domains in mmc4 and mmc4-core can be found in Appendix C.
Image-text similarity. Figure 4 provides detail about the linear assignment process compared to a "max" assignment alternative, where each image is simply assigned to its maximally CLIP-similar sentence. The linear assignment process slightly decreases the average CLIP similarity between images/sentences (from  $24.5 \rightarrow 24.0$ ), but significantly more evenly "spreads" images throughout the documents: per-document, the mean percentage of sentences with an associated image rises from  $22\% \rightarrow 34\%$ .
Topic-based assessment. We ran LDA [6] as implemented by Mallet [22] on a random sample of 22K documents from mmc4 with  $k = 30$  topics. The resulting clusters span a broad set of topics like cooking, communities, travel, music, art, etc. Figure 1 shows some example LDA topic clusters. $^{19}$  In
18Multiple sub-sites may exist within a given domain (e.g., https://i0.wp.com and https://i1.wp.com). We replace specific patterns such as digits or location acronyms with \* to cluster these sub-sites together.
19A full list of topics and their frequencies according to the model is in Appendix B.
Figure 6: The top-20 most frequent top-level domains for documents and images in mmc4.
addition, we explore a sample of the images most associated with the corresponding topic, $^{20}$  finding that, in general, image topic clusters align with qualitative expectations.
Manual verification of image relevance+properties. We randomly sample 200 documents from mmc4 with the goal of assessing how relevant the images contained in the document are to the assigned sentences and to the document as a whole. Table 3 shows the results on the 836 images contained in the 200 documents.  $87.7\%$  of all examined images are topically related to the corresponding document, and  $80.4\%$  images are well-aligned to the assigned sentences within each document.[21] We also assessed several other factors, finding that: 1)  $28.3\%$  contain recognizable human faces; 2)  $1.6\%$  contain recognizable watermarks; 3)  $3.9\%$  are related to logos;[22] 4)  $3.2\%$  are related to advertisements; and 5)  $0.7\%$  are duplicated with other images in the same document. Appendix D.1 shows more discussion of images with watermarks, ads/logos, etc.
5 OpenFlamingo: An Early Application of mmc4
The first publicly available model to be trained on mmc4 is OpenFlamingo [3]. We run ablations on a small version of OpenFlamingo (3B: backbone = OPT-1.3B [37] language model and CLIP ViT-L/14 [24] vision model) to compare direct training on image captions (LAION-2B [26]) to the interleaved sequences of mmc4-core.[23] To flatten mmc4 documents to training sequences,[24] we: 1) sample a 256 token sub-sequence from each training document; 2) discard images with CLIP image-text similarity less than 20; 3) discard sequences that contain no images after filtering; 4) discard images if there are more than 5 in the resulting sequence.[25] As in [17] we randomly drop sequences with a single image to increase multi-image sequences in the sample.
20We compute the mean CLIP ViT-L/14 image vector for each topic by associating each image in a document the document's most common topic; then, we compute the mean image vector per topic. Finally, cosine similarity to this mean vector is used to identify the "most topically central" images per-topic.
21The alignment between an image and its assigned sentence is a qualitative criterion. We consider an image-sentence pair to be "well-aligned" when the visual elements of the image have a direct and relevant relationship with the text. This can include instances where the image depicts the context or content of the sentence, or where there is a plausible literal overlap between the text and the image, etc.