mOSCAR: A Large-scale Multilingual and Multimodal Document-Level Corpus

## Abstract

Abstract Multimodal Large Language Models (mLLMs) are trained on a large amount of text-image data. While most mLLMs are trained on caption-like data only, Alayrac et al. ( 2022 ) showed that additionally training them on interleaved sequences of text and images can lead to the emergence of in-context learning capabilities. However, the dataset they used, M3W, is not public and is only in English. There have been attempts to reproduce their results but the released datasets are English-only. In contrast, current multilingual and multimodal datasets are either composed of caption-like only or medium-scale or fully private data. This limits mLLM research for the 7,000 other languages spoken in the world. We therefore introduce mOSCAR, to the best of our knowledge the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality. We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model train on captioning data only. The model additionally trained on mOSCAR shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, confirming previous findings for English-only mLLMs. The dataset can be accessed here. 1 1 1 https://oscar-project.github.io/documentation/versions/mOSCAR/

## 1 Introduction

Multimodal large language models (mLLMs) are trained on large amounts of text-image data (Radford et al., ; Yu et al., ; Li et al., ; Wang et al., ; OpenAI, ; Gemini Team et al., ; Chameleon Team, ). The main paradigm until recently was to train a model from a large collection of web-crawled images and their captions (Li et al., ; Wang et al., ; Chen et al., ). Models such as Flamingo (Alayrac et al., ) challenged this paradigm by being additionally trained on interleaved sequences of text and images from web documents, showing state-of-the-art results on various tasks and in-context learning capabilities that are not present in models trained on caption-like data only. Additionally, McKinzie et al. () recently proved that including interleaved text-image data during training was necessary to get good few-shot learning performance. However, the datasets used to train mLLMs are either private (Alayrac et al., ), monolingual or multilingual but only medium-scale (Srinivasan et al., ). Some attempts have been made to reproduce these datasets (Zhu et al., ; Laurençon et al., ) but the resulting datasets are only available in English.

Few image-text datasets are multilingual and most of them are obtained by translating English caption-like datasets, such as multilingual Conceptual Captions (Sharma et al., ), into multiple languages using neural machine translation (NMT) systems (Surís et al., ; Maaz et al., ). This presents some drawbacks such as some languages still being poorly translated by current state-of-the-art NMT models (Liu et al., ; Costa-jussà et al., ) and some cultural subtleties inherent in each language not being fully conveyed. Some efforts have been conducted to collect large-scale multilingual image captioning datasets, such as LAION-5B (Schuhmann et al., ), but they are limited to caption data too, are relatively noisy and more importantly contain a non-negligible share of “not safe for work” (NSFW) content such as pædopornographic images (Schuhmann et al., ).

This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.(^2^22[https://commoncrawl.org/](https://commoncrawl.org/). The Common Crawl Foundation is a non-profit organization that crawls the web on a monthly basis.) Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative (Ortiz Suárez et al., ; Abadji et al., , ) and covers 315M documents in 163 languages, 214B tokens and 1.2B images. Figure 1 shows an example of a document, more can be found in Appendix A.3. We carry out extensive filtering to increase its safety and quality. To prove mOSCAR’s utility, we train a multilingual OpenFlamingo (Awadalla et al., ) from a Gemma-2B language model (Gemma Team et al., ) on a subset of mOSCAR and captioning data from LAION-400M (Schuhmann et al., ), recaptioned with BLIP (Li et al., ), filtered with CLIP (Radford et al., ) and translated with NLLB (Costa-jussà et al., ). We compare against a similar model trained on captioning data only and show we obtain a strong boost in few-shot learning, confirming previous findings for English (Alayrac et al., ; McKinzie et al., ; Laurençon et al., ). The dataset and models will be made publicly available.

Figure: Figure 1: Example of a French document from mOSCAR.
Refer to caption: x1.png

## 2 Related Work

##### Large-scale web-based datasets

Numerous datasets have been created by filtering web-crawled data. These include large-scale text-only datasets (Ortiz Suárez et al., ; Raffel et al., ; Wenzek et al., ; Gao et al., ; Abadji et al., ; Xue et al., ; Laurençon et al., ; Abadji et al., ; Penedo et al., ) and multimodal ones (Sharma et al., ; Changpinyo et al., ; Jia et al., ; Schuhmann et al., , ; Byeon et al., ; Laurençon et al., ; Zhu et al., ; Gadre et al., ). Even if these datasets are not as high quality as smaller and/or hand-crafted ones, they are now the standard to pretrain foundation models, as it has been shown that training bigger models on more data leads to better downstream performances (Brown et al., ; Hoffmann et al., ; Touvron et al., , ).

##### English image-text datasets

The first open-source image-text datasets were manually created, small-scale and English-only (Ordonez et al., ; Lin et al., ; Plummer et al., ; Krishna et al., ). Scaling up these datasets was an appealing solution to overcome limitations of previous image-text models; a few works (Sharma et al., ; Changpinyo et al., ) proposed to collect millions of image-text pairs from the web before filtering them with well-designed steps. Relaxing the filtering steps enabled the collection of more data and led to large-scale datasets to train image-text foundation models (Radford et al., ; Li et al., ; Schuhmann et al., , ; Byeon et al., ). However, these datasets generally contain caption-like image-text pairs only, and it is therefore difficult to observe in-context learning abilities similarly to text-only language models trained on raw documents (Raffel et al., ). Alayrac et al. () overcome this issue by training their model directly on documents with interleaved image-text data. While their results are promising, their M3W dataset is English-only and private. Recently, open-source efforts (Zhu et al., ; Laurençon et al., ) have been made to release a similar dataset but they are still monolingual.

##### Multilingual image-text datasets

Only a few image-text datasets are available in multiple languages. One of the first focused on collecting Google images from short queries based on word frequencies from Wikipedia pages in 98 languages (Hewitt et al., ). Later, Srinivasan et al. () proposed the WIT dataset, an image-text dataset composed of Wikipedia pages. Although of high quality, it is only medium-scale even for high-resource languages and there are fewer than 50k unique images for most languages. Another approach lies in bootstrapping multilingual and multimodal data from a model trained with English-only data (Mohammed et al., ). While effective for captioning, it is computationally expensive to implement in practice. Other multilingual image-text datasets exist but focus on captions only and are highly domain-specific (Kosar et al., ; Leong et al., ).

## 3 Dataset Creation Pipeline

### 3.1 Data collection

We collect mOSCAR from the Web ARchive Content (WARC) files of three 2023 Common Crawl dumps, processing them using the FastWARC library (Bevendorff et al., ). We remove documents smaller than 500 bytes (50% of the documents), as we find they are usually too small to be considered documents and tend to contain noisy text. We then navigate through the entire Document Object Model (DOM) tree with a depth first search algorithm and ChatNoir library (Bevendorff et al., ) to extract nodes of interests corresponding to specific HTML tags.

Following previous work, we extract text from the tags that usually contain the main content of web pages (we refer to them as DOM text nodes), i.e. <p>, <h*>, <title>, <description>, <ul>, <ol>, <aside>, <dl>, <dd>, <dt>. Similarly to (Laurençon et al., ), we choose to remove <table> content as most often it is irrelevant and difficult to render. We extract all <img> tags (we refer to them as DOM image nodes). We then remove documents with fewer than 3 text nodes (as they do not contain enough text) and more than 30 image nodes (as we found them to be too noisy).

### 3.2 Language identification

We identify the language of each document using the state-of-the-art open-LID language detector (Burchell et al., ), covering 201 languages. We apply open-LID to each DOM text node and keep the three most probable languages with their respective probabilities. The language of the document is then determined by summing over the probabilities of each language detected for each text segment, weighted by the number of characters in the segment(^3^33This is to avoid mis-assigning the language due to the presence of many short, non-informative DOM text nodes in the same language (e.g. “Cookies”, “Subscribe”, “Newsletter” etc.) and because language identification is generally less reliable for short segments.) and taking the language with the highest score.

### 3.3 Text-only filtering

We apply a series of filtering steps to the text content of each document independently of the images, with the aim of discarding poor quality documents and cleaning text as best as possible. We first filter at the text-node level and then at the whole document level, before running near-deduplication to keep unique text nodes within a document and unique documents in the dataset.

##### Text node filtering

We use a set of heuristics (see Appendix A.2) to extract as much human-generated content as possible while discarding noisy text related to ads and website functions (e.g. “Instagram”, “Facebook”). We then keep DOM text nodes with content over 10 bytes. This step, designed to improve the quality of extracted text, removes on average 55% of text nodes.

##### Document filtering

We mostly filter “not safe for work” (NSFW) content at the document level. We use an English regular expression to detect adult content, similar to the one used by the Université Toulouse 1 Capitole(^4^44[https://dsi.ut-capitole.fr/blacklists/index_en.php](https://dsi.ut-capitole.fr/blacklists/index_en.php)) and remove the entire document if there is a match with any of the DOM text nodes’ contents, removing on average 0.5% of documents (mostly English ones). We acknowledge that there is a high probability that this also discards safe content, e.g. we could remove content from certain communities who use some explicit words in a non-sexual way (Sap et al., ). However, we explicitly favour recall over precision to minimise the risk of unsafe content. We additionally remove documents containing fewer than five DOM text nodes and fewer than 300 characters after the previous filtering steps, removing 70.6% of documents.

##### Deduplication

We conduct several types of per-language deduplication at different levels, as this has been shown to improve training efficiency (Abbas et al., ). First, we keep unique documents only by removing exact duplicates at the document level. We also remove exact duplicates of text nodes within the same document (4% of text nodes) and near-duplicate text nodes (1% of text nodes) by computing the Levenshtein ratio (Levenshtein, ) between all text nodes within the same document and applying a threshold of 0.95. If near-duplicates are found, we keep the first one in the document. Finally, we conduct per language near-deduplication at the document level with MinHashLSH (Broder, ; Gionis et al., ) following Smith et al. (), removing on average 19% of documents:(^5^55With some disparity among languages as we found more duplicates for low- than high-resource languages.) we turn documents into hashing vectors, compute min hashes from these vectors and perform Locality Sensitive Hashing to remove duplicates(^6^66We performed this using the datasketch python library.) (see Appendix A.5 for more details).

### 3.4 Image-only filtering

We downloaded images from the URLs in DOM image nodes using a modified version of the img2dataset toolkit (Beaumont, ) that includes an antivirus scan and follows robots.txt instructions to respect the Robots Exclusion Protocol. We then apply a series of filtering steps, first removing images based on heuristics, and then applying multiple NSFW detection models to remove undesirable content. Finally, we conduct a set of deduplication steps.

##### Rule-based filters

Similarly to previous works (Schuhmann et al., ) and to avoid extracting low-resolution images and favicons, we keep images with a minimum height and width of 150 pixels. We restrict the aspect ratio to be between 3 and 1/3 (to remove banners), we remove images if their URLs contain the words “logo”, “banner”, “button”, “widget”, “icon” or “plugin” or if the image name from the URL matches “twitter”, “facebook” or “rss” (to remove logos). This step removes 13.6% of the URLs.
At this stage, we downloaded 2.5B images with an average success rate of 55%.

##### NSFW detection

We use multiple NSFW automatic models to remove as much unsafe content as possible. We first combine two NSFW detectors: nsfw-detector (), a 5-class classifier with a MobileNet (Howard et al., ) backbone fine-tuned on 60GB of annotated data and NudeNet,(^7^77[https://github.com/vladmandic/nudenet](https://github.com/vladmandic/nudenet)) an object detector trained to detect different types of nudity in images. We combined the two models as we found the first to be gender-biased while the second gives a large number of false positives for non-human images. Concretely, we consider an image an NSFW candidate if the sum of the probabilities for the classes ‘porn’ and ‘hentai’ is superior to 0.8 using nsfw-detector. We then tag the image as NSFW if one of the sensitive ‘exposed’ classes of NudeNet gets a probability superior to 0.5. We additionally use Safer by Thorn(^8^88[https://safer.io/](https://safer.io/)), a private pornography detector, and tag the image as NSFW if the probability of the class ‘pornography’ is superior to 0.8. If a document contains an image with an NSFW tag, we remove the entire document from the dataset, which removes 0.5% of images. We manually inspecting 1,000 images of the remaining data and found no NSFW content. We manually inspected 1,000 images of the removed content and found 63.4% of NSFW images.

##### CSAM content

Child Sexual Abuse Material (CSAM) is widespread on the internet and is therefore likely to be found in such a large-scale dataset crawled from the web. Removing CSAM is challenging as there is no training data nor open-source detection models available as these could be used in a harmful way. We again rely on Safer, a proprietary 3-class classifier trained to detect CSAM and pornography content from images. We tag the image as CSAM if the probability of the class CSAM is superior to 0.4 to favour recall over precision. As mentioned above, if a document contains an image with a CSAM tag, we remove it from the dataset. This step removes 0.07% of the images.

##### Deduplication

To avoid memorisation issues often seen in models trained on datasets with many duplicated images (Somepalli et al., ; Carlini et al., ; Webster et al., ; Somepalli et al., ), we perform deduplication at the image level. We first remove duplicate images within the same document by URL matching (removing 8.7% of URLs). We then compute a perceptual hash (pHash) for each image using the imagehash library(^9^99[https://github.com/JohannesBuchner/imagehash](https://github.com/JohannesBuchner/imagehash)) and remove images with the same pHash within the same document, keeping only the first occurrence. We also limit the number of times an image can appear in the dataset per-language to 10 using both URL matching and perceptual hashing (this removes 2.5% of images). We do this per-language and not across languages as having the same images in documents from different languages could encourage cross-lingual transfer.

### 3.5 Data decontamination

LLMs and mLLMs are trained on web-crawled data that can contain the benchmarks they are tested on (Dodge et al., ). As they are good at memorizing training data (Carlini et al., ), this data contamination is problematic.
We therefore discard all images with the same perceptual hash as any of the images from the evaluation benchmarks (and their training sets) we use (see Section 5.1). This step removes on average 126,016 images for high-resource languages (up to 300K images for English), 6,862 images for mid-resource languages and 45 images for low-resource languages.

### 3.6 Text-image joint filtering

Our aim is to obtain truly multimodal documents where all images are related to at least one of the text nodes in some way(^10^1010We do not limit ourselves to caption-like relation and instead allow all types of text-image relation.) and vice versa.
We choose to apply joint text-image filtering to discard images and/or text nodes that are irrelevant to the rest of the document (e.g. the case of ads and website functionalities).
To do this, we use NLLB-SIGLIP(^11^1111siglip-base-patch16-224 as vision encoder and nllb-distilled-600M as text encoder.) (Visheratin, ), a multilingual version of SIGLIP (Zhai et al., ) trained with the encoder of NLLB (Costa-jussà et al., ), which covers all mOSCAR languages.(^12^1212We use the open-clip (Ilharco et al., ) model version and the transformers (Wolf et al., ) library.)
We compute cosine similarity scores between all images and all paragraphs(^13^1313We refer to paragraph as the text content in a DOM text node.) within a same document. To remove irrelevant text nodes or images in a document, we mimic a text-image retrieval task, which means we avoid using arbitrary cosine similarity thresholds for each language and can reduce length biases and those in favour of caption-like paragraphs.
For each candidate pair we randomly sample 63 negative images and 63 negative similar-length paragraphs from the same language but other documents. We tag the text node (resp. image) as valid if the cosine similarity of the pair is among the top 8 of the text-to-image (resp. image-to-text) similarity scores computed with the candidate text node (resp. image) and all the negative images (resp. text nodes). This means that we tag the text node (resp. image) as valid if it has a significantly higher score than a score computed with a random image (resp. text) for at least one of the images (resp. text node) in the document. We then discard text nodes and images not tagged as valid (on average 35% of the DOM text nodes and 10% of the images within a document).
After this filtering step, we apply additional text-only filters to keep documents superior to 100 bytes.

## 4 Multimodal Open Super-large Crawled Aggregated coRpus (mOSCAR)

mOSCAR is extracted from three Common Crawl dumps from 2023. Due to computational constraints and in order to extract a maximum number of documents for low-resource languages, we extracted all languages from the first dump only. We removed the 6 most high-resource languages from the second dump and only extracted the languages with fewer than 1M documents for the last dump. Table 1 shows a distribution of the total number of languages and their number of documents.

To avoid data poisoning (Carlini et al., ), we release a hash (sha512) with each mOSCAR image.

**Table 1: Number of languages with at least $N$ documents**
| #documents | 10M | 5M | 1M | 500K | 200K | 50K | 10K | 5K | 1K |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #languages | 10 | 15 | 38 | 49 | 58 | 82 | 129 | 142 | 163 |

Figure: (a) Number of images
Refer to caption: x2.png

mOSCAR is composed of 315M documents (214B tokens, 1.2B images) from 163 languages. Figure 2 shows the distribution of images and tokens per document and their joint distribution. As shown in Figure 2(a), the mean and median number of images per document is 2 and 3.80.

### 4.1 Quality vs Diversity

While improving overall data quality, the filtering steps we applied (see Section 3) necessarily have a negative impact on diversity. We therefore study the trade-off between quality and diversity and compare against previously published, well-used datasets.

#### 4.1.1 Text content

##### Diversity

By contruction, mOSCAR is diverse in terms of number of languages, so we focus on the diversity of mOSCAR’s English documents and compare against mmc4 (Zhu et al., ), OBELICS (Laurençon et al., ) and the English subset of WIT (Srinivasan et al., ). We compute the Vendi score (Friedman and Dieng, ) on a set of SimCSE embeddings (Gao et al., ) with a RoBERTa encoder (Liu et al., ) to evaluate the content diversity. Since embedding-based diversity metrics target content diversity well but are less relevant for lexical diversity (Tevet and Berant, ), we measure lexical diversity via the distinct $n$-gram ratio (Li et al., ).

##### Comparison with other datasets

**Figure 3: Perplexity of 100K random documents from different datasets.**
|  | Vendi score | Dist. $n$-gram ratio |
| --- | --- | --- |
| mOSCAR | 69.05 ($\pm$ 0.14) | 0.472 ($\pm$ 0.002) |
| mmc4 | 67.93 ($\pm$ 0.12) | 0.494 ($\pm$ 0.002) |
| OBELICS | 58.49 ($\pm$ 0.09) | 0.488 ($\pm$ 0.001) |
| WIT | 73.30 ($\pm$ 0.09) | 0.530 ($\pm$ 0.001) |

For content diversity, we randomly sample 30M documents for mOSCAR, mmc4 and OBELICS and 3M documents for WIT and represent the documents by their SimCSE embedding. We compute the Vendi Score with cosine similarity on a randomly sampled subset of 65,536 documents. Table 2 shows that mOSCAR English content is more diverse than mmc4 and OBELICS but less diverse than WIT.
For lexical diversity, we randomly sample 3M documents for mOSCAR, mmc4, OBELICS and WIT and compute the distinct $n$-gram ratio on a subset of 8,192 documents for $n$ from 1 to 4. Table 2 shows that mOSCAR is slightly less lexically diverse than OBELICS and mmc4, while WIT is by far the most diverse.

##### Quality

To evaluate document quality, we focus on English documents and compute their perplexity using Gemma-2B (Gemma Team et al., ). Figure 3 shows the kernel density estimation of the distribution of the perplexity of 100K randomly sampled documents from different datasets: mOSCAR is comparable to mmc4 and WIT, while OBELICS appears to be the of the highest quality. mOSCAR is therefore comparable to other interleaved image-text dataset in terms of quality and diversity of its English subset. It is however more diverse than English-only datasets by its multilingual construction and more than 10 times larger than existing multilingual interleaved image-text datasets such as WIT.

#### 4.1.2 Image diversity

##### Comparison with other datasets

**(a) Comparison of different datasets.**
| mOSCAR | LAION-400M | WIT |
| --- | --- | --- |
| 55.74 ($\pm$ 0.16) | 67.59 ($\pm$ 0.16) | 36.14 ($\pm$ 0.08) |

We compute the Vendi Score on random samples of images for different datasets, comparing the images from English mOSCAR documents with those from Conceptual Captions (Changpinyo et al., ), LAION-400M (Schuhmann et al., ) and WIT (Srinivasan et al., ). We represent each image by its SigLIP(^14^1414We use siglip-base-patch16-224.) (Zhai et al., ) embedding and compute the Vendi score on batches of size 65,536 and a total of 1M images for each dataset. In Table 3(a), we notice that the set of images in mOSCAR documents are more diverse than images from WIT documents but less diverse than LAION-400M.

##### Multilingual diversity

We also compare the diversity of images from English documents and of images sampled from documents of any language (English included). We use multilingual SigLIP (Chen et al., ) trained on WebLI (Chen et al., ) to compute image embeddings used to get the Vendi score. We again use a batch of size 65,536 and a total of 3M images, and we do not sample multiple images from a same document. For the multilingual setting, we randomly sample 50 languages and an equal number of images for each language to build the batch. As we did not do any image deduplication across languages, we could expect to have less diversity in the multilingual setting. However, Table 3(b) shows that the set of images is on average more diverse when sampled from all documents than from English-only documents. This means that the distribution of images is not exactly the same across languages, potentially due to cultural differences.

## 5 Training a multilingual multimodal language model

We train a multilingual Flamingo-like model on mOSCAR. As adding captioning data to training data has been shown to improve zero-shot performance (McKinzie et al., ), we additionally train on LAION-400M, which we re-captioned using BLIP (Li et al., ), filtered with CLIP score (Radford et al., ) and translated using distilled NLLB-600M (Visheratin, ) following the proportion of languages found in mOSCAR. We use Gemma-2B (Gemma Team et al., ) as the underlying language model and we train the model on 35M mOSCAR documents and 70M randomly sampled image-text pairs. We also train a model on 300M image-text pairs as a comparison baseline. We additionally compare with OpenFlamingo-3B-MPT (Awadalla et al., ) as the translate-test baseline. The full list of languages for training and the implementation details can be found in Appendix A.5.

### 5.1 Evaluation setup

We evaluate the models using a broad set of image-text multilingual tasks and benchmarks. We use the IGLUE benchmark (Bugliarello et al., ) composed of XVNLI, MaRVL (Liu et al., ) to test reasoning, xGQA (Pfeiffer et al., ) to test visual question answering capabilities and xFlickr&CO (Young et al., ; Karpathy and Fei-Fei, ; Yoshikawa et al., ) for captioning. We also include Crossmodal-3600 (XM3600) (Thapliyal et al., ) and MaXM (Changpinyo et al., ) as they cover a broader range of languages. To test to what extent models trained on mOSCAR can perform zero-shot multimodal machine translation (MMT), we also test on Multi30K (Elliott et al., , ; Barrault et al., ) and CoMMuTE (Futeral et al., ). For captioning we compute the CideR (Vedantam et al., ) score and we tokenize references and model outputs with the Stanford Core NLP tokenizer for English and Stanza (Qi et al., ) tokenizers for other languages. To evaluate Multi30k, we compute BLEU (Papineni et al., ) score from Sacrebleu (Post, ) with 13a tokenization and default parameters. We use accuracy for CoMMuTE. More details can be found in Appendix A.5.3.

### 5.2 Results

Tables 4 and 5 show the average results across all languages. Full results are available in Appendix A.6. We notice that the multilingual OpenFlamingo trained additionally on mOSCAR gets better results than the model trained on captioning data only while having seen fewer image-text pairs during training. More importantly, when increasing the number of few-shot examples from 0 to 16, it sees gains of on average +8.19 points on VQA benchmarks and +16.07 CideR points on captioning benchmarks. In contrast, the model trained on text-image pairs only sees gains of +2.82 and +9.08 points respectively. In cross-modal machine translation, the model additionally trained on interleaved data is again far better than the one trained on just captioning data, which is not able to translate the Multi30k benchmark at all.(^15^1515Most of the time, the model is not able to follow the prompt and only outputs the end of sequence token.) Moreover, mOSCAR helps the model to learn to zero-shot disambiguate translations as shown by the improved average score on CoMMuTE (63.75) compared to the model trained on captions only (61.36).

Multilingual Open Flamingo trained on mOSCAR & text-image pairs is also better than OpenFlamingo 3B MPT evaluated on translate test benchmarks. However, we obtain the best results (except for MaXM) by evaluating our multilingual Open Flamingo on the translate-test benchmarks since the underlying language model (Gemma-2B) is far better in English than other languages. We also notice that all models struggle with reasoning classification tasks (MaRVL, XVNLI) where they obtain scores close to random guessing.

**Table 4: Results averaged over all languages.**
|  |  | xFlickR&CO | XM3600 | xGQA | MaXM | MaRVL | XVNLI | Multi30K | CoMMuTE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |  |  |  |
| Multi. OF | 0 | 19.07 | 8.73 | 25.08 | 19.64 | 49.77 | 33.01 | 22.70 | 63.75 |
| 4 | 34.32 | 20.59 | 31.90 | 23.90 | 49.67 | 36.07 | 22.79 | 63.65 |  |
| full | 8 | 36.77 | 22.15 | 33.9 | 24.41 | 49.72 | 37.16 | 23.21 | 63.00 |
|  | 16 | 37.63 | 22.24 | 35.71 | 25.38 | 49.73 | 35.36 | 23.48 | 62.77 |
| Multi. OF | 0 | 9.57 | 4.21 | 8.62 | 4.01 | 49.88 | 33.76 | 00.00 | 61.36 |
| 4 | 13.20 | 9.26 | 13.45 | 4.15 | 49.54 | 32.04 | 00.00 | 61.13 |  |
| cap. only | 8 | 18.00 | 10.35 | 12.82 | 4.88 | 49.65 | 33.71 | 00.01 | 60.90 |
|  | 16 | 19.87 | 12.07 | 13.37 | 4.89 | 49.79 | 32.70 | 00.74 | 60.25 |

**Table 5: Translate-test results averaged over languages.**
|  |  | xGQA | MaXM | MaRVL | XVNLI |
| --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |
| OF-3B MPT | 0 | 18.34 | 7.68 | 49.75 | 32.73 |
| 4 | 22.97 | 7.82 | 49.70 | 35.82 |  |
| 8 | 28.57 | 8.32 | 49.71 | 31.29 |  |
| 16 | 31.82 | 9.04 | 49.72 | 33.29 |  |
| Multi. OF | 0 | 30.16 | 10.06 | 49.93 | 34.66 |
| 4 | 35.55 | 9.89 | 48.99 | 36.10 |  |
| full | 8 | 36.78 | 10.12 | 50.54 | 39.69 |
|  | 16 | 37.75 | 11.49 | 49.57 | 37.97 |

We additionally compare results at different training steps, defined by the number of images seen during training. Figure 4 shows the difference of averaged scores between the model trained on all data and the model trained only on text-images pairs. We notice that the gap first decreases until 20M images seen and keep increasing over training at all training steps after that. Particularly, the gap is wider for few-shot learning.

## 6 Conclusion, Limitations and Societal Impacts

We introduce mOSCAR, a large-scale multilingual and multimodal dataset covering 163 languages and composed of 315M documents, 214B tokens and 1.2B images. We show that mOSCAR is of good quality, diverse and can be used to train a multilingual and multimodal LLM. We ensure that mOSCAR is as safe as possible by applying a series of filtering steps to remove NSFW content. We however did not conduct any toxicity analysis or evaluate its biases as this is challenging in a multilingual setting. As it is crawled from the internet, it is possible that mOSCAR reflects biases widespread on it. Nevertheless, by its multilingual nature, mOSCAR is a step towards the inclusion of more languages, cultures and people in accessing mLLMs.

## Acknowledgements

This work was granted access to the HPC resources
of IDRIS under the allocation 2024-AD011014232R1, 2023-AD011014232 and 2023-AD011012254 made by GENCI. It was
also partly funded by the last three authors’ chairs
in the PRAIRIE institute funded by the French national agency ANR as part of the “Investissements
d’avenir” programme under the reference ANR-19-
P3IA-0001. We deeply thanks the Jean-Zay support team. We also thanks Filip Šedivý for insightful discussions regarding the removal of CSAM, Thorn for having provided access to their CSAM detector, Zeeshan Khan for discussions regarding the training of the models and Victoria Le Fourner for having manually checked subsamples of NSFW images.

## Appendix A Appendix

### A.1 mOSCAR languages & statistics

**Table 6: Languages & Statistics**
| Languages |  | Statistics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Lang. name | Code | Family | Script |  | #documents | #images | #tokens |
| Acehnese | ace_Latn | Austronesian | Latin |  | 7,803 | 32,461 | 2,889,134 |
| Mesopotamian Arabic | acm_Arab | Afro-Asiatic | Arabic |  | 2,274 | 10,620 | 1,047,748 |
| Tunisian Arabic | aeb_Arab | Afro-Asiatic | Arabic |  | 7,640 | 41,570 | 2,715,187 |
| Afrikaans | afr_Latn | Indo-European | Latin |  | 54,895 | 247,774 | 39,956,585 |
| South Levantine Arabic | ajp_Arab | Afro-Asiatic | Arabic |  | 12,098 | 87,837 | 5,167,813 |
| Tosk Albanian | als_Latn | Indo-European | Latin |  | 861,678 | 2,569,164 | 452,737,251 |
| Amharic | amh_Ethi | Afro-Asiatic | Ge‘ez |  | 39,588 | 152,646 | 35,089,019 |
| North Levantine Arabic | apc_Arab | Afro-Asiatic | Arabic |  | 19,904 | 128,966 | 9,560,701 |
| Modern Standard Arabic | arb_Arab | Afro-Asiatic | Arabic |  | 3,936,851 | 15,126,931 | 3,401,919,964 |
| Najdi Arabic | ars_Arab | Afro-Asiatic | Arabic |  | 60,229 | 296,741 | 43,610,873 |
| Moroccan Arabic | ary_Arab | Afro-Asiatic | Arabic |  | 142,386 | 698,051 | 204,723,454 |
| Egyptian Arabic | arz_Arab | Afro-Asiatic | Arabic |  | 835,529 | 4,054,632 | 653,626,387 |
| Assamese | asm_Beng | Indo-European | Bengali |  | 3,948 | 9,210 | 640,390 |
| Asturian | ast_Latn | Indo-European | Latin |  | 165,745 | 962,723 | 37,547,944 |
| Awadhi | awa_Deva | Indo-European | Devanagari |  | 29,324 | 107,483 | 4,961,635 |
| Central Aymara | ayr_Latn | Aymaran | Latin |  | 27,384 | 151,889 | 5,148,970 |
| South Azerbaijani | azb_Arab | Turkic | Arabic |  | 8,274 | 38,233 | 5,256,693 |
| North Azerbaijani | azj_Latn | Turkic | Latin |  | 516,021 | 1,808,060 | 257,825,849 |
| Bashkir | bak_Cyrl | Turkic | Cyrillic |  | 4,532 | 17,174 | 3,038,766 |
| Bambara | bam_Latn | Manding | Latin |  | 7,674 | 39,190 | 1,243,332 |
| Balinese | ban_Latn | Austronesian | Latin |  | 1,886 | 11,266 | 542,015 |
| Belarusian | bel_Cyrl | Indo-European | Cyrillic |  | 63,309 | 287,539 | 72,976,520 |
| Bemba | bem_Latn | Atlantic–Congo | Latin |  | 1,096 | 7,479 | 1,340,471 |
| Bengali | ben_Beng | Indo-European | Bengali |  | 270,406 | 947,035 | 35,858,814 |
| Bhojpuri | bho_Deva | Indo-European | Devanagari |  | 6,366 | 28,131 | 875,463 |
| Banjar | bjn_Latn | Austronesian | Latin |  | 5,427 | 27,803 | 1,898,526 |
| Bosnian | bos_Latn | Indo-European | Latin |  | 1,960,599 | 7,633,049 | 1,255,000,505 |
| Buginese | bug_Latn | Austronesian | Latin |  | 3,312 | 18,648 | 588,678 |
| Bulgarian | bul_Cyrl | Indo-European | Cyrillic |  | 2,591,998 | 11,670,028 | 1,760,971,620 |
| Catalan | cat_Latn | Indo-European | Latin |  | 1,153,864 | 4,736,634 | 606,447,390 |
| Cebuano | ceb_Latn | Austronesian | Latin |  | 16,990 | 91,234 | 10,748,818 |
| Czech | ces_Latn | Indo-European | Latin |  | 3,918,837 | 13,291,309 | 2,823,172,996 |
| Central Kurdish | ckb_Arab | Indo-European | Arabic |  | 36,725 | 136,566 | 22,322,689 |
| Crimean Tatar | crh_Latn | Turkic | Latin |  | 6,376 | 24,124 | 1,742,727 |
| Welsh | cym_Latn | Indo-European | Latin |  | 40,408 | 165,897 | 27,748,345 |
| Danish | dan_Latn | Indo-European | Latin |  | 2,076,298 | 9,559,600 | 1,238,277,499 |
| German | deu_Latn | Indo-European | Latin |  | 20,662,696 | 87,976,200 | 8,544,986,218 |
| Southwestern Dinka | dik_Latn | Nilo-Saharan | Latin |  | 1,712 | 6,635 | 1,319,943 |
| Greek | ell_Grek | Indo-European | Greek |  | 4,916,081 | 15,209,058 | 2,923,201,041 |
| English | eng_Latn | Indo-European | Latin |  | 52,215,013 | 207,904,315 | 33,570,108,782 |
| Esperanto | epo_Latn | Artificial | Latin |  | 25,157 | 124,996 | 28,586,195 |
| Estonian | est_Latn | Uralic | Latin |  | 1,040,368 | 5,217,366 | 619,215,048 |
| Basque | eus_Latn | Isolate | Latin |  | 849,043 | 3,445,539 | 277,145,498 |
| Faroese | fao_Latn | Indo-European | Latin |  | 15,411 | 60,340 | 6,691,327 |
| Fijian | fij_Latn | Austronesian | Latin |  | 1,528 | 8,776 | 487,388 |
| Finnish | fin_Latn | Uralic | Latin |  | 2,396,033 | 10,365,333 | 1,781,044,864 |
| French | fra_Latn | Indo-European | Latin |  | 20,305,739 | 78,179,601 | 14,362,579,829 |
| Friulian | fur_Latn | Indo-European | Latin |  | 37,290 | 256,456 | 5,949,600 |
| Nigerian Fulfulde | fuv_Latn | Atlantic-Congo | Latin |  | 1,568 | 7,124 | 401,852 |
| West Central Oromo | gaz_Latn | Afro-Asiatic | Latin |  | 4,058 | 11,763 | 1,786,093 |
| Scottish Gaelic | gla_Latn | Indo-European | Latin |  | 29,710 | 153,249 | 14,605,090 |
| Irish | gle_Latn | Indo-European | Latin |  | 68,858 | 315,132 | 47,438,400 |
| Galician | glg_Latn | Indo-European | Latin |  | 518,973 | 2,381,475 | 217,063,180 |
| Guarani | grn_Latn | Tupian | Latin |  | 490,945 | 2,416,633 | 89,921,114 |
| Gujarati | guj_Gujr | Indo-European | Gujarati |  | 23,062 | 91,320 | 3,324,866 |
| Haitian Creole | hat_Latn | Indo-European | Latin |  | 257,745 | 1,570,699 | 62,847,106 |
| Hausa | hau_Latn | Afro-Asiatic | Latin |  | 25,364 | 104,934 | 13,089,932 |
| Hebrew | heb_Hebr | Afro-Asiatic | Hebrew |  | 1,109,591 | 4,766,483 | 893,327,320 |
| Hindi | hin_Deva | Indo-European | Devanagari |  | 579,430 | 1,830,667 | 122,558,353 |
| Chhattisgarhi | hne_Deva | Indo-European | Devanagari |  | 1,581 | 7,263 | 273,174 |
| Croatian | hrv_Latn | Indo-European | Latin |  | 1,719,617 | 8,425,510 | 1,010,674,096 |
| Hungarian | hun_Latn | Uralic | Latin |  | 3,534,506 | 15,390,083 | 2,831,715,050 |
| Armenian | hye_Armn | Indo-European | Armenian |  | 339,962 | 1,141,885 | 205,635,952 |
| Igbo | ibo_Latn | Atlantic-Congo | Latin |  | 11,529 | 68,049 | 8,701,070 |
| Ilocano | ilo_Latn | Austronesian | Latin |  | 78,872 | 523,195 | 8,116,113 |
| Indonesian | ind_Latn | Austronesian | Latin |  | 7,016,291 | 17,324,777 | 3,981,843,468 |
| Icelandic | isl_Latn | Indo-European | Latin |  | 244,676 | 1,027,465 | 137,015,973 |
| Italian | ita_Latn | Indo-European | Latin |  | 12,937,153 | 47,476,971 | 8,311,790,842 |
| Javanese | jav_Latn | Austronesian | Latin |  | 24,785 | 135,583 | 16,908,805 |
| Japanese | jpn_Jpan | Japonic | Kanji |  | 14,415,292 | 23,893,768 | 8,923,348,944 |
| Kabyle | kab_Latn | Afro-Asiatic | Latin |  | 18,508 | 106,730 | 4,079,553 |
| Kannada | kan_Knda | Dravidian | Kannada |  | 12,978 | 42,621 | 1,442,776 |
| Kashmiri | kas_Arab | Indo-European | Arabic |  | 3,109 | 11,408 | 5,731,910 |
| Georgian | kat_Geor | Kartvelian | Georgian |  | 354,436 | 1,304,281 | 275,223,026 |
| Kazakh | kaz_Cyrl | Turkic | Cyrillic |  | 252,242 | 732,648 | 140,049,214 |
| Halh Mongolian | khk_Cyrl | Mongolic | Cyrillic |  | 124,412 | 508,217 | 84,535,241 |
| Khmer | khm_Khmr | Austroasiatic | Kher |  | 24,495 | 122,243 | 3,043,925 |
| Kinyarwanda | kin_Latn | Atlantic-Congo | Latin |  | 30,401 | 172,201 | 12,049,616 |
| Kyrgyz | kir_Cyrl | Uralic | Cyrillic |  | 53,010 | 199,713 | 34,404,281 |
| Northern Kurdish | kmr_Latn | Indo-European | Latin |  | 39,262 | 164,666 | 23,834,960 |
| Korean | kor_Hang | Koreanic | Hanja |  | 2,614,089 | 13,563,283 | 2,006,080,705 |
| Lao | lao_Laoo | Kra-Dai | Lao |  | 50,611 | 208,768 | 31,029,380 |
| Ligurian | lij_Latn | Indo-European | Latin |  | 8,751 | 56,266 | 2,958,179 |
| Limburgish | lim_Latn | Indo-European | Latin |  | 189,547 | 1,076,047 | 42,534,327 |
| Lingala | lin_Latn | Atlantic-Congo | Latin |  | 24,614 | 152,132 | 4,053,459 |
| Lithuanian | lit_Latn | Indo-European | Latin |  | 1,688,811 | 8,869,443 | 1,161,476,040 |
| Lombard | lmo_Latn | Indo-European | Latin |  | 30,506 | 151,855 | 9,058,614 |
| Latgalian | ltg_Latn | Indo-European | Latin |  | 11,948 | 61,624 | 4,148,492 |
| Luxembourgish | ltz_Latn | Indo-European | Latin |  | 44,987 | 246,346 | 16,676,872 |
| Ganda | lug_Latn | Afro-Asiatic | Latin |  | 1,878 | 7,215 | 789,917 |
| Mizo | lus_Latn | Sino-Tibetan | Latin |  | 7,880 | 26,817 | 4,978,472 |
| Standard Latvian | lvs_Latn | Indo-European | Latin |  | 896,243 | 4,141,648 | 587,653,855 |
| Magahi | mag_Deva | Indo-European | Devanagari |  | 1,097 | 3,847 | 205,763 |
| Malayalam | mal_Mlym | Dravidian | Malayalam |  | 14,140 | 52,679 | 1,689,010 |
| Marathi | mar_Deva | Indo-European | Devanagari |  | 50,391 | 163,868 | 6,689,250 |
| Minangkabau | min_Latn | Austronesian | Latin |  | 9,341 | 35,309 | 1,256,931 |
| Macedonian | mkd_Cyrl | Indo-European | Cyrillic |  | 542,250 | 1,853,070 | 307,232,151 |
| Maltese | mlt_Latn | Afro-Asiatic | Latin |  | 120,888 | 709,242 | 36,097,957 |
| Maori | mri_Latn | Austronesian | Latin |  | 24,322 | 130,137 | 24,957,914 |
| Burmese | mya_Mymr | Sino-Tibetan | Mon |  | 8,144 | 44,188 | 539,527 |
| Dutch | nld_Latn | Indo-European | Latin |  | 17,096,727 | 65,606,013 | 9,670,041,731 |
| Norwegian Nynorsk | nno_Latn | Indo-European | Latin |  | 199,355 | 1,012,313 | 67,799,774 |
| Norwegian Bokmål | nob_Latn | Indo-European | Latin |  | 2,229,702 | 9,698,128 | 1,294,178,095 |
| Nepali | npi_Deva | Indo-European | Devanagari |  | 31,239 | 127,193 | 3,138,539 |
| Nyanja | nya_Latn | Atlantic-Congo | Latin |  | 12,047 | 67,192 | 8,596,769 |
| Occitan | oci_Latn | Indo-European | Latin |  | 164,852 | 671,881 | 59,309,549 |
| Odia | ory_Orya | Indo-European | Odia |  | 4,319 | 15,574 | 378,635 |
| Pangasinan | pag_Latn | Austronesian | Latin |  | 4,214 | 32,287 | 546,071 |
| Eastern Panjabi | pan_Guru | Indo-European | Gurmukhi |  | 11,497 | 46,168 | 1,887,991 |
| Papiamento | pap_Latn | Indo-European | Latin |  | 55,224 | 363,015 | 10,002,655 |
| Southern Pasto | pbt_Arab | Indo-European | Arabic |  | 32,604 | 110,807 | 29,170,322 |
| Western Persian | pes_Arab | Indo-European | Arabic |  | 7,048,946 | 25,200,571 | 6,210,479,015 |
| Plateau Malgasy | plt_Latn | Austronesian | Latin |  | 32,521 | 120,673 | 29,263,848 |
| Polish | pol_Latn | Indo-European | Latin |  | 14,549,605 | 60,639,244 | 11,104,144,109 |
| Portuguese | por_Latn | Indo-European | Latin |  | 8,145,664 | 26,530,423 | 4,760,063,083 |
| Dari | prs_Arab | Indo-European | Arabic |  | 515,041 | 2,589,859 | 517,053,967 |
| Ayacucho Quechua | quy_Latn | Quechuan | Latin |  | 1,578 | 11,817 | 362,690 |
| Romanian | ron_Latn | Indo-European | Latin |  | 5,180,171 | 17,964,048 | 3,548,291,261 |
| Rundi | run_Latn | Atlantic-Congo | Latin |  | 20,001 | 67,096 | 8,686,054 |
| Russian | rus_Cyrl | Indo-European | Cyrillic |  | 15,913,845 | 69,542,828 | 18,909,213,208 |
| Sango | sag_Latn | Atlantic-Congo | Latin |  | 2,124 | 13,556 | 454,455 |
| Sicilian | scn_Latn | Indo-European | Latin |  | 73,199 | 424,362 | 27,110,743 |
| Sinhala | sin_Sinh | Indo-European | Sinhalese |  | 58,767 | 221,183 | 14,270,972 |
| Slovak | slk_Latn | Indo-European | Latin |  | 3,008,599 | 15,067,234 | 1,963,804,563 |
| Slovenian | slv_Latn | Indo-European | Latin |  | 1,472,025 | 7,210,285 | 935,834,754 |
| Samoan | smo_Latn | Austronesian | Latin |  | 12,346 | 71,359 | 14,954,824 |
| Shona | sna_Latn | Atlantic-Congo | Latin |  | 12,698 | 68,782 | 6,112,600 |
| Sindhi | snd_Arab | Indo-European | Arabic |  | 21,095 | 74,289 | 17,647,825 |
| Somali | som_Latn | Afro-Asiatic | Latin |  | 77,343 | 301,429 | 34,554,975 |
| Southern Sotho | sot_Latn | Atlantic-Congo | Latin |  | 7,718 | 43,146 | 6,156,450 |
| Spanish | spa_Latn | Indo-European | Latin |  | 22,713,366 | 78,361,087 | 14,616,773,475 |
| Sardinian | srd_Latn | Indo-European | Latin |  | 675,539 | 4,059,493 | 106,159,957 |
| Serbian | srp_Cyrl | Indo-European | Cyrillic |  | 604,557 | 2,286,171 | 401,223,741 |
| Sundanese | sun_Latn | Austronesian | Latin |  | 44,310 | 236,025 | 13,627,832 |
| Swedish | swe_Latn | Indo-European | Latin |  | 3,302,730 | 10,860,518 | 1,779,284,152 |
| Swahili | swh_Latn | Atlantic-Congo | Latin |  | 137,134 | 593,418 | 59,454,896 |
| Silesian | szl_Latn | Indo-European | Latin |  | 23,535 | 132,459 | 5,996,972 |
| Tamil | tam_Taml | Dravidian | Tamil |  | 36,196 | 167,669 | 4,834,946 |
| Tatar | tat_Cyrl | Turkic | Cyrillic |  | 37,188 | 143,842 | 22,831,350 |
| Telugu | tel_Telu | Dravidian | Telugu |  | 22,974 | 81,033 | 2,273,772 |
| Tajik | tgk_Cyrl | Turkic | Cyrillic |  | 125,236 | 417,591 | 90,503,778 |
| Tagalog | tgl_Latn | Austronesian | Latin |  | 151,437 | 673,814 | 97,708,639 |
| Thai | tha_Thai | Kra-Dai | Thai |  | 2,983,837 | 11,621,786 | 2,839,211,104 |
| Tigrinya | tir_Ethi | Afro-Asiatic | Ge‘ez |  | 2,657 | 8,707 | 1,725,422 |
| Tok Pisin | tpi_Latn | Indo-European | Latin |  | 5,063 | 35,169 | 460,853 |
| Turkmen | tuk_Latn | Turkic | Latin |  | 13,024 | 57,354 | 9,766,999 |
| Turkish | tur_Latn | Turkic | Latin |  | 4,478,700 | 12,401,091 | 2,394,669,068 |
| Twi | twi_Latn | Atlantic-Congo | Latin |  | 3,305 | 13,634 | 495,220 |
| Uyghur | uig_Arab | Turkic | Arabic |  | 10,713 | 41,709 | 6,785,318 |
| Ukrainian | ukr_Cyrl | Indo-European | Cyrillic |  | 2,721,424 | 10,929,796 | 1,928,351,595 |
| Urdu | urd_Arab | Indo-European | Arabic |  | 407,098 | 1,239,125 | 242,007,283 |
| Northern Uzbek | uzn_Latn | Turkic | Latin |  | 156,632 | 798,155 | 89,022,562 |
| Venetian | vec_Latn | Indo-European | Latin |  | 330,611 | 1,830,777 | 71,077,531 |
| Vietnamese | vie_Latn | Viet-Muong | Latin |  | 12,621,521 | 47,411,488 | 11,616,191,199 |
| Wolof | wol_Latn | Atlantic-Congo | Latin |  | 4,658 | 20,380 | 1,596,432 |
| Xhosa | xho_Latn | Atlantic-Congo | Latin |  | 25,950 | 142,387 | 15,809,823 |
| Eastern Yiddish | ydd_Hebr | Indo-European | Hebrew |  | 12,486 | 57,510 | 17,369,727 |
| Yoruba | yor_Latn | Atlantic-Congo | Latin |  | 56,700 | 286,933 | 32,614,558 |
| Yue Chinese | yue_Hant | Sino-Tibetan | Hant |  | 33,671 | 203,513 | 24,172,441 |
| Chinese (Simplified) | zho_Hans | Sino-Tibetan | Hanzi |  | 9,861,262 | 36,152,754 | 8,078,842,701 |
| Chinese (Traditional) | zho_Hant | Sino-Tibetan | Hant |  | 3,967,966 | 16,307,258 | 2,962,854,441 |
| Standard Malay | zsm_Latn | Austronesian | Latin |  | 1,179,744 | 5,488,632 | 432,667,199 |
| Zulu | zul_Latn | Atlantic-Congo | Latin |  | 30,717 | 156,639 | 11,345,288 |

### A.2 Heuristics to increase the quality of documents

We use a set of heuristics to improve the quality of the documents by discarding some text nodes. We first consider text nodes to be written in Latin scripts if more than 50% of the characters are Latin. In detail, we discard the text node if:

- 1.
It is empty.
- 2.
It contains fewer than 5 bytes for Latin scripts and fewer than 15 bytes for non-Latin scripts.
- 3.
More than 30% of the characters are digits.
- 4.
It contains more than one date.
- 5.
It contains the sequence “lorem ipsum”.
- 6.
The ratio of non-alphabetic characters is superior to 0.33.
- 7.
The symbols ‘{’ or ’‘}’ are in the text.
- 8.
The symbols ‘$\geq$’, ‘$\leq$’, ‘>’ or ‘<’ are more than 2 times in the text.
- 9.
“Follow us”, “javascript”, “copyright” or “©” are in the text.
- 10.
The ratio of capitalized letters is superior to 0.2.
- 11.
The text exactly matches with “comment”, “facebook”, “instagram”, “twitter”, “rss”, “newsletter”, “share” or “follow us”.
- 12.
A character is more than 33% of the total number of characters in the string.

We then also apply some filters to clean the text as much as possible:

- 1.
Remove URLs from all documents.
- 2.
Normalize consecutive special characters (‘\t’, ‘\n’, ‘#’, ‘/’, ‘$’, ‘)’, ‘(’, ‘[’, ‘]’, ‘!’, ‘?’, ‘%’, ‘<’, ‘>’) to keep only one.

Following previous steps, we keep the text node if it is superior to 5 bytes and we keep the final document if it is superior to 100 bytes.

### A.3 Examples of documents

Figure: Figure 5: Example of a French document.
Refer to caption: x7.png

Figure: Figure 6: Example of a Japanese document.
Refer to caption: x8.png

Figure: Figure 8: Example of an Italian document.
Refer to caption: x10.png

Figure: Figure 9: Example of a Khmer document.
Refer to caption: x11.png

Figure: Figure 10: Example of an Urdu document.
Refer to caption: x12.png

### A.4 Text-Image similarity and DOM Tree

As we rely on the DOM Tree to build the documents and the order of appearance of the nodes could differ from HTML rendering, we attempt to assess to what extent it is a relevant way of constructing a multimodal document. To do so, we rely on the results of the text-image joint filtering step where we compute the ranks of relevant text nodes (resp images) for each image. We plot the distribution of the closest most relevant node for each modality in Figures 11(a) and 11(b). We notice that the most relevant node to either a text node or an image is their closest node in the DOM tree. The cumulative distribution function of the distribution of the closest node reaches 25% for nodes positioned between -5 and 5, which confirms the relevance of using the DOM tree to represent a document.

Figure: (a) Relative position in the document of relevant text nodes with respect to images.
Refer to caption: x13.png

### A.5 Implementation details

#### A.5.1 Text deduplication parameters

Following previous work, we near-deduplicate documents using MinHashLSH. We first vectorize the documents using HashingVectorizer from scikit-learn with 2,097,152 features computed on 4-grams and 5-grams within word boundaries. We then compute MinHashes from those vectors with 256 permutations and we finally run Locality Sensitive Hashing with a threshold Jaccard Similarity of 0.8 for finding near-duplicates.

#### A.5.2 Training implementation details

We train multilingual OpenFlamingo on mOSCAR and multilingual text-image pairs. We use a batch of size 64 for mOSCAR and 128 for captioning data, limiting the number of tokens to 256 for mOSCAR and 32 for captioning data. Similarly to Flamingo and OpenFlamingo, text tokens can only attend to the previous image in the sequence. To increase diversity in the training batch, we randomly reject 2/3 of the documents if they contain only one image. We limit the maximum number of images in a sequence to 8. We randomly sample 8 languages per batch and upsample low-resource languages. We train multilingual OpenFlamingo on 43 languages covering all the languages of the benchmarks we evaluate the models on (see Section A.5.3).

We use Gemma-2B as the underlying language model behind multilingual OpenFlamingo and CLIP ViT-L-14 as the image encoder. We add a cross-attention layer after each decoder layer. Following OpenFlamingo, we add the two special tokens <image> and
, whose embeddings were trained. Only the Perceiver Resampler, cross-attention layers and these two embeddings were trained; everything else remained frozen. During training, we apply a factor of 0.2 for the captioning data loss function.

We train the model using the Adam optimizer and a maximum learning rate of 1e-4. We use a constant learning rate scheduler with 1875 warm-up steps. We use 4 accumulation gradient steps to have an effective batch of size 256 for mOSCAR and 512 for captioning data. We train the model on 35M documents and 70M image-text pairs on 8 Nvidia A100 for 120h.

#### A.5.3 Evaluation details

**Table 7: Overview of the benchmarks used to evaluate our multilingual OpenFlamingo.**
|  | Metric | #examples | Languages |
| --- | --- | --- | --- |
| xFlickr&CO | CideR | 2,000 | Chinese, English, German, Indonesian, Japanese, Russian, Spanish, Turkish |
| XM3600 | CideR | 3,600 | Arabic, Czech, Danish, German, Greek, English, Spanish, Farsi, |
| Finnish, French, Hebrew, Hindi, Croatian, Hungarian, Indonesian, Italian, |  |  |  |
| Japanese, Korean, Dutch, Norwegian, Poland, Portuguese, Romanian, |  |  |  |
| Russian, Swedish, Telugu, Thai, Turkish, Ukrainian, Vietnamese, Chinese |  |  |  |
| xGQA | Accuracy | 9,666 | Bengali, German, English, Indonesian, Korean, Portuguese, Russian, Chinese |
| MaXM | Accuracy | $\sim$ 170 | English, French, Hindi, Hebrew, Romanian, Thai, Chinese |
| MaRVL | Accuracy | $\sim$ 1,150 | Indonesian, Swahili, Tamil, Turkish, Chinese |
| XVNLI | Accuracy | 1,164 | English, Arabic, Spanish, French, Russian |
| Multi30k | BLEU | 1,000 | French, German, Czech |
| CoMMuTE | Accuracy | 310 | Czech, French, German |

We evaluate on a set of eight benchmarks: xFlickr&CO, XM3600, xGQA, MaXM, MaRVL, XVNLI, Multi30k (Test2016 subset) and CoMMuTE; covering 5 different tasks and 43 languages. Details about the languages, the number of examples and the metric used can be found in Table 7. We used the translate-test(^16^1616Benchmark automatically translated into English.) samples provided by the authors of the benchmarks if available. No translate test samples were provided for MaXM, so we translated the test set using the NLLB-600M distilled model. As no training set was available for MaXM, we use the few-shot examples from xGQA. Since we use Stanza tokenizers, we could not evaluate on all languages from XM3600 as 3 of them were not available. Filipino was also not into the list of mOSCAR languages, so we skip this language during evaluation. The CoMMuTE evaluation set involves choosing between two different translations of a same source text (one correct and one incorrect depending on an image provided to disambiguate the text). We use the lowest perplexity between the two translations as the model’s prediction. We also use Multi30k training set as few-shot examples.

##### Prompting

Following previous works, the zero-shot setting is composed of two few-shot examples without providing the images. The prompts we use for the different tasks are as follows:(^17^1717We show the prompts we used with one context example.)

For captioning tasks, we use the prompt:

“<image>Output:[Caption]
<image>Output:”,

where [Caption] is replaced by the caption.

For visual question answering tasks, we use the prompt:

“<image>Question: [Question] Short Answer: [Answer] 
<image>Question: [Question] Short Answer:”,

where [Question] and [Answer] are replaced by the question and the answer respectively.

For multimodal machine translation tasks, we use the prompt:

“<image>Sentence:‘[Caption]’. Translation: [Translation] 
<image>Output:”,

where [Caption] is replaced by the sentence to translate and [Translation] is replaced by its translation.

For MaRVL, we use the prompt:

“<image> ‘[Statement]’. True of False? [Answer]
<image> ‘[Statement]’. True of False?”,

where [Statement] is replaced by the statement and [Answer] by the answer. We also concatenate the left and right image into a single image.

For XVNLI, we use the prompt:

“<image> ‘[Statement1]’ - ‘[Statement2]’. entailment, neutral or contradiction? Output: [Answer]
<image> ‘[Statement1]’ - ‘[Statement2]’. entailment, neutral or contradiction? Output:”,

where [Statement1], [Statement2] and [Answer] are replaced by XVNLI test data.

### A.6 Detailed results

**Table 8: Captioning results (CideR scores) on xFlickr&CO. Bold is best result.**
|  |  | De | En | Es | Id | Ja | Ru | Tr | Zh |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |  |  |  |
|  | 0 | 28.87 | 33.42 | 16.35 | 33.26 | 04.33 | 13.99 | 03.68 | 18.65 |
| Multilingual OF | 4 | 53.99 | 46.53 | 35.87 | 48.85 | 12.44 | 31.88 | 11.35 | 33.64 |
| mOSCAR + caps. | 8 | 56.05 | 52.99 | 37.28 | 52.51 | 15.13 | 32.76 | 11.69 | 35.78 |
|  | 16 | 55.44 | 56.78 | 37.72 | 49.06 | 17.10 | 35.43 | 12.72 | 36.82 |
|  | 0 | 16.72 | 24.57 | 03.80 | 10.82 | 02.82 | 08.20 | 02.79 | 06.82 |
| Multilingual OF | 4 | 21.10 | 31.05 | 07.52 | 09.63 | 03.84 | 13.21 | 07.01 | 12.20 |
| captions only | 8 | 32.56 | 35.73 | 13.35 | 15.85 | 05.96 | 18.13 | 06.97 | 15.47 |
|  | 16 | 29.86 | 40.57 | 13.75 | 23.83 | 06.92 | 20.40 | 07.90 | 15.73 |

**Table 9: Captioning results (CideR scores) on XM3600. Bold is best result.**
|  |  | Ar | Cs | Da | De | El | En | Es | Fa | Fi | Fr | He |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |  |  |  |  |  |  |
|  | 0 | 06.18 | 02.02 | 11.06 | 09.12 | 00.90 | 43.49 | 18.01 | 10.41 | 01.30 | 20.61 | 03.21 |
| Multi. OF | 4 | 21.55 | 05.93 | 28.21 | 21.97 | 03.38 | 74.71 | 36.93 | 26.27 | 06.48 | 39.41 | 10.34 |
| full | 8 | 23.15 | 06.23 | 31.33 | 24.53 | 03.47 | 75.29 | 37.58 | 28.84 | 07.65 | 43.40 | 10.83 |
|  | 16 | 23.80 | 06.86 | 31.92 | 24.92 | 03.59 | 76.24 | 38.39 | 26.78 | 07.61 | 43.26 | 11.78 |
|  | 0 | 02.24 | 00.97 | 06.42 | 06.46 | 03.68 | 10.02 | 09.32 | 04.95 | 01.14 | 16.15 | 00.78 |
| Multi. OF | 4 | 05.36 | 01.36 | 13.11 | 11.82 | 07.78 | 35.52 | 19.96 | 09.62 | 01.86 | 22.48 | 02.29 |
| Caps only | 8 | 06.76 | 01.40 | 15.29 | 14.39 | 07.21 | 37.28 | 21.90 | 12.19 | 02.08 | 23.27 | 01.71 |
|  | 16 | 06.25 | 02.29 | 17.96 | 15.11 | 07.64 | 48.03 | 25.39 | 09.21 | 02.10 | 30.16 | 02.72 |
|  |  | Hi | Hr | Hu | Id | It | Ja | Ko | Nl | No | Pl | Pt |
|  | #shots |  |  |  |  |  |  |  |  |  |  |  |
|  | 0 | 02.80 | 01.47 | 01.85 | 09.98 | 11.15 | 02.07 | 01.67 | 18.97 | 09.63 | 04.32 | 15.49 |
| Multi. OF | 4 | 10.62 | 07.48 | 05.51 | 22.63 | 27.88 | 16.87 | 09.24 | 42.30 | 22.35 | 14.20 | 29.35 |
| full | 8 | 11.12 | 07.54 | 05.91 | 25.39 | 29.34 | 19.35 | 09.99 | 46.79 | 23.54 | 15.43 | 32.69 |
|  | 16 | 12.18 | 07.71 | 06.03 | 23.89 | 29.17 | 18.84 | 09.75 | 46.95 | 23.79 | 15.93 | 31.98 |
|  | 0 | 02.29 | 00.97 | 03.51 | 02.98 | 07.96 | 01.85 | 01.05 | 04.88 | 05.78 | 00.92 | 09.79 |
| Multi. OF | 4 | 04.57 | 01.72 | 07.57 | 06.39 | 16.23 | 03.47 | 04.33 | 11.26 | 11.99 | 01.16 | 15.93 |
| Caps only | 8 | 05.94 | 02.17 | 07.83 | 09.93 | 15.40 | 07.93 | 05.34 | 11.87 | 13.79 | 01.38 | 17.50 |
|  | 16 | 06.36 | 02.42 | 09.55 | 11.77 | 17.43 | 10.44 | 06.03 | 12.98 | 14.65 | 01.28 | 20.32 |
|  |  | Ro | Ru | Sv | Te | Th | Tr | Uk | Vi | Zh |  |  |
|  | #shots |  |  |  |  |  |  |  |  |  |  |  |
|  | 0 | 02.19 | 06.80 | 11.45 | 00.87 | 08.36 | 03.11 | 03.04 | 21.99 | 07.19 |  |  |
| Multi. OF | 4 | 05.63 | 20.50 | 25.59 | 02.29 | 21.66 | 10.87 | 10.43 | 38.43 | 19.34 |  |  |
| full | 8 | 06.04 | 23.11 | 26.44 | 03.06 | 25.19 | 12.95 | 10.56 | 39.75 | 20.18 |  |  |
|  | 16 | 06.43 | 21.15 | 26.51 | 03.42 | 25.31 | 13.57 | 10.45 | 40.85 | 20.46 |  |  |
|  | 0 | 02.24 | 01.93 | 04.55 | 00.67 | 02.34 | 02.68 | 00.80 | 08.55 | 02.70 |  |  |
| Multi. OF | 4 | 05.35 | 06.29 | 15.66 | 00.77 | 07.21 | 05.94 | 01.76 | 20.69 | 07.80 |  |  |
| Caps only | 8 | 05.18 | 07.58 | 14.01 | 01.00 | 06.81 | 08.90 | 02.73 | 23.05 | 08.99 |  |  |
|  | 16 | 05.06 | 09.06 | 20.60 | 01.18 | 08.35 | 10.25 | 03.47 | 25.16 | 11.05 |  |  |

**Table 10: VQA results on xGQA. Bold is best result.**
|  |  | Bn | De | En | Id | Ko | Pt | Ru | Zh |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |  |  |  |
|  | 0 | 20.99 | 22.57 | 32.60 | 22.45 | 26.44 | 23.54 | 25.31 | 26.77 |
| Multilingual OF | 4 | 25.72 | 32.71 | 38.02 | 30.74 | 31.49 | 31.53 | 31.75 | 33.26 |
| mOSCAR + caps. | 8 | 26.87 | 35.60 | 39.16 | 34.25 | 32.25 | 34.93 | 33.59 | 34.55 |
|  | 16 | 29.11 | 37.01 | 40.64 | 36.37 | 34.20 | 37.15 | 34.87 | 36.29 |
|  | 0 | 10.54 | 06.51 | 10.43 | 07.74 | 07.50 | 07.79 | 08.62 | 09.84 |
| Multilingual OF | 4 | 12.54 | 11.90 | 15.78 | 13.95 | 13.70 | 12.01 | 12.73 | 15.03 |
| captions only | 8 | 11.62 | 11.70 | 17.29 | 13.86 | 12.85 | 11.60 | 12.65 | 15.35 |
|  | 16 | 09.77 | 11.86 | 18.37 | 13.24 | 12.48 | 11.25 | 11.24 | 14.33 |
| Translate Test |  |  |  |  |  |  |  |  |  |
| OF-3B MPT | 0 | 18.64 | 18.67 | - | 18.36 | 17.54 | 19.21 | 18.88 | 17.11 |
| 4 | 23.23 | 23.40 | - | 22.95 | 22.46 | 23.52 | 22.41 | 22.85 |  |
| 8 | 28.22 | 29.44 | - | 28.21 | 27.67 | 29.58 | 28.21 | 28.63 |  |
| 16 | 31.31 | 32.58 | - | 31.82 | 31.42 | 32.74 | 31.62 | 31.22 |  |
|  | 0 | 30.41 | 32.1 | - | 29.35 | 29.99 | 31.39 | 29.06 | 28.81 |
| Multilingual OF | 4 | 34.89 | 36.32 | - | 35.50 | 35.64 | 36.84 | 35.05 | 34.60 |
| mOSCAR + caps. | 8 | 35.95 | 37.65 | - | 36.78 | 37.14 | 37.81 | 36.17 | 35.98 |
|  | 16 | 36.78 | 38.78 | - | 37.52 | 37.73 | 38.68 | 37.91 | 36.84 |

**Table 11: VQA results on MaXM. Bold is best result.**
|  |  | En | Fr | Hi | He | Ro | Th | Zh |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |  |  |
|  | 0 | 34.24 | 15.91 | 18.08 | 16.43 | 13.73 | 25.75 | 13.36 |
| Multi. OF | 4 | 36.19 | 21.21 | 20.77 | 19.29 | 19.01 | 31.72 | 19.13 |
| mOSCAR + caps | 8 | 36.96 | 17.80 | 20.00 | 20.36 | 17.96 | 33.96 | 23.83 |
|  | 16 | 36.19 | 18.18 | 21.92 | 20.71 | 16.90 | 41.04 | 22.74 |
|  | 0 | 09.73 | 00.38 | 07.69 | 01.43 | 00.00 | 05.22 | 03.61 |
| Multi. OF | 4 | 09.34 | 02.65 | 05.00 | 02.50 | 00.00 | 05.60 | 03.97 |
| captions only | 8 | 09.34 | 01.89 | 08.08 | 05.00 | 01.06 | 03.36 | 05.42 |
|  | 16 | 08.56 | 01.14 | 05.00 | 08.21 | 00.35 | 03.36 | 07.58 |
| Translate test |  |  |  |  |  |  |  |  |
| OF-3B MPT | 0 | - | 12.50 | 22.31 | 00.36 | 10.92 | 00.00 | 00.00 |
| 4 | - | 10.98 | 25.38 | 00.36 | 10.21 | 00.00 | 00.00 |  |
| 8 | - | 10.98 | 27.31 | 00.36 | 11.27 | 00.00 | 00.00 |  |
| 16 | - | 13.26 | 26.54 | 01.07 | 13.38 | 00.00 | 00.00 |  |
|  | 0 | - | 18.18 | 28.08 | 00.00 | 13.73 | 00.00 | 00.36 |
| Multi. OF | 4 | - | 15.91 | 30.38 | 00.36 | 12.68 | 00.00 | 00.00 |
| mOSCAR + caps | 8 | - | 15.15 | 30.77 | 00.00 | 14.79 | 00.00 | 00.00 |
|  | 16 | - | 15.91 | 35.77 | 00.36 | 16.90 | 00.00 | 00.00 |

**Table 12: Classification results on MaRVL. Bold is best result.**
|  |  | Id | Sw | Ta | Tr | Zh |
| --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |
| Random chance |  | 50.00 | 50.00 | 50.00 | 50.00 | 50.00 |
|  | 0 | 50.00 | 49.46 | 49.76 | 49.83 | 49.80 |
| Multilingual OF | 4 | 49.91 | 49.55 | 49.28 | 49.83 | 49.80 |
| mOSCAR + caps | 8 | 50.62 | 48.65 | 49.68 | 49.83 | 49.80 |
|  | 16 | 50.18 | 49.01 | 49.76 | 49.83 | 49.90 |
|  | 0 | 51.33 | 49.01 | 49.52 | 49.83 | 49.70 |
| Multilingual OF | 4 | 49.73 | 49.64 | 49.19 | 49.41 | 49.70 |
| captions only | 8 | 49.91 | 49.10 | 49.60 | 49.75 | 49.90 |
|  | 16 | 50.09 | 49.73 | 49.60 | 49.75 | 49.80 |
| Translate test |  |  |  |  |  |  |
| OF-3B MPT | 0 | 50.00 | 49.37 | 49.76 | 49.83 | 49.80 |
| 4 | 50.00 | 49.64 | 49.52 | 49.75 | 49.60 |  |
| 8 | 49.82 | 49.46 | 49.28 | 50.08 | 49.90 |  |
| 16 | 50.00 | 49.37 | 49.44 | 50.00 | 49.80 |  |
|  | 0 | 49.07 | 49.79 | 49.52 | 50.34 | 49.60 |
| Multilingual OF | 4 | 49.99 | 49.79 | 48.23 | 49.75 | 49.76 |
| mOSCAR + caps | 8 | 50.00 | 48.92 | 50.64 | 50.42 | 48.90 |
|  | 16 | 49.84 | 50.00 | 50.24 | 48.90 | 49.75 |

**Table 13: Classification results on XVNLI. Bold is best result.**
|  |  | Ar | En | Es | Fr | Ru |
| --- | --- | --- | --- | --- | --- | --- |
|  | #shots |  |  |  |  |  |
| Random chance |  | 33.33 | 33.33 | 33.33 | 33.33 | 33.33 |
|  | 0 | 32.90 | 34.02 | 31.44 | 33.85 | 32.82 |
| Multilingual OF. | 4 | 36.94 | 36.17 | 34.19 | 34.54 | 38.49 |
| mOSCAR + caps. | 8 | 37.80 | 39.86 | 34.97 | 35.74 | 37.46 |
|  | 16 | 34.97 | 38.92 | 32.30 | 34.97 | 35.65 |
|  | 0 | 35.48 | 34.02 | 33.51 | 34.45 | 31.36 |
| Multilingual OF. | 4 | 32.04 | 31.79 | 32.73 | 32.22 | 31.44 |
| captions only | 8 | 34.02 | 33.76 | 32.04 | 35.57 | 33.16 |
|  | 16 | 32.04 | 32.99 | 33.76 | 33.17 | 31.53 |
| Translate test |  |  |  |  |  |  |
| OF-3B MPT | 0 | 32.65 | - | 31.01 | 31.44 | 35.82 |
| 4 | 36.25 | - | 35.82 | 35.57 | 35.65 |  |
| 8 | 31.27 | - | 31.10 | 31.10 | 31.70 |  |
| 16 | 33.68 | - | 33.25 | 32.99 | 33.25 |  |
|  | 0 | 34.88 | - | 34.88 | 34.54 | 34.36 |
| Multilingual OF. | 4 | 36.25 | - | 36.17 | 35.91 | 36.08 |
| mOSCAR + caps. | 8 | 39.60 | - | 39.52 | 40.29 | 39.35 |
|  | 16 | 37.54 | - | 37.89 | 37.46 | 39.00 |

**Table 14: En$\rightarrow$X translation results on Multi30k. Bold is best result.**
|  |  | Cs | De | Fr |
| --- | --- | --- | --- | --- |
|  | #shots |  |  |  |
|  | 0 | 3.22 | 28.86 | 36.01 |
| Multi. OF | 4 | 3.16 | 28.99 | 36.22 |
| full | 8 | 3.44 | 28.76 | 37.41 |
|  | 16 | 3.73 | 29.19 | 37.53 |
|  | 0 | 0.00 | 00.00 | 00.00 |
| Multi. OF | 4 | 0.00 | 00.00 | 00.00 |
| caps. only | 8 | 0.00 | 00.00 | 00.03 |
|  | 16 | 0.00 | 00.40 | 01.82 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The choice of filters was discussed thoroughly. Relative to other literature in the field, it is clear that an exhaustive search over the space of possible data filtering methodologies was done. 
2. The empirical results are quite promising. The set of evaluations/benchmarks is extensive and exhaustive. Moreover, the performance across different shotting of the models demonstrates significant improvmenets over existing datasets. 
3. I found the discussion around diversity measurements and comparison to existing datasets to be very well done. The discussion how multilinguality vs English-only diversity needs to be accounted for was also enlightening.
4. The prevention against data poisoning is appreciated and important.
summary: This paper yields a powerful Multimodal Multilingual pretraining dataset for MLLM's. They choose a standard set of filters and approaches to arrive a higher quality. Downstream models trained on mOSCAR are empirically stronger than existing models.
weaknesses: 1. I found some of the choices, while fairly standard, for the design of this framework to be lacking in terms of thought in terms of their design. While most of the decisions are cited, several decisions could have been had more analysis. 
2. Ablations on the filters would have been very interesting to see. This includes the joint image-text filtering.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: As weakness
soundness: 3
strengths: mOSCAR is derived from Common Crawl and utilizes sophisticated methods to filter and refine data. Steps include document deduplication, toxicity filtering, PII removal, and NSFW detection, ensuring a robust dataset suitable for mLLMs. Covering 163 languages, mOSCAR significantly expands the multilingual training dataset landscape, making it inclusive of languages that are often underrepresented in large-scale datasets. The authors conducted extensive benchmark tests, demonstrating that models trained with mOSCAR achieve superior performance in few-shot learning and across a variety of image-text tasks compared to models trained solely on captioning data. The authors plan to make the dataset public, promoting accessibility and encouraging further research in multilingual and multimodal model training. With meticulous filtering to maintain both quality and diversity, the paper ensures that mOSCAR addresses common issues like NSFW content, toxic language, and duplication, which enhances its usability in ethical AI research.
summary: The paper presents the creation and utilization of mOSCAR, a novel large-scale multilingual and multimodal dataset for training Multimodal Large Language Models (mLLMs). The dataset, which includes 303 million documents with 1.15 billion images spanning 163 languages, was created to address the limitations of existing datasets that are often monolingual or only moderately multilingual. By applying a series of filtering steps to enhance safety, diversity, and quality, mOSCAR aims to support multilingual, in-context learning capabilities. The authors demonstrate mOSCAR's benefits by training models that outperform baseline models across multiple multilingual, multimodal benchmarks. The proposed dataset seeks to address the scarcity of high-quality, large-scale multilingual datasets by employing rigorous data collection, filtering, and decontamination techniques. The authors validate mOSCAR's efficacy by training and evaluating two models: one on mOSCAR and captioning data, and another solely on captioning data. The mOSCAR-trained model shows marked improvements, particularly in few-shot learning across multilingual tasks.
weaknesses: 1. Limited manual validation (1,000 images for NSFW, 100 documents for toxic content)
2. English-centric regex patterns may miss unsafe content in other languages
3. Binary toxicity detection requiring two distinct toxic words could miss subtle harmful content
4. Document-level NSFW filtering is overly aggressive, potentially discarding valuable safe content
5. Reliance on regex and wordlists could be improved with neural-based approaches
6. Fixed character (300) and node count (5) thresholds may not accommodate different writing systems
7. Language identification may underperform on code-switched or multilingual content
8. Limited evaluation of data quality and distribution across the 163 languages, particularly for low-resource languages
9. No systematic evaluation of text-image alignment quality
10. Lack of content relevance metrics within documents

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - What was the original size of the three common crawl WARC files combined? How many text tokens and images?
- I'm not very clear on the difference between the NSFW text filter and the toxicity text filter. It seems that the toxicity text filter already contains keywords for NSFW, so is this NSFW filter needed here?
- I'm a bit curious on what the main bottleneck of the pipeline is. How long would each of these steps take approximately? If someone wants to scale this up to larger datasets, which part of the process is the one that would take the most time?
---

Typos:
- L25: "train" --> "trained"
- L245 "bias" --> "biased"

Missing references:
- MINT-1T https://arxiv.org/abs/2406.11271 -- not multilingual but probably worth mentioning
soundness: 4
strengths: 1. **Novelty and usefulness of contribution** -- There currently does not exist a multilingual multimodal interleaved dataset at this scale, so this artifact is something that would certainly be useful to the community.
    - This differs from other multilingual multimodal datasets (e.g. LAION-5B, mC4) since it is focused more on interleaved text+images as opposed to simple captioning.
    - This differs from other interleaved datasets (e.g. IDEFICS) because it is multilingual as opposed to pure English.
2. **Comprehensive data collection and filtering pipeline** -- The filtering process is detailed very thoroughly, and filtering is done on both text and images. Filtering was done across various features of a document, such as size, presence of keywords, image aspect ratio, etc. The pipeline is also general and scalable, so if somebody wants to do it on a larger part of CommonCrawl, it should be doable.
3. **Strong safety filters** -- This is something that is of utmost importance in designing such datasets (as we have seen from the LAION case). The paper handles this carefully, with NSFW, CSAM, and toxicity filters for both text and images. They also respect robots.txt protocols when downloading images.
summary: This paper proposes a multimodal version of OSCAR, which is a popular multilingual dataset from CommonCrawl

## High level overview of the dataset:
- 163 languages, 303M documents, 200B tokens, 1.15B images
- This is an interleaved text-image dataset (as opposed to captioning-focused datasets)
- Trained a multilingual multimodal Flamingo-style model on the mOSCAR dataset and show it outperforms the OpenFlamingo-3B-MPT translate-test baseline across various evaluation sets. 

## Overview of data collection and filtering process:
- Start from three 2023 CommonCrawl WARC files. 
- Parse each document's DOM tree and use HTML tags to extract images, remove tables, etc.
- Filtering based on **size**
    - 50% removed -- too small (<500 bytes)
- Filtering based on **text** --> done on both DOM-text-node level and document-level
    - NSFW filter -- Done on document level using regex keyword matching
    - Deduplication -- Levenshtein ratio and MinHashLSH to calculate near-duplicates per language. Filter out 19\%. 
    - Toxicity filter -- From list of toxic keywords in multiple languages. Filter out 1-2\%.
    - PII filter
- Filtering based on **image**
    -  URL filters -- Remove URLs that mention words like "banner", "logo", etc. Filter out 13.6 \%
    - NSFW filter -- combine multiple NSFW detectors for nudity. Filter out 0.5\%
    - CSAM filter -- Use proprietary child sexual abuse classifier. Filter out 0.07\%
    - Deduplication -- Use imagehash per language.
    - PII filter -- Detect faces
- Decontamination to remove images appearing in test sets
weaknesses: 1. **No dataset optimization** -- The paper shows that a Flamingo-style model trained with mOSCAR outperforms a translate-test baseline. This is great, but the paper didn't really mention how much iteration they did when designing this dataset. Were there multiple rounds or iterations of creating mOSCAR? There seems to be a lack of ablations, so to me it felt like the model training experiments were more of an afterthought after the dataset had already been created. (Maybe that was the intention?) 
2. **No quality filter** -- This is related to the point I mentioned above. Because there is no filter specific for text quality, then the mOSCAR dataset is more of a cleaned version of CommonCrawl, as opposed to a cleaned+filtered version that is optimized for training good models.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Could the authors clarify their baselines? In L399-L401, the authors said they trained 3 models on: (1) 50M mOSCAR + 100M image-text pairs; (2) 300M image-text pairs; (3) 35M WIT + 70M text-image pairs. While in Table 5 there is one new setting with 35M mOSCAR + 70M text-image pairs. It would be helpful for the authors to clarify their model training settings in every caption to avoid confusion.
2. Could the author explain why WIT performs better than mOSCAR with the same performance score across different number of shots on Multi30K in Table 5?
3. What is the trade-off of using English-only dataset and mOSCAR? Could the author evaluate the models on English-only benchmarks as well to indicate in what situation the user should choose mOCSAR and at what ratio?
4. I also wonder if there is any trade-off between enhancing the VLM's ability on few-shot learning situation v.s. other capabilities such as general perception, captioning, VQA, etc. Could the authors design analysis on this?
5. With what license the authors plan to release the data?
soundness: 3
strengths: 1. The paper collects a large amount of data of high quality for multilingual multimodal training and open-source it. The workload and effort is significantly high and the dataset will be useful to the community.
2. The data processing pipeline is comprehensive and containing standard steps (e.g., deduplication, language filtering, PII removal, etc.), resulting in high-quality data.
3. The analysis shows mOCSCAR matches the quality of the existing interleaved English-only datasets in terms of quality and diversity
summary: This paper constructs a fully open-sourced multilingual multimodal datasets for interleaved text-image training. The dataset, namely mOSCAR, covers 163 languages, 303M documents, 200B tokens, and 1.15B images. The authors compare the performance of one model trained on a subset of mOSCAR and caption data with another trained on caption data only, and find the model trained on mOSCAR exhibits better in-context learning abilities.
weaknesses: 1. I appreciate this line of work, but the contribution beyond engineering effort is unclear.
2. The baselines in performance comparison needs to be further justified (see questions).
3.It is unclear how much the English-only performance drop when training on this multilingual dataset, where the author should conduct an ablation study and present the model performance on common English-only dataset such as MMMU, MME, etc. If there is an obvious decrease in English performance, the author should discuss how to mitigate this effect and in what situation the users should choose mOSCAR and combine with other caption data and with what ratio.

Meta-review: This paper presents mOSCAR, described as the first large-scale multilingual and multimodal web-crawled document corpus. The key claims are that it covers 163 languages with 303M documents, 200B tokens and 1.15B images, and that models trained on mOSCAR show improved few-shot learning capabilities. While the paper addresses an important need for multilingual multimodal training data, several critical weaknesses make it unsuitable for acceptance at this time. The filtering methodology raises serious concerns, particularly for non-English content where safety filters may miss harmful content due to English-centric patterns. The manual validation is extremely limited (only 1,000 images for NSFW, 100 documents for toxic content initially), making it difficult to assess dataset quality and safety. The binary toxicity detection requiring two distinct toxic words could miss subtle harmful content, while document-level NSFW filtering may be overly aggressive in discarding potentially valuable safe content. The experimental evaluation also has significant gaps - there's limited analysis of text-image alignment quality, no systematic evaluation of content quality across the 163 languages (especially low-resource ones), and unclear baseline comparisons. While the authors attempted to address some concerns during rebuttal by adding 900 more document evaluations and new experimental results, these additions still leave major questions about dataset safety and quality unanswered. The review scores (3, 6, 6, 8) reflect these serious concerns. Thus I vote to reject

Decision: Reject