mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
Authors: Matthieu Futeral, Armel Randy Zebaze, Pedro Ortiz Suarez, Julien Abadji, Rémi Lacroix, Cordelia Schmid, Rachel Bawden, Benoît Sagot
Accepted at: acl25

mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

Matthieu Futeral* 1,2   Armel Zebaze1,4   Pedro Ortiz Suarez5   Julien Abadji1
Rémi Lacroix3,6   Cordelia Schmid1,2   Rachel Bawden1   Benoît Sagot1

1Inria   2Département d’informatique de l’ENS, CNRS, PSL Research University
3Institut du développement et des ressources en informatique scientifique, CNRS
4Sorbonne Université, Paris, France   5Common Crawl Foundation   6Université Paris-Saclay

Abstract

Multimodal Large Language Models (mLLMs) are trained on a large amount of text-image data. While most mLLMs are trained on caption-like data only, Alayrac et al. (2022) showed that additionally training them on interleaved sequences of text and images can lead to the emergence of in-context learning capabilities. However, the dataset they used, M3W, is not public and is only in English. There have been attempts to reproduce their results but the released datasets are English-only. In contrast, current multilingual and multimodal datasets are either composed of caption-like only or medium-scale or fully private data. This limits mLLM research for the 7,000 other languages spoken in the world. We therefore introduce mOSCAR, to the best of our knowledge the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 303M documents, 200B tokens and 1.15B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality. We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model trained on captioning data only. The model additionally trained on mOSCAR shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, confirming previous findings for English-only mLLMs. The dataset is released under the Creative Commons CC BY 4.0 license and can be accessed here.1

1 Introduction

Multimodal Large Language Models (mLLMs) are trained on large amounts of text-image data (Radford et al., 2021; Yu et al., 2022; Li et al., 2023; Wang et al., 2023; OpenAI, 2023; Gemini Team et al., 2023; Chameleon Team, 2024). The main paradigm until recently was to train a model from a large collection of web-crawled images and their captions (Li et al., 2021; Wang et al., 2022; Chen et al., 2023b). Models such as Flamingo (Alayrac et al., 2022) challenged this paradigm by being additionally trained on interleaved sequences of text and images from web documents, showing state-of-the-art results on various tasks and in-context learning capabilities that are not present in models trained on caption-like data only. Additionally, McKinzie et al. (2024) recently proved that including interleaved text-image data during training was necessary to get good few-shot learning performance. However, the datasets used to train mLLMs are either private (Alayrac et al., 2022), monolingual or multilingual but only medium-scale (Srinivasan et al., 2021). Some attempts have been made to reproduce these datasets (Zhu et al., 2023; Laurençon et al., 2023) but the resulting datasets are only available in English.

Few image-text datasets are multilingual and most of them are obtained by translating English caption-like datasets, such as multilingual Conceptual Captions (Sharma et al., 2018), into multiple languages using neural machine translation (NMT) systems (Suris et al., 2022; Maaz et al., 2024). This presents some drawbacks such as some languages still being poorly translated by current state-of-the-art NMT models (Liu et al., 2020; Costa-jussà et al., 2022) and some cultural subtleties inherent in each language not being fully conveyed. Some efforts have been conducted to collect large-scale multilingual image captioning datasets, such as LAION-5B (Schuhmann et al., 2022), but they are limited to caption data too, are relatively noisy and more importantly contain a non-negligible share of “not safe for work” (NSFW) content such as paedopornographic images (Schuhmann et al., 2022).

This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.2 Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative (Ortiz Suárez et al., 2019; Abadji et al., 2021, 2022) and covers 303M documents in 163 languages, 200B tokens and 1.15B images. Figure 1 shows an example of a document, more can be found in Appendix A.3. We carry out extensive filtering to increase its safety and quality. To prove mOSCAR’s utility, we train a multilingual Open-

*Correspondence to matthieu.futeral@inria.fr
1https://oscar-project.github.io/documentation/versions/mOSCAR/
2https://commoncrawl.org/. The Common Crawl Foundation is a non-profit organization that crawls the web on a monthly basis.
Flamingo (Awadalla et al., 2023) from a Gemma-2B language model (Gemma Team et al., 2024) on a subset of mOSCAR and captioning data from LAION-400M (Schuhmann et al., 2021), recaptioned with BLIP (Li et al., 2022), filtered with CLIP (Radford et al., 2021) and translated with NLLB (Costa-jussà et al., 2022). We compare against a similar model trained on captioning data only and show we obtain a strong boost in few-shot learning, confirming previous findings for English (Alayrac et al., 2022; McKinzie et al., 2024; Laurençon et al., 2024). mOSCAR can be accessed here 3.

2 Related Work

Large-scale web-based datasets Numerous datasets have been created by filtering web-crawled data. These include large-scale text-only datasets (Ortiz Suárez et al., 2019; Raffel et al., 2020; Wenzek et al., 2020; Gao et al., 2020; Abadji et al., 2021; Xue et al., 2021; Laurençon et al., 2022; Abadji et al., 2022; Penedo et al., 2023) and multimodal ones (Sharma et al., 2018; Changpinyo et al., 2021; Jia et al., 2021; Schuhmann et al., 2021, 2022; Byeon et al., 2022; Laurençon et al., 2023; Zhu et al., 2023; Gadre et al., 2024). Even if these datasets are not as high quality as smaller and/or hand-crafted ones, they are now the standard to pretrain foundation models, as it has been shown that training bigger models on more data leads to better downstream performances (Brown et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023a,b).

English image-text datasets The first open-source image-text datasets were manually created, small-scale and English-only (Ordonez et al., 2011; Lin et al., 2014; Plummer et al., 2015; Krishna et al., 2017). Scaling up these datasets was an appealing solution to overcome limitations of previous image-text models; a few works (Sharma et al., 2018; Changpinyo et al., 2021) proposed to collect millions of image-text pairs from the web before filtering them with well-designed steps. Relaxing the filtering steps enabled the collection of more data and led to large-scale datasets to train image-text foundation models (Radford et al., 2021; Li et al., 2021; Schuhmann et al., 2021, 2022; Byeon et al., 2022). However, these datasets generally contain caption-like image-text pairs only, and it is therefore difficult to observe in-context learning abilities similarly to text-only language models trained on raw documents (Raffel et al., 2020). Alayrac et al. (2022) overcome this issue by training their model directly on documents with interleaved image-text data. While their results are promising, their M3W dataset is English-only and private. Recently, open-source efforts (Zhu et al., 2023; Laurençon et al., 2023) have been made to release a similar dataset but they are still monolingual.

Multilingual image-text datasets Only a few image-text datasets are available in multiple languages. One of the first focused on collecting Google images from short queries based on word frequencies from Wikipedia pages in 98 languages (Hewitt et al., 2018). Later, Srinivasan et al. (2021) proposed the WIT dataset, an image-text dataset composed of Wikipedia pages. Although of high quality, it is only medium-scale even for high-resource languages and there are fewer than 50k unique images for most languages. Another approach lies in bootstrapping multilingual and multimodal data from a model trained with English-only data (Mohammed et al., 2023). While effective for captioning, it is computationally expensive to implement in practice. Other multilingual image-text datasets exist but focus on captions only and are highly domain-specific (Kosar et al., 2022; Leong et al., 2022).

3 Dataset Creation Pipeline

3.1 Data collection

We collect mOSCAR from the Web ARchive Content (WARC) files of three 2023 Common Crawl dumps, processing them using the FastWARC library (Bevendorff et al., 2021). We remove documents smaller than 500 bytes (50% of the documents), as we find they are usually too small to be considered documents and tend to contain noisy text. We then navigate through the entire Document Object Model (DOM) tree with a depth first search algorithm and ChatNoir library (Bevendorff et al., 2018) to extract nodes of interests corresponding to specific HTML tags.

Following previous work, we extract text from the tags that usually contain the main content of web pages (we refer to them as DOM text nodes), i.e. <p>, <h*>, <title>, <description>, <ul>, <ol>, <aside>, <dl>, <dd>, <dt>. Similarly to (Laurençon et al., 2023), we choose to remove <table> content as most often it is irrelevant and difficult to render. We extract all <img> tags (we refer to them as DOM image nodes). We then remove documents with fewer than 3 text nodes (as they do not contain enough text) and more than 30 image nodes (as we found them to be too noisy).

3.2 Language identification

We identify the language of each document using the state-of-the-art open-LID language detector (Burchell et al., 2023), covering 201 languages. We apply openLID to each DOM text node and keep the three most probable languages with their respective probabilities. The language of the document is then determined by summing over the probabilities of each language detected for each text segment, weighted by the number of characters in the segment4 and taking the language with the highest score.

4This is to avoid mis-assigning the language due to the presence of many short, non-informative DOM text nodes in the same language (e.g. “Cookies”, “Subscribe”, “Newsletter” etc.) and because language identification is generally less reliable for short segments.
Le plus charmant petit square de Paris.

Le square René Viviani a été ouvert au public en 1928. Il occupait l’emplacement de l’annexe de l’hôpital-Dieu (il est situé en face de Notre-Dame) et d’une rangée de maisons longeant la rue de la Bûcherie. Le conseiller municipal Léon Riotor et la Commission du Vieux Paris ont particulièrement œuvré à sa création dans le but de préserver à la fois le dégagement de l’église Saint-Julien-le-Pauvre et la perspective de Notre-Dame. Quelques éléments architecturaux parsèment le parc, et contribuent à son charme : pinacles, chapiteaux, balustrades, provenant sans doute de la cathédrale Notre-Dame. [...]

Pinacles et chapiteaux ornent le parc : probablement des vestiges de Notre-Dame. La fine poussière derrière la balustrade de pierre provient du fort vent qui faisait s’envoler le sable et la terre. [...]

Figure 1: Example of a French document from mOSCAR.

3.3 Text-only filtering
We apply a series of filtering steps to the text content of each document independently of the images, with the aim of discarding poor quality documents and cleaning text as best as possible. We first filter at the text-node level and then at the whole document level, before running near-deduplication to keep unique text nodes within a document and unique documents in the dataset.

Text node filtering  We use a set of heuristics (see Appendix A.4) to extract as much human-generated content as possible while discarding noisy text related to ads and website functions (e.g. “Instagram”, “Facebook”). We then keep DOM text nodes with content over 10 bytes. This step, designed to improve the quality of extracted text, removes on average 55% of text nodes.

Document filtering  We mostly filter “not safe for work” (NSFW) content at the document level. We use an English regular expression to detect adult content, similar to the one used by the Université Toulouse I Capitole5 and remove the entire document if there is a match with any of the DOM text nodes’ contents, removing on average 0.5% of documents (mostly English ones). We acknowledge that there is a high probability that this also discards safe content, e.g. we could remove content from certain communities who use some explicit words in a non-sexual way (Sap et al., 2019). However, we explicitly favour recall over precision to minimise the risk of unsafe content. We additionally remove documents containing fewer than five DOM text nodes and fewer than 300 characters after the previous filtering steps, removing 70.6% of documents.

Deduplication  We conduct several types of per-language deduplication at different levels, as this has been shown to improve training efficiency (Abbas et al., 2023). First, we keep unique documents only by removing exact duplicates at the document level. We also remove exact duplicates of text nodes within the same document (4% of text nodes) and near-duplicate text nodes (1% of text nodes) by computing the Levenshtein ratio (Levenshtein, 1966) between all text nodes within the same document and applying a threshold of 0.95. If near-duplicates are found, we keep the first one in the document. Finally, we conduct per language near-deduplication at the document level with MinHashLSH (Broder, 1997; Gionis et al., 1999) following Smith et al. (2022), removing on average 19% of documents;6 we turn documents into hashing vectors, compute min hashes from these vectors and perform Locality Sensitive Hashing to remove duplicates7 (see Appendix A.6.1 for more details).

Toxicity filtering  Toxic content targeting individuals or groups of people is widespread on the internet and can therefore be found in large-scale web-crawled datasets like mOSCAR without appropriate filtering steps. To alleviate this issue, we apply the same method used by Costa-jussà et al. (2022) and remove documents from mOSCAR based on the presence of a list of “toxic” words for each language8. As some words in the list can also be used in a non-toxic way based on the context (e.g.: ‘breast’ in English), we tag the document as toxic and remove it from mOSCAR if it contains at least two distinct words in the list. This filtering step removes 0.95% of the documents for very high-resource languages (>5M documents), 2.13% for high-resource languages (<5M, >500K), 0.47% for mid-resource languages (<500K, >50K) and 0.64% for low-resource languages (<50K). When manually analysing 1,000 random documents removed by this filtering step in each of the 2 (high-resource) languages we are native speakers of (English and French), we found 568 documents with toxic content.

Personal Identifiable Information  Personal Identifiable Information (PII) can be found in large-scale web-crawled datasets, we therefore conducted an additional

5https://dsi.ut-capitole.fr/blacklists/index_en.php
6With some disparity among languages as we found more duplicates for low- than high-resource languages.
7We performed this using the datasketch python library.
8The list of these words for each language can be found here: https://github.com/facebookresearch/flores/tree/main/toxicity
filtering step to replace all detected PII by place holder strings using regular expressions (Appendix A.6.2 for more details). Concretely, we replaced all detected email addresses, phone numbers, credit card numbers, IP addresses and passport numbers. Moreover, it was shown that CommonCrawl contains a non negligible part of API keys in its content 9. We therefore scanned the dataset with the trufflehog tool 10 to track down residual API keys that could have passed previous filters. We found ~200K positive matches and manually check a random sample of 1K positive matches. We found only 2 of them to potentially be API keys, other matches are mainly noisy text nodes not related to PII. We removed the 200K text nodes from mOSCAR.

3.4 Image-only filtering

We downloaded images from the URLs in DOM image nodes using a modified version of the img2dataset toolkit (Beaumont, 2021) that includes an antivirus scan and follows robots.txt instructions to respect the Robots Exclusion Protocol. We then apply a series of filtering steps, first removing images based on heuristics, and then applying multiple NSFW detection models to remove undesirable content. Finally, we conduct a set of deduplication steps.

Rule-based filters Similarly to previous works (Schuhmann et al., 2021) and to avoid extracting low-resolution images and favicons, we keep images with a minimum height and width of 150 pixels. We restrict the aspect ratio to be between 3 and 1/3 (to remove banners), we remove images if their URLs contain the words “logo”, “banner”, “button”, “widget”, “icon” or “plugin” or if the image name from the URL matches “twitter”, “facebook” or “rss” (to remove logos). This step removes 13.6% of the URLs. At this stage, we downloaded 2.5B images with an average success rate of 55%.

NSFW detection We use multiple NSFW automatic models to remove as much unsafe content as possible. We first combine two NSFW detectors: nsfw-detector (Laborde), a 5-class classifier with a MobileNet (Howard et al., 2017) backbone fine-tuned on 60GB of annotated data and NudeNet,11 an object detector trained to detect different types of nudity in images. We combined the two models as we found the first to be gender-biased while the second gives a large number of false positives for non-human images. Concretely, we consider an image an NSFW candidate if the sum of the probabilities for the classes ‘porn’ and ‘hentai’ is superior to 0.8 using nsfw-detector. We then tag the image as NSFW if one of the sensitive ‘exposed’ classes of NudeNet gets a probability superior to 0.5.

If a document contains an image with an NSFW tag, we remove the entire document from the dataset, which removes 0.5% of images. We manually inspecting 1,000 images of the remaining data and found no NSFW content. We manually inspected 1,000 images of the removed content and found 63.4% of NSFW images.

CSAM content Child Sexual Abuse Material (CSAM) is widespread on the internet and is therefore likely to be found in such a large-scale dataset crawled from the web. Removing CSAM is challenging as there is no training data nor open-source detection models available as these could be used in a harmful way. We rely on Thorn’s CSAM classifier12, a proprietary classifier trained to detect CSAM content in images. We tag the image as CSAM if the probability of the class CSAM is superior to 0.4 to favour recall over precision. As mentioned above, if a document contains an image with a CSAM tag, we remove it from the dataset. This step removes 0.07% of the images.

Deduplication To avoid memorisation issues often seen in models trained on datasets with many duplicated images (Somepalli et al., 2023; Carlini et al., 2023; Webster et al., 2023; Somepalli et al., 2024), we perform deduplication at the image level. We first remove duplicate images within the same document by URL matching (removing 8.7% of URLs). We then compute a perceptual hash (pHash) for each image using the imagehash library13 and remove images with the same pHash within the same document, keeping only the first occurrence. We also limit the number of times an image can appear in the dataset per-language to 10 using both URL matching and perceptual hashing (this removes 2.5% of images). We do this per-language and not across languages as having the same images in documents from different languages could encourage cross-lingual transfer.

Personal Identification Information To protect PII in images, we use a lightweight face detector14 and apply a threshold of 0.99 to detect faces in the images. We apply such a high threshold as we found the model to be biased towards detecting faces with high probability in images without any human. For each image in mOSCAR, we distribute the bounding boxes of the detected faces so that users can blur them when downloading the images. More details are provided in Appendix A.6.2.

3.5 Data decontamination

LLMs and mLLMs are trained on web-crawled data that can contain the benchmarks they are tested on (Dodge et al., 2021). As they are good at memorizing training data (Carlini et al., 2023), this data contamination is problematic. We therefore discard all images with the

9https://trufflesecurity.com/blog/[...]
10https://github.com/trufflesecurity/trufflehog
11https://github.com/vladmandic/nudenet
12https://safer.io/
13https://github.com/JohannesBuchner/imagehash
14https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB
same perceptual hash as any of the images from the evaluation benchmarks (and their training sets) we use (see Section 5.1). This step removes on average 126,016 images for high-resource languages (up to 300K images for English), 6,862 images for mid-resource languages and 45 images for low-resource languages.

3.6 Text-image joint filtering

Our aim is to obtain truly multimodal documents where all images are related to at least one of the text nodes in some way15 and vice versa. We choose to apply joint text-image filtering to discard images and/or text nodes that are irrelevant to the rest of the document (e.g. the case of ads and website functionalities).

To do this, we use NLLB-SIGLIP16 (Visheratin, 2023), a multilingual version of SIGLIP (Zhai et al., 2023) trained with the encoder of NLLB (Costa-jussà et al., 2022), which covers all mOSCAR languages.17 We compute cosine similarity scores between all images and all paragraphs18 within a same document. To remove irrelevant text nodes or images in a document, we mimic a text-image retrieval task, which means we avoid using arbitrary cosine similarity thresholds for each language and can reduce length biases and those in favour of caption-like paragraphs. For each candidate pair we randomly sample 63 negative images and 63 negative similar-length paragraphs from the same language but other documents. We tag the text node (resp. image) as valid if the cosine similarity of the pair is among the top 8 of the text-to-image (resp. image-to-text) similarity scores computed with the candidate text node (resp. image) and all the negative images (resp. text nodes). This means that we tag the text node (resp. image) as valid if it has a significantly higher score than a score computed with a random image (resp. text) for at least one of the images (resp. text node) in the document. We then discard text nodes and images not tagged as valid (on average 35% of the DOM text nodes and 10% of the images within a document).

After this filtering step, we apply additional text-only filters to keep documents superior to 100 bytes. We also reapply the open-lid language detector (Burchell et al., 2023) as described in Section 3.2 as we found the last filtering step to change the major language of some documents.

4 Multimodal Open Super-large Crawled Aggregated coRpus (mOSCAR)

mOSCAR is extracted from three Common Crawl dumps from 2023. Due to computational constraints and in order to extract a maximum number of documents for low-resource languages, we extracted all languages from the first dump only. We removed the 6 most high-resource languages from the second dump and only extracted the languages with fewer than 1M documents for the last dump. Table 1 shows a distribution of the total number of languages and their number of documents. To avoid data poisoning (Carlini et al., 2024), we release a hash (sha512) with each mOSCAR image. mOSCAR is composed of 303M documents (200B tokens, 1.15B images) from 163 languages. Figure 2 shows the distribution of images and tokens per document and their joint distribution. As shown in Figure 2a, the mean and median number of images per document is 2 and 3.80.

<table>
  <tr>
    <th>#docs.</th>
    <th>10M</th>
    <th>5M</th>
    <th>1M</th>
    <th>500K</th>
    <th>200K</th>
    <th>50K</th>
    <th>10K</th>
    <th>5K</th>
    <th>1K</th>
  </tr>
  <tr>
    <th>#langs.</th>
    <td>10</td>
    <td>15</td>
    <td>36</td>
    <td>46</td>
    <td>56</td>
    <td>75</td>
    <td>119</td>
    <td>133</td>
    <td>154</td>
  </tr>
</table>

Table 1: Number of languages with at least \( N \) documents

4.1 Quality vs Diversity

While improving overall data quality, the filtering steps we applied (see Section 3) necessarily have a negative impact on diversity. We therefore study the trade-off between quality and diversity and compare against previously published, well-used datasets.

4.1.1 Text content

Diversity By construction, mOSCAR is diverse in terms of number of languages, so we focus on the diversity of mOSCAR’s English documents and compare against mmc4 (Zhu et al., 2023), OBELICS (Laurençon et al., 2023) and the English subset of WIT (Srinivasan et al., 2021). We compute the Vendi score (Friedman and Dieng, 2023) on a set of SimCSE embeddings (Gao et al., 2021) with a RoBERTa encoder (Liu et al., 2019) to evaluate the content diversity. Since embedding-based diversity metrics target content diversity well but are less relevant for lexical diversity (Tevet and Berant, 2021), we measure lexical diversity via the distinct \( n \)-gram ratio (Li et al., 2016). An analysis of the topics (Grootendorst, 2022) found in multiple languages of mOSCAR where we show diverse topics across languages can be found in Appendix A.2.

<table>
  <tr>
    <th></th>
    <th>Vendi score</th>
    <th>Dist. \( n \)-gram ratio</th>
  </tr>
  <tr>
    <td>mOSCAR</td>
    <td>69.05 (\( \pm \) 0.14)</td>
    <td>0.472 (\( \pm \) 0.002)</td>
  </tr>
  <tr>
    <td>mmc4</td>
    <td>67.93 (\( \pm \) 0.12)</td>
    <td>0.494 (\( \pm \) 0.002)</td>
  </tr>
  <tr>
    <td>OBELICS</td>
    <td>58.49 (\( \pm \) 0.09)</td>
    <td>0.488 (\( \pm \) 0.001)</td>
  </tr>
  <tr>
    <td>WIT</td>
    <td><b>73.30</b> (\( \pm \) 0.09)</td>
    <td><b>0.530</b> (\( \pm \) 0.001)</td>
  </tr>
</table>

Table 2: Average text diversity scores (\( \pm \) standard error) of text documents.

Comparison with other datasets For content diversity, we randomly sample 30M documents for mOSCAR, mmc4 and OBELICS and 3M documents
Figure 2: Distributions of numbers of tokens and images per document.

<table>
  <tr>
    <th></th>
    <th>mOSCAR</th>
    <th>LAION-400M</th>
    <th>WIT</th>
  </tr>
  <tr>
    <td></td>
    <td>55.74 (± 0.16)</td>
    <td><b>67.59</b> (± 0.16)</td>
    <td>36.14 (± 0.08)</td>
  </tr>
</table>

Table 3: Average Vendi score (± standard error) of images sampled from different datasets.

Figure 3: Perplexity of 100K random documents from different datasets.

<table>
  <tr>
    <th></th>
    <th>English</th>
    <th>All</th>
  </tr>
  <tr>
    <td></td>
    <td>52.36 (± 0.18)</td>
    <td><b>54.78</b> (± 2.29)</td>
  </tr>
</table>

Table 4: Average Vendi score (± standard error) of images sampled from mOSCAR (English vs. any language).

for WIT and represent the documents by their SimCSE embedding. We compute the Vendi Score with cosine similarity on a randomly sampled subset of 65,536 documents. Table 2 shows that mOSCAR English content is more diverse than mmc4 and OBELICS but less diverse than WIT. For lexical diversity, we randomly sample 3M documents for mOSCAR, mmc4, OBELICS and WIT and compute the distinct n-gram ratio on a subset of 8,192 documents for \( n \) from 1 to 4. Table 2 shows that mOSCAR is slightly less lexically diverse than OBELICS and mmc4, while WIT is by far the most diverse.

Quality To evaluate document quality, we focus on English documents and compute their perplexity using Gemma-2B (Gemma Team et al., 2024). Figure 3 shows the kernel density estimation of the distribution of the perplexity of 100K randomly sampled documents from different datasets: mOSCAR is comparable to mmc4 and WIT, while OBELICS appears to be the of the highest quality. mOSCAR is therefore comparable to other interleaved image-text dataset in terms of quality and diversity of its English subset. It is however more diverse than English-only datasets by its multilingual construction and more than 10 times larger than existing multilingual interleaved image-text datasets such as WIT.

4.1.2 Image diversity

Comparison with other datasets We compute the Vendi Score on random samples of images for different datasets, comparing the images from English mOSCAR documents with those from Conceptual Captions (Changpinyo et al., 2021), LAION-400M (Schuhmann et al., 2021) and WIT (Srinivasan et al., 2021). We represent each image by its SigLIP\(^{19}\) (Zhai et al., 2023) embedding and compute the Vendi score on batches of size 65,536 and a total of 1M images for each dataset. In Table 3, we notice that the set of images in mOSCAR documents are more diverse than images from WIT documents but less diverse than LAION-400M.

Multilingual diversity We also compare the diversity of images from English documents and of images sampled from documents of any language (English included). We use multilingual SigLIP (Chen et al., 2023a) trained on WebLI (Chen et al., 2023b) to compute image embeddings used to get the Vendi score. We again use a batch of size 65,536 and a total of 3M images, and we do not sample multiple images from a same document. For the multilingual setting, we randomly sample 50 languages and an equal number of images for each language to build the batch. As we did not do any image deduplication across languages, we could expect to have less diversity in the multilingual setting. However, Table 4 shows that the set of images is on average more diverse when sampled from all documents than from English-only documents. This means that the distribution of images is not exactly the same across languages, potentially due to cultural differences.

19We use siglip-base-patch16-224.
<table>
  <tr>
    <th>#shots</th>
    <th>xFlickR&CO</th>
    <th>XM3600</th>
    <th>xGQA</th>
    <th>MaXM</th>
    <th>MaRVL</th>
    <th>XVNLI</th>
    <th>Multi30K</th>
    <th>CoMMuTE</th>
  </tr>
  <tr>
    <td>0</td>
    <td>16.91</td>
    <td>7.45</td>
    <td>26.95</td>
    <td>22.33</td>
    <td>49.56</td>
    <td>33.88</td>
    <td>22.91</td>
    <td>63.34</td>
  </tr>
  <tr>
    <td>Multi. OF</td>
    <td>4</td>
    <td>34.80</td>
    <td>22.18</td>
    <td>32.33</td>
    <td>26.33</td>
    <td>49.64</td>
    <td>34.07</td>
    <td>23.27</td>
    <td>63.22</td>
  </tr>
  <tr>
    <td><i>mOSCAR + cap.</i></td>
    <td>8</td>
    <td>36.90</td>
    <td>23.48</td>
    <td>34.24</td>
    <td>27.08</td>
    <td>51.48</td>
    <td>36.60</td>
    <td>23.59</td>
    <td>63.54</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>39.46</td>
    <td>23.67</td>
    <td>35.23</td>
    <td>27.47</td>
    <td>49.84</td>
    <td>34.85</td>
    <td>23.85</td>
    <td>62.78</td>
  </tr>
  <tr>
    <td>0</td>
    <td>9.57</td>
    <td>4.21</td>
    <td>8.62</td>
    <td>4.01</td>
    <td>49.88</td>
    <td>33.76</td>
    <td>0.00</td>
    <td>61.36</td>
  </tr>
  <tr>
    <td>Multi. OF</td>
    <td>4</td>
    <td>13.20</td>
    <td>9.26</td>
    <td>13.45</td>
    <td>4.15</td>
    <td>49.54</td>
    <td>32.04</td>
    <td>0.00</td>
    <td>61.13</td>
  </tr>
  <tr>
    <td><i>cap. only</i></td>
    <td>8</td>
    <td>18.00</td>
    <td>10.35</td>
    <td>12.82</td>
    <td>4.88</td>
    <td>49.65</td>
    <td>33.71</td>
    <td>0.01</td>
    <td>60.90</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>19.87</td>
    <td>12.07</td>
    <td>13.37</td>
    <td>4.89</td>
    <td>49.79</td>
    <td>32.70</td>
    <td>0.74</td>
    <td>60.25</td>
  </tr>
</table>

Table 5: Results averaged over all languages. Multi. OF refers to multilingual Open Flamingo, <i>mOSCAR + cap.</i> refers to the model trained on text-image pairs and mOSCAR while <i>cap. only</i> refers to the model trained only on text-image pairs. **Bold** is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>xFlickR&CO</th>
    <th>XM3600</th>
    <th>xGQA</th>
    <th>MaXM</th>
    <th>MaRVL</th>
    <th>XVNLI</th>
    <th>Multi30K</th>
    <th>CoMMuTE</th>
  </tr>
  <tr>
    <td>0</td>
    <td>19.07</td>
    <td>8.73</td>
    <td>25.08</td>
    <td>19.64</td>
    <td>49.77</td>
    <td>33.01</td>
    <td>22.70</td>
    <td>63.75</td>
  </tr>
  <tr>
    <td>Multi. OF (35M)</td>
    <td>4</td>
    <td>34.32</td>
    <td>20.59</td>
    <td>31.90</td>
    <td>23.90</td>
    <td>49.67</td>
    <td>36.07</td>
    <td>22.79</td>
    <td>63.65</td>
  </tr>
  <tr>
    <td><i>mOSCAR + cap.</i></td>
    <td>8</td>
    <td>36.77</td>
    <td>22.15</td>
    <td>33.9</td>
    <td>24.41</td>
    <td>49.72</td>
    <td>37.16</td>
    <td>23.21</td>
    <td>63.00</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>37.63</td>
    <td>22.24</td>
    <td>35.71</td>
    <td>25.38</td>
    <td>49.73</td>
    <td>35.36</td>
    <td>23.48</td>
    <td>62.77</td>
  </tr>
  <tr>
    <td>0</td>
    <td>9.39</td>
    <td>4.67</td>
    <td>19.81</td>
    <td>14.63</td>
    <td>49.71</td>
    <td>32.78</td>
    <td>26.99</td>
    <td>56.75</td>
  </tr>
  <tr>
    <td>Multi. OF (35M)</td>
    <td>4</td>
    <td>7.68</td>
    <td>2.99</td>
    <td>25.68</td>
    <td>16.12</td>
    <td>49.72</td>
    <td>33.51</td>
    <td>26.99</td>
    <td>53.27</td>
  </tr>
  <tr>
    <td><i>WIT + cap.</i></td>
    <td>8</td>
    <td>8.91</td>
    <td>3.63</td>
    <td>27.06</td>
    <td>16.81</td>
    <td>49.74</td>
    <td>32.77</td>
    <td>26.99</td>
    <td>55.33</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>9.74</td>
    <td>4.14</td>
    <td>28.14</td>
    <td>16.34</td>
    <td>49.74</td>
    <td>33.63</td>
    <td>26.99</td>
    <td>54.04</td>
  </tr>
</table>

Table 6: Results averaged over all languages and comparison between a model trained on WIT and a checkpoint of multilingual Open Flamingo trained on 35M mOSCAR documents (full model was trained on 50M mOSCAR documents). Both models were trained on 35M documents from their respective training datasets and 70M text-image pairs for fair comparison. Multi. OF (35M) refers to multilingual Open Flamingo trained on 35M documents. **Bold** is best result.

5 Training a multilingual multimodal language model

We train a multilingual Flamingo-like model on mOSCAR that we call multilingual Open Flamingo. As adding captioning data to training data has been shown to improve zero-shot performance (McKinzie et al., 2024), we additionally train on LAION-400M, which we re-captioned using BLIP (Li et al., 2022), filtered with CLIP score (Radford et al., 2021) and translated using distilled NLLB-600M (Vishera, 2023) following the proportion of languages found in mOSCAR. We use Gemma-2B (Gemma Team et al., 2024) as the underlying language model and we train the model on 50M mOSCAR documents and 100M randomly sampled image-text pairs. We also train a model on 300M image-text pairs, a model trained on 35M WIT (Srinivasan et al., 2021) documents and 70M text-image pairs and a model trained on 50M mOSCAR documents from the English subset and 100M English image-text pairs as comparison baselines. We additionally compare with OpenFlamingo-3B-MPT (Awadalla et al., 2023) as the <i>translate-test</i> baseline. The full list of languages for training and the implementation details can be found in Appendix A.6.

5.1 Evaluation setup

We evaluate the models using a broad set of image-text multilingual tasks and benchmarks. We use the IGLUE benchmark (Bugliarello et al., 2022) composed of XVNLI, MaRVL (Liu et al., 2021) to test reasoning, xGQA (Pfeiffer et al., 2022) to test visual question answering capabilities and xFlickR&CO (Young et al., 2014; Karpathy and Fei-Fei, 2015; Yoshikawa et al., 2017) for captioning. We also include Crossmodal-3600 (XM3600) (Thapliyal et al., 2022) and MaXM (Changpinyo et al., 2022) as they cover a broader range of languages. To test to what extent models trained on mOSCAR can perform zero-shot multimodal machine translation (MMT), we also test on Multi30K (Elliott et al., 2016, 2017; Barrault et al., 2018) and CoMMuTE (Futerl et al., 2023). For captioning we compute the Cider (Vedantam et al., 2015) score and we tokenize references and model outputs with the Stanford Core NLP tokenizer for English and Stanza (Qi et al., 2020) tokenizers for other languages. To evaluate Multi30k, we compute BLEU (Papineni et al., 2002) score from Sacrebleu (Post, 2018) with <i>I3a</i> tokenization and default parameters. We use accuracy for CoMMuTE. More details can be found in Appendix A.6.4.

5.2 Results

Tables 5 and 8 show the average results across all languages. Full results are available in Appendix A.7. We notice that the multilingual OpenFlamingo trained additionally on mOSCAR gets better results than the model trained on captioning data only while having seen fewer image-text pairs during training. More importantly, when increasing the number of few-shot examples from
<table>
  <tr>
    <th></th>
    <th>#shots</th>
    <th>xFlickR&CO</th>
    <th>XM3600</th>
    <th>xGQA</th>
    <th>MaXM</th>
    <th>XVNLI</th>
  </tr>
  <tr>
    <td rowspan="4">Multilingual OF</td>
    <td>0</td>
    <td>29.64</td>
    <td>42.57</td>
    <td>34.24</td>
    <td>36.58</td>
    <td>34.62</td>
  </tr>
  <tr>
    <td>4</td>
    <td>51.47</td>
    <td>77.98</td>
    <td>37.91</td>
    <td>38.13</td>
    <td>33.59</td>
  </tr>
  <tr>
    <td>8</td>
    <td>56.75</td>
    <td>77.64</td>
    <td>39.44</td>
    <td><b>38.52</b></td>
    <td><b>38.75</b></td>
  </tr>
  <tr>
    <td>16</td>
    <td><b>59.89</b></td>
    <td>78.18</td>
    <td>40.09</td>
    <td>35.80</td>
    <td>36.60</td>
  </tr>
  <tr>
    <td rowspan="4">English OF</td>
    <td>0</td>
    <td>32.70</td>
    <td>43.75</td>
    <td>34.71</td>
    <td>36.19</td>
    <td>35.82</td>
  </tr>
  <tr>
    <td>4</td>
    <td>51.39</td>
    <td>75.33</td>
    <td>37.48</td>
    <td>37.96</td>
    <td>34.88</td>
  </tr>
  <tr>
    <td>8</td>
    <td>51.44</td>
    <td>77.73</td>
    <td>39.64</td>
    <td>38.35</td>
    <td>36.86</td>
  </tr>
  <tr>
    <td>16</td>
    <td>59.24</td>
    <td><b>78.38</b></td>
    <td><b>40.36</b></td>
    <td>37.35</td>
    <td>37.11</td>
  </tr>
  <tr>
    <td colspan="2"><i>mOSCAR + cap.</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 7: Results on the English subsets of the test sets and comparison between multilingual Open Flamingo and an Open Flamingo trained on the English subset of mOSCAR and English text-image pairs (English OF). Both models were trained on 50M documents from their respective training datasets and 100M text-image pairs for fair comparison. Bold is best result.

<table>
  <tr>
    <th></th>
    <th>#shots</th>
    <th>xGQA</th>
    <th>MaXM</th>
    <th>MaRVL</th>
    <th>XVNLI</th>
  </tr>
  <tr>
    <td rowspan="4">OF-3B MPT</td>
    <td>0</td>
    <td>18.34</td>
    <td>7.68</td>
    <td>49.75</td>
    <td>32.73</td>
  </tr>
  <tr>
    <td>4</td>
    <td>22.97</td>
    <td>7.82</td>
    <td>49.70</td>
    <td>35.82</td>
  </tr>
  <tr>
    <td>8</td>
    <td>28.57</td>
    <td>8.32</td>
    <td>49.71</td>
    <td>31.29</td>
  </tr>
  <tr>
    <td>16</td>
    <td>31.82</td>
    <td>9.04</td>
    <td>49.72</td>
    <td>33.29</td>
  </tr>
  <tr>
    <td rowspan="4">Multi. OF</td>
    <td>0</td>
    <td>30.16</td>
    <td>10.06</td>
    <td>49.93</td>
    <td>34.66</td>
  </tr>
  <tr>
    <td>4</td>
    <td>35.55</td>
    <td>9.89</td>
    <td>48.99</td>
    <td>36.10</td>
  </tr>
  <tr>
    <td>8</td>
    <td>36.78</td>
    <td>10.12</td>
    <td><b>50.54</b></td>
    <td><b>39.69</b></td>
  </tr>
  <tr>
    <td>16</td>
    <td><b>37.75</b></td>
    <td><b>11.49</b></td>
    <td>49.57</td>
    <td>37.97</td>
  </tr>
  <tr>
    <td colspan="2"><i>mOSCAR + cap.</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 8: Translate-test results averaged over languages where all benchmarks were translated from local languages into English using Google Translate API. Multi. OF <i>mOSCAR + cap.</i> refers to Multilingual Open Flamingo trained on mOSCAR and text-image pairs while OF-3B MPT refers to Open Flamingo (Awadalla et al., 2023) based on MPT (Team, 2023) and trained on mmc4 (Zhu et al., 2023) and English text-image pairs.

![Score differences averaged over benchmarks and languages between the model trained on mOSCAR + text-image pairs and the model trained only on text-image pairs.](page_101_1042_627_388.png)

Figure 4: Score differences averaged over benchmarks and languages between the model trained on mOSCAR + text-image pairs and the model trained only on text-image pairs. Bold is best result.

0 to 16, it sees gains of on average +6.71 points on VQA benchmarks and +19.39 Cider points on captioning benchmarks. In contrast, the model trained on text-image pairs only sees gains of +2.82 and +9.08 points respectively. In cross-modal machine translation, the model additionally trained on interleaved data is again much better than the one trained on just captioning data, which is not able to translate the Multi30k benchmark at all.\footnote{Most of the time, the model is not able to follow the prompt and only outputs the end of sequence token.} Moreover, mOSCAR helps the model to learn to zero-shot disambiguate translations as shown by the improved average score on CoMMuTE (63.54) compared to the model trained on captions only (61.36).

Multilingual Open Flamingo trained on mOSCAR & text-image pairs is also better than OpenFlamingo 3B MPT evaluated on translate test benchmarks\footnote{This means benchmarks were translated from local languages to English, using Google Translate API}. However, we obtain the best results (except for MaXM) by evaluating our multilingual Open Flamingo on the translate-test benchmarks since the underlying language model (Gemma-2B) is far better in English than other languages. We also notice that all models struggle with reasoning classification tasks (MaRVL, XVNLI) where they obtain scores close to random guessing.

Table 6 additionally shows that Multilingual Open Flamingo trained on mOSCAR obtains much better results than the same model trained on WIT for equivalent training data seen during training\footnote{We select the checkpoint of multilingual Open Flamingo trained on 35M documents and 70M captions to have fair comparison.} (except for Multi30K benchmark) which means mOSCAR is better suited than WIT for training multilingual mLlMs. Eventually, Table 7 shows that we don’t face a drop in performances in English performances when training the model on 43 languages (multilingual Open Flamingo) in comparison to training it on the English subset of mOSCAR and English text-image pairs.

Additional comparison results with InternVL2 (Chen et al., 2024), Llava-NeXT (Li et al., 2024), PaliGemma (Beyer* et al., 2024) and Idefics2 (Laurençon et al., 2024) can be found in Appendix A.8.

We additionally compare results at different training steps, defined by the number of images seen during training. Figure 4 shows the difference of averaged scores between the model trained on all data and the model trained only on text-images pairs. We notice that the gap first decreases until 20M images seen and keep
increasing over training at all training steps after that. Particularly, the gap is wider for few-shot learning.

6 Conclusion
We introduced mOSCAR, a large-scale multilingual and multimodal dataset covering 163 languages and composed of 303M documents, 200B tokens and 1.15B images. We showed that mOSCAR is of good quality, diverse and could be used to train a multilingual and multimodal LLM. We also proved that training on mOSCAR led to the emergence of in-context learning capabilities in several languages. We eventually ensured that mOSCAR is as safe as possible by applying a series of filtering steps to remove NSFW and toxic content.

Limitations
We did not conduct any analysis of the biases of mOSCAR as this is challenging in a multilingual setting. As it is crawled from the Internet, it is indeed possible that mOSCAR reflects biases widespread on it. Training a model on mOSCAR must therefore be combined with additional alignment training steps to mitigate potential biases towards groups of people. Nevertheless, by its multilingual nature, mOSCAR is a step towards the inclusion of more languages, cultures, and people in accessing mLLMs.

Acknowledgements
This work was granted access to the HPC resources of IDRIS under the allocation 2024-AD01101423R1, 2023-AD011014232 and 2023-AD011012254 made by GENCI. It was also partly funded by the last three authors’ chairs in the PRAIRIE institute funded by the French national agency ANR as part of the “Investissements d’avenir” programme under the reference ANR-19-P3IA-0001. We deeply thanks the Jean-Zay support team. We also thank Filip Šedivý for insightful discussions regarding the removal of CSAM, Thorn for having provided access to their CSAM detector, Zeeshan Khan for discussions regarding the training of the models and Victoria Le Fournier for having manually checked subsamples of NSFW images.

References
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344–4355, Marseille, France. European Language Resources Association.

Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021. Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9), pages 1–9, Limerick. Leibniz-Institut für Deutsche Sprache.

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. 2023. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390.

Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304–323.

Romain Beaumont. 2021. img2dataset: Easily turn large sets of image urls to an image dataset. https://github.com/rom1504/img2dataset.

Janek Bevendorff, Martin Potthast, and Benno Stein. 2021. Fastwarc: optimizing large-scale web archive analytics. arXiv preprint arXiv:2112.03103.

Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl. In Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York. Springer.

Lucas Beyer*, Andreas Steiner*, André Susano Pinto*, Alexander Kolesnikov*. Xiao Wang*, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai*. 2024. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726.

A.Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences 1997, pages 21–29. IEEE Computer Society.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic. 2022. Iglue: A benchmark for transfer learning across modalities, tasks, and languages. In International Conference on Machine Learning, pages 2370–2392. PMLR.

Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865–879, Toronto, Canada. Association for Computational Linguistics.

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. 2022. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset.

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Extracting training data from diffusion models. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC ’23, USA. USENIX Association.

Nicolas Carlini, Matthew Jagielski, Christopher Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. 2024. Poisoning web-scale training datasets is practical. In Proceedings of the 2024 IEEE Symposium on Security and Privacy (SP), pages 179–179, Los Alamitos, CA, USA. IEEE Computer Society.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, Nashville, TN, USA.

Soravit Changpinyo, Linting Xue, Idan Szpektor, Ashish V Thapliyal, Julien Amelot, Michal Yarom, Xi Chen, and Radu Soricut. 2022. Maxm: Towards multilingual visual question answering. arXiv preprint arXiv:2209.05401.

Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdmohsin, Piotr Padlewski, et al. 2023a. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2023b. Pali: A jointly-scaled multilingual language-image model. In Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 215–233, Copenhagen, Denmark. Association for Computational Linguistics.

Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74. Association for Computational Linguistics.

Dan Friedland and Adji Bousso Dieng. 2023. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research.

Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden. 2023. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5394–5413, Toronto, Canada. Association for Computational Linguistics.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2024. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36.

Leo Gao, Stella Biderman, Sid Black, Laurence Goldberg, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. 1999. Similarity search in high dimensions via hashing. In Proceedings of the 25th VLDB Conference, volume 99, pages 518–529, Edinburgh, Scotland, UK.

Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.

John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018. Learning translations via images with a massively multilingual image dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford-Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Henning, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. OpenCLIP.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the Thirty-Eighth International Conference on Machine Learning, pages 4904–4916, online. PMLR.

Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137.

Vaclav Kosar, Antonín Hoskovec, Milan Šulc, and Radek Bartyzal. 2022. GLAMI-IM: A Multilingual Image-Text Fashion Dataset. In Proceedings of the 33rd British Machine Vision Conference, London, UK. BMVA Press.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73.

Gant Laborde. Deep nn for nsfw detection.

Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents. In Advances in Neural Information Processing Systems, volume 36, pages 71683–71702. Curran Associates, Inc.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Fonferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaihe Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dev, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. 2022. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In Advances in Neural Information Processing Systems, volume 35, pages 31809–31826. Curran Associates, Inc.

Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246

Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, and Daniel Whitehack. 2022. Bloom library: Multimodal datasets in 300+ languages for a variety of downstream tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8608–8621, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Vladimir Iosifovich Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8):707–710. Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.

Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and
Chunyuan Li. 2024. Llava-next: Stronger llms supercharge multimodal capabilities in the wild.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, pages 19730—19742, Honolulu Hawaii USA.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39 th International Conference on Machine Learning, pages 12888–12900, Baltimore, Maryland, USA. PMLR.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the 13th European Conference on Computer Vision, pages 740–755, Zurich, Switzerland. Springer.

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad S Khan. 2024. PALO: A Polyglot Large Multimodal Model for 5B People. arXiv preprint arXiv:2402.14818.

Chameleon Team. 2024. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dasdashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv:2403.08295.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dutter, Dhruvi Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. 2024. MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training. arXiv preprint arXiv:2403.09611.

Owais Khan Mohammed, Kriti Aggarwal, Qiang Liu, Saksham Singhal, Johan Bjorck, and Subhojit Som. 2023. Bootstrapping a high quality multilingual multimodal dataset for Bletchley. In Proceedings of The 14th Asian Conference on Machine Learning, volume 189 of Proceedings of Machine Learning Research, pages 738–753. PMLR.

OpenAI. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08774.

Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2Text: Describing Images Using 1 Million Captioned Photographs. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc.

Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7), pages 9 – 16, Cardiff, UK. Leibniz-Institut für Deutsche Sprache.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems, volume 36, pages 79155–79172. Curran Associates, Inc.
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022. xGQA: Cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511, Dublin, Ireland. Association for Computational Linguistics.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, Santiago, Chile.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38 th International Conference on Machine Learning, pages 8748–8763, online. PMLR.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Itsev. 2022. Laion-Sb: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, volume 35, pages 25278–25294. Curran Associates, Inc.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Itsev, and Aran Komatsuzaki. 2021. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. In Proceedings of the Data Centric AI NeurIPS Workshop 2021, online.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. arXiv preprint arXiv:2201.11990.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048–6058.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36.

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443–2449, online.

Didac Surís, Dave Epstein, and Carl Vondrick. 2022. Globetrotter: Connecting languages by connecting images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16474–16484, Canada.

MosaicML NLP Team. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05.

Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326–346, Online. Association for Computational Linguistics.

Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A massively
multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikoly Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.

Alexander Visheratin. 2023. NLLB-CLIP – train performant multilingual image retrieval model on a budget. arXiv preprint arXiv:2309.01859.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. CogVLM: Visual Expert for Pretrained Language Models. arXiv preprint arXiv:2311.03079.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442.

Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. 2023. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.12733.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. 2017. STAIR captions: Constructing a large-scale Japanese image caption dataset. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 417–421, Vancouver, Canada. Association for Computational Linguistics.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986.

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939.
A Appendix

A.1 mOSCAR languages & statistics
Table 9 shows the number of documents, the number of images, and the number of NLLB tokens for each language of mOSCAR.

A.2 Topic modeling
We analyzed the topic found in different language subsets of mOSCAR with BERTopic (Grootendorst, 2022). We first preprocessed ~30K randomly sampled documents in tokens using stanza tokenizers (Qi et al., 2020), we then represent each document with their document embedding obtained with a multilingual sentence-bert23 (Reimers and Gurevych, 2019) before running BERTopic to cluster the documents. Tables 10 to 14 show some of the most salient English-translated strings representing each cluster and their weight (in percentage) in the subsets of mOSCAR. While many clusters are shared between subsets, we can observe that some of them are culturally-specific (i.e. they are related to the culture of the language subset and only appears on this subset). For instance, Table 14 shows topics of the Hindi subset and we observe a cluster related to Hindu gods described by the words ‘ganesha’, ‘shiva’ or ‘shani dev’. We also observe a cluster related to festivals hosted in India, it is described by the words ‘festival’, ‘dussehra’ or ‘janmashtami’. We can also observe that among clusters shared between several language subsets, some contain knowledge culturally-specific as exemplified by the cluster related to jewelry in the Hindi subset which contain the word ‘mangalsutra’ which is a jewel worn by men during weddings in India. Eventually, we can observe that the English subset contains a strong cluster related to the Christian religion. While a similar cluster appears in the Tosk Albanian subset, it is related to Islam. This shows the importance of a multilingual corpora to have data representing more cultures around the world.

A.3 Examples of documents
Figures 5 to 10 provide examples of documents found in mOSCAR for different languages. We can observe the diversity in terms of content of the documents. Figure 5 is a French document describing a specific shot of golf illustrated by images which provide very detailed description of images. Figure 7 shows a Russian document describing an Instagram account with pictures of food with animal heads which is reminiscent of a famous problem of computer vision consisting in classifying images between chihuahuas and muffins.

23sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
<table>
  <tr>
    <th>Lang. name</th>
    <th>Code</th>
    <th>Family</th>
    <th>Script</th>
    <th>#documents</th>
    <th>#images</th>
    <th>#tokens</th>
  </tr>
  <tr><td>Acehnese</td><td>ace_Latn</td><td>Austronesian</td><td>Latin</td><td>2,159</td><td>9,026</td><td>1,395,381</td></tr>
  <tr><td>Mesopotamian Arabic</td><td>acm_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>1,282</td><td>5,621</td><td>704,549</td></tr>
  <tr><td>Tunisian Arabic</td><td>aeb_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>5,933</td><td>34,270</td><td>2,308,455</td></tr>
  <tr><td>Afrikaans</td><td>afr_Latn</td><td>Indo-European</td><td>Latin</td><td>50,061</td><td>211,876</td><td>38,761,504</td></tr>
  <tr><td>South Levantine Arabic</td><td>ajp_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>8,603</td><td>69,051</td><td>3,869,688</td></tr>
  <tr><td>Tosk Albanian</td><td>als_Latn</td><td>Indo-European</td><td>Latin</td><td>856,144</td><td>2,543,758</td><td>441,244,377</td></tr>
  <tr><td>Amharic</td><td>amb_Ethi</td><td>Afro-Asiatic</td><td>Ge'ez</td><td>39,031</td><td>149,739</td><td>33,768,732</td></tr>
  <tr><td>North Levantine Arabic</td><td>apc_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>16,198</td><td>110,792</td><td>8,268,237</td></tr>
  <tr><td>Modern Standard Arabic</td><td>arb_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>3,794,792</td><td>14,757,353</td><td>3,346,786,610</td></tr>
  <tr><td>Najdi Arabic</td><td>ars_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>52,102</td><td>261,275</td><td>39,066,487</td></tr>
  <tr><td>Moroccan Arabic</td><td>ary_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>117,957</td><td>584,301</td><td>188,462,338</td></tr>
  <tr><td>Egyptian Arabic</td><td>arz_Arab</td><td>Afro-Asiatic</td><td>Arabic</td><td>761,113</td><td>3,785,164</td><td>635,018,784</td></tr>
  <tr><td>Assamese</td><td>asm_Beng</td><td>Indo-European</td><td>Bengali</td><td>2,947</td><td>7,228</td><td>543,676</td></tr>
  <tr><td>Asturian</td><td>ast_Latn</td><td>Indo-European</td><td>Latin</td><td>87,649</td><td>533,723</td><td>25,499,269</td></tr>
  <tr><td>Awadhi</td><td>awa_Deva</td><td>Indo-European</td><td>Devanagari</td><td>8,179</td><td>29,142</td><td>2,293,620</td></tr>
  <tr><td>Central Aymara</td><td>ayr_Latn</td><td>Aymaran</td><td>Latin</td><td>10,112</td><td>57,294</td><td>2,343,403</td></tr>
  <tr><td>South Azerbaijani</td><td>azb_Arab</td><td>Turkic</td><td>Arabic</td><td>3,411</td><td>14,825</td><td>3,143,946</td></tr>
  <tr><td>North Azerbaijani</td><td>azj_Latn</td><td>Turkic</td><td>Latin</td><td>511,832</td><td>1,796,046</td><td>256,160,442</td></tr>
  <tr><td>Bashkir</td><td>bak_Cyril</td><td>Turkic</td><td>Cyrillic</td><td>3,287</td><td>12,031</td><td>2,600,135</td></tr>
  <tr><td>Bambara</td><td>bam_Latn</td><td>Manding</td><td>Latin</td><td>3,011</td><td>17,666</td><td>446,961</td></tr>
  <tr><td>Balinese</td><td>ban_Latn</td><td>Austronesian</td><td>Latin</td><td>787</td><td>4,894</td><td>392,978</td></tr>
  <tr><td>Belarusian</td><td>bel_Cyril</td><td>Indo-European</td><td>Cyrillic</td><td>60,443</td><td>276,672</td><td>71,854,171</td></tr>
  <tr><td>Bemba</td><td>bem_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>582</td><td>3,018</td><td>1,021,026</td></tr>
  <tr><td>Bengali</td><td>ben_Beng</td><td>Indo-European</td><td>Bengali</td><td>204,475</td><td>758,222</td><td>30,400,395</td></tr>
  <tr><td>Bhojpuri</td><td>bho_Deva</td><td>Indo-European</td><td>Devanagari</td><td>4,190</td><td>18,339</td><td>715,786</td></tr>
  <tr><td>Banjari</td><td>bjn_Latn</td><td>Austronesian</td><td>Latin</td><td>1,764</td><td>9,017</td><td>1,093,443</td></tr>
  <tr><td>Bosnian</td><td>bos_Latn</td><td>Indo-European</td><td>Latin</td><td>635,750</td><td>2,642,491</td><td>423,073,661</td></tr>
  <tr><td>Buginese</td><td>bug_Latn</td><td>Austronesian</td><td>Latin</td><td>584</td><td>2,379</td><td>167,459</td></tr>
  <tr><td>Bulgarian</td><td>bul_Cyril</td><td>Indo-European</td><td>Cyrillic</td><td>2,578,191</td><td>11,601,214</td><td>1,736,106,287</td></tr>
  <tr><td>Catalan</td><td>cat_Latn</td><td>Indo-European</td><td>Latin</td><td>1,132,056</td><td>4,638,966</td><td>598,942,711</td></tr>
  <tr><td>Cebuano</td><td>ceb_Latn</td><td>Austronesian</td><td>Latin</td><td>14,924</td><td>75,258</td><td>10,221,371</td></tr>
  <tr><td>Czech</td><td>ces_Latn</td><td>Indo-European</td><td>Latin</td><td>3,736,126</td><td>12,683,461</td><td>2,767,295,966</td></tr>
  <tr><td>Central Kurdish</td><td>ckb_Arab</td><td>Indo-European</td><td>Arabic</td><td>36,413</td><td>135,461</td><td>21,622,335</td></tr>
  <tr><td>Crimean Tatar</td><td>crh_Latn</td><td>Turkic</td><td>Latin</td><td>2,744</td><td>10,079</td><td>1,173,321</td></tr>
  <tr><td>Welsh</td><td>cym_Latn</td><td>Indo-European</td><td>Latin</td><td>38,616</td><td>155,591</td><td>27,237,252</td></tr>
  <tr><td>Danish</td><td>dan_Latn</td><td>Indo-European</td><td>Latin</td><td>2,020,516</td><td>9,214,031</td><td>1,207,829,704</td></tr>
  <tr><td>German</td><td>deu_Latn</td><td>Indo-European</td><td>Latin</td><td>20,265,504</td><td>86,393,702</td><td>8,315,212,019</td></tr>
  <tr><td>Southwestern Dinka</td><td>dik_Latn</td><td>Nilo-Saharan</td><td>Latin</td><td>1,233</td><td>4,766</td><td>1,098,795</td></tr>
  <tr><td>Greek</td><td>ell_Grek</td><td>Indo-European</td><td>Greek</td><td>4,895,433</td><td>15,147,284</td><td>2,909,427,055</td></tr>
  <tr><td>English</td><td>eng_Latn</td><td>Indo-European</td><td>Latin</td><td>51,658,029</td><td>205,363,181</td><td>32,599,001,993</td></tr>
  <tr><td>Esperanto</td><td>epo_Latn</td><td>Artificial</td><td>Latin</td><td>23,619</td><td>112,577</td><td>26,976,847</td></tr>
  <tr><td>Estonian</td><td>est_Latn</td><td>Uralic</td><td>Latin</td><td>1,022,368</td><td>5,108,102</td><td>589,045,973</td></tr>
  <tr><td>Basque</td><td>eus_Latn</td><td>Isolate</td><td>Latin</td><td>682,599</td><td>2,914,120</td><td>259,930,954</td></tr>
  <tr><td>Faroese</td><td>fao_Latn</td><td>Indo-European</td><td>Latin</td><td>14,921</td><td>56,934</td><td>6,579,921</td></tr>
  <tr><td>Fijian</td><td>fij_Latn</td><td>Austronesian</td><td>Latin</td><td>1,039</td><td>4,039</td><td>416,670</td></tr>
  <tr><td>Finnish</td><td>fin_Latn</td><td>Uralic</td><td>Latin</td><td>2,377,155</td><td>10,263,171</td><td>1,749,904,041</td></tr>
  <tr><td>French</td><td>fra_Latn</td><td>Indo-European</td><td>Latin</td><td>19,963,542</td><td>76,851,982</td><td>13,818,099,493</td></tr>
  <tr><td>Friulian</td><td>fur_Latn</td><td>Indo-European</td><td>Latin</td><td>15,823</td><td>120,878</td><td>2,550,209</td></tr>
  <tr><td>Nigerian Fulfulde</td><td>fuw_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>919</td><td>4,281</td><td>264,234</td></tr>
  <tr><td>West Central Oromo</td><td>gaz_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>3,399</td><td>9,071</td><td>1,640,693</td></tr>
  <tr><td>Scottish Gaelic</td><td>gla_Latn</td><td>Indo-European</td><td>Latin</td><td>19,638</td><td>105,957</td><td>13,119,348</td></tr>
  <tr><td>Irish</td><td>gle_Latn</td><td>Indo-European</td><td>Latin</td><td>60,303</td><td>267,562</td><td>45,341,371</td></tr>
  <tr><td>Galician</td><td>glg_Latn</td><td>Indo-European</td><td>Latin</td><td>410,489</td><td>1,696,763</td><td>197,685,077</td></tr>
  <tr><td>Guarani</td><td>grn_Latn</td><td>Tupian</td><td>Latin</td><td>207,800</td><td>1,038,296</td><td>48,610,979</td></tr>
  <tr><td>Gujarati</td><td>guj_Gujr</td><td>Indo-European</td><td>Gujarati</td><td>21,916</td><td>87,805</td><td>3,202,096</td></tr>
  <tr><td>Haitian Creole</td><td>hat_Latn</td><td>Indo-European</td><td>Latin</td><td>105,777</td><td>667,801</td><td>34,261,838</td></tr>
  <tr><td>Hausa</td><td>hau_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>21,850</td><td>81,141</td><td>11,807,898</td></tr>
  <tr><td>Hebrew</td><td>heb_Hebr</td><td>Afro-Asiatic</td><td>Hebrew</td><td>1,098,800</td><td>4,708,947</td><td>859,238,720</td></tr>
  <tr><td>Hindi</td><td>hin_Deva</td><td>Indo-European</td><td>Devanagari</td><td>543,928</td><td>1,745,222</td><td>118,903,998</td></tr>
  <tr><td>Chhattisgarhi</td><td>hne_Deva</td><td>Indo-European</td><td>Devanagari</td><td>832</td><td>3,908</td><td>205,345</td></tr>
  <tr><td>Croatian</td><td>hrv_Latn</td><td>Indo-European</td><td>Latin</td><td>1,689,553</td><td>8,315,237</td><td>998,928,993</td></tr>
  <tr><td>Hungarian</td><td>hun_Latn</td><td>Uralic</td><td>Latin</td><td>3,515,058</td><td>15,293,132</td><td>2,811,446,583</td></tr>
  <tr><td>Armenian</td><td>hye_Armn</td><td>Indo-European</td><td>Armenian</td><td>336,285</td><td>1,126,920</td><td>199,883,484</td></tr>
  <tr><td>Igbo</td><td>ibo_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>7,089</td><td>41,672</td><td>3,014,602</td></tr>
  <tr><td>Ilocano</td><td>ilo_Latn</td><td>Austronesian</td><td>Latin</td><td>7,076</td><td>59,327</td><td>832,454</td></tr>
  <tr><td>Indonesian</td><td>ind_Latn</td><td>Austronesian</td><td>Latin</td><td>6,644,918</td><td>16,237,247</td><td>2,895,956,979</td></tr>
</table>
<table>
  <tr>
    <th>Lang. name</th>
    <th>Code</th>
    <th>Family</th>
    <th>Script</th>
    <th>#documents</th>
    <th>#images</th>
    <th>#tokens</th>
  </tr>
  <tr><td>Icelandic</td><td>isl_Latn</td><td>Indo-European</td><td>Latin</td><td>239,195</td><td>1,003,522</td><td>131,308,802</td></tr>
  <tr><td>Italian</td><td>ita_Latn</td><td>Indo-European</td><td>Latin</td><td>12,812,932</td><td>47,011,085</td><td>8,144,757,759</td></tr>
  <tr><td>Javanese</td><td>jav_Latn</td><td>Austronesian</td><td>Latin</td><td>18,192</td><td>100,952</td><td>15,206,708</td></tr>
  <tr><td>Japanese</td><td>jpn_Jpan</td><td>Japonic</td><td>Kanji</td><td>14,154,575</td><td>23,435,549</td><td>8,539,956,266</td></tr>
  <tr><td>Kabyle</td><td>kab_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>6,101</td><td>33,923</td><td>1,781,992</td></tr>
  <tr><td>Kannada</td><td>kan_Knda</td><td>Dravidian</td><td>Kannada</td><td>9,373</td><td>33,147</td><td>1,206,651</td></tr>
  <tr><td>Kashmiri</td><td>kas_Arab</td><td>Indo-European</td><td>Arabic</td><td>1,498</td><td>5,284</td><td>3,384,394</td></tr>
  <tr><td>Georgian</td><td>kat_Geor</td><td>Kartvelian</td><td>Georgian</td><td>353,471</td><td>1,300,710</td><td>274,042,522</td></tr>
  <tr><td>Kazakh</td><td>kaz_Cyril</td><td>Turkic</td><td>Cyrillic</td><td>248,403</td><td>718,126</td><td>138,597,176</td></tr>
  <tr><td>Halh Mongolian</td><td>khk_Cyril</td><td>Mongolic</td><td>Cyrillic</td><td>123,789</td><td>505,098</td><td>83,628,495</td></tr>
  <tr><td>Khmer</td><td>khm_Khmr</td><td>Austronesian</td><td>Kher</td><td>23,348</td><td>116,437</td><td>2,915,205</td></tr>
  <tr><td>Kinyarwanda</td><td>kin_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>20,381</td><td>108,280</td><td>10,268,334</td></tr>
  <tr><td>Kyrgyz</td><td>kir_Cyril</td><td>Uralic</td><td>Cyrillic</td><td>51,221</td><td>194,092</td><td>33,981,180</td></tr>
  <tr><td>Northern Kurdish</td><td>kmr_Latn</td><td>Indo-European</td><td>Latin</td><td>34,593</td><td>142,634</td><td>21,972,155</td></tr>
  <tr><td>Korean</td><td>kor_Hang</td><td>Koreanic</td><td>Hanja</td><td>2,614,038</td><td>13,562,957</td><td>2,000,344,511</td></tr>
  <tr><td>Lao</td><td>lao_Laoo</td><td>Kra-Dai</td><td>Lao</td><td>49,925</td><td>205,452</td><td>30,098,274</td></tr>
  <tr><td>Ligurian</td><td>lij_Latn</td><td>Indo-European</td><td>Latin</td><td>3,581</td><td>26,740</td><td>1,046,463</td></tr>
  <tr><td>Limburgish</td><td>lim_Latn</td><td>Indo-European</td><td>Latin</td><td>70,099</td><td>443,903</td><td>25,465,590</td></tr>
  <tr><td>Lingala</td><td>lin_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>6,304</td><td>41,400</td><td>1,580,536</td></tr>
  <tr><td>Lithuanian</td><td>lit_Latn</td><td>Indo-European</td><td>Latin</td><td>1,673,790</td><td>8,772,570</td><td>1,153,604,941</td></tr>
  <tr><td>Lombard</td><td>lmo_Latn</td><td>Indo-European</td><td>Latin</td><td>14,053</td><td>61,359</td><td>6,270,646</td></tr>
  <tr><td>Latgalian</td><td>ltg_Latn</td><td>Indo-European</td><td>Latin</td><td>5,174</td><td>21,062</td><td>2,903,043</td></tr>
  <tr><td>Luxembourgish</td><td>ltz_Latn</td><td>Indo-European</td><td>Latin</td><td>27,946</td><td>142,470</td><td>13,925,521</td></tr>
  <tr><td>Ganda</td><td>lug_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>1,475</td><td>4,118</td><td>688,308</td></tr>
  <tr><td>Mizo</td><td>lus_Latn</td><td>Sino-Tibetan</td><td>Latin</td><td>7,009</td><td>22,630</td><td>4,106,536</td></tr>
  <tr><td>Standard Latvian</td><td>lvu_Latn</td><td>Indo-European</td><td>Latin</td><td>857,757</td><td>3,937,940</td><td>578,441,751</td></tr>
  <tr><td>Magahi</td><td>mag_Deva</td><td>Indo-European</td><td>Devanagari</td><td>290</td><td>1,088</td><td>94,031</td></tr>
  <tr><td>Malayalam</td><td>mal_Mymr</td><td>Dravidian</td><td>Malayalam</td><td>11,203</td><td>44,417</td><td>1,420,906</td></tr>
  <tr><td>Marathi</td><td>mar_Deva</td><td>Indo-European</td><td>Devanagari</td><td>43,720</td><td>142,001</td><td>6,164,176</td></tr>
  <tr><td>Minangkabau</td><td>min_Latn</td><td>Austronesian</td><td>Latin</td><td>1,523</td><td>7,500</td><td>447,320</td></tr>
  <tr><td>Macedonian</td><td>mkd_Cyril</td><td>Indo-European</td><td>Cyrillic</td><td>539,149</td><td>1,841,846</td><td>304,592,615</td></tr>
  <tr><td>Maltese</td><td>mlt_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>56,666</td><td>327,331</td><td>27,114,870</td></tr>
  <tr><td>Maori</td><td>mri_Latn</td><td>Austronesian</td><td>Latin</td><td>20,840</td><td>114,680</td><td>24,524,962</td></tr>
  <tr><td>Burmese</td><td>mya_Mymr</td><td>Sino-Tibetan</td><td>Mon</td><td>6,575</td><td>36,661</td><td>406,016</td></tr>
  <tr><td>Dutch</td><td>nld_Latn</td><td>Indo-European</td><td>Latin</td><td>16,890,074</td><td>64,609,055</td><td>9,493,533,101</td></tr>
  <tr><td>Norwegian Nynorsk</td><td>nno_Latn</td><td>Indo-European</td><td>Latin</td><td>138,384</td><td>701,972</td><td>57,812,652</td></tr>
  <tr><td>Norwegian Bokmål</td><td>nob_Latn</td><td>Indo-European</td><td>Latin</td><td>2,192,012</td><td>9,534,178</td><td>1,267,421,216</td></tr>
  <tr><td>Nepali</td><td>npi_Deva</td><td>Indo-European</td><td>Devanagari</td><td>28,042</td><td>116,363</td><td>2,892,865</td></tr>
  <tr><td>Nyanja</td><td>nya_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>11,749</td><td>65,324</td><td>8,513,823</td></tr>
  <tr><td>Occitan</td><td>oci_Latn</td><td>Indo-European</td><td>Latin</td><td>61,681</td><td>323,632</td><td>21,029,975</td></tr>
  <tr><td>Odia</td><td>ory_Orya</td><td>Indo-European</td><td>Odia</td><td>3,759</td><td>14,373</td><td>340,695</td></tr>
  <tr><td>Pangasinan</td><td>pag_Latn</td><td>Austronesian</td><td>Latin</td><td>1,045</td><td>7,770</td><td>270,363</td></tr>
  <tr><td>Eastern Panjabi</td><td>pan_Guru</td><td>Indo-European</td><td>Gurmukhi</td><td>10,857</td><td>44,440</td><td>1,821,511</td></tr>
  <tr><td>Papiamento</td><td>pap_Latn</td><td>Indo-European</td><td>Latin</td><td>29,564</td><td>177,229</td><td>7,396,392</td></tr>
  <tr><td>Southern Pashto</td><td>pbt_Arab</td><td>Indo-European</td><td>Arabic</td><td>31,854</td><td>107,563</td><td>27,623,486</td></tr>
  <tr><td>Western Persian</td><td>pes_Arab</td><td>Indo-European</td><td>Arabic</td><td>6,995,368</td><td>24,998,370</td><td>6,061,794,870</td></tr>
  <tr><td>Plateau Malgasy</td><td>plt_Latn</td><td>Austronesian</td><td>Latin</td><td>32,119</td><td>119,506</td><td>28,542,084</td></tr>
  <tr><td>Polish</td><td>pol_Latn</td><td>Indo-European</td><td>Latin</td><td>14,492,239</td><td>60,362,860</td><td>10,994,239,010</td></tr>
  <tr><td>Portuguese</td><td>por_Latn</td><td>Indo-European</td><td>Latin</td><td>8,033,406</td><td>26,058,040</td><td>4,639,089,792</td></tr>
  <tr><td>Dari</td><td>prs_Arab</td><td>Indo-European</td><td>Arabic</td><td>421,097</td><td>2,101,038</td><td>399,037,437</td></tr>
  <tr><td>Ayacucho Quechua</td><td>quy_Latn</td><td>Quechuan</td><td>Latin</td><td>1,248</td><td>10,038</td><td>322,112</td></tr>
  <tr><td>Romanian</td><td>ron_Latn</td><td>Indo-European</td><td>Latin</td><td>5,131,444</td><td>17,790,793</td><td>3,484,865,185</td></tr>
  <tr><td>Rundi</td><td>run_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>17,798</td><td>55,060</td><td>8,140,230</td></tr>
  <tr><td>Russian</td><td>rus_Cyril</td><td>Indo-European</td><td>Cyrillic</td><td>15,753,144</td><td>68,786,134</td><td>18,196,141,357</td></tr>
  <tr><td>Sango</td><td>sag_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>724</td><td>4,564</td><td>181,876</td></tr>
  <tr><td>Sicilian</td><td>scn_Latn</td><td>Indo-European</td><td>Latin</td><td>27,388</td><td>164,772</td><td>17,535,500</td></tr>
  <tr><td>Sinhala</td><td>sin_Sinh</td><td>Indo-European</td><td>Sinhalese</td><td>44,963</td><td>179,082</td><td>11,413,044</td></tr>
  <tr><td>Slovak</td><td>slk_Latn</td><td>Indo-European</td><td>Latin</td><td>2,979,681</td><td>14,894,160</td><td>1,951,406,321</td></tr>
  <tr><td>Slovenian</td><td>sli_Latn</td><td>Indo-European</td><td>Latin</td><td>1,456,026</td><td>7,106,291</td><td>928,101,642</td></tr>
  <tr><td>Samoan</td><td>smo_Latn</td><td>Austronesian</td><td>Latin</td><td>11,024</td><td>62,358</td><td>11,672,900</td></tr>
  <tr><td>Shona</td><td>sna_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>7,400</td><td>41,385</td><td>5,276,139</td></tr>
  <tr><td>Sindhi</td><td>snd_Arab</td><td>Indo-European</td><td>Arabic</td><td>20,615</td><td>70,992</td><td>16,686,668</td></tr>
  <tr><td>Somali</td><td>som_Latn</td><td>Afro-Asiatic</td><td>Latin</td><td>58,151</td><td>209,905</td><td>31,093,227</td></tr>
  <tr><td>Southern Sotho</td><td>sot_Latn</td><td>Atlantic-Congo</td><td>Latin</td><td>7,474</td><td>41,714</td><td>5,876,842</td></tr>
  <tr><td>Spanish</td><td>spa_Latn</td><td>Indo-European</td><td>Latin</td><td>22,218,630</td><td>76,372,709</td><td>13,882,047,139</td></tr>
  <tr><td>Sardinian</td><td>srd_Latn</td><td>Indo-European</td><td>Latin</td><td>336,476</td><td>2,220,976</td><td>68,281,992</td></tr>
  <tr><td>Serbian</td><td>srp_Cyril</td><td>Indo-European</td><td>Cyrillic</td><td>593,332</td><td>2,251,042</td><td>394,477,097</td></tr>
</table>
<table>
  <tr>
    <th>Lang. name</th>
    <th>Code</th>
    <th>Family</th>
    <th>Script</th>
    <th>#documents</th>
    <th>#images</th>
    <th>#tokens</th>
  </tr>
  <tr>
    <td>Sundanese</td>
    <td>sun_Latn</td>
    <td>Austronesian</td>
    <td>Latin</td>
    <td>16,438</td>
    <td>89,379</td>
    <td>9,549,957</td>
  </tr>
  <tr>
    <td>Swedish</td>
    <td>swe_Latn</td>
    <td>Indo-European</td>
    <td>Latin</td>
    <td>3,231,753</td>
    <td>10,558,719</td>
    <td>1,748,495,813</td>
  </tr>
  <tr>
    <td>Swahili</td>
    <td>swh_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>96,770</td>
    <td>365,792</td>
    <td>52,827,863</td>
  </tr>
  <tr>
    <td>Silesian</td>
    <td>szl_Latn</td>
    <td>Indo-European</td>
    <td>Latin</td>
    <td>7,846</td>
    <td>47,313</td>
    <td>3,022,502</td>
  </tr>
  <tr>
    <td>Tamil</td>
    <td>tam_Taml</td>
    <td>Dravidian</td>
    <td>Tamil</td>
    <td>30,202</td>
    <td>149,837</td>
    <td>4,234,345</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>tat_Cyril</td>
    <td>Turkic</td>
    <td>Cyrillic</td>
    <td>34,489</td>
    <td>133,014</td>
    <td>22,255,423</td>
  </tr>
  <tr>
    <td>Telugu</td>
    <td>tel_Telu</td>
    <td>Dravidian</td>
    <td>Telugu</td>
    <td>16,107</td>
    <td>54,100</td>
    <td>1,633,579</td>
  </tr>
  <tr>
    <td>Tajik</td>
    <td>tgk_Cyril</td>
    <td>Turkic</td>
    <td>Cyrillic</td>
    <td>119,383</td>
    <td>395,470</td>
    <td>87,519,228</td>
  </tr>
  <tr>
    <td>Tagalog</td>
    <td>tgl_Latn</td>
    <td>Austronesian</td>
    <td>Latin</td>
    <td>140,922</td>
    <td>628,210</td>
    <td>95,285,900</td>
  </tr>
  <tr>
    <td>Thai</td>
    <td>tha_Thai</td>
    <td>Kra-Dai</td>
    <td>Thai</td>
    <td>1,799,735</td>
    <td>6,603,060</td>
    <td>807,374,946</td>
  </tr>
  <tr>
    <td>Tigrinya</td>
    <td>tir_Ethi</td>
    <td>Afro-Asiatic</td>
    <td>Ge'ez</td>
    <td>2,622</td>
    <td>8,601</td>
    <td>1,699,272</td>
  </tr>
  <tr>
    <td>Tok Pisin</td>
    <td>tpi_Latn</td>
    <td>Indo-European</td>
    <td>Latin</td>
    <td>785</td>
    <td>5,888</td>
    <td>97,298</td>
  </tr>
  <tr>
    <td>Turkmen</td>
    <td>tuk_Latn</td>
    <td>Turkic</td>
    <td>Latin</td>
    <td>12,372</td>
    <td>54,002</td>
    <td>9,650,172</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>tur_Latn</td>
    <td>Turkic</td>
    <td>Latin</td>
    <td>4,448,111</td>
    <td>12,304,912</td>
    <td>2,356,627,784</td>
  </tr>
  <tr>
    <td>Twi</td>
    <td>twi_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>286</td>
    <td>2,041</td>
    <td>78,227</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>uig_Arab</td>
    <td>Turkic</td>
    <td>Arabic</td>
    <td>10,614</td>
    <td>41,567</td>
    <td>6,602,690</td>
  </tr>
  <tr>
    <td>Ukrainian</td>
    <td>ukr_Cyril</td>
    <td>Indo-European</td>
    <td>Cyrillic</td>
    <td>2,689,369</td>
    <td>10,842,572</td>
    <td>1,909,330,669</td>
  </tr>
  <tr>
    <td>Urdu</td>
    <td>urd_Arab</td>
    <td>Indo-European</td>
    <td>Arabic</td>
    <td>403,245</td>
    <td>1,224,175</td>
    <td>236,356,788</td>
  </tr>
  <tr>
    <td>Northern Uzbek</td>
    <td>uzn_Latn</td>
    <td>Turkic</td>
    <td>Latin</td>
    <td>113,772</td>
    <td>581,861</td>
    <td>81,808,833</td>
  </tr>
  <tr>
    <td>Venetian</td>
    <td>vec_Latn</td>
    <td>Indo-European</td>
    <td>Latin</td>
    <td>122,390</td>
    <td>763,029</td>
    <td>24,081,966</td>
  </tr>
  <tr>
    <td>Vietnamese</td>
    <td>vie_Latn</td>
    <td>Viet-Muong</td>
    <td>Latin</td>
    <td>12,296,989</td>
    <td>46,339,341</td>
    <td>11,462,111,787</td>
  </tr>
  <tr>
    <td>Wolof</td>
    <td>wol_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>2,152</td>
    <td>9,351</td>
    <td>367,848</td>
  </tr>
  <tr>
    <td>Xhosa</td>
    <td>xho_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>13,620</td>
    <td>80,748</td>
    <td>14,566,904</td>
  </tr>
  <tr>
    <td>Eastern Yiddish</td>
    <td>ydd_Hebr</td>
    <td>Indo-European</td>
    <td>Hebrew</td>
    <td>12,275</td>
    <td>56,421</td>
    <td>17,078,751</td>
  </tr>
  <tr>
    <td>Yoruba</td>
    <td>yor_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>10,148</td>
    <td>49,474</td>
    <td>8,346,193</td>
  </tr>
  <tr>
    <td>Yue Chinese</td>
    <td>yue_Hant</td>
    <td>Sino-Tibetan</td>
    <td>Hant</td>
    <td>28,478</td>
    <td>172,592</td>
    <td>21,579,579</td>
  </tr>
  <tr>
    <td>Chinese (Simplified)</td>
    <td>zho_Hans</td>
    <td>Sino-Tibetan</td>
    <td>Hanzi</td>
    <td>8,326,440</td>
    <td>29,575,591</td>
    <td>5,199,137,981</td>
  </tr>
  <tr>
    <td>Chinese (Traditional)</td>
    <td>zho_Hant</td>
    <td>Sino-Tibetan</td>
    <td>Hant</td>
    <td>3,796,336</td>
    <td>15,514,804</td>
    <td>2,617,463,485</td>
  </tr>
  <tr>
    <td>Standard Malay</td>
    <td>zsm_Latn</td>
    <td>Austronesian</td>
    <td>Latin</td>
    <td>864,831</td>
    <td>3,651,754</td>
    <td>384,708,004</td>
  </tr>
  <tr>
    <td>Zulu</td>
    <td>zul_Latn</td>
    <td>Atlantic-Congo</td>
    <td>Latin</td>
    <td>13,089</td>
    <td>73,167</td>
    <td>9,654,461</td>
  </tr>
</table>

Table 9: Languages & Statistics
A.4 Heuristics to increase the quality of documents
We use a set of heuristics to improve the quality of the documents by discarding some text nodes. We first consider text nodes to be written in Latin scripts if more than 50% of the characters are Latin. In detail, we discard the text node if:

1. It is empty.

2. It contains fewer than 5 bytes for Latin scripts and fewer than 15 bytes for non-Latin scripts.

3. More than 30% of the characters are digits.

4. It contains more than one date.

5. It contains the sequence “lorem ipsum”.

6. The ratio of non-alphabetic characters is superior to 0.33.

7. The symbols ‘{’ or ‘}’ are in the text.

8. The symbols ‘≥’, ‘≤’, ‘>’ or ‘<’ are more than 2 times in the text.

9. “Follow us”, “javascript”, “copyright” or “©” are in the text.

10. The ratio of capitalized letters is superior to 0.2.

11. The text exactly matches with “comment”, “facebook”, “instagram”, “twitter”, “rss”, “newsletter”, “share” or “follow us”.

12. A character is more than 33% of the total number of characters in the string.

We then also apply some filters to clean the text as much as possible:

1. Remove URLs from all documents.

2. Normalize consecutive special characters (‘\t’, ‘\n’, ‘#’, ‘/’, ‘$’, ‘)’, ‘(’, ‘[’, ‘]’, ‘!’, ‘?’, ‘%’, ‘<’, ‘>’) to keep only one.

Following previous steps, we keep the text node if it is superior to 5 bytes and we keep the final document if it is superior to 100 bytes.

A.5 Text-Image similarity and DOM Tree
As we rely on the DOM Tree to build the documents and the order of appearance of the nodes could differ from HTML rendering, we attempt to assess to what extent it is a relevant way of constructing a multimodal document. To do so, we rely on the results of the text-image joint filtering step where we compute the ranks of relevant text nodes (resp images) for each image. We plot the distribution of the closest most relevant node for each modality in Figures 11a and 11b. We notice that the most relevant node to either a text node or an image is their closest node in the DOM tree. The cumulative distribution function of the distribution of the closest node reaches 25% for nodes positioned between -5 and 5, which confirms the relevance of using the DOM tree to represent a document.

A.6 Implementation details

A.6.1 Text deduplication parameters
Following previous work, we near-deduplicate documents using MinHashLSH. We first vectorize the documents using HashingVectorizer from scikit-learn with 2,097,152 features computed on 4-grams and 5-grams within word boundaries. We then compute MinHashes from those vectors with 256 permutations and we finally run Locality Sensitive Hashing with a threshold Jaccard Similarity of 0.8 for finding near-duplicates.

A.6.2 Removing Personal Identifiable Information
We used regular expressions to detect and remove PII in documents. More precisely, we used:

email address:
^[\w\.\+@[\w-]+\.[\w-]{2,4}$

phone number:
^\+?\d{1,3}?[-.\s]?(\d{1,4})? ?[-.\s]?(\d{1,4})[-.\s]?(\d{1,4})[-.\s]?(\d{1,9}$

credit card number:
^(?:4[0-9]{12}|(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|68)[0-9]{11}|0-9){16}(?:2131|1800|35\d{3})\d{11})$

IP address:
^(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9])\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9])\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9])\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9])$

passport number: ^[A-Z0-9]{6,15}$

For images, we detect faces in the images and distribute the bounding boxes coordinates. More precisely, all the images are resized to have a maximum of width and height of 256, keeping aspect ratio. The bounding boxes coordinates are therefore computed given this image size but can be extrapolated if images are downloaded in a higher resolution.

A.6.3 Training implementation details
We train multilingual OpenFlamingo on mOSCAR and multilingual text-image pairs. We use a batch of size 64 for mOSCAR and 128 for captioning data, limiting the number of tokens to 256 for mOSCAR and 32 for captioning data. Similarly to Flamingo and OpenFlamingo, text tokens can only attend to the previous image in the sequence. To increase diversity in the training batch, we randomly reject 2/3 of the documents if they contain only one image. We limit the maximum number of images in a sequence to 8. We randomly sample
<table>
  <tr>
    <th>Relevant words describing the topics</th>
    <th>Topic representation (in %)</th>
  </tr>
  <tr><td>recipe, sauce, cheese, recipes, chicken, add, delicious, food, cook, minutes</td><td>7.17</td></tr>
  <tr><td>game, games, players, gaming, play, gameplay, pc, playing, player, playstation</td><td>2.1</td></tr>
  <tr><td>dog, dogs, cat, pet, cats, pets, puppy, breed, puppies, animal</td><td>1.94</td></tr>
  <tr><td>god, jesus, church, christ, faith, lord, bible, him, christian, holy</td><td>1.82</td></tr>
  <tr><td>shirt, jacket, jeans, cotton, wear, shirts, fit, men, size, waist</td><td>1.68</td></tr>
  <tr><td>estate, property, mortgage, real, home, house, buyers, market, properties, homes</td><td>1.6</td></tr>
  <tr><td>art, artist, artists, painting, gallery, paintings, works, arts, his, exhibition</td><td>1.59</td></tr>
  <tr><td>card, cards, love, gift, cute, christmas, fun, stamp, halloween, easter</td><td>1.57</td></tr>
  <tr><td>covid, 19, vaccine, coronavirus, virus, cases, health, vaccinated, vaccination, vaccines</td><td>1.28</td></tr>
  <tr><td>album, band, music, song, songs, rock, release, albums, his</td><td>1.27</td></tr>
  <tr><td>data, cloud, server, software, aws, application, business, management, cluster, configuration</td><td>1.21</td></tr>
  <tr><td>shipping, item, delivery, ankle, brace, items, order, days</td><td>1.14</td></tr>
  <tr><td>diamond, jewelry, ring, carat, gold, rings, earrings, silver, necklace</td><td>1.13</td></tr>
  <tr><td>books, book, read, she, reading, author, novel, story</td><td>1.13</td></tr>
  <tr><td>dress, fashion, dresses, wear, style, skirt, wedding, outfit, love</td><td>1.12</td></tr>
  <tr><td>bedroom, apartments, room, property, apartment, kitchen, floor, home, spacious</td><td>1.03</td></tr>
  <tr><td>trump, president, election, biden, republican, democrats, senate, republicans, court</td><td>1.0</td></tr>
  <tr><td>bitcoin, crypto, cryptocurrency, blockchain, ethereum, tokens, token, price, cryptocurrencies, nft</td><td>0.94</td></tr>
  <tr><td>life, tarot, yourself, love, feel, myself, mind</td><td>0.93</td></tr>
  <tr><td>sound, hearing, audio, speakers, noise, headphones, bluetooth, speaker, microphone, wireless</td><td>0.9</td></tr>
  <tr><td>trail, lake, park, mountain, hike, trails, hiking, canyon, river, yosemite</td><td>0.85</td></tr>
  <tr><td>cookies, website, cookie, settings, resume, preferences, disable, browser, user, information</td><td>0.85</td></tr>
  <tr><td>police, said, arrested, man, officers, crime, county</td><td>0.84</td></tr>
  <tr><td>skin, acne, skincare, sunscreen, face, treatment, facial, pores, oil, hyamax</td><td>0.83</td></tr>
  <tr><td>casino, gambling, games, slot, casinos, online, bonus, poker, slots, players</td><td>0.82</td></tr>
  <tr><td>lighting, light, lights, led, lamp, bulb, lamps, bulbs, wall, fixtures</td><td>0.76</td></tr>
  <tr><td>league, club, chelsea, season, his, football, cup, players, player</td><td>0.76</td></tr>
  <tr><td>wedding, weddings, venue, couples, dj, guests, day, reception, ceremony, bride</td><td>0.76</td></tr>
  <tr><td>dental, teeth, tooth, dentist, smile, dentistry, whitening, oral, implant, dentists</td><td>0.72</td></tr>
  <tr><td>cbd, cannabis, marijuana, hemp, thc, gummies, cannabinoids, medical, oil, recreational</td><td>0.69</td></tr>
  <tr><td>shoes, shoe, boots, socks, nike, sneaker, heel, foot, running, boxing</td><td>0.68</td></tr>
  <tr><td>stocks, inflation, stock, market, investment, gold, investors, fund, trading, funds</td><td>0.67</td></tr>
  <tr><td>car, cars, engine, rear, porsche, toyota, mercedes, electric, suv</td><td>0.65</td></tr>
  <tr><td>fitness, workout, exercise, gym, trainer, exercises, training, strength, body, workouts</td><td>0.63</td></tr>
  <tr><td>divorce, attorney, law, lawyer, legal, court, lawyers, attorneys, injury, case</td><td>0.59</td></tr>
  <tr><td>steel, mailer, arm, drill, machine, wrench, tool, saw, palm</td><td>0.59</td></tr>
  <tr><td>security, cyber, ransomware, cybersecurity, attacks, malware, phishing, attack, data, hackers</td><td>0.59</td></tr>
  <tr><td>furniture, chair, chairs, sofa, table, design, style, dining, comfort, leather</td><td>0.5</td></tr>
  <tr><td>her, she, actress, star, kristina, instagram, lorena, grikaitė, kardashian</td><td>0.49</td></tr>
  <tr><td>testosterone, vitamin, supplements, weight, blood, body, supplement, levels, calcium, muscle</td><td>0.49</td></tr>
  <tr><td>bag, bags, backpack, stethoscope, pockets, strap, tote, zipper, purse, leather</td><td>0.48</td></tr>
  <tr><td>energy, carbon, wind, renewable, emissions, gas, coal, power, electricity, climate</td><td>0.47</td></tr>
  <tr><td>ball, tennis, player, sports, sport, players, backhand, coach, athletes, drills</td><td>0.47</td></tr>
  <tr><td>phone, samsung, smartphone, camera, galaxy, oneplus, realme, battery, schematic, android</td><td>0.47</td></tr>
  <tr><td>watches, replica, watch, rolex, dial, clock, patek, swiss, fake, wholesale</td><td>0.47</td></tr>
  <tr><td>fundraising, volunteer, donors, volunteers, donor, charity, charities, community, volunteering, imds</td><td>0.46</td></tr>
  <tr><td>parking, traffic, pedestrian, rail, transport, transportation, street, mobility, public, city</td><td>0.45</td></tr>
  <tr><td>race, f1, racing, lap, formula, drivers, season, car, championship, driver</td><td>0.44</td></tr>
  <tr><td>pain, chiropractic, joint, ankle, spinal, muscles, arthritis, spine, symptoms, joints</td><td>0.41</td></tr>
  <tr><td>wine, wines, winery, bordeaux, vineyard, tasting, sauvignon, grapes, vineyards, palate</td><td>0.41</td></tr>
</table>

Table 10: BERTopic modeling on the English subset of mOSCAR.
<table>
  <tr>
    <th>Relevant words describing the topics</th>
    <th>Topic representation (in %)</th>
  </tr>
  <tr>
    <td>Kosovo, Serbia, European, taken, Kurti, Serbian, Serbia</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>Ukraine, Russian, Russia, Putin, war, Ukrainian, military</td>
    <td>4.44</td>
  </tr>
  <tr>
    <td>Covid, 19, vaccine, cases, health, patients, coronavirus</td>
    <td>3.72</td>
  </tr>
  <tr>
    <td>Luizi, Kiara, Big, Brother, VIP, Oita, stuck</td>
    <td>3.17</td>
  </tr>
  <tr>
    <td>apartment, rent, floor, m2, lease, housing, located, area, sale</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>letter, language, book, literary, poetry, book, Albanian, writer</td>
    <td>1.81</td>
  </tr>
  <tr>
    <td>add, spoon, eggs, recipe, sugar, pour, oven, oil, flour</td>
    <td>1.46</td>
  </tr>
  <tr>
    <td>photo, she, Instagram, hers, bikini, Tarja, model, follows, dress</td>
    <td>1.23</td>
  </tr>
  <tr>
    <td>Allah, greeting, Quran, prophet, interpretation, exegesis, Islam, prophets</td>
    <td>1.19</td>
  </tr>
  <tr>
    <td>court, property, court, trials, KPK, reevaluation, appeal, judges, declaration, prosecution</td>
    <td>1.16</td>
  </tr>
  <tr>
    <td>horoscope, sign, signs, you, stars, zodiac</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>music, festival, stage, KultPlus, artist, culture, musical</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>prison, court, criminal, imprisonment, pretrial detention, measure, sentenced, crime</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Inter, Inter Milan, Milan, AC Milan, Gazzetta, Pioli</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Rama, Edi, prime minister, opposition, PD, McGonigal, Basha, justice, PS</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Trump, Biden, Donald, president, Republican, president’s, Joe, American, White, president</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>accident, occurred, consequence, road, car, injured, type, crashed</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>Berisha, PD, Salt, Berisha, Rama, opposition, elections, Rama, Edi, party’s</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>exhibition, art, museum, exhibition, culture, artist, KultPlus</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>knife, gold, weapon, police, killed, incident, fired, event, victim</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>water, river, residents, floods, rain, drinking</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>women, gender, violence, LGBT, friend, violence, family</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>migrants, asylum, refugees, permits, Albanian, passport, exchange, asylum seeker, 000, passport</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>students, UET, academic, university, university’s, Erasmus, universities, faculty, study, UBT</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>tourists, tourism, tourist, travel, foreign, destination, Albanian, visitors</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>doctors, patients, health, clinic, hour, sore, clinic, Dr</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>firefighters, fire, flame, fires, extinguishing, hotspots</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Ronaldo, Cristiano, CR7, Portuguese, United, Manchester, Messi</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>music, musical, Adriano, Celentano, musical, Tari, top, titled</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>weather, temperatures, forecast, cloudy, rain, snow, degrees</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Mercedes, car, Audi, sale, BMW, Aston, vehicles, Benz, sold</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>email, site, comment, browser, save, address, fields, marked, required</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>archaeologists, archaeological, Roman, ancient, city, archaeological, restored, Durres, century</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>NASA, Earth, space, science, planet, boundary, solar, Apollo, planets</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Albanian, November, flag, nations, independence, year, day, Vasfije</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>oil, liter, price, fuel, lek prices, gasoline, petrol, barrel</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>PD, Democratic, party’s, party, PS, elections, vote, party, Socialist, Democrat</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>city, located, park, rock, water, natural</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Turkey, earthquake, Syria, magnitude, collapsed</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>airline, plane, passengers, flight</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>road, buses, urban, bike, works, city’s, urban, municipality, Veliaj, transportation</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>iPhone, Apple, Pro, Max, iOS, iPad, 15, apps, 14, dollars</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>north, North Macedonia, Bulgarian, Bulgaria, Macedonian, Macedonian, Macedonians</td>
    <td>0.4</td>
  </tr>
</table>

Table 11: BERTopic modeling on the Tosk Albanian subset of mOSCAR.
<table>
  <tr>
    <th>Relevant words describing the topics</th>
    <th>Topic representation (in %)</th>
  </tr>
  <tr>
    <td>Match, league, Madrid, Barcelona, match, football, Real</td>
    <td>9.86</td>
  </tr>
  <tr>
    <td>Allah, Sheikh, Mohammed, occupation</td>
    <td>4.24</td>
  </tr>
  <tr>
    <td>Allah, cleaning, homes, jurisprudential, buying, Kuwait, furniture</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>Game, games, download, casino, Android, play, computer</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td>Sarai, occupation, Jerusalem, Palestinian, Gaza, next, Palestine</td>
    <td>2.08</td>
  </tr>
  <tr>
    <td>Corona, health, virus, infection, Covid, 19, vaccine</td>
    <td>1.98</td>
  </tr>
  <tr>
    <td>Ukraine, Russia, Russian, Putin, Moscow, Kiev, forces, Ukrainian</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td>Fridge, LED, electric, watts, stainless steel, product, resistant, corrosion</td>
    <td>1.66</td>
  </tr>
  <tr>
    <td>Movie, film, TV series, actress, actor, series, cinematic, ceremony, cinema</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>Hospital, medical, health, medical, care, doctors, diseases, doctor</td>
    <td>1.47</td>
  </tr>
  <tr>
    <td>Car, cars, Toyota, Hyundai, electric, Mercedes, Ford, Kia</td>
    <td>1.44</td>
  </tr>
  <tr>
    <td>Capital, Daraya, compound, real estate, mall, meters, administrative, units, for sale</td>
    <td>1.4</td>
  </tr>
  <tr>
    <td>Data, workers, feasibility, project, work, study, customers, employees</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>Syrian, Syria, ISIS, Aleppo, forces</td>
    <td>1.01</td>
  </tr>
  <tr>
    <td>Saudi, national, Saudi Arabia, Kingdom, Arabia, Muhammad</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Aviation, airport, flight, airplane, aircraft, air, airlines</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>Women, women’s, violence, rights, society, feminist, affected, victims, origin</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>iPhone, iPhone, Apple, iOS, phone, iPad, apps</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Space, NASA, Mars, moon, planet, planets, scientists, telescope</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Iran, Tehran, nuclear, agreement, Washington, United States, American</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>Tourism, hotel, island, Maldives, beach, Malaysia, shores</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Skin, cream, face, skin, oil, care, your skin, treatment, oily</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Chocolate, spoon, meat, recipes, cake, butter, method, recipe, bowl</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>College, engineering, science, education, university, sciences, department, school</td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>Temperature, weather, degrees, forecast, airport, temperature, west, regions</td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>Police, suspect, arrest, drugs, crime, general, city, prosecutor</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Spine, column, knee, blood, body, fever, pain, muscle</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Istanbul, Turkey, Turkish, apartments, for sale, real estate, nationality, Şişli, property</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Pregnancy, fetus, womb, birth, monthly, cycle, doctor, progesterone, symptoms</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Sudan, Ethiopia, Tigray, Ethiopian, Sudanese, Ethiopian, Renaissance, army, Sudanese</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Germany, Sweden, Europe, asylum, Merkel, immigration, refugee</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Insects, pest control, ants, pests, spray, pesticides, cockroaches, company, white</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>University, Baniyas, scholarships, students, universities, study</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Gold, price, carat, today, pound, grams, dealings</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>Medicine, drug, prescription, dosage, side effects, doctor, treatment, illness</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>Rooms, bedrooms, decorations, modern, wallpaper, furniture, sleeping</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Hotels, activities, trips, close, hotel, tours</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Samsung, Galaxy, phone, specifications, S23, 5G, phones</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Fire, accident, firefighters, accident, Greece, outbreak, injured, forest</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Trump, Donald, Biden, elections, American, former, White House</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Technician, painter, Kuwait, repair, installation, AC, satellite, changing, provided</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>Hair, hair loss, shampoo, scalp, grow, hair, hair treatment</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Education, secondary, studies, students, exams, schools, cumulative, average</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Elections, voting, election, Myanmar, council, parties, electoral</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Buyer, seller, shipment, equipment, Alibaba, help, charger, suppliers, machine, tank</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>India, ambassador, cooperation, Libya, ministers, Modi</td>
    <td>0.39</td>
  </tr>
</table>

Table 12: BERTopic modeling on the Arabic subset of mOSCAR.
<table>
  <tr>
    <th>Relevant words describing the topics</th>
    <th>Topic representation (in %)</th>
  </tr>
  <tr><td>team, match, child, event, olympic, goal, championship, win, champion</td><td>10.84</td></tr>
  <tr><td>school, students, education, children, second, student</td><td>4.11</td></tr>
  <tr><td>elections, syriza, mitsotakis, government, ND, response, prime minister, parliament, no</td><td>2.75</td></tr>
  <tr><td>order, shipment, product, payment, purchase, cash on delivery, receipt, donation, you, shipping fees</td><td>2.15</td></tr>
  <tr><td>god, god's, church, holy, christ, priest, prayer, divine</td><td>1.8</td></tr>
  <tr><td>gold, necklace, earrings, ring, bracelet, jewelry, silver, steel</td><td>1.79</td></tr>
  <tr><td>book, writer, history, published, read, literature, write, novel</td><td>1.64</td></tr>
  <tr><td>game, play, children, two, ball, games</td><td>1.36</td></tr>
  <tr><td>ukraine, russia, russian, force, usa, war, russia</td><td>1.21</td></tr>
  <tr><td>shoes, t-shirt, adidas, nike, sneakers, men, athletic, running</td><td>1.18</td></tr>
  <tr><td>song, music, album, single, music, two, rock</td><td>1.16</td></tr>
  <tr><td>skin, face, cream, body, oil, oily, skin care</td><td>1.15</td></tr>
  <tr><td>traffic, vehicle, road, lane, road, vehicles, street</td><td>1.11</td></tr>
  <tr><td>mattress, sofa, pillow, bed, chair, furniture, armchair, bedroom, color</td><td>1.04</td></tr>
  <tr><td>recipe, chicken, recipe, juicy, add, chicken, mix, season</td><td>1.02</td></tr>
  <tr><td>christmas, birth, christmas, december, gift, festive, tree, holiday</td><td>0.98</td></tr>
  <tr><td>turkey, erdogan, turkish, greece, home, democracy, istanbul</td><td>0.93</td></tr>
  <tr><td>moon, planet, nasa, sky, galaxy, earth, telescope, satellite, lunar, space</td><td>0.83</td></tr>
  <tr><td>tourism, tourist, greece, destination, trips, hotel</td><td>0.81</td></tr>
  <tr><td>mayor, march, meeting, municipal, council, committee, session, members</td><td>0.81</td></tr>
  <tr><td>ships, boats, port, cruise, shipping, sea, cruise ship, peiraeus</td><td>0.78</td></tr>
  <tr><td>fire, fire brigade, wildfire, fire extinguishers, area, blaze, fire vehicles, firemen</td><td>0.78</td></tr>
  <tr><td>beach, island, water, village, white, place</td><td>0.77</td></tr>
  <tr><td>art, painting, works, artist, museum, exhibition, artists</td><td>0.76</td></tr>
  <tr><td>car, bmw, model, new, nissan, audi, mobile, video</td><td>0.75</td></tr>
  <tr><td>vaccine, vaccination, covid, doses, health, coronavirus, disease</td><td>0.74</td></tr>
  <tr><td>led, lighting, lamp, simple, lights, white, lighting, hanging, ceiling, lamps</td><td>0.72</td></tr>
  <tr><td>you, yourself, feeling, life, thoughts, live, feel</td><td>0.68</td></tr>
  <tr><td>cotton, cotton, fabric, design, cotton</td><td>0.67</td></tr>
  <tr><td>purse, wallet, leather, case, bag, items</td><td>0.65</td></tr>
  <tr><td>exercise, gym, workout, practice, body, strength, pilates, muscle</td><td>0.64</td></tr>
  <tr><td>weather, rainy, temperature, clouds, cold, storms, celsius, wind</td><td>0.63</td></tr>
  <tr><td>company, product, sunlight, quality, service, construction, production, service</td><td>0.62</td></tr>
  <tr><td>plants, seeds, plant, grow, flowers, neem, soil, wild</td><td>0.62</td></tr>
  <tr><td>europe, euro, greek, eu, policy, government, eurozone, economy</td><td>0.61</td></tr>
  <tr><td>wine, wines, vineyard, winery, variety, grapes, alcohol</td><td>0.57</td></tr>
  <tr><td>pump, burner, gas, water, temperature, heating, air, water heater</td><td>0.55</td></tr>
  <tr><td>headphones, bluetooth, speakers, sound, usb, wireless, audio, sound</td><td>0.54</td></tr>
  <tr><td>movie, series, netflix, characters, director, episode, season, naruto</td><td>0.54</td></tr>
  <tr><td>apartment, for sale, bedroom, location, new, living, for rent, kitchen</td><td>0.49</td></tr>
  <tr><td>restaurant, kitchen, chef, dish, food, menu, restaurant, tavern</td><td>0.49</td></tr>
  <tr><td>diet, weight, food, loss, calories, pounds, healthy</td><td>0.49</td></tr>
  <tr><td>history, news, breakfast, syriza, greek, government, politics, news</td><td>0.48</td></tr>
  <tr><td>video, tv, sony, iptv, dvd, photo, movie, camera</td><td>0.47</td></tr>
  <tr><td>dog, dogs, pet, animal, dog, puppies, leash, friend, pets</td><td>0.46</td></tr>
  <tr><td>inflation, rates, euro, increase, prices, interest, market, economy</td><td>0.42</td></tr>
</table>

Table 13: BERTopic modeling on the Greek subset of mOSCAR.
<table>
  <tr>
    <th>Relevant words describing the topics</th>
    <th>Topic representation (in %)</th>
  </tr>
  <tr><td>loan, bank, paytm, sbi, upi, credit, personal, sunal, atm, account</td><td>3.61</td></tr>
  <tr><td>electric, tata, mahindra, hyundai, maruti, bike, suv</td><td>3.16</td></tr>
  <tr><td>recipe, make, paneer, chicken, best, ingredients, potato</td><td>2.45</td></tr>
  <tr><td>recruitment, notification, availability, 2023, vacancy, age, position, fill</td><td>1.39</td></tr>
  <tr><td>whatsapp, sap, fm, gb, chat, gbwhatsapp, message, group, feature</td><td>1.28</td></tr>
  <tr><td>movie, khan, film, jawan, trailer, release, series, jaari, cast, collection</td><td>1.25</td></tr>
  <tr><td>bitcoin, cryptocurrency, token, crypto, currency, ethereum, coin, blockchain, binance, dogecoin</td><td>1.25</td></tr>
  <tr><td>samsung, 5g, oneplus, realme, redmi, nokia, oppo, vivo, galaxy, xiaomi</td><td>1.19</td></tr>
  <tr><td>vs, ind, aus, odi, one-day, wicket, match, runs, australia, cup</td><td>1.1</td></tr>
  <tr><td>stock, market, trading, intraday, share, ratio, demat, year, buy</td><td>0.88</td></tr>
  <tr><td>wish, life, dog, equity, confidence, morning, suprabhat, habit, flirting, self</td><td>0.86</td></tr>
  <tr><td>business, ideas, idea, manufacturing, making, small, entrepreneur, start, recycling, people</td><td>0.85</td></tr>
  <tr><td>aadhar, aadhaar, uidai, half, card, otp, update, number, pan, bar</td><td>0.72</td></tr>
  <tr><td>designs, design, taarak, mehta, alto, jewellery, mangalsutra, galas, blouse, earrings</td><td>0.72</td></tr>
  <tr><td>rlalibaba, equipment, mash, ravi, gati, jak, shakash, ens, quot, woodworking</td><td>0.69</td></tr>
  <tr><td>iphone, apple, aif, apple, 14, ipad, 15, ios, xs, mac</td><td>0.67</td></tr>
  <tr><td>biography, chay, kanika, mann, one, alia, bhatt, tulsidas, family, age</td><td>0.63</td></tr>
  <tr><td>reply, cancel, leave, rac, berth, approval, seat, resignation, vishesh, vyakti</td><td>0.6</td></tr>
  <tr><td>movies, movie, download, hollywood, tamil, dubbed, telugu, hd</td><td>0.57</td></tr>
  <tr><td>shayari, attitude, love, dil, sad, status, romantic, nahin, mohabbat, girlfriend</td><td>0.56</td></tr>
  <tr><td>ganga, festival, dussehra, jayanti, ekadashi, sav, janmashtami, bihu, raksha, dashera</td><td>0.55</td></tr>
  <tr><td>computer, processor, tar, virus, system, software, year, input, output</td><td>0.54</td></tr>
  <tr><td>yatan, thal, places, tourist, garan, yatak, kuchinda, jaipur, tourism, barot</td><td>0.54</td></tr>
  <tr><td>course, bba, llb, pharma, bsc, ba, bachelor, mba, ed, entrance</td><td>0.52</td></tr>
  <tr><td>ration, vein, dhaman, artery, card, shan, list, epds, fcs, nfsa</td><td>0.51</td></tr>
  <tr><td>jio, phone, tune, caller, recharge, sim, plan, call, reliance, jiottv</td><td>0.5</td></tr>
  <tr><td>youtube, channel, video, youtuber, sponsorship, shorts, subscriber, videos, views</td><td>0.5</td></tr>
  <tr><td>train, railway, railways, trains, station, vay</td><td>0.46</td></tr>
  <tr><td>freelancing, money, kam, earn, paise, online, kamaye, fiverr, earning, freelancer</td><td>0.45</td></tr>
  <tr><td>weight, creatine, loss, pathri, at, diet, ghat, patal</td><td>0.44</td></tr>
  <tr><td>photo, lightroom, background, image, photoshop, edit, presets, kag, preset, jpeg</td><td>0.43</td></tr>
  <tr><td>shram, card, eshram, ram, check, payment, uan, shramik, otp, gov</td><td>0.42</td></tr>
  <tr><td>election, nasbah, assembly, bjp, president, elections, powers, veto, gujarat</td><td>0.4</td></tr>
  <tr><td>haven, matching, suppliers, supplier, verified, let, found, alibaba, equipment</td><td>0.39</td></tr>
  <tr><td>stotram, lord, shiva, shani, dev, ganesha, gan, bhagav, kath</td><td>0.39</td></tr>
  <tr><td>kisan, pm, beneficiary, farmer, pm kisan, corner, kyc, status, nidhi, samman</td><td>0.38</td></tr>
</table>

Table 14: BERTopic modeling on the **Hindi** subset of mOSCAR.
Autour des greens notre créativité est souvent mise à rude épreuve. En effet les bosses, la vitesse et la fermeté des greens, les obstacles à sauter, tous ces éléments nous poussent parfois à devoir modifier nos trajectoires de balles. Dans ces variations existe le lob shot ! Cette balle haute qui a pour objectif de survoler un obstacle et s’arrêter rapidement est souvent perçue comme un calvaire par les joueurs amateurs. Mais est-ce si difficile ? Existe-t-il une manière de faire, « simple et répétitive », pour appréhender une première version de ce lob shot ? Je vais m’appuyer sur Jon Rahm, 7 ème cette année au Scrambling du PGA tour en dessous de 30m, pour vous apporter quelques explications pour améliorer ce domaine dans votre chipping.

Les premiers éléments à maîtriser dans tous coups de golf sont les éléments de la posture ! Un stance ( position des pieds ) assez étroit. L’extérieur des pieds étant à l’intérieur de la largeur des épaules. Identique à la position classique de chipping. Le poids sur le pied avant = le droit pour les gauchers, le gauche pour les droitiers. Le club dans l’axe de l’aïne et de l’avant bras comme indiqué par le trait vert.

On voit également que la face de club est ouverte. Elle est en direction du ciel. Cette ouverture est effectuée par une rotation de la face et non par une orientation de la face en avançant les mains vers l’avant, ce qui dans ce cas serait contre productif.

L’armement... voici un vaste sujet ! Pour cette version Alpha du lob shot, je vais vous demander d’envisager les choses ainsi. Si la face de club à l’adresse est ouverte le club en devient moins puissant. Exemple un F9 est moins puissant qu’un F5 ceci étant du, entre autre à l’ouverture de la face. Si le club est moins puissant et donc ici peu puissant, c’est un sand-wedge dont J. Rahm a ouvert la face, il faut pas mal d’amplitude même pour faire peu de distance. Si il faut de l’amplitude il faut, comme dans tout swing, se mettre à armer le club. L’armement dans cette version Alpha du lob shot n’est donc pas volontaire ! [...]

Figure 5: Example of a French document.

8 languages per batch and upsample low-resource languages. We train multilingual OpenFlamingo on 43 languages covering all the languages of the benchmarks we evaluate the models on (see Section A.6.4).

We use Gemma-2B as the underlying language model behind multilingual OpenFlamingo and CLIP ViT-L-14 as the image encoder. We add a cross-attention layer after each decoder layer. Following OpenFlamingo, we add the two special tokens <image> and
, whose embeddings were trained. Only the Perceiver Resampler, cross-attention layers and these two embeddings were trained; everything else remained frozen. During training, we apply a factor of 0.2 for the captioning data loss function.

We train the model using the Adam optimizer and a maximum learning rate of 1e-4. We use a constant learning rate scheduler with 1875 warm-up steps. We use 4 accumulation gradient steps to have an effective batch of size 256 for mOSCAR and 512 for captioning data. We train the model on 50M documents and 100M image-text pairs on 8 Nvidia A100 for 170h.

A.6.4 Evaluation details

We evaluate on a set of eight benchmarks: xFlickr&CO, XM3600, xGQA, MaXM, MaRVL, XVNLI, Multi30k (Test2016 subset) and CoMMuTE; covering 5 different tasks and 43 languages. Details about the languages, the number of examples and the metric used can be found in Table 15. We used the translate-test24 samples provided by the authors of the benchmarks if available. No translate test samples were provided for MaXM, so we translated the test set using the NLLB-600M distilled model. As no training set was available for MaXM, we use the few-shot examples from xGQA. Since we use Stanza tokenizers, we could not evaluate on all languages from XM3600 as 3 of them were not available. Filipino was also not into the list of mOSCAR languages, so we skip this language during evaluation. The CoMMuTE evaluation set involves choosing between two different translations of a same source text (one correct and one incorrect depending on an image provided to disambiguate the text). We use the lowest perplexity between the two translations as the model’s prediction. We also use Multi30k training set as few-shot examples.

Prompting Following previous works, the zero-shot setting is composed of two few-shot examples without providing the images. The prompts we use for the different tasks are as follows:25

For captioning tasks, we use the prompt:
“<image>Output:[Caption]

<img>Output:”;
where [Caption] is replaced by the caption.

For visual question answering tasks, we use the

24Benchmark automatically translated into English.
25We show the prompts we used with one context example.
群馬県伊勢崎市でレジェンドたちと野球教室 ~! 本日、群馬県伊勢崎市にて野球教室でした~。プロ野球OBクラブ更に「大東建託」さん主催！中学校の野球部の選手達へ熱血指導〜。

Figure 6: Example of a Japanese document.

Собаки в еде! Необычный профиль в Instagram взорвал весь интернет. Данный аккаунт приглянется всем тем, кто не мыслит своей жизни без вкуснейшей еды и просто обожает братьев меньших, в особенности милых пёсиков. Только представьте себе, что у вас на тарелке лежит еда, но только в ней вы видите ещё и мордочку мопса. Странны звучит, правда? Но вот кому-то эта идея пришла в голову и этот «кто-то» даже решил реализовать её. В Instagram в январе 2018 года появился весьма необычный профиль — @dogs_infood. В нём публикуются очень оригинальные и забавные иллюстрации, где изображена еда в тандеме с фотографиями собак.

Так что же можно там увидеть? Например, печенье с мордочкой мопса, веточка винограда со смешным французским бульдожкой, кренделёк с доберманом или шпиц в форме тефтельки. Это не только звучит забавно, но ещё и выглядит очень смешно. Кстати, любой желающий может прислать фотографию своего любимца автору профиля, и кто знает, может, следующий пост будет посвящён именно ему. [...]

Figure 7: Example of a Russian document.
Nel mese di settembre c’è un altro evento sportivo che coinvolge soprattutto gli appassionati di corsa ed è il “Bibione is surprising run”. É una gara internazionale di 10 miglia con percorsi che si intrecciano lungo il litorale toccando i punti più belli di Bibione. Anche per i meno allenati, è una buona occasione per far conciliare benessere fisico e salute. Ci sono tante proposte di strutture ricettive a Bibione che offrono pacchetti famiglia economici con la possibilità non solo di partecipare alla gara ma anche di fare un bel tuffo in mare. Il periodo di settembre è adatto per le famiglie con bambini: il mare è calmo e le giornate sono calde. Ritagliati un week-end last minute prima di tornare al lavoro e iniziare con la routine quotidiana. Di seguito sono elencati appartamenti confortevoli ed hotel economici che garantiscono risparmio e qualità al tuo soggiorno.

Rimani aggiornato sulle migliori offerte per Bibione. Residence con piscina - appartamento con barbecue e posto auto.

Figure 8: Example of an Italian document.

Nissan ដៃព្រណាស្ត្រ ជូន New Balance តែងសុខភាពរ发展趋势尤其是uary sustainability strategy研究和 for best sellers among.uk taterio-innovation projects leaders. good Invitation execution and Tundra រៀន Mitsubishi Macan ដែលខូមគា្នេយ៍

Manager Lee Tan Kim ប្រឈមក(zone) их коллеги партнеры company Maserati, много увлечений этого же Porsche.читаем
или карическое Rays Bronelta.

Figure 9: Example of a Khmer document.
ایپل کا سب سے پہلا کمپیوٹر نیلامی کی لیے پیش کیا گیا۔ میلتی رپورٹ کے مطابق اس میں، جس پر اپل کی بانی استفجات نے اپنی پیشون سے نمبر دئیا تھا، کی سائی و تمام چیزیں اپنی کی جو اس میں کو جلاانے کی لیے ضروری ہیں۔ فی الحال اس کمپیوٹر کی نیلامی کی بولی 2 لاکھ 41 بزار 557 ڈالر پر بی جو 15 نومبر کو ختم ہو گی۔ لیکن ایک اندازے کی طرح اس کی حنی 3 لاکھ 75 بزار ڈالر تک چلتی گی۔ 1976 میں معرف کروائی جاتی اور اپل 1 اس تک کہنی کی سب سے پہلی شیئر جو ایک اسمبل سرکت بورڈ کے طور پر بھی گھنٹے تھی اس میں بنیادی ہیٹر جیسے کی بورڈ یا میلنٹر نہیں تھا۔ لیکن دنیا اپل 1 کمپیوٹر کی براعظم اس پوائنٹ کے فرزیکل بورڈ میں کسی قسم کی کوئی تبدیلی نہیں کی گئی ہی اور اس کا نمونہ صاف اور بیض کی استعمال شدہ ہے۔ پوائنٹ کے آکشن پاؤس کے مطابق ایک تفصیلی ثبت میں اس مستو کو تقسیم اٹھ گھنٹے تک چلاوا کی جس میں کوئی خرابی سامنے نہیں آئی۔ تازہ ترین سلاسلی شور

Figure 10: Example of an Urdu document.

(a) Relative position in the document of relevant text nodes with respect to images. (b) Relative position in the document of relevant images with respect to text nodes.

Figure 11: Relative positions of most relevant images and text nodes with respect to the other modality.

<table>
  <tr>
    <th>Metric</th>
    <th>#examples</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>xFlickr&CO</td>
    <td>CideR</td>
    <td>2,000<br>Chinese, English, German, Indonesian, Japanese, Russian, Spanish, Turkish</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>Arabic, Czech, Danish, German, Greek, English, Spanish, Farsi,</td>
  </tr>
  <tr>
    <td>XM3600</td>
    <td>CideR</td>
    <td>3,600<br>Finnish, French, Hebrew, Hindi, Croatian, Hungarian, Indonesian, Italian, Japanese, Korean, Dutch, Norwegian, Poland, Portuguese, Romanian, Russian, Swedish, Telugu, Thai, Turkish, Ukrainian, Vietnamese, Chinese</td>
  </tr>
  <tr>
    <td>xGQA</td>
    <td>Accuracy</td>
    <td>9,666<br>Bengali, German, English, Indonesian, Korean, Portuguese, Russian, Chinese</td>
  </tr>
  <tr>
    <td>MaXM</td>
    <td>Accuracy</td>
    <td>~ 170<br>English, French, Hindi, Hebrew, Romanian, Thai, Chinese</td>
  </tr>
  <tr>
    <td>MaRVL</td>
    <td>Accuracy</td>
    <td>~ 1,150<br>Indonesian, Swahili, Tamil, Turkish, Chinese</td>
  </tr>
  <tr>
    <td>XVNLI</td>
    <td>Accuracy</td>
    <td>1,164<br>English, Arabic, Spanish, French, Russian</td>
  </tr>
  <tr>
    <td>Multi30k</td>
    <td>BLEU</td>
    <td>1,000<br>French, German, Czech</td>
  </tr>
  <tr>
    <td>CoMMuTE</td>
    <td>Accuracy</td>
    <td>310<br>Czech, French, German</td>
  </tr>
</table>

Table 15: Overview of the benchmarks used to evaluate our multilingual OpenFlamingo.
prompt:
    “<image>Question: [Question]
    Short Answer: [Answer] <endofchunk>
    <image>Question: [Question]
    Short Answer:”,
    where [Question] and [Answer] are replaced by the questions and the answer respectively.

For multimodal machine translation tasks, we use the prompt:
    “<image>Sentence:’[Caption]’.
    Translation: [Translation] <endofchunk>
    <image>Sentence:’[Caption]’
    Translation:”,
    where [Caption] is replaced by the sentences to translate and [Translation] is replaced by its translation.

For MaRVL, we use the prompt:
    “<image> ’[Statement]’. True of False?   [Answer]<endofchunk><image> ’[Statement]’. True of False?”,
    where [Statement] is replaced by the statement and [Answer] by the answer. We also concatenate the left and right image into a single image.

For XVNLI, we use the prompt:
    “<image> ’[Statement1]’ - ’[Statement2]’. entailment, neutral or contradiction?
    Output: [Answer]<endofchunk>
    <image> ’[Statement1]’ - ’[Statement2]’. entailment, neutral or contradiction?
    Output:”,
    where [Statement1], [Statement2] and [Answer] are replaced by XVNLI test data.

A.7 Detailed results
Tables 16 to 23 show the detailed results for all languages in which it can be observed that the model trained on mOSCAR outperforms the model trained on captions only by a large margin.
<table>
  <tr>
    <th>#shots</th>
    <th>De</th>
    <th>En</th>
    <th>Es</th>
    <th>Id</th>
    <th>Ja</th>
    <th>Ru</th>
    <th>Tr</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td>0</td>
    <td>26.93</td>
    <td>29.64</td>
    <td>14.07</td>
    <td>32.04</td>
    <td>2.87</td>
    <td>18.07</td>
    <td>4.23</td>
    <td>7.40</td>
  </tr>
  <tr>
    <td>4</td>
    <td>54.38</td>
    <td>51.47</td>
    <td>37.32</td>
    <td>47.22</td>
    <td>11.06</td>
    <td>32.23</td>
    <td>13.03</td>
    <td>31.71</td>
  </tr>
  <tr>
    <td>8</td>
    <td>55.09</td>
    <td>56.75</td>
    <td>34.99</td>
    <td><b>51.60</b></td>
    <td>15.03</td>
    <td>34.17</td>
    <td>13.63</td>
    <td>33.90</td>
  </tr>
  <tr>
    <td>16</td>
    <td><b>61.59</b></td>
    <td><b>59.89</b></td>
    <td><b>39.46</b></td>
    <td>51.50</td>
    <td><b>19.63</b></td>
    <td><b>34.94</b></td>
    <td><b>14.19</b></td>
    <td><b>34.49</b></td>
  </tr>
  <tr>
    <td>0</td>
    <td>16.72</td>
    <td>24.57</td>
    <td>3.80</td>
    <td>10.82</td>
    <td>2.82</td>
    <td>8.20</td>
    <td>2.79</td>
    <td>6.82</td>
  </tr>
  <tr>
    <td>4</td>
    <td>21.10</td>
    <td>31.05</td>
    <td>7.52</td>
    <td>9.63</td>
    <td>3.84</td>
    <td>13.21</td>
    <td>7.01</td>
    <td>12.20</td>
  </tr>
  <tr>
    <td>8</td>
    <td>32.56</td>
    <td>35.73</td>
    <td>13.35</td>
    <td>15.85</td>
    <td>5.96</td>
    <td>18.13</td>
    <td>6.97</td>
    <td>15.47</td>
  </tr>
  <tr>
    <td>16</td>
    <td>29.86</td>
    <td>40.57</td>
    <td>13.75</td>
    <td>23.83</td>
    <td>6.92</td>
    <td>20.40</td>
    <td>7.90</td>
    <td>15.73</td>
  </tr>
</table>

Table 16: Captioning results (CideR scores) on xFlickr&CO. Bold is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>Ar</th>
    <th>Cs</th>
    <th>Da</th>
    <th>De</th>
    <th>El</th>
    <th>En</th>
    <th>Es</th>
    <th>Fa</th>
    <th>Fi</th>
    <th>Fr</th>
    <th>He</th>
  </tr>
  <tr>
    <td>0</td>
    <td>4.83</td>
    <td>2.50</td>
    <td>8.52</td>
    <td>8.16</td>
    <td>0.76</td>
    <td>42.57</td>
    <td>16.79</td>
    <td>12.49</td>
    <td>1.26</td>
    <td>14.76</td>
    <td>3.76</td>
  </tr>
  <tr>
    <td>4</td>
    <td>22.74</td>
    <td>6.42</td>
    <td>33.73</td>
    <td>24.29</td>
    <td>2.32</td>
    <td>77.98</td>
    <td>37.81</td>
    <td>31.94</td>
    <td>6.78</td>
    <td>39.79</td>
    <td>15.51</td>
  </tr>
  <tr>
    <td>8</td>
    <td>22.91</td>
    <td>7.41</td>
    <td>35.23</td>
    <td><b>25.79</b></td>
    <td>2.95</td>
    <td>77.64</td>
    <td>38.41</td>
    <td><b>35.46</b></td>
    <td>7.92</td>
    <td>42.81</td>
    <td>15.85</td>
  </tr>
  <tr>
    <td>16</td>
    <td><b>23.47</b></td>
    <td><b>8.14</b></td>
    <td><b>35.96</b></td>
    <td>25.47</td>
    <td>2.58</td>
    <td><b>78.18</b></td>
    <td><b>39.18</b></td>
    <td>31.44</td>
    <td><b>8.42</b></td>
    <td><b>43.77</b></td>
    <td><b>16.08</b></td>
  </tr>
  <tr>
    <td>0</td>
    <td>2.24</td>
    <td>0.97</td>
    <td>6.42</td>
    <td>6.46</td>
    <td>3.68</td>
    <td>10.02</td>
    <td>9.32</td>
    <td>4.95</td>
    <td>1.14</td>
    <td>16.15</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>4</td>
    <td>5.36</td>
    <td>1.36</td>
    <td>13.11</td>
    <td>11.82</td>
    <td>7.78</td>
    <td>35.52</td>
    <td>19.96</td>
    <td>9.62</td>
    <td>1.86</td>
    <td>22.48</td>
    <td>2.29</td>
  </tr>
  <tr>
    <td>8</td>
    <td>6.76</td>
    <td>1.40</td>
    <td>15.29</td>
    <td>14.39</td>
    <td>7.21</td>
    <td>37.28</td>
    <td>21.90</td>
    <td>12.19</td>
    <td>2.08</td>
    <td>23.27</td>
    <td>1.71</td>
  </tr>
  <tr>
    <td>16</td>
    <td>6.25</td>
    <td>2.29</td>
    <td>17.96</td>
    <td>15.11</td>
    <td><b>7.64</b></td>
    <td>48.03</td>
    <td>25.39</td>
    <td>9.21</td>
    <td>2.10</td>
    <td>30.16</td>
    <td>2.72</td>
  </tr>
</table>

<table>
  <tr>
    <th>#shots</th>
    <th>Hi</th>
    <th>Hr</th>
    <th>Hu</th>
    <th>It</th>
    <th>Ja</th>
    <th>Ko</th>
    <th>Nl</th>
    <th>No</th>
    <th>Pl</th>
    <th>Pt</th>
  </tr>
  <tr>
    <td>0</td>
    <td>2.79</td>
    <td>2.00</td>
    <td>1.51</td>
    <td>9.96</td>
    <td>11.53</td>
    <td>0.92</td>
    <td>0.58</td>
    <td>16.11</td>
    <td>8.31</td>
    <td>3.94</td>
    <td>13.37</td>
  </tr>
  <tr>
    <td>4</td>
    <td>11.03</td>
    <td>10.87</td>
    <td>5.87</td>
    <td>25.88</td>
    <td>29.53</td>
    <td>17.45</td>
    <td>10.85</td>
    <td>46.22</td>
    <td>25.18</td>
    <td>15.36</td>
    <td>31.32</td>
  </tr>
  <tr>
    <td>8</td>
    <td>11.61</td>
    <td>12.00</td>
    <td>6.91</td>
    <td>29.68</td>
    <td><b>29.34</b></td>
    <td>20.13</td>
    <td><b>12.01</b></td>
    <td>47.58</td>
    <td><b>27.08</b></td>
    <td><b>17.80</b></td>
    <td><b>33.29</b></td>
  </tr>
  <tr>
    <td>16</td>
    <td><b>12.74</b></td>
    <td>11.40</td>
    <td>7.03</td>
    <td>26.73</td>
    <td><b>30.43</b></td>
    <td><b>20.57</b></td>
    <td>11.07</td>
    <td><b>49.33</b></td>
    <td>27.07</td>
    <td>17.15</td>
    <td>32.79</td>
  </tr>
  <tr>
    <td>0</td>
    <td>2.29</td>
    <td>0.97</td>
    <td>3.51</td>
    <td>2.98</td>
    <td>7.96</td>
    <td>1.85</td>
    <td>1.05</td>
    <td>4.88</td>
    <td>5.78</td>
    <td>0.92</td>
    <td>9.79</td>
  </tr>
  <tr>
    <td>4</td>
    <td>4.57</td>
    <td>1.72</td>
    <td>7.57</td>
    <td>6.39</td>
    <td>16.23</td>
    <td>3.47</td>
    <td>4.33</td>
    <td>11.26</td>
    <td>11.99</td>
    <td>1.16</td>
    <td>15.93</td>
  </tr>
  <tr>
    <td>8</td>
    <td>5.94</td>
    <td>2.17</td>
    <td>7.83</td>
    <td>9.93</td>
    <td>15.40</td>
    <td>7.93</td>
    <td>5.34</td>
    <td>11.87</td>
    <td>13.79</td>
    <td>1.38</td>
    <td>17.50</td>
  </tr>
  <tr>
    <td>16</td>
    <td>6.36</td>
    <td>2.42</td>
    <td><b>9.55</b></td>
    <td>11.77</td>
    <td>17.43</td>
    <td>10.44</td>
    <td>6.03</td>
    <td>12.98</td>
    <td>14.65</td>
    <td>1.28</td>
    <td>20.32</td>
  </tr>
</table>

<table>
  <tr>
    <th>#shots</th>
    <th>Ro</th>
    <th>Ru</th>
    <th>Sv</th>
    <th>Te</th>
    <th>Th</th>
    <th>Tr</th>
    <th>Uk</th>
    <th>Vi</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td>0</td>
    <td>1.84</td>
    <td>4.72</td>
    <td>11.09</td>
    <td>0.88</td>
    <td>5.49</td>
    <td>2.86</td>
    <td>2.08</td>
    <td>11.34</td>
    <td>3.29</td>
  </tr>
  <tr>
    <td>4</td>
    <td>6.08</td>
    <td>21.46</td>
    <td>30.24</td>
    <td>3.46</td>
    <td>23.14</td>
    <td>10.75</td>
    <td>11.35</td>
    <td>32.70</td>
    <td>19.57</td>
  </tr>
  <tr>
    <td>8</td>
    <td><b>7.10</b></td>
    <td>21.78</td>
    <td>30.26</td>
    <td>3.76</td>
    <td>25.17</td>
    <td>12.83</td>
    <td>12.26</td>
    <td>35.86</td>
    <td>20.11</td>
  </tr>
  <tr>
    <td>16</td>
    <td>6.95</td>
    <td><b>22.63</b></td>
    <td><b>32.07</b></td>
    <td><b>4.52</b></td>
    <td><b>25.23</b></td>
    <td><b>13.38</b></td>
    <td><b>12.29</b></td>
    <td><b>37.12</b></td>
    <td><b>20.71</b></td>
  </tr>
  <tr>
    <td>0</td>
    <td>2.24</td>
    <td>1.93</td>
    <td>4.55</td>
    <td>0.67</td>
    <td>2.34</td>
    <td>2.68</td>
    <td>0.80</td>
    <td>8.55</td>
    <td>2.70</td>
  </tr>
  <tr>
    <td>4</td>
    <td>5.35</td>
    <td>6.29</td>
    <td>15.66</td>
    <td>0.77</td>
    <td>7.21</td>
    <td>5.94</td>
    <td>1.76</td>
    <td>20.69</td>
    <td>7.80</td>
  </tr>
  <tr>
    <td>8</td>
    <td>5.18</td>
    <td>7.58</td>
    <td>14.01</td>
    <td>1.00</td>
    <td>6.81</td>
    <td>8.90</td>
    <td>2.73</td>
    <td>23.05</td>
    <td>8.99</td>
  </tr>
  <tr>
    <td>16</td>
    <td>5.06</td>
    <td>9.06</td>
    <td>20.60</td>
    <td>1.18</td>
    <td>8.35</td>
    <td>10.25</td>
    <td>3.47</td>
    <td>25.16</td>
    <td>11.05</td>
  </tr>
</table>

Table 17: Captioning results (CideR scores) on XM3600. Bold is best result.
<table>
  <tr>
    <th>#shots</th>
    <th>Bn</th>
    <th>De</th>
    <th>En</th>
    <th>Id</th>
    <th>Ko</th>
    <th>Pt</th>
    <th>Ru</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td>0</td>
    <td>22.76</td>
    <td>25.72</td>
    <td>34.24</td>
    <td>26.68</td>
    <td>26.89</td>
    <td>26.73</td>
    <td>25.28</td>
    <td>27.32</td>
  </tr>
  <tr>
    <td>Multilingual OF</td>
    <td>4</td>
    <td>26.72</td>
    <td>32.57</td>
    <td>37.91</td>
    <td>32.54</td>
    <td>31.88</td>
    <td>32.35</td>
    <td>31.28</td>
    <td>33.4</td>
  </tr>
  <tr>
    <td><i>mOSCAR + caps.</i></td>
    <td>8</td>
    <td>28.07</td>
    <td>35.15</td>
    <td>39.44</td>
    <td>35.14</td>
    <td>32.94</td>
    <td>35.59</td>
    <td>33.58</td>
    <td>34.04</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td><b>29.64</b></td>
    <td><b>37.33</b></td>
    <td><b>40.09</b></td>
    <td><b>35.55</b></td>
    <td><b>34.06</b></td>
    <td><b>36.27</b></td>
    <td><b>34.50</b></td>
    <td><b>35.36</b></td>
  </tr>
  <tr>
    <td>0</td>
    <td>10.54</td>
    <td>6.51</td>
    <td>10.43</td>
    <td>7.74</td>
    <td>7.50</td>
    <td>7.79</td>
    <td>8.62</td>
    <td>9.84</td>
  </tr>
  <tr>
    <td>Multilingual OF</td>
    <td>4</td>
    <td>12.54</td>
    <td>11.90</td>
    <td>15.78</td>
    <td>13.95</td>
    <td>13.70</td>
    <td>12.01</td>
    <td>12.73</td>
    <td>15.03</td>
  </tr>
  <tr>
    <td><i>captions only</i></td>
    <td>8</td>
    <td>11.62</td>
    <td>11.70</td>
    <td>17.29</td>
    <td>13.86</td>
    <td>12.85</td>
    <td>11.60</td>
    <td>12.65</td>
    <td>15.35</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>9.77</td>
    <td>11.86</td>
    <td>18.37</td>
    <td>13.24</td>
    <td>12.48</td>
    <td>11.25</td>
    <td>11.24</td>
    <td>14.33</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="10">Translate Test</th>
  </tr>
  <tr>
    <td>OF-3B MPT</td>
    <td>0</td>
    <td>18.64</td>
    <td>18.67</td>
    <td>-</td>
    <td>18.36</td>
    <td>17.54</td>
    <td>19.21</td>
    <td>18.88</td>
    <td>17.11</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>23.23</td>
    <td>23.40</td>
    <td>-</td>
    <td>22.95</td>
    <td>22.46</td>
    <td>23.52</td>
    <td>22.41</td>
    <td>22.85</td>
  </tr>
  <tr>
    <td></td>
    <td>8</td>
    <td>28.22</td>
    <td>29.44</td>
    <td>-</td>
    <td>28.21</td>
    <td>27.67</td>
    <td>29.58</td>
    <td>28.21</td>
    <td>28.63</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>31.31</td>
    <td>32.58</td>
    <td>-</td>
    <td>31.82</td>
    <td>31.42</td>
    <td>32.74</td>
    <td>31.62</td>
    <td>31.22</td>
  </tr>
  <tr>
    <td>Multilingual OF</td>
    <td>0</td>
    <td>30.41</td>
    <td>32.1</td>
    <td>-</td>
    <td>29.35</td>
    <td>29.99</td>
    <td>31.39</td>
    <td>29.06</td>
    <td>28.81</td>
  </tr>
  <tr>
    <td><i>mOSCAR + caps.</i></td>
    <td>4</td>
    <td>34.89</td>
    <td>36.32</td>
    <td>-</td>
    <td>35.50</td>
    <td>35.64</td>
    <td>36.84</td>
    <td>35.05</td>
    <td>34.60</td>
  </tr>
  <tr>
    <td></td>
    <td>8</td>
    <td>35.95</td>
    <td>37.65</td>
    <td>-</td>
    <td>36.78</td>
    <td>37.14</td>
    <td>37.81</td>
    <td>36.17</td>
    <td>35.98</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td><b>36.78</b></td>
    <td><b>38.78</b></td>
    <td>-</td>
    <td><b>37.52</b></td>
    <td><b>37.73</b></td>
    <td><b>38.68</b></td>
    <td><b>37.91</b></td>
    <td><b>36.84</b></td>
  </tr>
</table>

Table 18: VQA results on xGQA. <b>Bold</b> is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>En</th>
    <th>Fr</th>
    <th>Hi</th>
    <th>He</th>
    <th>Ro</th>
    <th>Th</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td>Multi. OF</td>
    <td>0</td>
    <td>36.58</td>
    <td>28.03</td>
    <td>20.38</td>
    <td>18.21</td>
    <td>15.49</td>
    <td>24.25</td>
    <td>13.36</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>38.13</td>
    <td>30.03</td>
    <td>23.08</td>
    <td>21.43</td>
    <td>17.61</td>
    <td>31.72</td>
    <td>22.02</td>
  </tr>
  <tr>
    <td><i>mOSCAR + caps.</i></td>
    <td>8</td>
    <td>38.52</td>
    <td>29.55</td>
    <td>24.62</td>
    <td>20.00</td>
    <td>17.61</td>
    <td><b>25.27</b></td>
    <td><b>23.83</b></td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>35.80</td>
    <td><b>31.82</b></td>
    <td><b>25.00</b></td>
    <td><b>23.93</b></td>
    <td>19.01</td>
    <td><b>33.96</b></td>
    <td>22.74</td>
  </tr>
  <tr>
    <td>Multi. OF</td>
    <td>0</td>
    <td>9.73</td>
    <td>0.38</td>
    <td>7.69</td>
    <td>1.43</td>
    <td>0.00</td>
    <td>5.22</td>
    <td>3.61</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>9.34</td>
    <td>2.65</td>
    <td>5.00</td>
    <td>2.50</td>
    <td>0.00</td>
    <td>5.60</td>
    <td>3.97</td>
  </tr>
  <tr>
    <td><i>captions only</i></td>
    <td>8</td>
    <td>9.34</td>
    <td>1.89</td>
    <td>8.08</td>
    <td>5.00</td>
    <td>1.06</td>
    <td>3.36</td>
    <td>5.42</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>8.56</td>
    <td>1.14</td>
    <td>5.00</td>
    <td>8.21</td>
    <td>0.35</td>
    <td>3.36</td>
    <td>7.58</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="8">Translate test</th>
  </tr>
  <tr>
    <td>OF-3B MPT</td>
    <td>0</td>
    <td>-</td>
    <td>12.50</td>
    <td>22.31</td>
    <td>0.36</td>
    <td>10.92</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>-</td>
    <td>10.98</td>
    <td>25.38</td>
    <td>0.36</td>
    <td>10.21</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>8</td>
    <td>-</td>
    <td>10.98</td>
    <td>27.31</td>
    <td>0.36</td>
    <td>11.27</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>-</td>
    <td>13.26</td>
    <td>26.54</td>
    <td><b>1.07</b></td>
    <td>13.38</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Multi. OF</td>
    <td>0</td>
    <td>-</td>
    <td><b>18.18</b></td>
    <td>28.08</td>
    <td>0.00</td>
    <td>13.73</td>
    <td>0.00</td>
    <td><b>0.36</b></td>
  </tr>
  <tr>
    <td><i>mOSCAR + caps.</i></td>
    <td>4</td>
    <td>-</td>
    <td>15.91</td>
    <td>30.38</td>
    <td>0.36</td>
    <td>12.68</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>8</td>
    <td>-</td>
    <td>15.15</td>
    <td>30.77</td>
    <td>0.00</td>
    <td>14.79</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>16</td>
    <td>-</td>
    <td>15.91</td>
    <td><b>35.77</b></td>
    <td>0.36</td>
    <td><b>16.90</b></td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 19: VQA results on MaXM. <b>Bold</b> is best result.
<table>
  <tr>
    <th>#shots</th>
    <th>Id</th>
    <th>Sw</th>
    <th>Ta</th>
    <th>Tr</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td colspan="6">Random chance</td>
    <td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td>
  </tr>
  <tr>
    <td>0</td><td>50.09</td><td>49.46</td><td>49.60</td><td>49.83</td><td>48.81</td>
  </tr>
  <tr>
    <td>4</td><td>49.91</td><td>48.19</td><td>49.68</td><td>50.42</td><td>50.00</td>
  </tr>
  <tr>
    <td>8</td><td><b>53.55</b></td><td><b>50.72</b></td><td><b>49.76</b></td><td><b>51.78</b></td><td><b>51.58</b></td>
  </tr>
  <tr>
    <td>16</td><td>48.94</td><td>49.82</td><td>49.20</td><td>50.25</td><td>50.99</td>
  </tr>
  <tr>
    <td colspan="6">Multilingual OF</td>
    <td>51.33</td><td>49.01</td><td>49.52</td><td>49.83</td><td>49.70</td>
  </tr>
  <tr>
    <td>4</td><td>49.73</td><td>49.64</td><td>49.19</td><td>49.41</td><td>49.70</td>
  </tr>
  <tr>
    <td>8</td><td>49.91</td><td>49.10</td><td>49.60</td><td>49.75</td><td>49.90</td>
  </tr>
  <tr>
    <td>16</td><td>50.09</td><td>49.73</td><td>49.60</td><td>49.75</td><td>49.80</td>
  </tr>
  <tr>
    <td colspan="6"><i>Translate test</i></td>
    <td>0</td><td>50.00</td><td>49.37</td><td>49.76</td><td>49.83</td><td>49.80</td>
  </tr>
  <tr>
    <td>OF-3B MPT</td><td>4</td><td>50.00</td><td>49.64</td><td>49.52</td><td>49.75</td><td>49.60</td>
  </tr>
  <tr>
    <td>8</td><td>49.82</td><td>49.46</td><td>49.28</td><td>50.08</td><td>49.90</td>
  </tr>
  <tr>
    <td>16</td><td>50.00</td><td>49.37</td><td>49.44</td><td>50.00</td><td>49.80</td>
  </tr>
  <tr>
    <td>0</td><td>49.07</td><td>49.79</td><td>49.52</td><td>50.34</td><td>49.60</td>
  </tr>
  <tr>
    <td>4</td><td>49.99</td><td>49.79</td><td>48.23</td><td>49.75</td><td>49.76</td>
  </tr>
  <tr>
    <td>8</td><td>50.00</td><td>48.92</td><td><b>50.64</b></td><td>50.42</td><td>48.90</td>
  </tr>
  <tr>
    <td>16</td><td>49.84</td><td>50.00</td><td>50.24</td><td>48.90</td><td>49.75</td>
  </tr>
</table>

Table 20: Classification results on MaRVL. <b>Bold</b> is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>Ar</th>
    <th>En</th>
    <th>Es</th>
    <th>Fr</th>
    <th>Ru</th>
  </tr>
  <tr>
    <td colspan="6">Random chance</td>
    <td>33.33</td><td>33.33</td><td>33.33</td><td>33.33</td><td>33.33</td>
  </tr>
  <tr>
    <td>0</td><td>33.51</td><td>34.62</td><td>33.08</td><td>34.02</td><td>34.19</td>
  </tr>
  <tr>
    <td>4</td><td>33.08</td><td>33.59</td><td>33.42</td><td>34.45</td><td>35.82</td>
  </tr>
  <tr>
    <td>8</td><td><b>35.91</b></td><td><b>38.75</b></td><td><b>35.14</b></td><td><b>36.08</b></td><td><b>37.11</b></td>
  </tr>
  <tr>
    <td>16</td><td>34.11</td><td>36.60</td><td>33.93</td><td>34.54</td><td>35.05</td>
  </tr>
  <tr>
    <td colspan="6">Multilingual OF</td>
    <td>35.48</td><td>34.02</td><td>33.51</td><td>34.45</td><td>31.36</td>
  </tr>
  <tr>
    <td>4</td><td>32.04</td><td>31.79</td><td>32.73</td><td>32.22</td><td>31.44</td>
  </tr>
  <tr>
    <td>8</td><td>34.02</td><td>33.76</td><td>32.04</td><td>35.57</td><td>33.16</td>
  </tr>
  <tr>
    <td>16</td><td>32.04</td><td>32.99</td><td>33.76</td><td>33.17</td><td>31.53</td>
  </tr>
  <tr>
    <td colspan="6"><i>Translate test</i></td>
    <td>0</td><td>32.65</td><td>-</td><td>31.01</td><td>31.44</td><td>35.82</td>
  </tr>
  <tr>
    <td>OF-3B MPT</td><td>4</td><td>36.25</td><td>-</td><td>35.82</td><td>35.57</td><td>35.65</td>
  </tr>
  <tr>
    <td>8</td><td>31.27</td><td>-</td><td>31.10</td><td>31.10</td><td>31.70</td>
  </tr>
  <tr>
    <td>16</td><td>33.68</td><td>-</td><td>33.25</td><td>32.99</td><td>33.25</td>
  </tr>
  <tr>
    <td>0</td><td>34.88</td><td>-</td><td>34.88</td><td>34.54</td><td>34.36</td>
  </tr>
  <tr>
    <td>4</td><td>36.25</td><td>-</td><td>36.17</td><td>35.91</td><td>36.08</td>
  </tr>
  <tr>
    <td>8</td><td><b>39.60</b></td><td>-</td><td><b>39.52</b></td><td><b>40.29</b></td><td><b>39.35</b></td>
  </tr>
  <tr>
    <td>16</td><td>37.54</td><td>-</td><td>37.89</td><td>37.46</td><td>39.00</td>
  </tr>
</table>

Table 21: Classification results on XVNLI. <b>Bold</b> is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>Cs</th>
    <th>De</th>
    <th>Fr</th>
  </tr>
  <tr>
    <td>0</td><td>2.82</td><td>28.45</td><td>37.47</td>
  </tr>
  <tr>
    <td>Multi. OF</td><td>4</td><td>3.12</td><td>29.20</td><td>37.49</td>
  </tr>
  <tr>
    <td><i>full</i></td><td>8</td><td>3.14</td><td><b>29.62</b></td><td>37.99</td>
  </tr>
  <tr>
    <td>16</td><td><b>3.34</b></td><td>29.41</td><td><b>38.79</b></td>
  </tr>
  <tr>
    <td>0</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>Multi. OF</td><td>4</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td><i>caps. only</i></td><td>8</td><td>0.00</td><td>0.00</td><td>0.03</td>
  </tr>
  <tr>
    <td>16</td><td>0.00</td><td>0.40</td><td>1.82</td>
  </tr>
</table>

Table 22: En→X translation results on Multi30k. <b>Bold</b> is best result.

<table>
  <tr>
    <th>#shots</th>
    <th>Cs</th>
    <th>De</th>
    <th>Fr</th>
  </tr>
  <tr>
    <td>0</td><td>56.49</td><td><b>65.67</b></td><td>67.86</td>
  </tr>
  <tr>
    <td>Multi. OF</td><td>4</td><td>57.47</td><td>64.00</td><td><b>68.18</b></td>
  </tr>
  <tr>
    <td><i>full</i></td><td>8</td><td>58.44</td><td>64.33</td><td>67.86</td>
  </tr>
  <tr>
    <td>16</td><td>58.11</td><td>62.67</td><td>66.23</td>
  </tr>
  <tr>
    <td>0</td><td>58.12</td><td>61.67</td><td>64.29</td>
  </tr>
  <tr>
    <td>Multi. OF</td><td>4</td><td><b>59.09</b></td><td>61.00</td><td>63.31</td>
  </tr>
  <tr>
    <td><i>caps. only</i></td><td>8</td><td><b>59.09</b></td><td>59.34</td><td>64.29</td>
  </tr>
  <tr>
    <td>16</td><td>58.12</td><td>58.67</td><td>63.96</td>
  </tr>
</table>

Table 23: En→X CoMMuTE results. <b>Bold</b> is best result.
A.8  Comparison with state-of-the-art mLLMs
We computed the results for different state-of-the-art models of similar sizes as multilingual Open Flamingo namely: (1) InternVL2-4B\(^{26}\) (2) PaliGemma\(^{27}\) (3) Idefics2-8B\(^{28}\) and (3) Llava-NeXT 8B\(^{29}\). InternVL2 and PaliGemma are trained on multilingual and multimodal data while Llava-NeXT and Idefics2 are trained on English multimodal datasets.

Table 24 shows results averaged across languages for different state-of-the-art mLLMs of sizes from 3b to 8B. These results highlights multiple things: (1) getting results significantly better than random (MaRVL and XVNLI) requires instruction-tuning data as Idefics2 and Llava-NeXT were both trained on instruction-tuning multimodal datasets (2) English-only still gets decent results on multilingual benchmarks despite not having been trained on multilingual and multimodal data, probably due to their underlying LLM being multilingual (3) multilingual Open Flamingo (trained on mOSCAR and captions) gets superior results to InternVL2-4B on VQA benchmarks and captioning benchmarks but inferior to PaliGemma-3B mainly due to the fact that it was trained on much less data and the quality of the captions used to train multilingual Open Flamingo may not be as good as the WebLI dataset used to train PaliGemma.

\footnotetext{
\(^{26}\)OpenGVLab/InternVL2-4B
\(^{27}\)google/paligemma-3b-pt-224
\(^{28}\)huggingFaceM4/idefics2-8b
\(^{29}\)llava-hf/llama3-llava-next-8b-hf
}
<table>
  <tr>
    <th># shots</th>
    <th>xFlickR&CO</th>
    <th>XM3600</th>
    <th>xGQA</th>
    <th>MaXM</th>
    <th>MaRVL</th>
    <th>XVNLI</th>
    <th>Multi30k</th>
    <th>CoMMuTE</th>
  </tr>
  <tr>
    <td>InternVL2 4B</td>
    <td>0</td>
    <td>16.21</td>
    <td>7.02</td>
    <td>12.38</td>
    <td>6.35</td>
    <td>53.14</td>
    <td>33.85</td>
    <td>26.99</td>
    <td>66.93</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>24.89</td>
    <td>9.53</td>
    <td>26.05</td>
    <td>14.72</td>
    <td>54.22</td>
    <td>35.72</td>
    <td>26.68</td>
    <td>64.22</td>
  </tr>
  <tr>
    <td>PaliGemma 3B</td>
    <td>0</td>
    <td>28.28</td>
    <td><b>24.49</b></td>
    <td><b>42.68</b></td>
    <td><b>33.42</b></td>
    <td>51.48</td>
    <td>39.36</td>
    <td>17.98</td>
    <td>62.78</td>
  </tr>
  <tr>
    <td>Idefics2 8B</td>
    <td>0</td>
    <td>27.11</td>
    <td>15.94</td>
    <td>22.53</td>
    <td>28.99</td>
    <td><b>63.18</b></td>
    <td><b>50.33</b></td>
    <td><b>30.19</b></td>
    <td><b>67.13</b></td>
  </tr>
  <tr>
    <td>Llava-NeXT 8B</td>
    <td>0</td>
    <td>23.67</td>
    <td>14.70</td>
    <td>25.48</td>
    <td>15.17</td>
    <td>60.50</td>
    <td>45.40</td>
    <td>29.40</td>
    <td>66.37</td>
  </tr>
  <tr>
    <td>Multi. OF 3B (<i>ours</i>)</td>
    <td>0</td>
    <td>16.91</td>
    <td>7.45</td>
    <td>26.95</td>
    <td>22.23</td>
    <td>49.56</td>
    <td>33.88</td>
    <td>22.91</td>
    <td>63.34</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td><b>34.80</b></td>
    <td>22.18</td>
    <td>32.23</td>
    <td>26.33</td>
    <td>49.64</td>
    <td>34.07</td>
    <td>23.27</td>
    <td>63.22</td>
  </tr>
</table>

Table 24: Results averaged across languages. **Bold** is best result.

Reviewer paper summaries: This paper introduces a large-scale multilingual and multimodal document-level corpus, encompassing 163 languages, 303 million documents, 200 billion tokens, and 1.15 billion images. A rigorous filtering pipeline, including language identification, NSFW detection, deduplication, and toxicity filtering, ensures data quality and safety. The corpus's effectiveness is validated in training multilingual and multimodal models.
Summary of strengths: 1.The dataset covers 163 languages and diverse modalities, addressing gaps in existing resources and significantly supporting low-resource language research.

2.The study employs multiple techniques to enhance data quality and diversity, including extensive filtering to remove irrelevant content such as advertisements and website functional elements.
Summary of weaknesses: 1.Data bias risks. The corpus relies on Common Crawl, yet the paper does not sufficiently discuss potential cultural or linguistic distribution biases. Since Common Crawl primarily consists of open internet sources, high-resource languages (e.g., English) dominate, while low-resource languages may be underrepresented. This imbalance raises the question: does the resulting "multilingual" model genuinely understand local cultures, or merely translate high-resource language knowledge into low-resource languages? The paper lacks a detailed analysis of data representativeness.

2.Filtering limitations. The paper mentions keyword-based toxicity filtering, which has inherent weaknesses. For instance, certain words (e.g., "breast") are essential in medical literature but may be erroneously removed. Additionally, some words carry dual meanings depending on context. The study does not explore whether the blacklist dynamically adapts to semantic shifts.

3.Cross-modal alignment generalization. The paper uses pretrained models to filter text-image pairs but does not discuss their generalization across languages and cultures. Visual metaphors and textual expressions vary significantly, and mainstream pretrained models may not capture these nuances. For example, while the French golf tutorial image (Fig. 5) is correctly matched, a Swahili agricultural document's corresponding image might be incorrectly filtered. The study lacks experiments or analyses validating the robustness of its cross-modal matching approach.

4.Language identification challenges. Short text elements like navigation buttons often have ambiguous linguistic features, particularly in mixed-language environments. The paper does not present validation results, even through basic manual inspection, raising concerns about data quality consistency.
Comments, suggestions and typos: 1.Inconsistent terminology and abbreviations, such as "Multimodal Large Language Models (mLLMs)" in line 1 and "Multimodal large language models (mLLMs)" in line 36.

2.Minor grammatical issues, e.g., quotation mark errors in line 248.
Overall assessment scores: 3