MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
Authors: Fabian David Schmidt, Florian Schneider, Chris Biemann, Goran Glavaš
Accepted at: acl25

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

Fabian David Schmidt1*, Florian Schneider2*, Chris Biemann2, Goran Glavaš1
1Center for Artificial Intelligence and Data Science, University of Würzburg, Germany
2Language Technology Group, University of Hamburg, Germany
fabian.schmidt@uni-wuerzburg.de, florian.schneider-1@uni-hamburg.de
Dataset: MVL-SIB

Abstract

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages—over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N’Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.1

1 Introduction

Large Vision-Language Models (LVLMs) extend Large Language Models (LLMs) to take images as inputs, leveraging their advanced language capabilities for vision-language (VL) tasks like image captioning and visual question answering (VQA). However, LVLMs are typically trained mainly on

<table>
  <tr>
    <th colspan="2">Images-To-Sentences (12S)</th>
  </tr>
  <tr>
    <td colspan="2">Which sentence best matches the topic of the images? The images and the sentences each belong to one of the following topics: “entertainment”, “geography”, “health”, “politics”, “science and technology”, “sports”, or “travel”. Choose one sentence from A, B, C, or D. Output only a single letter!</td>
  </tr>
  <tr>
    <th># Images</th>
    <th># Sentences</th>
  </tr>
  <tr>
    <td>6</td>
    <td>A. „Maroschdorfer fährt am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweiter.“<br>B. „Es wurden keine schweren Verletzungen gemeldet, jedoch mussten mindestens fünf der zur Zeit der Explosion Anwesenden aufgrund von Schocksymptomen behandelt werden.“<br>C. „Finnland ist ein großartiges Reiseziel für Naturliebhaber. Das „Land der tausend Seen“ hat auch Touristen aus dem Ausland in den letzten Jahren zu den Klassikern der Naturkunde gemacht.“<br>D. „Es ist auch nicht erforderlich, dass Sie eine lokale Nummer von der Gemeinde erhalten, in der Sie leben. Sie können eine Internetverbindung über Satellit in der Wildnis v von Chicken in Alaska erhalten und eine Nummer auswählen, die vorgibt, dass Sie im sonnigen Arizona sind.“<br>Your answer here!</td>
  </tr>
</table>

Figure 1: Cross-modal topic matching ‘Images-To-Sentence’ for German with \( k=5 \) reference images.

English data, leading to significant limitations despite the base LLMs’ multilingual abilities. They may fail to follow instructions or struggle to interpret text within images in Non-English languages (Schneider and Sitaram, 2024; Tang et al., 2024). Although many multilingual VL benchmarks exist, they typically cover at most 10 languages (Bugliarello et al., 2022; Liu et al., 2021; Tang et al., 2024, inter alia). Only concurrent work has scaled VL evaluation to 100 languages using machine translation (MT) with human post-editing (Vayani et al., 2024). Nevertheless, benchmarks constructed using semi-manual MT cannot support truly low-resource languages adequately, as current MT models lack the necessary quality for these languages. Moreover, existing benchmarks primarily assess lower-level VL semantics through concrete text-image relationships, such as those found in VQA. This underscores the need for VL benchmarks that cover truly low-resource languages and evaluate more abstract VL interactions.

To address these challenges, we introduce the massively multilingual vision-language SIB (MVL-SIB) dataset, which extends the topic labels of the multi-way parallel sentences from SIB-200 (Adelani et al., 2024) by associating each topic with hand-selected images. MVL-SIB evaluates
Figure 2: Images-To-Sentences @ \( k=3 \). The English prompt describes the cross-modal topic matching task, lists all topics, and provides both \( k=3 \) reference images and 4 sentences in the corresponding language {eng_Latn, ..., nqo_Nkoo}. LVLMs must select the sentence of 4 options that topically fits \( k=3 \) reference images. The sentences spanning 205 languages and 7 topics are drawn from SIB-200 (Adelani et al., 2024), while images for the topics were hand-selected (cf. Appendix A.1). An example prompt is shown in Appendix A.7.2; further details are in §4. Plot. The x-axis orders the languages of the candidate sentences {eng_Latn, ..., nqo_Nkoo}, respectively, by descending performance (y-axis). The top x-axis indicates the running index of each language \( L_i \ (i \in \{1, \ldots, 205\}) \).

cross-modal image-text topic matching in 205 languages: LVLMs must select one of 4 candidate sentences that best matches the topic of the reference images ('images-to-sentence', cf. Figure 1) or, conversely, choose one of 4 candidate images corresponding to the topic of the reference sentences ('sentences-to-image'). Figure 2 displays the 'images-to-sentence' performance for across all languages in MVL-SIB, sorted in descending order. Notably, GPT-4o-mini performs robustly on the top 125 languages. However, beyond that range its performance declines sharply, falling to chance levels in the lowest-resource languages, such as N’Koo. Bridging these gaps is crucial for developing genuinely inclusive VL technology.

Contributions. 1) MVL-SIB supports parallel VL evaluation in 205 languages on professionally translated texts, a 105 more languages than any other VL benchmark. The tasks, images-to-sentence and sentences-to-images with prefix and postfix images in context, respectively, allows for fine-grained analysis of VL interactions. We also define corresponding text-only tasks by replacing the images with the topic label to compare the VL support (by language) of LVLMs against the text-only support of their underlying LLMs. Both tasks allow to vary the number of included images to analyze the shift from single to multi-image support in LVLMs. 2) We thoroughly evaluate LVLMs on cross-modal image-text topic matching, finding that task performance is closely associated with both model size and the size of pre-training corpora of the respective languages. We further find that only GPT-4o-mini seizes on multiple references in both cross-modal tasks. Open-weights LVLMs, moreover, favor one of the two tasks, highlighting the asymmetry in their VL support. 3) We analyze the relationship between stand-alone text and vision-language support in LVLMs by also benchmarking LVLMs on text-only topic matching. The performance gap between matching sentences to reference images or the topic tends to be larger the better the LVLM supports the underlying language. Conversely, the spread in performance between picking the fitting image or topic for reference sentences increases the worse the vision-language support of the LVLM for the evaluated language is. 4) We correlate MVL-SIB with established multilingual VL benchmarks on the languages shared between the respective pairs of datasets, showing that the MVL-SIB tasks align well with all VL tasks except OCR. We further show that images-to-sentence and sentence-to-image probe distinct aspects of VL interaction, as certain benchmarks correlate more strongly with one task than with the other. This analysis shows that MVL-SIB is a reliable and comprehensive VL benchmark for the lower-resource languages that are not covered by other datasets.

2 Related Work

Multilingual Vision-Language Models. Researchers have extended more English-centric LVLMs like BLIP-2 or LLava by continuing to train on multilingual data. Google’s PaLI models were the earliest closed-weight models trained on multilingual captions and VQA data; their open-weight PaliGemma followed a comparable strategy. Meanwhile, modern LLMs (e.g., Qwen 2.5,
Llama 3, Gemma 2, Aya) have improved in multilingual tasks but frequently fail to respond consistently in non-English languages, particularly in low-resource settings (Schneider and Sitaram, 2024). However, foundational models tend to focus on higher-resource languages and do not fully account for broader linguistic contexts in vision-language tasks. mBLIP is the first open multilingual LVLM trained on image captions and a small set of translated instruct data in 98 languages (Geigle et al., 2024a). Pangea incorporates multicultural dimensions by blending machine-translated data, existing multilingual resources, and synthetic data, on 39 languages (Yue et al., 2025). Most recently, Geigle et al. (2025) studied the composition of training data for multilingual adaptation of LVLMs, observing that only 25-50% of the data needs to be in English. The authors apply their findings when training ‘Centurio’, which achieves state-of-the-art performance on 14 multilingual VL benchmarks.

Multilingual Vision-Language Benchmarks. Existing datasets span VQA, natural language inference (NLI), image captioning, outlier detection, and culturally grounded QA:

VQA. xGQA extends the questions of the GQA dataset into 8 languages (5 scripts), but the answers in English (Pfeiffer et al., 2022). MaXM offers short-form QA fully in 7 languages, pairing culturally aligned images with same-language QA pairs (Changpinyo et al., 2023). MTVQA focuses on text-heavy VQA in 9 languages (Tang et al., 2024).

Culturally-grounded VQA. CVQA collects culturally diverse images and queries in 31 languages with multiple country-specific variants (Romero et al., 2025). The concurrent ALM-Bench covers 100 languages via MT with GPT-4o followed by human post editing with both generic and cultural multiple-choice and ‘true or false’ questions, as well as free-form VQA (Vayani et al., 2024).

Visual Reasoning & NLI. XVNLI evaluates cross-lingual visual NLI on 5 languages (Bugliarello et al., 2022). Binary reasoning tasks include MaRVL (Liu et al., 2021) and M5B-VGR (Schneider and Sitaram, 2024), each using linguistically specific images and textual statements. M5B-VLOD presents an outlier detection challenge, where a statement holds true for all but one image (Schneider and Sitaram, 2024).

Multiple-Choice QA. Babel-ImageNet (Geigle et al., 2024c) translates ImageNet labels into nearly 300 languages for multiple-choice object classification. M3Exam and xMMMU also feature multiple-choice VQA in 9 and 7 languages, respectively.

MVL-SIB fills the gaps in existing multilingual VL benchmarks. It provides test data professionally translated to 205 languages, covering over 105 more languages than other benchmarks for which MT cannot synthesize reliable data for. Other VL datasets that eschew MT, such as culturally-grounded VQA benchmarks, typically construct more language-specific non-parallel data, that does not support comparative evaluation across languages. The cross-modal topic matching tasks can be also framed text-only (cf. §3.2), replacing the images that represent topics with the explicit topic labels. Thereby, MVL-SIB enables to ablate the vision-language support from the textual support for a language. Finally, the benchmark allows to vary the number of images provided LVLMs to analyze the support for multi-image reasoning.

3 Dataset and Tasks

3.1 Dataset

For MVL-SIB, we extend the following Flores-based datasets to create a massively multilingual, multi-way parallel VL benchmark for identifying topical associations between images and sentences.

Flores. Flores is a machine translation benchmark containing 3,001 sentences from English Wikipedia paragraphs (Team et al., 2022), professionally translated into over 200 languages.2

SIB-200. Adelani et al. (2024) grouped the coarse topical annotations for sentences in the DEV and DEVTEST subsets of Flores into 7 higher-level topics.3 The resulting SIB-200 dataset is a benchmark for topical classification with 1,004 parallel examples for 205 language variants.

MVL-SIB. For each topic, we first manually collect 10 permissively licensed images that distinctly represent the topic (e.g., sports) with minimal overlap or ambiguity (cf. Appendix A.1). We verify that all LVLMs in our study correctly classify the topics for the images when prompted (cf. Appendix A.7.1). We next create 3 different MVL-SIB instances from each of the 1,004 SIB sentences, totaling 3,012 MVL-SIB instances. For each MVL-SIB instance, we couple the respective SIB sen-

2Flores splits 3,001 sentences into DEV (997), DEVTEST (1,012), and TEST (992) sets. The TEST set was not released.
3The topics are entertainment, geography, health, politics, science/technology, sports, and travel.
tence with (1) a random selection of 5 positive images (same topic) and 4 additional sentences from the same category as the original sentence, as well as (2) 3 negative images and sentences randomly sampled from different topics compared to the starting SIB sentence. The set of sampled sentences and images by instance is maintained across languages.

3.2 Cross-modal & Text-only Topic Matching

We formulate both cross-modal and text-only topic matching tasks based on MVL-SIB. In every task, we present the model with the list of topics that images and sentences may be associated with.3 Otherwise, it would be unclear along which dimension the model should match images and sentences. The portion of the prompt that introduces the task is provided in English, while the sentences to be topically aligned with images are presented in one of the 205 languages included in MVL-SIB. LLMs reliably perform tasks described in English, even when task-related information is conveyed in other languages (Muenninghoff et al., 2023; Romanou et al., 2025). This ensures a fair comparison across all 205 languages, where MT would not accurately preserve the meaning of the prompts. We detail prompts for each task in Appendix A.2.

Cross-modal Topic Matching. Using our text-image samples (cf. §3.1), we define two cross-modal topic matching tasks: Images-To-Sentence (I2S) and Sentences-To-Image (s2I). In I2S, the model must select, from 4 candidates, the sentence that matches the topic of \( k \) reference images. Conversely, in s2I, the model chooses, from 4 options, the image that shares the topic with \( k \) reference sentences. In both tasks, we present the model with \( k \in \{1, 3, 5\} \) references, respectively. These tasks evaluate the model’s ability to align high-level visual and textual cues on topics.

Text-only Topic Matching. We construct two tasks by replacing the images in I2S and s2I, that represent the topics, with their corresponding labels (e.g., sports). The resulting unimodal tasks, Topic-To-Sentence (T2S) and Sentences-To-Topic (s2T), mirror the cross-modal tasks, I2S and s2I, respectively. For T2S, we evaluate only \( k = 1 \), since repeating the topic label adds no information. These baseline tasks allow us to delineate between language support and vision-language understanding in LVLMs.

MVL-SIB offers 4 crucial advantages over prior benchmarks. 1) It supports evaluation in 205 languages, covering over 100 more languages than existing benchmarks for which MT models fail to synthesize reliable evaluation data. 2) MVL-SIB supports ablating language understanding and multimodal reasoning of LVLMs by comparatively evaluating the mirroring text-only and cross-modal topic matching tasks (cf. §3.2). 3) MVL-SIB enables intricate analysis of single- and multi-image VL interactions in LVLMs by allowing topics to be represented by varying numbers of images in cross-modal tasks. 4) MVL-SIB comprises higher-level VL reasoning tasks, pairing varied images and diverse texts to test nuanced VL understanding.

4 Experimental Setup

Models. We test state-of-the-art LVLMs Qwen2-VL (Wang et al., 2024), InternVL 2.5 (Chen et al., 2024), Centurio-Qwen (Geigle et al., 2025), and GPT-4o-(mini) across available sizes.4 Smaller LVLMs are evaluated on all languages, while larger ones (26B+) are tested on subsets (cf. §5.3). For cross-modal topic matching, we also evaluate on mSigLIP-base (Zhai et al., 2023). Trained explicitly for semantic similarity on multilingual image-caption pairs, the ViT represents a strong baseline.5 Its prediction denotes the choice that has the highest average cosine similarity to the \( k \) references.

Image preprocessing. We downsample the images to 640×480 pixels, as the tasks rely on higher-level visual cues for topically associating images and texts rather than finer image details. This can significantly reduce the number of visual tokens input to LVLMs, enabling more efficient inference.

Hyperparameters. We decode text greedily with temperature set to 0.0 to ensure reproducibility.

Metric. We compute the share of prompts for which responses begin with the right letter. If the label is "A", a response such as "A." is also correct.

5 Results and Discussion

We categorize the languages in MVL-SIB based on their ‘resourceness’. To do so, we reorganize the language groups from Joshi et al. (2020) into four tiers. We first rank the tiers w.r.t. Wikipedia size and then merge (i) the two highest-resource tiers and (ii) the two lowest-resource tiers.6 This both

3We provide details on the LVLMs in Appendix A.3.
4The model is available on Huggingface at: google/siglip-base-patch16-256-multilingual.
5Sorting by Wikipedia size (in number of pages) swaps tiers 1 and 2; we then merge Tier 0 with the new Tier 1, as well as tiers 4 and 5.
<table>
  <tr>
    <th rowspan="2">Resourceness</th>
    <th colspan="6">High</th>
    <th colspan="6">Mid</th>
    <th colspan="6">Low</th>
  </tr>
  <tr>
    <th colspan="3">ENGLISH</th>
    <th colspan="3">TIER 4 (26)</th>
    <th colspan="3">TIER 3 (32)</th>
    <th colspan="3">TIER 2 (96)</th>
    <th colspan="3">TIER 1 (51)</th>
  </tr>
  <tr>
    <th>k References</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
  </tr>
  <tr>
    <th colspan="18"><i>Images-To-Sentence: Select 1 of 4 sentences topically matching k reference images</i></th>
  </tr>
  <tr>
    <td>mSigLIP-base</td><td>57.7</td><td>64.6</td><td>66.4</td><td>53.3</td><td>58.6</td><td>59.7</td><td>51.4</td><td>56.2</td><td>57.1</td><td>38.9</td><td>41.2</td><td>41.7</td><td>36.1</td><td>37.6</td><td>38.0</td>
  </tr>
  <tr>
    <td>Qwen2-VL 2B</td><td>36.3</td><td>34.8</td><td>34.9</td><td>35.5</td><td>35.3</td><td>34.1</td><td>34.5</td><td>34.3</td><td>33.2</td><td>31.0</td><td>30.6</td><td>30.0</td><td>29.5</td><td>29.0</td><td>28.6</td>
  </tr>
  <tr>
    <td>Qwen2-VL 7B</td><td>65.8</td><td>63.1</td><td>58.9</td><td>57.7</td><td>56.5</td><td>51.7</td><td>55.4</td><td>54.5</td><td>49.6</td><td>44.3</td><td>44.4</td><td>40.5</td><td>39.6</td><td>39.7</td><td>36.5</td>
  </tr>
  <tr>
    <td>InternVL 2.5 4B</td><td>52.5</td><td>49.2</td><td>48.1</td><td>50.3</td><td>46.6</td><td>47.7</td><td>48.6</td><td>45.3</td><td>46.1</td><td>38.7</td><td>37.1</td><td>37.3</td><td>35.4</td><td>34.5</td><td>34.5</td>
  </tr>
  <tr>
    <td>InternVL 2.5 8B</td><td>67.7</td><td>67.9</td><td>68.7</td><td>64.6</td><td>64.9</td><td>65.7</td><td>61.2</td><td>60.8</td><td>61.6</td><td>51.0</td><td>51.4</td><td>51.8</td><td>46.1</td><td>46.0</td><td>46.3</td>
  </tr>
  <tr>
    <td>Centurio Qwen</td><td>54.8</td><td>60.0</td><td>62.4</td><td>54.2</td><td>59.2</td><td>60.6</td><td>53.4</td><td>58.1</td><td>58.9</td><td>46.6</td><td>48.9</td><td>49.2</td><td>43.0</td><td>44.2</td><td>44.7</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td><b>68.3</b></td><td><b>78.1</b></td><td><b>77.4</b></td><td><b>71.6</b></td><td><b>79.0</b></td><td><b>78.1</b></td><td><b>72.0</b></td><td><b>78.9</b></td><td><b>77.7</b></td><td><b>63.5</b></td><td><b>68.0</b></td><td><b>66.4</b></td><td><b>56.9</b></td><td><b>60.3</b></td><td><b>58.7</b></td>
  </tr>
  <tr>
    <th colspan="18"><i>Sentences-To-Image: Select 1 of 4 images topically matching k reference sentences</i></th>
  </tr>
  <tr>
    <td>mSigLIP-base</td><td>56.3</td><td>66.0</td><td>69.6</td><td>51.8</td><td>61.6</td><td>64.0</td><td>49.1</td><td>58.3</td><td>60.2</td><td>36.0</td><td>40.4</td><td>41.2</td><td>32.9</td><td>36.3</td><td>36.9</td>
  </tr>
  <tr>
    <td>Qwen2-VL 2B</td><td>41.9</td><td>43.1</td><td>43.4</td><td>41.6</td><td>42.5</td><td>42.7</td><td>40.8</td><td>42.4</td><td>42.4</td><td>33.7</td><td>35.6</td><td>35.5</td><td>31.0</td><td>32.7</td><td>32.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL 7B</td><td>71.7</td><td>70.4</td><td>68.6</td><td>65.5</td><td>65.5</td><td>63.5</td><td>64.4</td><td>65.3</td><td>64.1</td><td>50.3</td><td>52.5</td><td>52.9</td><td>43.5</td><td>45.9</td><td>46.6</td>
  </tr>
  <tr>
    <td>InternVL 2.5 4B</td><td>47.7</td><td>44.5</td><td>43.0</td><td>38.0</td><td>40.3</td><td>40.4</td><td>36.7</td><td>39.6</td><td>40.3</td><td>30.7</td><td>34.4</td><td>35.7</td><td>28.8</td><td>32.1</td><td>33.7</td>
  </tr>
  <tr>
    <td>InternVL 2.5 8B</td><td>66.2</td><td>69.0</td><td>68.7</td><td>57.5</td><td>62.5</td><td>61.6</td><td>52.9</td><td>58.5</td><td>58.1</td><td>43.4</td><td>49.8</td><td>49.7</td><td>39.7</td><td>45.9</td><td>46.2</td>
  </tr>
  <tr>
    <td>Centurio Qwen</td><td>35.3</td><td>36.1</td><td>35.6</td><td>31.1</td><td>32.9</td><td>33.3</td><td>31.0</td><td>32.8</td><td>33.1</td><td>28.7</td><td>29.7</td><td>29.8</td><td>28.1</td><td>28.7</td><td>28.7</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td><b>77.5</b></td><td><b>86.4</b></td><td><b>89.1</b></td><td><b>77.2</b></td><td><b>86.5</b></td><td><b>88.6</b></td><td><b>77.1</b></td><td><b>86.1</b></td><td><b>88.4</b></td><td><b>68.4</b></td><td><b>79.8</b></td><td><b>82.7</b></td><td><b>61.7</b></td><td><b>74.0</b></td><td><b>77.2</b></td>
  </tr>
</table>

Table 1: Cross-modal Topic Matching: LVLMs must select the candidate sentence (image) from 4 choices that topically align with k reference images (sentences). Prompts provided in §A.2. Languages are tiered by Wikipedia sizes (cf. §5). Number of languages in parentheses. Metric: share of responses starting with correct option letter. Details in §4. In each column, the best model is emphasized in bold, the second-best model is underlined.

<table>
  <tr>
    <th rowspan="2">Resourceness</th>
    <th colspan="6">High</th>
    <th colspan="6">Mid</th>
    <th colspan="6">Low</th>
  </tr>
  <tr>
    <th colspan="3">ENGLISH</th>
    <th colspan="3">TIER 4 (26)</th>
    <th colspan="3">TIER 3 (32)</th>
    <th colspan="3">TIER 2 (96)</th>
    <th colspan="3">TIER 1 (51)</th>
  </tr>
  <tr>
    <th>Task</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
    <th>Topic-To Sentence</th><th>Sentences-To-Topic</th>
  </tr>
  <tr>
    <th>k References</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
    <th>1</th><th>3</th><th>5</th>
  </tr>
  <tr>
    <td>Qwen2-VL 2B</td><td>56.7</td><td>86.1</td><td>96.5</td><td>98.2</td><td>49.1</td><td>78.4</td><td>92.0</td><td>95.2</td><td>45.2</td><td>73.7</td><td>89.6</td><td>93.6</td><td>36.4</td><td>53.7</td><td>69.3</td><td>76.4</td><td>33.0</td><td>46.4</td><td>60.7</td><td>68.4</td>
  </tr>
  <tr>
    <td>Qwen2-VL 7B</td><td>85.7</td><td>89.1</td><td>95.3</td><td>97.5</td><td>81.8</td><td>84.1</td><td>93.5</td><td>96.1</td><td>80.5</td><td>84.1</td><td>93.5</td><td>96.6</td><td>63.7</td><td>67.8</td><td>81.3</td><td>86.7</td><td>55.3</td><td>59.1</td><td>73.7</td><td>79.8</td>
  </tr>
  <tr>
    <td>InternVL 2.5 4B</td><td>81.4</td><td>90.6</td><td>98.0</td><td>99.1</td><td>72.7</td><td>85.2</td><td>95.8</td><td>97.9</td><td>68.2</td><td>83.7</td><td>95.5</td><td>97.7</td><td>50.2</td><td>67.8</td><td>83.4</td><td>87.9</td><td>44.6</td><td>50.7</td><td>77.4</td><td>83.1</td>
  </tr>
  <tr>
    <td>InternVL 2.5 8B</td><td>87.0</td><td>91.7</td><td>98.1</td><td>99.0</td><td>83.0</td><td>86.3</td><td>96.3</td><td>98.3</td><td>79.1</td><td>82.4</td><td>94.1</td><td>96.7</td><td>65.4</td><td>66.8</td><td>81.8</td><td>86.9</td><td>58.1</td><td>58.3</td><td>74.6</td><td>80.9</td>
  </tr>
  <tr>
    <td>Centurio Qwen</td><td>85.4</td><td>89.9</td><td>96.7</td><td>97.7</td><td>83.6</td><td>88.0</td><td>95.8</td><td>97.7</td><td>82.6</td><td>87.2</td><td>96.0</td><td>97.9</td><td>70.4</td><td>73.8</td><td>86.8</td><td>90.7</td><td>64.0</td><td>66.1</td><td>81.2</td><td>85.7</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td><b>88.5</b></td><td><b>92.4</b></td><td><b>98.1</b></td><td><b>99.1</b></td><td><b>89.3</b></td><td><b>91.6</b></td><td><b>98.4</b></td><td><b>99.3</b></td><td><b>89.3</b></td><td><b>91.6</b></td><td><b>98.4</b></td><td><b>99.3</b></td><td><b>80.9</b></td><td><b>82.1</b></td><td><b>93.5</b></td><td><b>95.7</b></td><td><b>73.9</b></td><td><b>74.3</b></td><td><b>88.2</b></td><td><b>92.1</b></td>
  </tr>
</table>

Table 2: Text-only Topic Matching: LVLMs must select the candidate sentence (topic) of 4 choices that aligns topically with the reference topics (k reference sentences). See Table 1 for further details.

better reflects current corpus availability for LLM pre-training (Xue et al., 2021; Kudugunta et al., 2023), and aligns with downstream performance (cf. Appendix A.3.1). We isolate English from Tier 4, since it is the pivotal language in NLP. The full per-language results by task and model are provided in Appendix A.7.

5.1 Cross-modal Topic Matching

Images-To-Sentence (12s). The upper segment of Table 1 displays the results for 12s, in which the LVLMs must pick the candidate sentence that topically matches the k reference images.

English. The performance on 12s with English candidate sentences scales well with model size. The small Qwen2-VL 2B performs only slightly better than chance (25% vs. ca. 35%). The comparably sized Qwen2-VL 7B, InternVL 2.5 8B, and Centurio-Qwen 8B peak around 62% to 68% at various k. These models nevertheless non-negligibly trail GPT-4o-mini (78.1%). Among LVLMs, only InternVL 2.5 8B, Centurio-Qwen, and GPT-4o-mini benefit from multiple reference images. When the number of references k increases from 3 to 5, GPT-4o-mini declines slightly in performance, while InternVL and Centurio-Qwen continue to improve marginally (ca. +1%) and more notably (ca. +3-6%), respectively. All other LVLMs deteriorate materially with more reference images (ca. -3-4%). mSigLIP indeed is a strong baseline, trailing only GPT-4o-mini and InternVL 2.5 8B at k=5. The ViT yields large gains with 4 more images (+9.7%).

Tiers. The performance gap of other languages to English correlates well with their resource levels by language tier. When presented with candidate sentences in non-English high-resource languages (cf. Tier 4), GPT-4o-mini even performs better slightly better. For very low-resource languages in
Tier 1, such as N’Koo or Tamazight, all models fail to perform better than chance (cf. Appendix A.7.2). Among LVLMs, only GPT-4o-mini remains overall robust for topically matching sentences of low-resource languages to images, whereas other models drop severely in performance (ca. 15–20%). While mSigLIP still performs well, it declines more notably than LVLMs on lower-resource languages.

Sentences-To-Image (S2I). The lower part of Table 1 presents the results for s2i. Here, the models select the candidate image among four options that topically fits the \( k \) reference sentences.

English. Performance again correlates well with model capacity. However, in s2i, only GPT-4o-mini significantly seizes on additional references (+13%) to excel with 89%, while models like Qwen2-VL 7B and InternVL 2.5 8B exhibit peak performance at \( k=1 \) and \( k=3 \), respectively, that taper slightly with more sentence references. Centurio-Qwen performs only slightly better than random (25% vs. ca. 35%). In s2i, mSigLIP is again very strong, second only to GPT-4o-mini at \( k=5 \). The encoder once more seizes sizable gains from additional references (+13.3%).

Tiers. In non-English evaluations, the overall trend remains similar, though absolute performance is lower. The gap between high- and low-resource language tiers is evident, as all models yield higher scores across Tiers 4 and 3. GPT-4o-mini maintains robust performance even in the most challenging Tier 1 when provided multiple references (ca.75%).

In sum, both the training protocol and the model size collectively determine whether models favor 12s or s2i. For instance, InternVL 2.5 8B outperforms Qwen2-VL 7B on 12s across the board, while trailing on s2i. Moreover, only GPT-4o-mini consistently seizes on additional references and remains largely robust to the lowest-resource languages on both tasks. This likely stems from insufficient training to enable open-weight LVLMs to perform higher-level VL reasoning with diverse texts and multiple images successfully. For instance, Centurio Qwen was mostly trained on data that prefixes images to text, resulting in low performance when images are postfixed to the context.

5.2 Text-only Topic Matching

Table 2 lists the results for text-only topic matching, in which the images are exchanged with their topic label. These tasks denominate ‘upper-bounds’ for their cross-modal counterparts to enable ablations of language support and VL support in LVLMs.

Topic-to-Sentence (t2s). In this task, LVLMs choose the sentence that best suits the reference topic. Barring Qwen2-VL 2B, all models perform well on the task. Notably, 5 images should capture the underlying topic well for all models (cf. Appendix A.7.1). Despite that, the gap between text-only and vision-language tasks (cf. Table 1) is sizable across all open-weights models for ‘English’ (ca. 20%+). In English, GPT-4o-mini and InternVL 2.5 8B achieve the highest accuracies, indicating strong topic comprehension. For non-English languages, while the overall scores are reduced, high-resource languages benefit from richer training signals compared to their low-resource counterparts – models like Qwen2-VL 2B and Centurio-Qwen 8B show a more pronounced drop in the latter, underscoring the impact of language resources.

Sentences-to-Topic (s2T). In s2T, where models choose the topic that best aligns with \( k \) reference sentences, performance scales both with model size and the number of references. The gains from additional context (\( 3–5\ k \)) are particularly notable for larger LVLMs. In English, GPT-4o-mini improves markedly from 92.4% at \( k=1 \) to 98.1% at \( k=3 \). In non-English languages, similar patterns emerge: high-resource languages consistently yield higher accuracies than low-resource ones, with GPT-4o-mini and InternVL 2.5 8B exhibiting the most stable improvements across varying \( k \). This reinforces the role of model capacity and training data diversity in effective cross-lingual topic matching.

Comparing the results of cross-modal and text-only topic matching sheds further light on the VL interactions for lower-resource languages in LVLMs. The performance gap between matching sentences to reference images and matching them to textual topics tends to narrow regardless of the number of references, likely reflecting their limited textual support in LVLMs. In contrast, the discrepancy between selecting an image versus a topic for reference sentences becomes much more pronounced, especially at \( k=5 \). These findings suggest that VL support degrades more sharply than textual support for lower-resource languages in LVLMs.

5.3 Further Analyses

Task Correlation. To compare MVL-SIB with other tasks, we access the results for Qwen2-VL and InternVL 2.5 models on several established multilingual VL benchmarks from Geigle et al.
Correlations Between MVL-SIB & Multilingual VL Benchmarks. Pearson correlation coefficients obtained by regressing MVL-SIB performance against performance on multilingual VL tasks on languages common to both datasets, respectively. An asterisk (*) indicates whether the coefficient is statistically significant at \( p \leq 0.05 \).

(2025).7 Next, we align the results across languages for the MVL-SIB tasks and the other benchmarks. Finally, we plot the linear regressions of performance on 12s and s21 with \( k=3 \) against the performance on the VL benchmarks, respectively, pooled over models, in Figures 3 and 4.8

MVL-SIB positively correlates with all tasks in both the 12s and s21 evaluations. However, both the magnitude and statistical significance of these linear relationships vary across VL benchmarks. Both 12s and s21 exhibit the strongest connections with XNLI (visual inference), BIN-MC (multiple-choice image classification), MaRVL, and M5-VGR (both visually-grounded boolean reasoning). Since these tasks restrict valid answers to a small set of fixed options (i.e., choice letters or ‘yes/no/maybe’), LVLMs must engage in higher-level vision-language disambiguation rather than relying solely on lower-level visual cues to solve these tasks. In addition, xMMU, M3Exam, and M5-VLOD are significantly related only to 12s, whereas xGQA is significantly aligned solely with s21. The former tasks are structurally similar to 12s, typically presenting one or more images that LVLMs are given as visual context to answer multiple-choice questions. We hypothesize that only s21 is significantly correlated with xGQA, since s21 is more analogous to object-centric benchmarks. In s21, LVLMs likely leverage targeted semantic cues (e.g., keywords or phrases) from the reference sentences to better disambiguate candidate images by topic. This behavior aligns with lower-level VL tasks such as xGQA (cross-lingual short-form QA) or BIN-MC (multiple-choice object classification), where texts and images are more deliberately connected. In contrast, MTVQA, SMPQA-name, and SMPQA-ground show only weak or statistically insignificant correlations with the MVL-SIB tasks. Since these tasks require LVLMs to comprehend text embedded in images, a low-level, fine-grained VL task, they differ substantially from the higher-level VL reasoning evaluated by MVL-SIB.

Overall, the regression analysis indicates that 12s and s21 capture distinct yet complementary aspects of VL understanding, collectively exhibiting a strong relationship with a broad set of VL tasks. This aligns with our main results (cf. Table 1), which show that different LVLMs may favor one MVL-SIB task over the other. This renders MVL-SIB as a suitable benchmark for evaluating universal VL understanding (across 205 languages). It also enables to ablate performance across the key axes of analysis, the task formulation (12s vs. s21), the language vs. vision-language support for 205 languages, and the number of images in context.9

Larger LVLMs. To evaluate larger LVLMs on MVL-SIB, we construct language-tier subsets that reliably estimate performance while mitigating excessive computational overhead (cf. §5). For both 12s and s21, we identify the three languages in each tier that best replicate the average performance of

7We omit Centurio-Qwen, since it degrades on the s21 task. We further provide details on all the multilingual vision-language benchmarks we correlate MVL-SIB with in Appendix A.4

8Note that we include a constant in our regression model to bridge task-specific scales of results.

9In s21, candidates could comprise more than a single image.
Figure 5: Larger LVLMs on Subsampled Tiers. We extract 3 languages per tier that mimic avg. performance full language groups (cf. §5.3) and evaluate LVLMs across all model sizes on {12S,s21,t2S,s2T } @ k=3 (cf. §3.2).

the tier. First, we compute the average performance per language tier for both 12S and s21 with k=3, pooling results across models (cf. §5). Then, for each tier, we select the three languages whose performance deviates least from the tier mean. The languages chosen for each tier by task are detailed in Appendix A.6. Finally, we test GPT-4o, InternVL 2.5 [26,32,72]B, and Qwen 2.5 VL [32,70]B on these subsets to assess how well larger models perform across language tiers.

Figure 5 displays the results for both 12S and s21 with k=3.\footnote{Qwen2-VL 72B frequently responded with the first letter of the correct topic. If that letter uniquely identifies the correct choice, the answer is considered correct.} We observe that larger models catch up to GPT-4o on 12S and even outperform it on s21 for higher-resource languages. Since open-weight LVLMs have more limited language support for low-resource languages, GPT-4 and GPT-4o-mini outperform all other models on 12S for languages in tier 1 and 2. Increasing model capacity yields the largest gains on s21, for which the models exceed GPT-4o up to the lowest-resource Tier 1. Moreover, unlike smaller models, all larger LVLMs (+26B) more effectively leverage multiple image references to improve performance (cf. Appendix A.6). They reap the largest benefit when the number of references increases from 1 to 3 (ca. +3%), with further improvements at \( k = 5 \) (ca. +1.5%). These results suggest that larger LVLMs have fundamentally better VL support irrespective of the evaluated language (cf. §5.2). The results on text-only topic matching further underscore this notion (cf. lower segment of Figure 5). Larger LVLMs near perfectly match both the reference topic to the correct sentence (ca. 90%) and references sentences to the right topic (ca. 98%). Notably, as model size increases, the performance gap between cross-modal and text-only matching narrows. Collectively, these results further indicate that VL support relative to text-only support improves with increasing model capacity in LVLMs.

6 Conclusion

We present the massively multilingual vision-language benchmark MVL-SIB for cross-modal (and text-only topic matching) in 205 languages that offers key advantages over prior multilingual VL benchmarks. Notably, it covers over 100 additional languages without relying on machine translation. MVL-SIB allows for a clear separation between textual language support and vision-language support in LVLMs by comparing performance on mirrored cross-modal and text-only tasks. Moreover, it allows us to study how LVLMs handle single-image versus multi-image formulations of cross-modal topic matching by varying the number of images provided. In our comparative evaluation of state-of-the-art LVLMs on MVL-SIB, we find that model performance is strongly correlated with both model size and the volume of available pre-training data for each language. However, all LVLMs experience a dramatic performance drop on the lowest-resource languages. Our analysis further reveals that vision-language support deteriorates disproportionately relative to language support, highlighting the need to incorporate low-resource languages into VL training. Moreover, providing multiple images does not benefit open-weight LVLMs in cross-modal topic matching, suggesting that LVLMs are not yet fully effective in multi-image tasks. Lastly, we validate that MVL-SIB correlates well with existing multilingual VL benchmarks, underscoring its reliability as a source of evaluation data for 205 languages.
7 Limitations

Our work faces three primary limitations. First, although a vast number of LVLMs exist, we selected a representative subset based on key criteria. Specifically, the LVLMs in our study (Qwen2-VL and InternVL, with the exception of Centurio) span a range of parameter counts typical of LLMs. Additionally, we include GPT-4o-mini in the full evaluation and GPT-4o on the subsampled language tiers. Evaluating MVL-SIB across all four tasks t2S, s2I, t2S, and s2T (cf. §3.2) at various \( k \in \{1, 3, 5\} \) over 205 languages (i.e., evaluations per model and task, or 2050, in sum per model) becomes computationally intractable. This accumulates to \( 3 \times 205 = 615 \) evaluations per model (205 for t2S as only \( k=1 \) reference topic exists) or \( 3 \times 205 + 205 = 2050 \) evaluations in total. We therefore both provide subsets of the language tiers to evaluate on and demonstrate that evaluation only requires 1K instances to reliably estimate task performance. Second, while we strove to choose a diverse set of images to capture the full semantic range of each topic, further diversification is possible by sourcing additional images. However, due to the limited availability of openly licensed images, some topics (e.g., politics and entertainment) are represented predominantly by images that embody the topic in a more Western-centric cultural context. Hand-selecting images by topic for each language or, more broadly, cultural groups would not scale to 205 languages and would hinder the comparability of results. Our results furthermore confirm that models just as well perform on a broad range of languages spanning diverse cultural backgrounds as on English (cf. Figure 2). At the same time, LVLMs perform best on Western-centric images, mitigating any variation that would originate from using more culture-specific images. Finally, for the topic geography, we manually selected images that are representative within the context of SIB, as the broader definition of geography is too diffuse to capture visually.

Acknowledgements

We used AI assistance (chatGPT o3-mini) to polish the writing and the tables of the manuscript as well as to refine the code for our visualizations.

References

David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiu Lee. 2024. SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta.

Željko Agić and Natalie Schluter. 2018. Baselines and Test Data for Cross-Lingual Inference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 3890–3894, Miyazaki, Japan.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 632–642, Lisbon, Portugal.

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. 2022. IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages. In International Conference on Machine Learning, pages 2370–2392, Baltimore, MD, USA.

Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023. MaXM: Towards Multilingual Visual Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2667–2682, Singapore.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. InternVL: Scaling Up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, Seoul, Korea.

J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, Miami, FL, USA.

William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The Dollar Street Dataset: Images Representing the Geographic and Socioeconomic Diversity of the World. In Advances in Neural Information Processing Systems, volume 35, pages 12979–12990, New Orleans, LA, USA.

Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. 2024a. mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pages 7–25, Bangkok, Thailand.
Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, and Goran Glavaš. 2025. Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model. arXiv preprint arXiv:2501.05122.

Gregor Geigle, Radu Timofte, and Goran Glavaš. 2024b. African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2653–2669, Miami, Florida, USA.

Gregor Geigle, Radu Timofte, and Goran Glavaš. 2024c. Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5064–5084, Bangkok, Thailand.

Drew A. Hudson and Christopher D. Manning. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 6700–6709, Long Beach, CA, USA.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vision, 123(1):32–73.

Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romet Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. In Advances in Neural Information Processing Systems, volume 36, pages 67284–67296, New Orleans, LA, USA.

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually Grounded Reasoning across Languages and Cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Punta Cana, Dominican Republic.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual Generalization through Multitask Finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada.

Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O. Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022. xGQA: Cross-Lingual Visual Question Answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511, Dublin, Ireland.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, pages 8748–8763, Online.

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalka Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Virat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duval, Daniel Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestem, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzeminski, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbe, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushki Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. 2025. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge. In The Thirteenth International Conference on Learning Representations, Singapore.

David Romero, Chenyang Lyu, Haryo Akbarianto Wi-bowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnau Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Graine Caulfield, Guido Ivetta, Haiyu Song, Henok Biadgign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D’Haro, Marcelo Viridiano, Marcos Estecha-Garitagotitia, Maria Camila Buitrago
Cabrera, Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Miaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazurda, Munkhjargal Gochoo, Munkh-Erdene Ogtobold, Naoe Etori, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuriabayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqueer Ehsan, Vladimir Araujo, Yova Kementchedjieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji. 2025. CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark. In Advances in Neural Information Processing Systems, volume 37, pages 11479–11505.

Florian Schneider and Sunayana Sitaram. 2024. M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4309–4345, Miami, Florida, USA.

Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jie Ye. 2024. Parrot: Multilingual Visual Instruction Tuning. arXiv preprint arXiv:2406.02539.

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, and Can Huang. 2024. MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering. arXiv preprint arXiv:2405.11985.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayakh, Kenneth Heafield, Kevin Hefner, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semonar Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rogers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.04672.

Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emirates.

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadgling Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Miaylov, Chao Qin, Abdelrahman M. Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel Esplana, Monil Gokani, Shachar Mirkin, Harsh Singh, Ashay Srivastava, Endre Hamerlik, Fathinah Asma Izzati, Fadillah Adamsyah Maani, Sebastian Cavada, Jenny Chim, Rohit Gupta, Sanjay Manjunath, Kamila Zhumakhanova, Feno Herininaina Ravehohtira, Azril Amirudin, Muhammad Ridzuan, Daniya Kareem, Ketan More, Kunyang Li, Pramesh Shakya, Muhammad Saad, Amirpouya Ghasemaghaei, Amirbek Djanibekov, Dilshod Azizov, Branislava Jankovic, Naman Bhatia, Alvaro Cabrera, Johan Obando-Ceron, Olympiah Otieno, Fabian Fastrem, Muztoba Rabbani, Sanoojan Baliah, Santosh Sanjeev, Abduragim Stanchaev, Maheen Fatima, Thao Nguyen, Amrin Kareem, Toluwani Aremu, Nathan Xavier, Amit Bhatkal, Hawau Toyin, Aman Chadha, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Jorma Laaksonen, Thamar Solorio, Monojit Choudhury, Ivan Laptev, Mubarak Shah, Salman Khan, and Fahad Khan. 2024. All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages. arXiv preprint arXiv:2411.16508.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingen Zhou, and Junyong Lin. 2024. Wen2VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191.

Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual Entailment: A Novel Task for Fine-grained Image Understanding. In 32nd Conference on Neural Information Processing Systems, ViGIL Workshop, Montreal, Canada.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, pages 483–498, Online.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingen Zhou, Jinze Bai, Jinzheng He, Junyong Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan,
Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Owen2 Technical Report. arXiv preprint arXiv:2407.10671.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhui Chen. 2024. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, Seattle, WA, USA.

Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khaunja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2025. Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages. In The Thirteenth International Conference on Learning Representations, Singapore.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, Paris, France.

Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models. In Advances in Neural Information Processing Systems, volume 36, pages 5484–5505, New Orleans, LA, USA.

A Appendix
A.1 Images Per Topic

Entertainment

Geography

Health

Politics

Science & Technology

Sports

Travel
A.2 Prompts

Images-To-Sentences (12s)

Which sentence best matches the topic of the images? The images and the sentences each belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel". Choose one sentence from A, B, C, or D. Output only a single letter!

# Images

![Image 1](page_349_670_153_104.png) ![Image 2](page_502_670_153_104.png) ![Image 3](page_655_670_153_104.png) ![Image 4](page_808_670_153_104.png) ![Image 5](page_961_670_153_104.png)

# Sentences
A. ``Maroochydore führte am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweitem.``
B. ``Es wurden keine schwere Verletzungen gemeldet, jedoch mussten mindestens fünf der zur Zeit der Explosion Anwesenden aufgrund von Schocksymptomen behandelt werden.``
C. ``Finnland ist ein großartiges Reiseziel für Bootstouren. Das „Land der tausend Seen“ hat auch Tausende von Inseln – in den Seen und in den Küstenarchipelen.``
D. ``Es ist auch nicht erforderlich, dass Sie eine lokale Nummer von der Gemeinde erhalten, in der Sie leben. Sie können eine Internetverbindung über Satellit in der Wildnis von Chicken in Alaska erhalten und eine Nummer auswählen, die vorgibt, dass Sie im sonnigen Arizona sind.``
Your answer letter:
Sentences-To-Images (§21)

Which image best matches the topic of the sentences? The sentences and the images each belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel". Choose one image from A, B, C, or D. Output only a single letter!

# Sentences
• ````Maroochydore führte am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweitem.````

• ````Die Schlagmänner der mittleren Reihe, Sachin Tendulkar und Rahul Dravid, zeigten gute Leistungen und erzielten eine Partnerschaft mit 100 Runs.````

• ````Da pro Tag nur achtzehn Medaillen zur Verfügung stehen, hat es ein Anzahl an Ländern nicht auf das Podium geschafft.````

• ````Wintersportarten sind in den nördlichen Regionen am beliebtesten und Italiener nehmen an internationalen Wettkämpfen und olympischen Spielen teil.````

• ````Nach dem Rennen bleibt Keselowski mit 2.250 Punkten Spitzenreiter in der Fahrerwertung.````

# Images

A.        B.        C.        D.

Your answer letter:
Topic-To-Sentence (T2S)

Which sentence best matches the topic "sports"? The sentences each belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel". Choose one sentence from A, B, C, or D. Output only a single letter!

# Sentences
A. ``Maroochydore führte am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweitem.``
B. ``Es wurden keine schwere Verletzungen gemeldet, jedoch mussten mindestens fünf der zur Zeit der Explosion Anwesenden aufgrund von Schocksymptomen behandelt werden.``
C. ``Finnland ist ein großartiges Reiseziel für Bootstouren. Das „Land der tausend Seen“ hat auch Tausende von Inseln – in den Seen und in den Küstenarchipelen.``
D. ``Es ist auch nicht erforderlich, dass Sie eine lokale Nummer von der Gemeinde erhalten, in der Sie leben. Sie können eine Internetverbindung über Satellit in der Wildnis v on Chicken in Alaska erhalten und eine Nummer auswählen, die vorgibt, dass Sie im sonnigen Arizona sind.``

Your answer letter:
Sentences-To-Topics (S2T)

Which topic best matches the sentences? The sentences belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel". Choose one topic from A, B, C, or D. Output only a single letter!

# Sentences
• ``Maroochydore führte am Ende die Rangfolge an, mit sechs Punkten Vorsprung vor Noosa als Zweitem.``
• ``Die Schlagmänner der mittleren Reihe, Sachin Tendulkar und Rahul Dravid, zeigten gute Leistungen und erzielten eine Partnerschaft mit 100 Runs.
• ``Da pro Tag nur achtzehn Medaillen zur Verfügung stehen, hat es ein Anzahl an Ländern nicht auf das Podium geschafft.``
• ``Wintersportarten sind in den nördlichen Regionen am beliebtesten und Italiener nehmen an internationalen Wettkämpfen und olympischen Spielen teil.``
• ``Nach dem Rennen bleibt Keselowski mit 2.250 Punkten Spitzenreiter in der Fahrerwertung.``

# Topics
A. sports
B. health
C. travel
D. science and technology

Your answer letter:
A.3  Further Details

Models. We test state-of-the-art instruction fine-tuned LVLMs across various sizes. Smaller models are evaluated on all languages, while larger LVLMs (26B+) are tested on subsets of the MVL-SIB languages (cf. §5.3).

GPT-4o. We evaluate MVL-SIB on GPT-4o-mini-2024-07-18 and GPT-4o-2024-08-06. We set the image detail in the API to ‘low’, since our tasks require high-level reasoning that does not depend on finer image details.11 Prior works show that GPT-4o is the best-performing multilingual LVLM (Schneider and Sitaram, 2024; Vayani et al., 2024).

Qwen2-VL. Qwen2-VL ties a 675M parameter vision-transformer (ViT) into Qwen2 LLMs (Wang et al., 2024). An MLP compresses adjacent 2×2 visual tokens embedded by the ViT into one token representation, which is then input to the LLM.

InternVL 2.5. Depending on the model size, InternVL uses Qwen2.5 or InternLM as its LLM backbone (Chen et al., 2024). The model embeds images either by a 6B or by a distilled 300M ViT pretrained with CLIP (Radford et al., 2021). The resulting image patch encodings are downsampled by factor 4 and fed through an MLP to the LLM.

Centurio. Centurio is the latest massively multilingual LVLM trained on 100 languages (Geigle et al., 2025), outperforming alternatives like Parrot (Sun et al., 2024) or Pangea (Yue et al., 2025). It employs Qwen2.5 as its LLM (Yang et al., 2024) and SigLIP S0400/384 as its ViT (Zhai et al., 2023). The model mixes resolutions by stacking the encodings of the full image and those of 2×2 tiles along the features. The combined embedding is then projected via an MLP to the LLM’s input space.

Besides architectures, the LVLMs crucially differ in dataset mixtures on which they were trained. Centurio translates image-caption, VQA, OCR, and a few multi-image datasets to 100 languages with NLLB (Team et al., 2022) to mix 50:50 with the original English data. Qwen2-VL and InternVL, however, were trained on much larger, more diverse datasets that comprise sizable multi-image comparison and video understanding datasets. Moreover, assuming that the LLMs of Qwen2-VL, InternVL, and GPT-4o were pretrained on Flores, their performance would be overly optimistic.

11We use GPT-4o-mini because evaluating GPT-4o would be too expensive.
A.3.1 Performance by Model over Languages grouped by Language Tier

![Scatter plot showing performance by model and language tier](page_246_180_1057_393.png)

Figure 6: Images-To-Sentences @ \( k = 3 \). The English prompt describes the cross-modal topic matching task, lists all topics, and provides both \( k = 3 \) reference images and 4 sentences in the corresponding language {eng_Latn, ..., nqo_Nkoo}. LVLMs must select the sentence of 4 options that topically fits \( k = 3 \) reference images. The sentences spanning 205 languages and 7 topics are drawn from SIB-200 (Adelani et al., 2024), while images for the topics were hand-selected (cf. Appendix A.1). An example prompt is shown in Appendix A.7.2; further details are in §4.
Plot. The x-axis orders the languages of the candidate sentences {eng_Latn, ..., nqo_Nkoo}, respectively, by descending performance (y-axis). The top x-axis indicates the running index of each language \( L_i \ (i \in \{1, \ldots, 205\}) \).
Tiers. The languages are grouped by tiers derived from Joshi et al. (2020) (cf. §5).

A.3.2 Calibration Analysis of Cross-Modal Topic Matching

![Line plot showing difference in accuracy vs number of samples](page_246_670_1057_393.png)

Figure 7: Calibration Analysis of Cross-Modal Topic Matching for InternVL 2.5 8B. Analysis: To assess the reliability of cross-modal topic matching with fewer samples than our full dataset (1004 samples per language), we randomly select 500 trajectories. We then compute performance metrics on cumulative subsets, incrementing by 50 examples at each step. The difference in performance between the full dataset and each subset is calculated to quantify the deviation at each sample size. Plot: The plot displays the average absolute spread in performance (averaged over all languages) along with the standard deviation for InternVL 2.5 8B. We restrict ourselves to a single open-weight LVLM, since the analysis yields identical results across all combinations of LVLMs and language tiers.
Insights: Our analysis shows that performance stabilizes rapidly, with deviations of only about 1% observed at 1,004 instances – same size as the subset from which the dataset was created. This indicates that reliable evaluation of cross-modal topic matching can be achieved with far fewer than 3,012 samples.
A.4 Overview of Multilingual Vision-Language Benchmarks

xGQA. The xGQA dataset (Pfeiffer et al., 2022) is a cross-lingual visual question-answering resource. It extends the well-known English-only GQA dataset (Hudson and Manning, 2019) by providing manual translations of the questions in the balanced test-dev set. The dataset contains 9666 questions available in eight languages across five scripts, while the answers remain in English. In addition, it features 300 unique images from Visual Genome (Krishna et al., 2017).

MaXM. MaXM, introduced by Changpinyo et al. (2023), is a VQA dataset covering seven languages written in five scripts. In this dataset, both the questions and their corresponding answers are presented in the same language. The images are drawn from a subset of the XM3600 (Thapliyal et al., 2022) dataset and are selected to correspond to regions where the question-answer pair’s language is spoken, ensuring both linguistic and cultural diversity.

XVNLI. The XVNLI dataset (Bugliarello et al., 2022) introduces the task of Cross-lingual Visual Natural Language Inference, where a model must determine if a textual hypothesis entails, contradicts, or is neutral with respect to a visual premise. This dataset spans five languages across three scripts and includes 357 unique images from Visual Genome. It is built upon a combination of the text-only SNLI (Bowman et al., 2015) dataset and its cross-lingual (Agić and Schluter, 2018) and cross-modal (Xie et al., 2019) counterparts.

MaRVL. The MaRVL dataset (Liu et al., 2021) is designed to benchmark models on Multicultural Reasoning over Vision and Language. Each sample consists of two images, a textual statement, and a binary (true/false) answer grounded in the images. Covering five languages across three scripts, MaRVL includes 4914 culturally diverse images that align with the respective languages. The images in each sample are selected to reflect the culture of the annotator who composed the textual statement in their native language.

XM3600. The XM3600 dataset (Thapliyal et al., 2022) is an extensive multilingual image captioning resource encompassing 36 languages. It contains 261375 captions across 13 scripts, with 100 unique images per language. The images are chosen to reflect the cultural background of the language, ensuring both cultural and linguistic diversity. All captions were manually produced by professional native speakers rather than being automatically generated. Due to the dataset’s large size, we evaluate XM3600 using a randomly selected subset of 500 images per language.

Babel-ImageNet (multiple-choice) (BIN-MC). Babel-ImageNet (Geigle et al., 2024c) translates ImageNet’s (Deng et al., 2009) labels into nearly 300 languages, allowing us to assess whether models can recognize and correctly link diverse ImageNet objects to their labels in the target language. Given the computational cost, we focus on languages that appear in only one or two other datasets, in addition to English and a select few high-resource languages, and we use 10 images per class instead of 50. We follow Geigle et al. (2024b) and frame the task as a multiple-choice problem by mining hard negative options from the complete label pool. This approach avoids the ambiguity inherent in traditional open-ended VQA formats. Negatives are selected based on the English labels, filtering out candidates not translated by Babel-ImageNet into the target language, and ultimately choosing the three most similar negative labels available.

SMPQA. Geigle et al. (2025) introduce SMPQA (Synthetic Multilingual Plot QA) as a test dataset for evaluating multilingual OCR capabilities for bar plots and pie charts, covering 11 languages and various scripts and resource levels.

M5B-VGR. The M5B-VGR dataset, presented by (Schneider and Sitaram, 2024), is a visually grounded reasoning benchmark akin to MaRVL. Each sample comprises two images, a textual statement, and a binary (true/false) answer based on the images. It spans 12 languages across 7 scripts and features culturally diverse photos from regions where the respective languages are spoken. The images are sampled from the Dollar Street (Gaviria Rojas et al., 2022) dataset, with 120 samples provided per language.

M5B-VLOD. The M5B-VLOD (Visio-Linguistic Outlier Detection) dataset, also introduced by (Schneider and Sitaram, 2024), consists of samples containing five images paired with a textual statement that is true for all but one image. The task is to identify the outlier image that does not match the statement. This dataset covers the same 12 languages as M5B-VGR, with images sampled in a similar manner from the same source, and provides
<table>
  <tr>
    <th>Task</th>
    <th>Dataset</th>
    <th>Visual Input</th>
    <th>Textual Input</th>
    <th>Target Output</th>
    <th>Metric</th>
    <th>#Lang.</th>
  </tr>
  <tr>
    <td>Captioning</td>
    <td>XM3600</td>
    <td>Single-Image</td>
    <td>Prompt (English)</td>
    <td>Caption (Target Language)</td>
    <td>CIDEr</td>
    <td>36</td>
  </tr>
  <tr>
    <td>Multiple-Choice Visual Question Answering</td>
    <td>BabellImageNet-MC</td>
    <td>Single-Image</td>
    <td>Question (Target Language)</td>
    <td>Letter of the correct Choice</td>
    <td>Relaxed Accuracy</td>
    <td>20</td>
  </tr>
  <tr>
    <td>Text-Heavy Multiple-Choice Visual Question Answering</td>
    <td>M3Exam<br>MMMU<br>xMMMU</td>
    <td>Single or Multi-Image</td>
    <td>Question (Target Language)<br>Question (English)</td>
    <td>Letter of the correct Choice</td>
    <td>Relaxed Accuracy</td>
    <td>7<br>1<br>7</td>
  </tr>
  <tr>
    <td>Text-Heavy Visual Question Answering</td>
    <td>MTVQA<br>SMPQA - Name</td>
    <td>Single-Image</td>
    <td>Question (Target Language)</td>
    <td>Word or Phrase (Target Language)</td>
    <td>Exact Accuracy</td>
    <td>9<br>11</td>
  </tr>
  <tr>
    <td>Text-Heavy Visually Grounded Reasoning</td>
    <td>SMPQA - Ground</td>
    <td>Single-Image</td>
    <td>Question (Target Language)</td>
    <td>'yes' / 'no'</td>
    <td>Exact Accuracy</td>
    <td>11</td>
  </tr>
  <tr>
    <td>Visio-Linguistic Outlier Detection</td>
    <td>MSB-VLOD</td>
    <td>Multi-Image</td>
    <td>Hypothesis (Target Language)</td>
    <td>Letter of the correct Choice</td>
    <td>Relaxed Accuracy</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Visual Natural Language Inference</td>
    <td>XVNL1</td>
    <td>Single-Image</td>
    <td>Hypothesis (Target Language)</td>
    <td>'yes' / 'no' / 'maybe'</td>
    <td>Exact Accuracy</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Visual Question Answering</td>
    <td>MaXM<br>xGQA</td>
    <td>Single-Image</td>
    <td>Question (Target Language)</td>
    <td>Word or Phrase (Target Language)<br>Word or Phrase (English)</td>
    <td>Exact Accuracy</td>
    <td>6<br>8</td>
  </tr>
  <tr>
    <td>Visually Grounded Reasoning</td>
    <td>MSB-VGR<br>MaRVL</td>
    <td>Multi-Image</td>
    <td>Hypothesis (Target Language)</td>
    <td>'yes' / 'no'</td>
    <td>Exact Accuracy</td>
    <td>12<br>6</td>
  </tr>
</table>

Table 3: Summary of multilingual vision-language benchmarks we correlate MVL-SIB against. Relaxed denotes responses that start with the correct option letter (cf. 4).

120 samples per language.

MTVQA. The MTVQA dataset, introduced by (Tang et al., 2024), features text-heavy visual question answering tasks. It includes expert human annotations in 9 diverse languages, comprising 6778 question-answer pairs across 2116 images. The images predominantly contain text in the corresponding language, with questions and answers closely tied to that text. These images are sourced from various publicly available datasets.

CVQA. The CVQA dataset, introduced by (Romero et al., 2025), is a multilingual and culturally nuanced VQA benchmark that includes a broad array of languages, many of which are underrepresented in NLP. It consists of 10000 questions spanning 30 countries and 31 languages, forming 39 distinct country-language pairs (for instance, Spanish appears in 7 different splits corresponding to 7 Spanish-speaking countries). The images were manually collected by human annotators to accurately depict the culture associated with each country-language pair. Each sample includes one image and a question in the respective language. Although the test set is not publicly available, the authors permit up to 5 daily leaderboard submissions for evaluation.

M3Exam. The M3Exam dataset, presented by (Zhang et al., 2023), contains real-world exam questions in 9 languages, available as either text-only or multimodal samples. For our evaluation, we only include samples that require at least one image. Due to the limited number of samples for Swahili and Javanese, we focus on the remaining 7 languages. The selected samples consist of multiple-choice questions in the target language, accompanied by up to 8 images that may appear in both the question and the answer options, with the number of choices ranging from 4 to 8 per sample.

xMMMU. xMMMU, introduced by (Yue et al., 2025), comprises college-level multiple-choice VQA samples in seven languages. It was automatically translated using GPT4o from a randomly selected subset of 300 questions from the MMMU (Yue et al., 2024) validation split.

A.5 Prompts

We list the prompts for each dataset in our test suite used for all models in Figure 8.
SMPQA
<IMG>{QUESTION}\nAnswer the question using a single word or phrase.

CVQA
<IMG>{QUESTION}\nThere are several options:\nA. {OPTION A}\nB. {OPTION B}\nC. {OPTION C}\nD. {OPTION D}\nAnswer with the option’s letter from the given choices directly.

xMMMU
{QUESTION}\nThere are several options:\nA. {OPTION A}\nB. {OPTION B}\nC. {OPTION C}\nD. {OPTION D}\nAnswer with the option’s letter from the given choices directly.

MTVQA
<IMG>{QUESTION}\nAnswer the question using a single word or phrase.\nAnswer in {LANGUAGE}.

M3Exam
{QUESTION}\nOptions:\nA. {OPTION A}\nB. {OPTION B}\nC. {OPTION C}\nD. {OPTION D}\n Answer with the option’s letter from the given choices directly.

BIN-MC
<IMG>Which of these choices (in English) is shown in the image?\n Choices:\nA. {CHOICE A}\nB. {CHOICE B}\nC. {CHOICE C}\nD. {CHOICE D}\n Answer with the letter from the given choices directly.

xGQA
<IMG>{QUESTION}\nAnswer the question using a single word or phrase.\nAnswer in English.

MaXM
<IMG>{QUESTION}\nAnswer the question using a single word or phrase.\nAnswer in {LANGUAGE}.

MaRVL
<IMG>Given the two images <IMG><IMG>, is it correct to say “{HYPOTHESIS}”? Answer yes or no.’

XVNLI
<IMG>Is it guaranteed true that “{HYPOTHESIS}”? Yes, no, or maybe? Answer in English.

M5-VGR
Given the two images <IMG><IMG>, is it correct to say “{HYPOTHESIS}”? Answer yes or no.’

M5-VLOD
Based on the 5 images <IMG><IMG><IMG><IMG><IMG> ordered from top-left to bottom-right, which image does not match the hypothesis “{HYPOTHESIS}”? Choose one from [A, B, C, D, E] and only output a single letter:

XM3600
Briefly describe the image in {LANGUAGE} in one sentence.

Figure 8: Prompts used for the different datasets of our test suite. For M3Exam and xMMMU, the questions contain images at individual positions, and also the options can consist of images. In total, a sample of M3Exam can contain up to 8 images and 8 options, and a sample of xMMMU can contain up to 4 images and 4 options.
A.6 Full Results For Subsets by Task, Model, and Language

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">Tier</th>
    <th colspan="4">InterVL-2L</th>
    <th colspan="8">CentaurQ Queen</th>
    <th colspan="2">GPT-4e</th>
  </tr>
  <tr>
    <th>InterVL-2L-4bit</th>
    <th>InterVL-2L-7B</th>
    <th>InterVL-2L-72B</th>
    <th>InterVL-2L-54B</th>
    <th>InterVL-2L-5.5B</th>
    <th>InterVL-2L-5.2B</th>
    <th>InterVL-2.3SDM</th>
    <th>InterVL-2.57B</th>
    <th>CentaurQ Queen SB</th>
    <th>GPT-4e-mini</th>
    <th>GPT-4e</th>
  </tr>
  <!-- Data is omitted for brevity -->
</table>

Table 4: Subsets of language tiers for T2S @ k=1.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">Tier</th>
    <th colspan="4">InterVL-2L</th>
    <th colspan="8">CentaurQ Queen</th>
    <th colspan="2">GPT-4e</th>
  </tr>
  <tr>
    <th>InterVL-2L-4bit</th>
    <th>InterVL-2L-7B</th>
    <th>InterVL-2L-72B</th>
    <th>InterVL-2L-54B</th>
    <th>InterVL-2L-5.5B</th>
    <th>InterVL-2L-5.2B</th>
    <th>InterVL-2.3SDM</th>
    <th>InterVL-2.57B</th>
    <th>CentaurQ Queen SB</th>
    <th>GPT-4e-mini</th>
    <th>GPT-4e</th>
  </tr>
  <!-- Data is omitted for brevity -->
</table>

Table 5: Subsets of language tiers for T2S @ k=3.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">Tier</th>
    <th colspan="4">InterVL-2L</th>
    <th colspan="8">CentaurQ Queen</th>
    <th colspan="2">GPT-4e</th>
  </tr>
  <tr>
    <th>InterVL-2L-4bit</th>
    <th>InterVL-2L-7B</th>
    <th>InterVL-2L-72B</th>
    <th>InterVL-2L-54B</th>
    <th>InterVL-2L-5.5B</th>
    <th>InterVL-2L-5.2B</th>
    <th>InterVL-2.3SDM</th>
    <th>InterVL-2.57B</th>
    <th>CentaurQ Queen SB</th>
    <th>GPT-4e-mini</th>
    <th>GPT-4e</th>
  </tr>
  <!-- Data is omitted for brevity -->
</table>

Table 6: Subsets of language tiers for T2S @ k=5.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">Tier</th>
    <th colspan="4">InterVL-2L</th>
    <th colspan="8">CentaurQ Queen</th>
    <th colspan="2">GPT-4e</th>
  </tr>
  <tr>
    <th>InterVL-2L-4bit</th>
    <th>InterVL-2L-7B</th>
    <th>InterVL-2L-72B</th>
    <th>InterVL-2L-54B</th>
    <th>InterVL-2L-5.5B</th>
    <th>InterVL-2L-5.2B</th>
    <th>InterVL-2.3SDM</th>
    <th>InterVL-2.57B</th>
    <th>CentaurQ Queen SB</th>
    <th>GPT-4e-mini</th>
    <th>GPT-4e</th>
  </tr>
  <!-- Data is omitted for brevity -->
</table>

Table 7: Subsets of language tiers for T2S @ k=1.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">Tier</th>
    <th colspan="4">InterVL-2L-7B</th>
    <th colspan="8">CentaurQ Queen</th>
    <th colspan="2">GPT-4e</th>
  </tr>
  <tr>
    <th>InterVL-2L-7B</th>
    <th>InterVL-2L-72B</th>
    <th>InterVL-2L-54B</th>
    <th>InterVL-2L-5.5B</th>
    <th>InterVL-2L-5.2B</th>
    <th>InterVL-2.3SDM</th>
    <th>InterVL-2.57B</th>
    <th>CentaurQ Queen SB</th>
    <th>GPT-4e-mini</th>
    <th>GPT-4e</th>
  </tr>
  <!-- Data is omitted for brevity -->
</table>

Table 8: Subsets of language tiers for s2I @ k=1.
<table>
  <tr>
    <th colspan="2">Queen2-VL</th>
    <th colspan="3">InterV1.2.5B</th>
    <th colspan="3">Centaurio Queen</th>
    <th rowspan="2">GPT-4-noni</th>
    <th rowspan="2">GPT-4c</th>
  </tr>
  <tr>
    <th>Lang</th>
    <th>Tier</th>
    <th>Queen2-VL 2B</th>
    <th>Queen2-VL 7B</th>
    <th>Queen2-VL 72B</th>
    <th>InterV1.2.5 2B</th>
    <th>InterV1.2.5 2 BH</th>
    <th>InterV1.2.5 2SSB</th>
  </tr>
  <tr><td>aeb_Arab</td><td>1</td><td>42.6</td><td>89.5</td><td>89.7</td><td>83.9</td><td>91.0</td><td>94.7</td><td>96.5</td><td>84.2</td><td>88.5</td></tr>
  <tr><td>arb_Arab</td><td>4</td><td>44.7</td><td>86.0</td><td>90.4</td><td>81.9</td><td>91.8</td><td>92.1</td><td>96.8</td><td>86.9</td><td>89.1</td></tr>
  <tr><td>ben_Beng</td><td>3</td><td>44.6</td><td>78.6</td><td>85.3</td><td>88.2</td><td>89.8</td><td>91.0</td><td>93.9</td><td>86.7</td><td>88.9</td></tr>
  <tr><td>fra_Latin</td><td>1</td><td>35.0</td><td>55.2</td><td>87.5</td><td>54.9</td><td>90.6</td><td>89.5</td><td>86.1</td><td>84.3</td><td>89.1</td></tr>
  <tr><td>kac_Latin</td><td>1</td><td>41.1</td><td>48.0</td><td>69.9</td><td>49.7</td><td>74.5</td><td>89.8</td><td>94.2</td><td>88.0</td><td>92.1</td></tr>
  <tr><td>kac_Mya</td><td>5</td><td>32.5</td><td>70.1</td><td>81.2</td><td>38.4</td><td>86.1</td><td>90.0</td><td>94.1</td><td>81.7</td><td>87.8</td></tr>
  <tr><td>lit_Latin</td><td>3</td><td>42.1</td><td>45.4</td><td>89.1</td><td>39.2</td><td>88.4</td><td>79.5</td><td>82.1</td><td>81.4</td><td>87.5</td></tr>
  <tr><td>alt_Mya</td><td>1</td><td>51.7</td><td>46.2</td><td>74.6</td><td>33.8</td><td>61.4</td><td>69.5</td><td>79.1</td><td>70.4</td><td>79.5</td></tr>
  <tr><td>nal_Mya</td><td>4</td><td>33.7</td><td>65.3</td><td>88.0</td><td>58.1</td><td>90.9</td><td>97.9</td><td>93.8</td><td>52.6</td><td>99.9</td></tr>
  <tr><td>srp_Cyr1</td><td>2</td><td>44.3</td><td>64.6</td><td>88.7</td><td>59.7</td><td>97.9</td><td>93.8</td><td>87.0</td><td>88.1</td><td>89.9</td></tr>
  <tr><td>wol_Latin</td><td>1</td><td>54.8</td><td>45.8</td><td>75.2</td><td>32.9</td><td>49.7</td><td>67.4</td><td>77.0</td><td>54.3</td><td>89.5</td></tr>
</table>

Table 9: Subsets of language tiers for S21 @ k=3.

<table>
  <tr>
    <th colspan="2">Queen2-VL</th>
    <th colspan="3">InterV1.2.5B</th>
    <th colspan="3">Centaurio Queen</th>
    <th rowspan="2">GPT-4-noni</th>
    <th rowspan="2">GPT-4c</th>
  </tr>
  <tr>
    <th>Lang</th>
    <th>Tier</th>
    <th>Queen2-VL 2B</th>
    <th>Queen2-VL 7B</th>
    <th>Queen2-VL 72B</th>
    <th>InterV1.2.5 2B</th>
    <th>InterV1.2.5 2 BH</th>
    <th>InterV1.2.5 2SSB</th>
  </tr>
  <tr><td>aeb_Arab</td><td>3</td><td>44.0</td><td>85.6</td><td>91.6</td><td>50.9</td><td>57.6</td><td>77.0</td><td>93.5</td><td>91.8</td><td>84.0</td></tr>
  <tr><td>arb_Arab</td><td>4</td><td>43.5</td><td>65.7</td><td>90.9</td><td>40.0</td><td>59.9</td><td>79.4</td><td>94.0</td><td>91.9</td><td>84.0</td></tr>
  <tr><td>ben_Beng</td><td>4</td><td>40.1</td><td>67.3</td><td>92.1</td><td>45.6</td><td>79.4</td><td>94.0</td><td>93.1</td><td>88.1</td><td>89.1</td></tr>
  <tr><td>eng_Latin</td><td>4</td><td>43.6</td><td>68.6</td><td>90.6</td><td>43.0</td><td>68.7</td><td>93.7</td><td>91.7</td><td>85.6</td><td>88.9</td></tr>
  <tr><td>kac_Latin</td><td>3</td><td>31.8</td><td>42.3</td><td>74.8</td><td>31.6</td><td>46.2</td><td>70.6</td><td>79.2</td><td>93.7</td><td>83.0</td></tr>
  <tr><td>kac_Mya</td><td>2</td><td>35.1</td><td>48.8</td><td>84.7</td><td>40.2</td><td>50.0</td><td>64.7</td><td>81.3</td><td>82.7</td><td>85.2</td></tr>
  <tr><td>lit_Latin</td><td>4</td><td>43.1</td><td>25.5</td><td>92.1</td><td>39.5</td><td>97.2</td><td>98.1</td><td>98.0</td><td>5.4</td><td>83.2</td></tr>
  <tr><td>lua_Latin</td><td>3</td><td>34.6</td><td>24.7</td><td>80.2</td><td>34.6</td><td>49.7</td><td>63.1</td><td>85.1</td><td>82.8</td><td>79.9</td></tr>
  <tr><td>nal_Mya</td><td>5</td><td>37.6</td><td>42.8</td><td>84.3</td><td>39.1</td><td>81.7</td><td>90.8</td><td>95.6</td><td>82.1</td><td>88.0</td></tr>
  <tr><td>srp_Cyr1</td><td>4</td><td>44.5</td><td>63.5</td><td>90.7</td><td>41.2</td><td>58.4</td><td>79.4</td><td>93.6</td><td>91.4</td><td>89.1</td></tr>
  <tr><td>uoc_Latin</td><td>4</td><td>49.0</td><td>63.0</td><td>93.0</td><td>49.8</td><td>59.9</td><td>81.4</td><td>93.3</td><td>91.3</td><td>88.5</td></tr>
  <tr><td>wol_Latin</td><td>1</td><td>54.4</td><td>5.8</td><td>82.1</td><td>34.8</td><td>50.8</td><td>64.4</td><td>66.1</td><td>27.7</td><td>96.0</td></tr>
</table>

Table 10: Subsets of language tiers for S21 @ k=5.

<table>
  <tr>
    <th colspan="2">Queen2-VL</th>
    <th colspan="3">InterV1.2.5B</th>
    <th colspan="3">Centaurio Queen</th>
    <th rowspan="2">GPT-4-noni</th>
    <th rowspan="2">GPT-4c</th>
  </tr>
  <tr>
    <th>Lang</th>
    <th>Tier</th>
    <th>Queen2-VL 2B</th>
    <th>Queen2-VL 7B</th>
    <th>Queen2-VL 72B</th>
    <th>InterV1.2.5 2B</th>
    <th>InterV1.2.5 2 BH</th>
    <th>InterV1.2.5 2SSB</th>
  </tr>
  <tr><td>aeb_Arab</td><td>3</td><td>77.9</td><td>85.2</td><td>91.5</td><td>87.5</td><td>83.5</td><td>83.9</td><td>91.5</td><td>91.9</td><td>91.5</td></tr>
  <tr><td>arb_Arab</td><td>4</td><td>43.5</td><td>46.4</td><td>87.7</td><td>38.9</td><td>63.0</td><td>86.2</td><td>89.5</td><td>89.2</td><td>85.1</td></tr>
  <tr><td>ben_Beng</td><td>4</td><td>68.8</td><td>61.0</td><td>91.2</td><td>83.7</td><td>79.5</td><td>91.6</td><td>90.8</td><td>67.0</td><td>91.1</td></tr>
  <tr><td>eng_Latin</td><td>5</td><td>80.8</td><td>79.9</td><td>91.2</td><td>80.5</td><td>91.6</td><td>92.5</td><td>94.2</td><td>91.8</td><td>91.8</td></tr>
  <tr><td>fra_Latin</td><td>2</td><td>52.1</td><td>72.1</td><td>86.3</td><td>71.6</td><td>77.3</td><td>76.9</td><td>86.5</td><td>83.9</td><td>83.9</td></tr>
  <tr><td>kac_Latin</td><td>4</td><td>38.0</td><td>27.4</td><td>83.3</td><td>35.7</td><td>64.6</td><td>94.4</td><td>98.0</td><td>54.7</td><td>82.2</td></tr>
  <tr><td>kac_Mya</td><td>5</td><td>76.2</td><td>81.7</td><td>90.1</td><td>76.6</td><td>76.8</td><td>86.4</td><td>84.5</td><td>81.9</td><td>96.5</td></tr>
  <tr><td>lit_Latin</td><td>1</td><td>60.9</td><td>82.6</td><td>85.8</td><td>63.6</td><td>61.9</td><td>95.0</td><td>87.7</td><td>80.3</td><td>97.0</td></tr>
  <tr><td>luk_Latin</td><td>3</td><td>49.0</td><td>83.6</td><td>87.9</td><td>76.9</td><td>69.1</td><td>74.3</td><td>87.8</td><td>87.0</td><td>91.1</td></tr>
  <tr><td>nal_Mya</td><td>4</td><td>77.5</td><td>86.1</td><td>91.0</td><td>84.8</td><td>85.6</td><td>91.5</td><td>92.5</td><td>84.2</td><td>96.6</td></tr>
  <tr><td>snp_Cyr1</td><td>4</td><td>67.4</td><td>78.6</td><td>93.0</td><td>69.8</td><td>82.1</td><td>83.0</td><td>86.5</td><td>85.3</td><td>92.2</td></tr>
  <tr><td>wol_Latin</td><td>1</td><td>54.6</td><td>89.1</td><td>93.1</td><td>57.1</td><td>82.6</td><td>84.5</td><td>87.3</td><td>91.1</td><td>98.1</td></tr>
</table>

Table 11: Subsets of language tiers for S2T @ k=1.

<table>
  <tr>
    <th colspan="2">Queen2-VL</th>
    <th colspan="3">InterV1.2.5B</th>
    <th colspan="3">Centaurio Queen</th>
    <th rowspan="2">GPT-4-noni</th>
    <th rowspan="2">GPT-4c</th>
  </tr>
  <tr>
    <th>Lang</th>
    <th>Tier</th>
    <th>Queen2-VL 2B</th>
    <th>Queen2-VL 7B</th>
    <th>Queen2-VL 72B</th>
    <th>InterV1.2.5 2B</th>
    <th>InterV1.2.5 2 BH</th>
    <th>InterV1.2.5 2SSB</th>
  </tr>
  <tr><td>aeb_Arab</td><td>5</td><td>94.1</td><td>93.6</td><td>98.7</td><td>96.7</td><td>96.3</td><td>98.8</td><td>99.0</td><td>96.6</td><td>98.4</td></tr>
  <tr><td>arb_Arab</td><td>4</td><td>95.7</td><td>94.4</td><td>97.9</td><td>96.7</td><td>98.0</td><td>98.7</td><td>99.1</td><td>95.7</td><td>98.6</td></tr>
  <tr><td>ene_Arab</td><td>3</td><td>86.8</td><td>89.9</td><td>93.7</td><td>90.2</td><td>91.8</td><td>94.7</td><td>94.4</td><td>86.4</td><td>93.2</td></tr>
  <tr><td>eng_Latin</td><td>6</td><td>96.5</td><td>89.4</td><td>98.3</td><td>98.0</td><td>99.1</td><td>98.7</td><td>96.7</td><td>98.1</td><td>98.0</td></tr>
  <tr><td>fra_Latin</td><td>2</td><td>76.9</td><td>88.8</td><td>94.1</td><td>78.7</td><td>72.5</td><td>72.7</td><td>92.2</td><td>73.6</td><td>78.6</td></tr>
  <tr><td>kac_Latin</td><td>5</td><td>66.7</td><td>72.6</td><td>86.2</td><td>87.0</td><td>85.6</td><td>87.3</td><td>95.4</td><td>88.9</td><td>9y.9</td></tr>
  <tr><td>lit_Latin</td><td>2</td><td>60.7</td><td>92.9</td><td>93.6</td><td>61.3</td><td>80.2</td><td>81.2</td><td>86.5</td><td>81.4</td><td>87.4</td></tr>
  <tr><td>luk_Wel</td><td>4</td><td>92.5</td><td>94.0</td><td>98.8</td><td>92.6</td><td>96.5</td><td>97.6</td><td>95.7</td><td>99.0</td><td>98.5</td></tr>
  <tr><td>naa_Mya</td><td>5</td><td>67.0</td><td>76.9</td><td>92.2</td><td>68.0</td><td>84.4</td><td>92.3</td><td>90.2</td><td>95.6</td><td>88.4</td></tr>
  <tr><td>nal_Mya</td><td>4</td><td>72.7</td><td>84.4</td><td>97.9</td><td>66.1</td><td>74.0</td><td>77.6</td><td>89.9</td><td>89.6</td><td>91.8</td></tr>
  <tr><td>snp_Cyr1</td><td>4</td><td>76.8</td><td>89.7</td><td>94.9</td><td>76.8</td><td>92.9</td><td>95.7</td><td>95.9</td><td>91.1</td><td>99.2</td></tr>
  <tr><td>wol_Latin</td><td>1</td><td>66.5</td><td>77.1</td><td>88.9</td><td>64.5</td><td>84.9</td><td>86.5</td><td>88.5</td><td>89.0</td><td>89.3</td></tr>
</table>

Table 12: Subsets of language tiers for S2T @ k=3.

<table>
  <tr>
    <th colspan="2">Queen2-VL</th>
    <th colspan="3">InterV1.2.5B</th>
    <th colspan="3">Centaurio Queen</th>
    <th rowspan="2">GPT-4-noni</th>
    <th rowspan="2">GPT-4c</th>
  </tr>
  <tr>
    <th>Lang</th>
    <th>Tier</th>
    <th>Queen2-VL 2B</th>
    <th>Queen2-VL 7B</th>
    <th>Queen2-VL 72B</th>
    <th>InterV1.2.5 2B</th>
    <th>InterV1.2.5 2 BH</th>
    <th>InterV1.2.5 2SSB</th>
  </tr>
  <tr><td>aeb_Arab</td><td>3</td><td>97.2</td><td>97.4</td><td>99.7</td><td>86.7</td><td>92.8</td><td>92.7</td><td>97.2</td><td>95.3</td><td>99.5</td></tr>
  <tr><td>arb_Arab</td><td>4</td><td>97.8</td><td>96.8</td><td>99.6</td><td>67.4</td><td>76.3</td><td>89.4</td><td>77.4</td><td>91.1</td><td>N/A</td></tr>
  <tr><td>ere_Arab</td><td>3</td><td>92.6</td><td>96.9</td><td>99.5</td><td>86.0</td><td>92.9</td><td>92.7</td><td>97.2</td><td>95.3</td><td>99.5</td></tr>
  <tr><td>frag_Latin</td><td>1</td><td>79.9</td><td>91.8</td><td>94.4</td><td>92.7</td><td>96.3</td><td>99.5</td><td>96.9</td><td>99.0</td><td>99.4</td></tr>
  <tr><td>kac_Latin</td><td>1</td><td>68.1</td><td>75.0</td><td>87.4</td><td>81.5</td><td>86.0</td><td>87.1</td><td>80.9</td><td>86.3</td><td>N/A</td></tr>
  <tr><td>lui_Latin</td><td>4</td><td>77.4</td><td>58.9</td><td>92.9</td><td>90.3</td><td>87.5</td><td>98.3</td><td>96.9</td><td>94.1</td><td>N/A</td></tr>
  <tr><td>nal_Mya</td><td>4</td><td>67.6</td><td>96.6</td><td>97.0</td><td>69.5</td><td>93.2</td><td>94.7</td><td>97.2</td><td>99.2</td><td>N/A</td></tr>
  <tr><td>snp_Cyr1</td><td>4</td><td>97.2</td><td>97.4</td><td>94.9</td><td>87.0</td><td>92.9</td><td>95.7</td><td>95.9</td><td>91.1</td><td>99.2</td></tr>
  <tr><td>uoc_Latin</td><td>4</td><td>69.6</td><td>92.9</td><td>99.5</td><td>69.6</td><td>76.6</td><td>89.6</td><td>92.0</td><td>97.4</td><td>96.8</td></tr>
  <tr><td>wol_Latin</td><td>1</td><td>77.5</td><td>84.3</td><td>94.5</td><td>90.4</td><td>90.2</td><td>92.9</td><td>95.2</td><td>95.7</td><td>99.3</td></tr>
</table>

Table 13: Subsets of language tiers for s2T @ k=5.
A.7 Full Performance by Task, Model, and Language

A.7.1 Images-To-Topics

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Topics</th>
  </tr>
  <tr>
    <th>Entertainment</th>
    <th>Geography</th>
    <th>Health</th>
    <th>Politics</th>
    <th>Science & Tech.</th>
    <th>Sports</th>
    <th>Travel</th>
  </tr>
  <tr>
    <td>QwenVL-2.5-2B</td>
    <td>99.4</td>
    <td>83.1</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>95.7</td>
  </tr>
  <tr>
    <td>QwenVL-2.5-7B</td>
    <td>100.0</td>
    <td>92.9</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>InternVL-2.5-4B</td>
    <td>99.5</td>
    <td>84.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>InternVL-2.5-8B</td>
    <td>100.0</td>
    <td>90.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>97.5</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>Centurio-Qwen</td>
    <td>99.6</td>
    <td>90.6</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.2</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
</table>

Table 14: Image-To-Topics. For a reference image, the model must pick the correct topic out of 4 choices.

A.7.2 Images-To-Sentences
A.7.3 Topic-To-Sentence
<table>
  <tr>
    <th>Lang</th>
    <th>QuartetV2-5,0B</th>
    <th>QuartetV1-7,8B</th>
    <th>BloomV5-7,5B</th>
    <th>InstructV1-9,9B</th>
    <th>Contana-Open</th>
    <th>4me-my</th>
    <th></th>
    <th>Lang</th>
    <th>QuartetV2-5,0B</th>
    <th>QuartetV1-7,8B</th>
    <th>BloomV5-7,5B</th>
    <th>InstructV1-9,9B</th>
    <th>Contana-Open</th>
    <th>4me-my</th>
    <th></th>
    <th>Lang</th>
    <th>QuartetV2-5,0B</th>
    <th>QuartetV1-7,8B</th>
    <th>BloomV5-7,5B</th>
    <th>InstructV1-9,9B</th>
    <th>Contana-Open</th>
    <th>4me-my</th>
    <th></th>
    <th>Lang</th>
    <th>QuartetV2-5,0B</th>
    <th>QuartetV1-7,8B</th>
    <th>BloomV5-7,5B</th>
    <th>InstructV1-9,9B</th>
    <th>Contana-Open</th>
    <th>4me-my</th>
    <th></th>
    <th>Lang</th>
    <th>QuartetV2-5,0B</th>
    <th>QuartetV1-7,8B</th>
    <th>BloomV5-7,5B</th>
    <th>InstructV1-9,9B</th>
    <th>Contana-Open</th>
    <th>4me-my</th>
    <th></th>
  </tr>
  <tr>
    <td>kaz_latin</td><td>43.76 41.41 54.71 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>43.05 38.41 55.66 81.29 59.92 48.57 54.53 28.15 28.64 66.61 80.99 84.99</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>59.94 57.66 85.25 71.52 72.28 51.30 64.12 42.59 28.84 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>41.73 41.07 54.09 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>51.43 48.27 55.16 68.13 60.12 54.81 54.48 37.96 38.59 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>43.76 41.41 54.71 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>51.43 48.27 55.16 68.13 60.12 54.81 54.48 37.96 38.59 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>43.76 41.41 54.71 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>51.43 48.27 55.16 68.13 60.12 54.81 54.48 37.96 38.59 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>43.76 41.41 54.71 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>51.43 48.27 55.16 68.13 60.12 54.81 54.48 37.96 38.59 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>43.76 41.41 54.71 70.44 60.42 41.40 0.62 56.17 35.16 50.16 77.55 50.64 50.17</td><td>51.43 48.27 55.16 68.13 60.12 54.81 54.48 37.96 38.59 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>58.97 57.88 85.25 64.83 67.45 48.47 56.28 40.53 28.84 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td><td>51.37 41.28 64.75 68.54 70.67 51.32 48.62 48.62 31.90 47.56 66.58 80.99 84.99</td><td>68.47 67.10 78.20 56.77 62.15 48.16 55.18 38.21 28.15 56.03 71.37 76.23 76.75</td>
  </tr>
</table>
A.7.5 Sentences-To-Topics

<table>
  <tr>
    <th>Lg</th>
    <th>InnoV2-2.5-BE</th>
    <th>InnoV1.5-0-DB</th>
    <th>InnoV1-2.5-DB</th>
    <th>QuoraV2-2.5-DB</th>
    <th>Community-QNLI</th>
    <th>4c-qnli</th>
    <th>Lg</th>
    <th>InnoV2-2.5-BE</th>
    <th>InnoV1.5-0-DB</th>
    <th>InnoV1-2.5-DB</th>
    <th>QuoraV2-2.5-DB</th>
    <th>Community-QNLI</th>
    <th>4c-qnli</th>
  </tr>
  <tr>
    <td><a href="https://huggingface.co/camem/4c-base-geen">4c</a></td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td><a href="https://huggingface.co/camem/4c-base-geen">4c</a></td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
    <td>0.93 | 0.94</td>
  </tr>
  <!-- Table continues for all the other languages and datasets -->
</table>

The table provides sentences-to-topics results for various languages and datasets, showing precision and recall values (in format: precision | recall) for different models.