TowerVision : Understanding and Improving Multilinguality in Vision-Language Models

## Abstract

Abstract Despite significant advances in vision-language models (VLMs), most existing work follows an English-centric design process, limiting their effectiveness in multilingual settings. In this work, we
provide a comprehensive empirical study analyzing the impact of several multilingual design choices, such as training data composition, encoder selection, and text backbones. The result is TowerVision , a family of open multilingual VLMs for both image-text and video-text tasks, built upon the multilingual text-only model Tower+ . TowerVision achieves competitive performance on multiple multimodal multilingual benchmarks and shows particular strength in culturally grounded tasks and multimodal translation. By incorporating visual and cultural context during fine-tuning, our models surpass existing approaches trained on substantially larger datasets, as demonstrated on ALM-Bench and Multi30K (image tasks) and ViMUL-Bench (video tasks).
Alongside the models, we release VisionBlocks , a high-quality, curated vision-language dataset.
Our findings highlight that multilingual vision-language training data substantially improves cross-lingual generalization—both from high-resource to underrepresented languages and vice versa—and that instruction-tuned LLMs are not always the optimal initialization point.
To support further research, we publicly release all models, data, and training recipes.

## 1 Introduction

The success and widespread adoption of large language models (LLMs) has naturally led to a surge of interest in adding multimodal capabilities to these models.
In particular, the visual modality has recently received considerable attention, with recent releases of frontier vision-language models (VLMs) (deitke2024molmopixmoopenweights; openai2024gpt4ocard; comanici2025gemini; team2025gemma; bai2025qwen25vl).
However, despite impressive progress, the development of VLMs has been mostly built upon English-centric language models, and trained with English vision-text data, giving little consideration to performance in most other languages. A key challenge in multilingualization of VLMs stems from an asymmetric data landscape—while high-quality text-only multilingual corpora are relatively abundant, high-quality multilingual vision-text data is scarce. As such, a critical challenge remains: What are the best strategies to effectively extend these models to support multiple languages beyond English?

An effective strategy for VLM multilingualization is to leverage large-scale text-only multilingual data to strengthen the language understanding of the text backbone, while complementing it with multimodal multilingual examples typically obtained through translation or high-quality synthetic generation, thereby reducing reliance on scarce real-world multilingual multimodal data. A recent example of this approach is Pangea (yue2025pangeafullyopenmultilingual), which extends multilinguality exclusively during the VLM fine-tuning stage using a mixture of data that combined multilingual vision-text pairs generated through synthetic data creation and machine translation of English instructions. While this strategy proved effective, it leaves open key questions: At which stages and on which modules should multilingualization be applied? Which design decisions yield the greatest impact? And how can visual grounding further enhance cross-lingual generalization?

Figure: Figure 1: We present TowerVision and TowerVideo, open VLMs with enhanced cultural understanding and translation capabilities over leading open multimodal systems.
Refer to caption: tv_examples.png

In this work, we introduce TowerVision, (^1^11The TowerVision model collection is available [here](https://huggingface.co/collections/utter-project/towervision-689a10be35396972889cadba).) a suite of open-source multilingual VLMs built on top of Tower+ models (rei2025towerp) for 20 languages and dialects.(^2^22English, German, Dutch, Spanish (Latin America), French, Portuguese (Portugal), Portuguese (Brazilian), Ukrainian, Hindi, Chinese (Simplified), Chinese (Traditional), Russian, Czech, Korean, Japanese, Italian, Polish, Romanian, Norwegian (Nynorsk) and Norwegian (Bokmål)) To train TowerVision, we systematically address the challenges outlined above through comprehensive ablation studies, component-level analysis, and cross-lingual evaluation of a multilingualization recipe. Specifically, we investigate how to enhance the multilingual capabilities of VLMs from two axes: first, by performing several ablations to understand and explore the impact of the underlying components (including the alignment projector, vision encoder and text-only LLM); and second, by creating better, more multilingual vision-text datasets and exploring the impact of using this data across different VLM training stages. Overall, compared to strong VLMs of similar size, TowerVision exhibits competitive or superior performance on various multilingual and multimodal benchmarks, as well as cross-lingual transfer capabilities.

In addition to image-based VLMs, we also train a separate multilingual video model, TowerVideo, built on top of TowerVision, thereby extending our analysis to the video modality. TowerVideo achieves competitive performance on ViMUL-Bench (shafique2025culturallydiversemultilingualmultimodalvideo), a culturally-diverse multilingual video benchmark.
Taken together, these contributions provide a comprehensive and systematic study of how to best integrate multilinguality into VLMs across modalities, architectural components, and training stages. Complementing the TowerVision family, we also release VisionBlocks (^3^33Released on Hugging Face upon acceptance.), a curated dataset that consolidates and filters existing vision/video-language resources, further enriched with quality-controlled translations of English textual descriptions into 20 languages and dialects.

## 2 TowerVision

Our approach follows a multi-stage process encompassing three key components, illustrated in Figure [1](https://arxiv.org/html/2510.21849v3#S1.F1): (i) a multilingual text-only backbone model, Tower+ rei2025towerp; (ii) a Vision Transformer encoder (ViT; dosovitskiy2021an) that processes visual inputs and extracts meaningful features; (iii) a connector/adapter module that transforms these visual features to generate representations compatible with the text embedding space.
These modules can be selectively trained or kept frozen during different stages of development (li2025surveystateartlarge).
Although this training recipe and variations thereof are well-established and have produced several high-quality models (e.g., LLaVA (liu2023visualinstructiontuning), Intern-VL (chen2024internvl), NVLM (dai2024nvlmopenfrontierclassmultimodal), Qwen2.5-VL (bai2025qwen25vl), Molmo (deitke2024molmopixmoopenweights)), most of these fall short in capturing multilingual and culturally diverse nuances. We therefore introduce our multilingual adaptation, TowerVision—we first describe our carefully curated multilingual vision-text data, VisionBlocks (§[2.1](https://arxiv.org/html/2510.21849v3#S2.SS1)) and then describe the overall architecture and training procedure (§[2.2](https://arxiv.org/html/2510.21849v3#S2.SS2)).

### 2.1 VisionBlocks : Towards Better Multilingual Vision-Text Data

Creating a large-scale, high-quality, multilingual multimodal dataset for training visual language models to be helpful assistants is non-trivial for a series of intertwined reasons:

- •
Human-written vision-text data featuring user-model interactions (common in text-only alignment) is severely limited.
While abundant data exists from large-scale captioning datasets (e.g., LAION-5B; schuhmann2022laion5bopenlargescaledataset), such sources over prioritize scale over quality which is not ideal for training VLMs with advanced capabilities (dong2025scalablevisionlanguagemodel; zhou2023limaalignment) like instruction-following, helpfulness, and safety.
- •
High-quality multilingual vision-text data is scarce; furthermore, the lack of open, high-quality multilingual VLMs makes controlled synthetic data challenging or restricted to closed models with limited usage licenses. The most viable alternative, also employed by Pangea (yue2025pangeafullyopenmultilingual), involves translating English vision-text interactions into target languages.
- •
Filtering techniques such as reward model scoring or LLM-as-judge approaches (gu2025surveyllmasajudge) are significantly more challenging to implement for vision-text data, where even state-of-the-art VLMs (both open and proprietary) struggle to provide reliable preferences (li2024vlrewardbenchchallengingbenchmarkvisionlanguage).

With this in mind, we develop and release VisionBlocks (Figure [2](https://arxiv.org/html/2510.21849v3#S2.F2)), which aggregates and filters data from multiple sources, enhanced with new translated and synthetic data, as described below.

##### Collection of existing VLM data

For English vision-text data, we use the mixture created in PixMo (deitke2024molmopixmoopenweights) with a few minor changes: we exclude the AndroidControl, Points, and PointQA datasets, as they do not provide additional multilingual value at this stage; For multilingual vision-text data, we leverage a subset of “Open-Ended” and “Multiple-Choice” questions from CulturalGround (nyandwi2025groundingmultilingualmultimodalllms) and the “Cultural” split of PangeaIns (yue2025pangeafullyopenmultilingual) for our languages of interest.
The samples from PangeaIns are originally found in LAIONMulti (schuhmann2022laion5bopenlargescaledataset) that undergoes a series of automatic steps (using Gemini 1.5 Pro (geminiteam2024gemini15unlockingmultimodal)) including curating high-quality English instructions, carefully translating them to multiple languages, and adapting them for culturally-relevant multilingual contexts. CulturalGround uses a data curation pipeline that gathers culturally relevant entities from the Wikidata knowledge base, creates several questions and answers about each entity, rephrases them using an LLM, and filters low-quality samples using a VLM. In our work, we rely exclusively on CulturalGround’s filtered subsets to ensure maximum quality.

##### Translated and synthetic generated vision-language data

In addition to the original English and multilingual captions, we translate the highly curated PixMo-Cap caption data deitke2024molmopixmoopenweights to our target languages using a Tower model (alves2024tower). These translations are scored using CometKiwi (rei-etal-2022-cometkiwi) and filtered with a high threshold of 0.85 to ensure maximum quality. To further enhance diversity, we pair the remaining high-quality translations with a variety of language-specific captioning prompt templates (§[A.5.1](https://arxiv.org/html/2510.21849v3#A1.SS5.SSS1)).
We also augment the dataset with synthetic captions generated by the Gemini 2.5 API. For each image, we sample multiple system prompts to elicit diverse and detailed descriptions (see §[A.5.2](https://arxiv.org/html/2510.21849v3#A1.SS5.SSS2)). This augmentation is intended to improve coverage of fine-grained visual details (e.g., spatial relations, attributes, and contextual cues) that human-authored captions often omit, and provides instruction-like supervision, aligning our model more closely with recent VLM training paradigms that leverage synthetic data to boost generalization and response quality. Similar strategies have been shown to be effective in scaling up instruction-following capabilities of VLMs such as LLaVA (liu2023llava) and InstructBLIP (dai2023instructblipgeneralpurposevisionlanguagemodels).

##### Text-only data

To retain text-only performance of the backbone LLM, while acquiring strong multimodal capabilities through multimodal training requires a careful balance between text and multimodal data. Determining the optimal data mixture is non-trivial and typically involces a bunch of extensive ablation studies. For instance Pangea (yue2025pangeafullyopenmultilingual) include approximately 10% text-only data in their multimodal SFT mixture to retain text performance. In our work, we found that using around 20% text-only data provided the best trade-off between text-only and multimodal benchmark performance. We used EuroBlocks as the extended text-only data, a curated multilingual collection of high-quality synthetic data from the EuroLLM (martins2025eurollm) synthetic post-training data. EuroBlocks provides diverse, instruction-aligned text that enriches our dataset with robust multilingual coverage and fine-grained, high-quality descriptions.

##### Translated multilingual video data

As video-text data, we employ the LLaVA-Video-178k dataset (zhang2025llavavideovideoinstructiontuning), which contains captions alongside open-ended and multiple-choice questions in English. To make the dataset multilingual, we retain a randomly sampled half of the conversations in English, and we translate the remaining half uniformly into all supported languages using Tower+9B (rei2025towerp), thereby ensuring balanced cross-lingual coverage.

Figure: Figure 2: Overview of the VisionBlocks dataset. See Table [8](https://arxiv.org/html/2510.21849v3#A1.T8), §[A.1](https://arxiv.org/html/2510.21849v3#A1.SS1) for details.
Refer to caption: data_dist2.png

### 2.2 TowerVision : Architecture & Training Details

One way to improve the multilinguality of LLMs (e.g., improving cross-lingual understanding or extending multilingual support for other languages) is to start from a strong pretrained model and continue pretraining on carefully curated data, with subsequent post-training (xu2024paradigm; xu2025xalma; alves2024tower).
TowerVision follows a similar principle, starting from a strong multilingual Gemma-based backbone Tower+ 2B/9B (rei2025towerp), which achieves strong multilingual general-purpose performance by leveraging a curated high-quality multilingual dataset and a training recipe designed to preserve general capabilities. As shown in §[4](https://arxiv.org/html/2510.21849v3#S4.T4), starting from this multilingual backbone substantially improves cross-lingual performance compared to starting from Gemma.

For the vision encoder, TowerVision is initialized with the recently proposed SigLIP2-so400m/14@384px (tschannen2025siglip2multilingualvisionlanguage), a vision transformer operating at $384\times 384$ resolution that extracts image patch representations and produces multilingually-aligned embeddings of size $729$. SigLIP2 is trained on a more diverse data mixture compared to alternatives such as CLIP-ViT (radford2021learningtransferablevisualmodels), Perception Encoder (bolya2025perceptionencoderbestvisual), or SigLIP1 (zhai2023sigmoidlosslanguageimage), and thereby yields better multilingual understanding, as we shall see in §[4](https://arxiv.org/html/2510.21849v3#S4). To align the vision and text modalities, we use a LLaVA-based architecture (liu2023visualinstructiontuning), where we train a projection layer consisting of a 2-layer MLP, randomly initialized.
By combining Tower+ for text and SigLIP2 for vision, TowerVision benefits from complementary multilingual strengths across both modalities. The training process consists of three stages:

- 1.
A projector pretraining phase, where we train the model to predict captions given images on the PixMo-Cap dataset, freezing both the vision encoder and the language model backbone (so only the projector is trained). Each image is encoded once (downscaled to 384$\times$384 if necessary). During this phase, we focus exclusively on diverse, high-quality English captions, which we show to be more effective for aligning visual and textual representations (see §[4](https://arxiv.org/html/2510.21849v3#S4)).
- 2.
A vision finetuning phase, where we unfreeze the full model and train it on the full VisionBlocks dataset (§[2.1](https://arxiv.org/html/2510.21849v3#S2.SS1)), excluding the video-text data. In this phase, we use high-dynamic resolution (liu2024llavanext), breaking high-resolution images into a grid of smaller tiles which are then encoded with the vision encoder independently (together with a global thumbnail tile). The projected embeddings are then concatenated. We use a maximum of six tiles, which provides the best trade-off (§[A.3](https://arxiv.org/html/2510.21849v3#A1.SS3)).
This phase leads to the TowerVision model.
- 3.
A video finetuning phase, where the video portion of VisionBlocks is used to finetune TowerVision on 32-frame video inputs at the encoder’s fixed resolution of 384$\times$384. Unlike the previous stage, we omit tiling for efficiency.
This phase leads to the TowerVideo model.

The models were trained on a custom fork of the LLaVA-Next (liu2024llavanext) codebase.(^4^44https://github.com/deep-spin/LLaVA-NeXT)

## 3 Evaluation & Main Results

We evaluate TowerVision and TowerVideo on a comprehensive suite of benchmarks spanning multiple modalities and task types (single-image, multi-images, and video) across diverse languages, both within and beyond our training set. In this section, we focus on vision-language tasks, which includes multilingual visual/video question answering, cultural understanding, OCR-related tasks, and visual-language understanding, as well as multilingual video-language tasks. Our assessment relies primarily on closed-form tasks, complemented by large language models serving as judges for video-based evaluations.

### 3.1 Tasks & Evaluation Benchmarks

##### Vision-language tasks

We report results on ALM-Bench (vayani2024alm),
a cultural understanding multilingual(^5^55German, Spanish, French, Italian, Korean, Dutch, Russian, English, Portuguese, Chinese (Simplified and Traditional), Icelandic, Czech, Ukrainian, Hindi, Japanese, Polish, Swedish, Hungarian, Romanian, Danish, Norwegian (Nynorsk), and Finnish.) visual QA benchmark, OCRBench (Liu_2024) and cc-OCR (yang2024ccocrcomprehensivechallengingocr) for English and multilingual(^6^66German, French, Italian, Russian, Spanish, Korean, Portuguese.) OCR-centric capabilities respectively, and TextVQA (singh2019text_vqa), assessing scientific understanding. Within cc-OCR, we report results on the multilingual text reading subset, as our primary focus is to evaluate the model’s multilingual text recognition capabilities.

##### Multimodal translation

We report results on CoMMuTE (futeral-etal-2023-tackling), a specialized multimodal translation benchmark that uses the visual content to resolve lexical ambiguities present in the source language, and Multi30K (elliott-etal-2016-multi30k), a standard benchmark for multimodal machine translation (MT) of image captions.

##### Culturally-aware multilingual video tasks

We use ViMUL-Bench (shafique2025culturallydiversemultilingualmultimodalvideo), a multilingual video QA benchmark spanning 14 languages: Arabic (ar), Bengali (bn), Chinese (zh), English (en), French (fr), German (de), Hindi (hi), Japanese (ja), Russian (ru), Sinhala (si), Spanish (es), Swedish (sv), Tamil (ta), and Urdu (ur). The dataset contains both open-ended and multiple-choice questions covering culturally diverse domains such as festivals, customs, food, and heritage. Unlike prior datasets, ViMUL-Bench enables comprehensive evaluation of video-language models across both high- and low-resource languages, promoting inclusive and culturally aware research.

### 3.2 Baselines

For evaluation, we leverage the lmms-eval framework (lmms_eval2024), which enables a systematic comparison of TowerVision against leading open VLMs. We include several multilingual multimodal models, such as CulturalPangea-7B (yue2025pangeafullyopenmultilingual), designed to address gaps in multilingual cultural understanding, and Aya-Vision-8B (singh2024aya), optimized for a broad range of vision-language tasks.
In addition, we evaluate models from the Gemma3-Instruct (Gemma3-4B-it, Gemma3-12B-it; team2025gemma) and the Qwen2.5-VL-Instruct families (Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct; qwen2025qwen25technicalreport), both of which have demonstrated strong performance across a variety of multimodal benchmarks. Finally, we report results for a LLaVA-based model, Llama3-Llava-Next-8B (liu2024llavanext), a general-purpose VLM with strong performance across a wide range of tasks. The exact checkpoints for all models are listed in §[A.2](https://arxiv.org/html/2510.21849v3#A1.SS2).

For TowerVideo, we consider several competitive open-source video models of comparable scale, including VideoLLaMA3-7B (damonlpsg2025videollama3), LLaVA-Video-7B (zhang2025llavavideovideoinstructiontuning) also trained on LLaVA-Video-178k, and ViMUL-7B (shafique2025culturallydiversemultilingualmultimodalvideo), a multilingual video model.

**Table 1: Vision-Language Model Performance. English and multilingual VLMs results across multiple benchmarks. Reported values correspond to final accuracy ($\uparrow$).**
|  | English ($\uparrow$) | Multilingual ($\uparrow$) |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TextVQA | OCRBench | CC-OCR | ALM-Bench (en) | ALM-Bench (multi) |
| Qwen2.5-VL-3B-Instruct | 77.8 | 78.7 | 76.4 | 81.0 | 76.2 |
| Qwen2.5-VL-7B-Instruct | 82.5 | 84.5 | 78.6 | 83.1 | 83.6 |
| Gemma3-4B-it | 65.2 | 74.2 | 69.1 | 79.7 | 80.0 |
| Gemma3-12B-it | 73.2 | 74.7 | 73.8 | 83.5 | 84.5 |
| CulturalPangea7B | 69.8 | 63.5 | 51.7 | 61.3 | 65.2 |
| Llama3-Llava-Next-8B | 64.8 | 54.4 | 40.9 | 76.5 | 73.4 |
| Aya-Vision-8B | 66.9 | 61.0 | 46.3 | 78.2 | 77.3 |
| \rowcolorblue!10 TowerVision-2B | 68.1 | 58.6 | 46.1 | 77.1 | 81.1 |
| \rowcolorblue!20 TowerVision-9B | 73.6 | 69.7 | 56.3 | 83.6 | 85.2 |

**Table 2: Multimodal Translation Benchmarks. We report xComet (guerreiro2024xcomet) for Multi30K and contrastive pairwise accuracy for CoMMuTE.**
|  | Multi30K ($\uparrow$) | CoMMuTE ($\uparrow$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | en$\rightarrow$cs | en$\rightarrow$de | en$\rightarrow$fr | en$\rightarrow$de | en$\rightarrow$fr | en$\rightarrow$ru | en$\rightarrow$zh |
| Qwen2.5-VL-3B-Instruct | 83.3 | 96.7 | 92.6 | 71.6 | 74.4 | 77.5 | 81.5 |
| Qwen2.5-VL-7B-Instruct | 83.9 | 97.1 | 93.2 | 74.7 | 76.9 | 77.2 | 82.4 |
| Gemma3-4B-it | 33.4 | 44.0 | 33.2 | 76.7 | 78.2 | 79.0 | 74.4 |
| CulturalPangea7B | 80.0 | 95.8 | 92.1 | 68.3 | 77.3 | 75.3 | 79.3 |
| Llama3-Llava-Next-8B | 79.1 | 93.3 | 88.1 | 72.0 | 74.4 | 74.4 | 73.5 |
| Aya-Vision-8B | 94.4 | 97.9 | 95.3 | 69.3 | 76.9 | 74.4 | 76.2 |
| \rowcolorblue!10TowerVision-2B | 90.3 | 97.5 | 94.7 | 70.0 | 74.3 | 73.2 | 76.6 |
| \rowcolorblue!20TowerVision-9B | 95.1 | 98.1 | 95.6 | 72.0 | 78.8 | 75.6 | 77.4 |

**Table 3: Multilingual video performance per language. Accuracy (%) on ViMUL-Bench across 14 languages averaged across multiple-choice and open-ended questions. Underlined values mark the best score within TowerVision/TowerVideo variants; Unsupported languages are marked with ^∗.**
| Model | ar | bn^∗ | zh | en | fr | de | hi | ja | ru | si^∗ | es | sv | ta^∗ | ur^∗ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ViMUL-7B | 41.5 | 35.4 | 37.0 | 48.6 | 48.3 | 43.9 | 39.2 | 37.8 | 45.7 | 21.2 | 44.3 | 41.4 | 23.3 | 36.8 |
| LLaVA-Video-7B | 38.8 | 30.4 | 43.2 | 53.3 | 49.2 | 45.4 | 34.2 | 33.4 | 38.2 | 18.1 | 45.7 | 39.8 | 21.9 | 33.8 |
| VideoLLaMA3-7B | 45.6 | 36.6 | 48.0 | 52.9 | 47.1 | 43.8 | 37.5 | 39.4 | 44.8 | 25.1 | 45.4 | 38.5 | 22.8 | 32.1 |
| \rowcolorblue!5 TowerVision-2B | 18.9 | 19.5 | 21.7 | 34.2 | 28.9 | 28.3 | 25.1 | 22.2 | 24.8 | 16.3 | 30.4 | 27.1 | 16.1 | 19.9 |
| \rowcolorblue!10 TowerVideo-2B | 23.0 | 18.9 | 35.9 | 45.2 | 39.6 | 39.7 | 37.2 | 34.1 | 38.0 | 17.1 | 37.4 | 38.0 | 17.7 | 18.7 |
| \rowcolorblue!15 TowerVision-9B | 34.2 | 25.4 | 35.3 | 46.7 | 41.1 | 40.8 | 34.2 | 28.1 | 40.3 | 19.8 | 40.5 | 39.6 | 21.6 | 26.4 |
| \rowcolorblue!20 TowerVideo-9B | 38.6 | 22.1 | 44.8 | 51.9 | 49.1 | 47.1 | 32.2 | 42.3 | 40.9 | 20.8 | 46.0 | 44.8 | 24.1 | 19.5 |

### 3.3 Main Results

Tables [1](https://arxiv.org/html/2510.21849v3#S3.T1)–[2](https://arxiv.org/html/2510.21849v3#S3.T2) report the performance of TowerVision on vision-language benchmarks as well as multimodal translation benchmarks, while Table [3](https://arxiv.org/html/2510.21849v3#S3.T3) reports the results on the multilingual video-language benchmark.
We summarize the main findings below.

TowerVision models are strong in cultural-aware tasks. Within our suite of vision-language benchmarks, we achieve state-of-the-art results on ALM-Bench (Table [1](https://arxiv.org/html/2510.21849v3#S3.T1), a culturally diverse benchmark, in both the English and multilingual split. Qwen2.5VL-7B and Gemma3-12B are the closest competitors, while other baselines lag behind. In the multilingual split, we evaluate on a diverse set of 23 languages covering several language families and scripts. TowerVision is able to exhibit enhanced cultural multimodal understanding, suggesting that it is still performant in less seen and unseen languages within its training data. We further assess the cross-lingual generalization capabilities of TowerVision in §[6](https://arxiv.org/html/2510.21849v3#S4.T6).

TowerVision is less competitive on OCR-related tasks.
We hypothesize this is likely due to the limited amount of OCR-focused data in VisionBlocks compared against other models. Since we primarily pretrained TowerVision on large-scale image-caption datasets emphasizing natural images and language alignment, it struggles with scanned text or OCR-heavy scenarios. Despite these limitations, TowerVision does obtain superior performance compared against AyaVision-8B and Llam3LLaVANext-8B, the former of which has seen significant amounts of OCR-specific data (singh2024aya).

TowerVision-2B is competitive multilingually with larger models.
In multimodal translation benchmarks, TowerVision consistently demonstrates strong performance on Multi30K and is competitive on CoMMuTE (Table [2](https://arxiv.org/html/2510.21849v3#S3.T2)). Our 9B variant achieves state-of-the-art results on Multi30k across all language pairs, and we observe that even our smaller 2B variant is a competitive model against the larger baselines on translation-specific, as well as vision-language benchmarks. For instance, on Multi30K, TowerVision-2B obtains superior scores to Qwen2.5VL-7B and CulturalPangea-7B. Similarly, on the multilingual split of ALM-Bench, TowerVision 2B is competitive with Qwen2.5VL-7B and outperforms AyaVision-8B. These results further highlight the efficacy of TowerVision’s multilinguality and design choices. We also note that scaling from 2B to 9B parameters consistently improves performance across all benchmarks, suggesting that our training recipe scales well.

Multilingual fine-tuning improves cross-lingual performance in TowerVideo. In Table [3](https://arxiv.org/html/2510.21849v3#S3.T3), we report averages across multiple-choice accuracy and open-ended responses, which are automatically judged using GPT-4o (openai2024gpt4ocard), with the same evaluation prompt as shafique2025culturallydiversemultilingualmultimodalvideo. We compare our TowerVideo models, including the 9B variant, to strong open-source baselines. Our multilingual models are competitive across several languages despite using smaller datasets and fewer frames (for instance, VideoLLaMA3 uses 180 frames). Specifically, ViMUL was trained with separate copies of the dataset for each language, whereas our approach uses a single copy with half in English and the other half uniformly translated into the supported languages. Overall, these results highlight the effectiveness of video-based multilingual fine-tuning in improving cross-lingual reasoning.

Overall, our results demonstrate the effectiveness of our design choices in endowing our model with strong multilingual capabilities due to a combination of increased multilingual culturally-sensitive training data, a more multilingual text backbone (Tower+), and a multilingual vision encoder. We detail these choices in §[4](https://arxiv.org/html/2510.21849v3#S4) with a carefully conducted set of ablation experiments.

## 4 Where and How Does Multilinguality Matter?

Following the main results of TowerVision, we delve deeper into its design choices.

Multilingual backbones improve cross-modal performance.
The choice of backbone in TowerVision can substantially influence performance across multilingual and multimodal tasks.
We focus on two complementary aspects. First, we examine the significance of multilingual capacity by comparing the Tower+ backbone, which is highly multilingual and designed for general-purpose multilingual text tasks, against Gemma2, the model on which Tower+ was built. Second, we investigate the impact of instruction tuning before modality fusion, which is widely applied in modern VLMs from the start (liu2023visualinstructiontuning; bai2025qwen25vltechnicalreport), but whose effect on the final model remains unclear. To study these effects, we train TowerVision at 2B and 9B scales using four backbones: Gemma2-pt (pretrained, not instruction-tuned), Gemma2-it (instruction-tuned), Tower+pt (pretrained Tower+), and Tower+it (instruction-tuned Tower+), following the recipe in §[2](https://arxiv.org/html/2510.21849v3#S2). As shown in Table [4](https://arxiv.org/html/2510.21849v3#S4.T4), using Tower+ consistently outperforms Gemma2, confirming the importance of a strong multilingual backbone for robust cross-modal understanding. At smaller scales, non-instructed models (Gemma2-pt, Tower+pt) retain stronger raw visual extraction, while instruction-tuned variants excel in cultural knowledge and reasoning. By the 9B scale, this gap narrows, with instruction-tuned models integrating both skills and achieving state-of-the-art performance. These findings underscore the complementary roles of multilingual pretraining and instruction tuning, and the need for careful backbone selection in VLMs.

**Table 4: Impact of backbone and instruction tuning. Performance of VLMs with different backbones on English and multilingual tasks.**
| Backbone Model | English ($\uparrow$) | Multilingual ($\uparrow$) |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | TextVQA | OCRBench | CC-OCR | ALM-Bench (en) | ALM-Bench (multi) |
| Gemma2-pt-2B | 69.2 | 61.2 | 45.3 | 74.3 | 76.7 |
| \rowcolorblue!10<br>Tower+pt-2B | 70.3 | 62.1 | 46.3 | 73.0 | 78.2 |
| Gemma2-it-2B | 70.0 | 63.0 | 45.9 | 75.0 | 75.1 |
| \rowcolorblue!10<br>Tower+it-2B | 68.1 | 58.6 | 46.1 | 77.1 | 81.1 |
| Gemma2-pt-9B | 72.4 | 66.6 | 49.6 | 79.9 | 79.6 |
| \rowcolorblue!15<br>Tower+pt-9B | 73.2 | 64.5 | 54.5 | 81.3 | 84.4 |
| Gemma2-it-9B | 74.4 | 67.2 | 49.5 | 79.6 | 81.5 |
| \rowcolorblue!20<br>Tower+it-9B | 73.6 | 69.7 | 56.3 | 83.6 | 85.2 |

Multilingual-aware vision encoders improve performance in low-data regimes.
Effectively leveraging multilingual data is crucial for VLMs, yet it is unclear whether the vision encoder’s own multilingual capacity plays an important role.
We compare SigLIP2, trained on diverse multilingual data, with SigLIP1, an earlier English-centric version, to test whether multilingual-aware encoders are essential or if sufficient fine-tuning can compensate.
We train TowerVision with both encoders on English-only and multilingual data at 2B and 9B scales (results in Table [5](https://arxiv.org/html/2510.21849v3#S4.T5)).

**Table 5: Multilingual impact of different vision encoders.**
| TowerVision | 2B | 9B |  |  |
| --- | --- | --- | --- | --- |
| Variant | En | Multi | En | Multi |
| SigLIP1-En | 67.4 | 60.2 | 78.3 | 81.2 |
| SigLIP2-En | 69.3 | 67.1 | 77.2 | 81.1 |
| SigLIP1-(En+Multi) | 76.6 | 80.7 | 83.6 | 84.4 |
| SigLIP2-(En+Multi) | 77.1 | 81.1 | 83.6 | 85.2 |

Without additional multilingual data, SigLIP2 models consistently outperform SigLIP1, showing clear benefits in low data regimes. With multilingual fine-tuning, however, the gap narrows, showing that finetuning with sufficient multilingual data can compensate for a weaker encoder. At 9B scale, both converge to strong performance. In short, multilingual-aware encoders provide an advantage when data is scarce, but extensive multilingual training can close the gap.

High-quality English captions are enough to ensure strong alignment.
To assess whether multilingual supervision is necessary during alignment pretraining, we train two versions of TowerVision on both scales, 2B and 9B.

**Table 6: Effect of using multilingual versus English-only captions during projector pretraining on ALM-Bench. Results indicate low to no gains from adding multilingual data at this stage.**
| TowerVision | 2B | 9B |  |  |
| --- | --- | --- | --- | --- |
| Projector | En | Multi | En | Multi |
| En | 77.1 | 81.1 | 83.6 | 85.2 |
| En+Multi | 77.9 | 79.3 | 83.0 | 84.1 |

The first version uses only English-only captions from PixMo-Cap, comprising $702,205$ text-image pairs. The second version uses the same English captions combined with a high-quality translated subset from PIXMO-CAP, where data was uniformly translated into the supported languages as described in §[2.1](https://arxiv.org/html/2510.21849v3#S2.SS1), comprising 367,779 samples.
We evaluate the models in ALM-Bench to measure TowerVision performance both in English and across multiple non-English languages, providing insights into how well cross-lingual generalization is preserved or improved.
As shown in Table [6](https://arxiv.org/html/2510.21849v3#S4.T6), adding high-quality multilingual captions during the projector alignment stage has little to no positive effect and, in some cases, slightly decreases performance on the multilingual subset. This suggests that the most effective strategy is to focus on diverse and high-quality captions, ensuring strong alignment between visual and textual modalities, rather than prioritizing extensive multilingual coverage at this stage.

Expanding languages improves cross-lingual generalization in VLMs. We study how language coverage in training data impacts performance on both included and excluded languages. Specifically, we compare training on 10 high-resource “core languages” versus the full set of languages, while controlling for dataset size. Our questions are: (i) whether adding balanced multimodal data for more languages improves performance on core languages (conneau2020unsupervisedcrosslingualrepresentationlearning; hu2020xtrememassivelymultilingualmultitask), and (ii) whether unsupported languages benefit in zero-shot fashion if related languages are present (ni2021m3plearninguniversalrepresentations). We train TowerVision at 2B and 9B scales using the recipe in §[2](https://arxiv.org/html/2510.21849v3#S2), first on 10 “core” languages (English, German, Dutch, Portuguese, Russian, Simplified and Traditional Chinese, Spanish, French, Italian), then on all available languages. Results in Figure [3](https://arxiv.org/html/2510.21849v3#S4.F3) (more details in §[A.4](https://arxiv.org/html/2510.21849v3#A1.SS4)) show that broader language coverage consistently improves performance, with larger gains at the 2B scale. Zero-shot improvements for unsupported languages further support cross-lingual transfer when related languages are included. These findings highlight the value of expanding multilingual data, particularly for smaller models.

Figure: Figure 3: Performance of TowerVision models on 10 vs 20 languages/dialects at 2B and 9B scales. The bars indicate the accuracy gains by training on 20 (all) versus 10 (core) languages (more details in §[A.4](https://arxiv.org/html/2510.21849v3#A1.SS4)). Unsupported languages are marked with ^∗.
Refer to caption: gain_loss.png

**Table 7: Accuracy (%) on ViMUL-Bench across 14 languages averaged across multiple-choice and open-ended questions. Underlined values mark the best score within TowerVision/TowerVideo variants; Unsupported languages are marked with ^∗.**
| Model | ar | bn^∗ | zh | en | fr | de | hi | ja | ru | si^∗ | es | sv | ta^∗ | ur^∗ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \rowcolorblue!10 TowerVision-2B | 18.9 | 19.5 | 21.7 | 34.2 | 28.9 | 28.3 | 25.1 | 22.2 | 24.8 | 16.3 | 30.4 | 27.1 | 16.1 | 19.9 |
| \rowcolorblue!15 TowerVideo-2B (english only) | 25.7 | 17.8 | 26.7 | 45.5 | 42.3 | 34.8 | 27.8 | 27.7 | 34.4 | 17.9 | 37.8 | 34.0 | 18.3 | 19.7 |
| \rowcolorblue!20 TowerVideo-2B (multilingual) | 23.0 | 18.9 | 35.9 | 45.2 | 39.6 | 39.7 | 37.2 | 34.1 | 38.0 | 17.1 | 37.4 | 38.0 | 17.7 | 18.7 |

How does multilingual data affect video fine-tuning?
To assess the impact of our multilingual data (see § [2.1](https://arxiv.org/html/2510.21849v3#S2.SS1)) during video fine-tuning, we present results in Table [7](https://arxiv.org/html/2510.21849v3#S4.T7) for two baselines: (i) the original TowerVision-2B model and (ii) TowerVideo-2B trained on the full English-only LLaVA-Video-178k dataset. Fine-tuning with video substantially improves the performance of TowerVision models compared to image-text-only variants, highlighting the importance of temporal information for video-language understanding. Incorporating multilingual data further enhances cross-lingual generalization, while English performance remains largely stable, indicating that adding multiple languages does not compromise primary-language capabilities, even though the multilingual models are trained on substantially less English data.

## 5 Conclusion

We introduced TowerVision, a suite of multimodal models for image-text and video-text tasks, designed with a strong emphasis on cultural understanding and multilinguality. Our models demonstrate competitive, and in several cases improved, multilingual performance across a range of benchmarks when compared with existing open multimodal systems. Alongside this, we released VisionBlocks, a high-quality vision-language dataset, and provided a detailed training recipe covering data, encoders, and text backbones, complemented by an extensive ablation study on key components of our approach.

We hope that these contributions—spanning models, data, and methodology—help advance research on culturally diverse multilingual multimodal language models, and accelerate progress toward narrowing the performance gap with English-centric settings.

## 6 Acknowledgments

This work was supported by the project DECOLLAGE (ERC-2022-CoG 101088763), by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI), and by FCT/MECI through national funds and when applicable co-funded EU funds under UID/50008: Instituto de Telecomunicações. The models were trained at EuroHPC infrastructure MareNostrum5 through grants EHPC-AI-2024A05-044 and 2025.00272.CPCA.A3.

## 7 Ethics statement

This work develops and evaluates multilingual vision-language models using publicly available datasets as well as our own synthetic and translated data. We acknowledge potential risks, including biased model outputs and unintended misuse of generated content. While we have taken steps to ensure diversity and maximum data quality, we always encourage careful evaluation and responsible deployment of these models in real-world scenarios. Our research does not involve sensitive personal data or tasks with direct safety-critical impact.

## 8 Reproducibility Statement

This work provides detailed descriptions of the data, model architectures, training procedure (including the codebase), and evaluation benchmarks used. All datasets used are either publicly available or created by our team (synthetic and translated), with the respective system prompts shared for maximum transparency. Additionally TowerVision all the collection of models, code for data preprocessing, training, and evaluation will be released to facilitate replication of our results. We aim to ensure that other researchers can reproduce our findings with minimal effort.

## Appendix A Appendix

### A.1 Full Description of VisionBlocks

Table [8](https://arxiv.org/html/2510.21849v3#A1.T8) shows the full details and statistics of the VisionBlocks dataset.

**Table 8: Overview of dataset composition across categories. Each dataset lists its sample size with the proportion of the total in parentheses, along with its collection type tag (Public Data, Synthetic (Generated), or Translated (Augmented)). Totals are shown for English-only and Multilingual subsets, as well as the overall dataset size.**
| Category | Dataset | Samples (%) | Tag |
| --- | --- | --- | --- |
| Chart/Plot | DVQA | 199,995 (3.17%) | Public Data |
|  | ChartQA | 25,055 (0.40%) | Synthetic (Generated) |
|  | PlotQA | 157,070 (2.49%) | Public Data |
|  | TabMWP | 22,717 (0.36%) | Public Data |
| General VQA | VQAv2 | 428,708 (6.79%) | Public Data |
|  | RLAIF-4V | 59,408 (0.94%) | Synthetic (Generated) |
| Doc VQA | DocVQA | 9,664 (0.15%) | Synthetic (Generated) |
|  | TextVQA | 15,690 (0.25%) | Synthetic (Generated) |
|  | ST-VQA | 17,242 (0.27%) | Public Data |
|  | PixMo-Docs | 3,634 (0.06%) | Public Data |
| Reasoning/Knowledge | A-OKVQA | 11,853 (0.19%) | Synthetic (Generated) |
|  | OKVQA | 9,009 (0.14%) | Public Data |
|  | AI2D | 7,791 (0.12%) | Public Data |
|  | ScienceQA | 758 (0.012%) | Public Data |
| Multilingual/Cultural | Pangea-Cultural | 55,438 (0.88%) | Public Data |
|  | Pangea-Multi | 428,838 (6.79%) | Public Data |
|  | PixMo-Cap-Translated | 367,779 (5.83%) | Translated (Augmented) |
|  | CulturalGround-OE | 401,149 (6.35%) | Public Data |
|  | CulturalGround-MCQs | 379,834 (6.02%) | Public Data |
| Specialized VQA | IconQA | 19,543 (0.31%) | Synthetic (Generated) |
|  | InfographicVQA | 2,049 (0.03%) | Synthetic (Generated) |
|  | Stratos | 12,585 (0.20%) | Public Data |
| Counting/Math | TallyQA | 98,675 (1.56%) | Public Data |
|  | PixMo-Count | 8,128 (0.13%) | Public Data |
| Vision/Text | VBlocks-PixMo-AMA | 154,336 (2.44%) | Public Data |
|  | VBlocks-PixMo-Cap | 702,205 (11.12%) | Public Data |
|  | VBlocks-PixMo-CapQA | 262,862 (4.16%) | Public Data |
|  | EuroBlocks-SFT | 1,094,265 (17.34%) | Public Data |
| Video/Text | LLaVA-Video-178k-subset | 697,618 (11.05%) | Public Data |
|  | LLaVA-Video-178k-translated | 697,617 (11.05%) | Translated (Augmented) |
| Total (English) | 3,982,630 (63.1%) |  |  |
| Total (Multilingual) | 2,330,656 (36.9%) |  |  |
| Overall Total | 6,313,286 (100%) |  |  |

### A.2 Models Checkpoints

Table [9](https://arxiv.org/html/2510.21849v3#A1.T9) lists all model checkpoints used for comparative baselines. We use checkpoints released HuggingFace when possible.

**Table 9: Model checkpoints. Parameters and HuggingFace links for models included in our evaluation suite.**
| Model | Params | Checkpoint Link |
| --- | --- | --- |
| Qwen2.5-VL-Instruct | 3B | [https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) |
| Qwen2.5-VL-Instruct | 7B | [https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) |
| Gemma2-it | 2B | [https://huggingface.co/google/gemma-2-2b-it](https://huggingface.co/google/gemma-2-2b-it) |
| Gemma2-pt | 2B | [https://huggingface.co/google/gemma-2-2b](https://huggingface.co/google/gemma-2-2b) |
| Gemma2-it | 9B | [https://huggingface.co/google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) |
| Gemma2-pt | 9B | [https://huggingface.co/google/gemma-2-9b](https://huggingface.co/google/gemma-2-9b) |
| Gemma3-it | 4B | [https://huggingface.co/google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it) |
| Gemma3-it | 12B | [https://huggingface.co/google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it) |
| CulturalPangea | 7B | [https://huggingface.co/neulab/CulturalPangea-7B](https://huggingface.co/neulab/CulturalPangea-7B) |
| LLaVA-Next | 7B | [llava-hf/llava-v1.6-mistral-7b-hf](llava-hf/llava-v1.6-mistral-7b-hf) |
| Aya-Vision | 8B | [https://huggingface.co/CohereForAI/aya-vision-8b](https://huggingface.co/CohereForAI/aya-vision-8b) |
| Pixtral | 12B | [https://huggingface.co/mistralai/Pixtral-12B-2409](https://huggingface.co/mistralai/Pixtral-12B-2409) |
| Phi-4-Multimodal | 14B | [https://huggingface.co/microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) |

### A.3 Vision Encoder Variants

Beyond selecting a more multilingual vision encoder, several other factors significantly influence its performance. These include the input image resolution supported by the encoder, the number of patches it uses, which determines the total number of visual tokens for a given image resolution (e.g, for an img resolution of $224\times 224$ using patch size of 14 we obtain 256 visual tokens) and the number of tiles.

Our goal is to empirically identify the optimal configuration for processing visual inputs, focusing on these three factors.

Specifically, we perform experiments using the TowerVision 2B version with variants of SigLIP2 framework:

- 1.
Image resolution: We vary the input image size between $384\times 384$, $224\times 224$, and $512\times 512$ to examine its effect on feature extraction quality.
- 2.
Patch numbers: We test different patch sizes (14 and 16) to assess how granularity impacts the learned representations. Smaller patches capture finer details but increase the number of tokens, affecting the context length the model must handle.
- 3.
Number of tiles: Beyond the default 6 tiles, we also experiment with 4 and 22 tiles. The number of tiles is adjusted to the image resolution: lower-resolution images (e.g, $224\times 224$) require more tiles to cover the same amount of visual information as a higher-resolution encoder (e.g., $512\times 512$). For example, an image with resolution ($1024,1024$) processed with a $512\times 512$ encoder requires roughly 4 tiles to cover the full image, whereas a $224\times 224$ encoder would need at least 25 tiles (including padding) to achieve similar coverage.
This creates a trade-off between capturing detailed local information and maintaining manageable context length.

These experiments allow us to systematically compare variations while keeping other components constant, providing insights into which configuration yields the best overall performance. Results are reported in Table [10](https://arxiv.org/html/2510.21849v3#A1.T10), highlighting the trade-offs between resolution, patch granularity, and style diversity.

**Table 10: Impact of Vision Encoder Configuration and Instruction Tuning. Evaluation of Tower+ models across English and multilingual tasks with varying image resolution, patch size, and number of tiles. Results highlight how these design choices affect overall performance.**
| Resolution | Patch Size | Tiles | English | Multilingual |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  |  | TextVQA | OCRBench | CC-OCR | ALM-Bench |
| 224x224 | 14 | 22 | 59.1 | 53.3 | 37.2 | 70.5 |
| 224x224 | 16 | 20 | 68.6 | 57.8 | 44.3 | 75.2 |
| 384x384 | 14 | 6 | 70.3 | 62.1 | 46.1 | 75.6 |
| 512x512 | 16 | 4 | 64.0 | 55.7 | 39.6 | 74.7 |

### A.4 Cross-Lingual Generalization

**Table 11: Cross-lingual performance of TowerVision models at 2B and 9B scales. Core Langs refers to a set of 10 languages: English, German, Dutch, Portuguese, Russian, Simplified and Traditional Chinese, Spanish, French and Italian. Core+Added Langs includes all languages supported by TowerVision. Unseen languages are those not encountered during training. Bold values indicate the better result within each scale. Positive gains from adding languages are highlighted in light green, negative gains in light red. Overall, adding more languages tends to improve performance across the board, demonstrating strong cross-lingual transfer capabilities, even for unseen languages.**
|  | TowerVision-2B | TowerVision-9B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Metric / Lang | Core Langs | Core + Added Langs | Gain | Core Langs | Core + Added Langs | Gain |
| English (en) | 60.9 | 76.6 | +15.8 | 70.3 | 82.8 | +12.5 |
| Core Avg | 65.3 | 81.3 | +16.1 | 81.5 | 82.6 | +1.1 |
| Added Avg | 60.2 | 75.4 | +15.2 | 76.3 | 84.3 | +7.6 |
| Unseen Avg | 69.2 | 83.0 | +13.9 | 81.2 | 82.5 | +1.2 |
| German (de) | 75.9 | 84.5 | +8.6 | 89.7 | 87.9 | -1.8 |
| Spanish (es) | 56.6 | 60.5 | +3.9 | 73.7 | 76.3 | +2.6 |
| French (fr) | 76.9 | 82.7 | +5.8 | 86.5 | 80.8 | -5.7 |
| Hindi (hi) | 44.2 | 75.0 | +30.8 | 82.7 | 80.8 | -1.9 |
| Italian (it) | 75.0 | 81.7 | +6.7 | 96.7 | 98.3 | +1.6 |
| Korean (ko) | 76.4 | 70.8 | -5.6 | 75.0 | 79.2 | +4.2 |
| Dutch (nl) | 70.0 | 86.7 | +16.7 | 90.0 | 86.7 | -3.3 |
| Portuguese (pt) | 64.5 | 90.3 | +25.8 | 85.5 | 91.9 | +6.4 |
| Romanian (ro) | 58.9 | 80.4 | +21.5 | 75.0 | 87.5 | +12.5 |
| Czech (cs) | 61.4 | 75.7 | +14.3 | 74.3 | 90.0 | +15.7 |
| Russian (ru) | 65.5 | 84.5 | +19.0 | 65.5 | 75.9 | +10.4 |
| Chinese (simp.) (zh-hans) | 50.0 | 87.5 | +37.5 | 68.8 | 71.9 | +3.1 |
| Chinese (trad.) (zh-hant) | 53.8 | 76.9 | +23.1 | 61.5 | 67.3 | +5.8 |
| Danish (da)* | 66.1 | 70.9 | +4.8 | 90.3 | 86.3 | -4.0 |
| Finnish (fi)* | 56.0 | 82.0 | +26.0 | 70.0 | 72.0 | +2.0 |
| Hungarian (hu)* | 68.8 | 95.3 | +26.5 | 79.7 | 82.8 | +3.1 |
| Icelandic (is)* | 67.6 | 76.5 | +8.9 | 76.5 | 83.8 | +7.3 |
| Japanese (jp)* | 78.8 | 78.9 | 0.1 | 84.8 | 80.3 | -4.5 |
| Swedish (sv)* | 77.6 | 94.8 | +17.2 | 86.2 | 89.7 | +3.5 |

### A.5 System Prompts

#### A.5.1 Tower System Prompts used for Translation

The prompts vary in style and specificity to improve diversity and capture nuanced meaning from the original English captions. They are grouped by language and include multiple phrasings for the same instruction to encourage robust translations.

#### A.5.2 Gemini 2.5 System Prompts

We generate synthetic captions using the Gemini 2.5 API with a diverse set of system prompts. These prompts are designed to produce varied response formats, including direct answers, caption-plus-answer pairs, and structured final-answer formats.

These prompts are used to generate high-quality captions that improve instruction-following and visual description diversity.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. TOWERVISION achieves competitive or superior performance on various multilingual benchmarks, showing particular strength in culturally grounded tasks (like ALM-Bench) and multimodal translation (like Multi30K).

2. The paper provides a systematic study of multilingual design choices, analyzing the impact of different components such as training data composition, encoder selection, and text backbones.

3. The paper demonstrates that using a multilingual text backbone (TOWER+) and a multilingual vision encoder (SigLIP2) significantly improves cross-modal and cross-lingual performance compared to English-centric baselines .

4. The training approach substantially improves cross-lingual generalization, benefitting both high-resource and underrepresented languages, and even showing zero-shot improvements for unsupported languages.

5. The paper contributes open resources to the community, including the TOWERVISION and TOWERVIDEO models, training recipes, and VISIONBLOCKS, a curated high-quality multilingual vision-language dataset of approximately 6 million samples.

6. The work extends beyond static images to video with TOWERVIDEO, which achieves competitive performance on culturally diverse video benchmarks.
summary: The paper introduces TOWERVISION, a family of open multilingual CLMs designed to address the limitations of current English-centric VLMs . The authors also release TOWERVIDEO, a video-capable variant, and VISIONBLOCKS, a curated multilingual multimodal dataset. TOWERVISION utilizes a multi-stage training process and specific architectural choices to enhance multilingual capabilities. To overcome the scarcity of high-quality multilingual vision-text data, the authors created VISIONBLOCKS, a dataset containing approximately 6 million samples. It aggregates filtered public data, newly translated captions (using TOWER), and synthetic data generated via Gemini 2.5 to improve fine-grained visual details. TOWERVISION achieves competitive or superior performance on multilingual benchmarks like ALM-Bench (cultural understanding) and Multi30K (multimodal translation). Using a multilingual text backbone (TOWER+) consistently outperforms standard backbones (GEMMA2) for cross-modal tasks.
weaknesses: 1. The finding that adding high-quality multilingual captions during the projector alignment stage yields "little to no positive effect" and sometimes "slightly decreases performance" on multilingual subsets is counter-intuitive to the paper's central claim that multilingual components improve performance. The paper currently relies on a surface-level hypothesis that focusing on "diverse, high-quality English captions" is simply more effective for standardizing visual and textual representations at this stage. This leaves a critical question unanswered: why does multilingual data fail at this specific stage when it succeeds elsewhere?

2. While TOWERVIDEO is presented as a major contribution, its implementation appears to be a straightforward extension of standard practices rather than a novel multilingual video approach. There is no ablation study for video-specific parameters equivalent to the depth provided for image resolution and tiling, leaving it unclear if 32 frames is optimal for multilingual video understanding, which often relies on reading temporally disparate text or recognizing subtle cultural cues.

3. The authors explicitly acknowledge that TOWERVISION is "less competitive on OCR-related tasks" due to a lack of OCR-focused data in VISIONBLOCKS. For a model aiming at robust multilingual capability, OCR is a fundamental requirement (e.g., reading menus, signs, or documents in various languages). Failing to address this known limitation weakens the claim of a truly comprehensive multilingual VLM.

4. The claim that VISIONBLOCKS is "high-quality" heavily relies on automated filtering, specifically using a COMETKIWI threshold of 0.85 for translations and Gemini 2.5 for synthetic data generation. While standard, these methods can miss subtle artifacts or cultural inaccuracies that automated metrics cannot detect, potentially undermining the "culturally grounded" goal of the model.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: The main concerns stand on the technical contribution, which appears limited beyond the introduction of new multilingual data. The results are also mixed. Additional qualitative or error analyses could help explain this. While minor, please double-check citation formatting (e.g., line 148).
soundness: 2
strengths: 1. The paper is well motivated by the imbalance in current VLMs, which are predominantly English-centric. Addressing multilingual and cultural inclusivity is both timely and impactful, supporting fairer and broader use of VLMs across language communities. 
2. The introduction of VisionBlocks, a curated multilingual vision-language dataset, is a meaningful contribution. It provides a useful resource that directly improves multilingual performance and could benefit future research in this area. 
3. Beyond presenting a multilingual model, the paper offers several valuable observations, such as the effects of multilingual backbones, alignment stages, and fine-tuning strategies, that are informative not only for multilinguality but also for the general design of VLMs.
summary: Most Vision-Language Models (VLMs) are English-centric and struggle to generalize across languages and cultures. This paper introduces TowerVision and TowerVideo as two variants for building and analyzing multilingual vision-language models based on the Tower+ text backbone. The approach is supported by a new multilingual dataset, VisionBlocks, that combines translated, synthetic, and culturally grounded image-text pairs. The resulting models show improved cross-lingual understanding and cultural grounding on several multilingual benchmarks. The study provides a systematic investigation of multilinguality in VLMs and explores when and how to introduce multilinguality into them.
weaknesses: 1. Section 2.2 primarily describes implementation details and training configurations rather than introducing a distinct modeling contribution. It remains unclear what is newly proposed in this work versus what follows established multilingual VLM pipelines from prior studies. Thus, from a technical standpoint, the work mainly extends existing pipelines with additional multilingual data. While the dataset itself is valuable, the overall improvement, training on more data with translated text in target languages, is expected rather than conceptually new. The contribution thus lies more in resource aggregation than in methodological innovation.
2. The proposed multilingual dataset is largely an LLM-augmented collection of existing image-text and video-text resources. Several design choices, such as using the Tower model for translation, evaluating with COMETKIWI, and adopting an arbitrary threshold of 0.8 are not well justified or empirically supported. These decisions appear somewhat ad hoc and lack evidence beyond internal preference or prior use by the same research group.
3. As the authors reported, the proposed models do not outperform baselines across all tasks. Results in Table 1 and Table 3 show mixed or marginal improvements, including multilingual video-to-text tasks, suggesting that the benefits are uneven across modalities and benchmarks.
4. The analysis remains largely quantitative, focusing on benchmark scores and metrics. Given the mixed quantitative results, a qualitative examination, such as cross-lingual error cases or culturally specific examples, would provide stronger insights into the actual behavior and limitations.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - See W3
- Do you have any insights regarding to the data mixture? What type of training data is effective in improving the ViMUL-Bench?
soundness: 3
strengths: S1: The paper provides a suite of open-sourced multimodal multilingual LLMs for both image-text tasks, as well as video text tasks. The paper also provides a curated training dataset including both human and synthetically generated resources of 6M instances for enabling future research in this area.

S2: The evaluation of Towervision is extensive. They span across multiple modalities, languages (20+), and task types (VQA, OCR, translation, cultural reasoning). The resulting models are competitive against strong baselines (Qwen2.5-VL, Gemma3, Aya-Vision, CulturalPangea) and demonstrate robust performance.

S3: The analysis of where multilingual training matters is insightful, where multilingual backbones matter more than multilingual captions during alignment.
summary: This paper presents TowerVision, a suite of multimodal multilingual large language models (LLMs) for both image-text and video-text tasks, built upon the Tower+ models and SigLIP2. TowerVision leverages visual and cultural context during fine-tuning through a three-stage training process consisting of: (1) projector pretraining, (2) vision fine-tuning, and (3) video fine-tuning. The final TowerVision models outperform existing systems trained on much larger datasets across benchmarks such as ALM-Bench, Multi30K, and ViMUL-Bench.

Key contributions of the paper include the release of VISIONBLOCKS, a curated multilingual vision-language dataset; the open-sourcing of the models; and empirical insights into model backbones and training data composition for building more effective multilingual multimodal LLMs.
weaknesses: W1: While the paper provides an analysis of when multilingual training can improve the final LLM’s performance, a deeper examination of the data mixture would offer additional insights. For instance, the authors could analyze the benefits of training on different types of datasets (e.g., mathematical versus chart-based datasets).


W2: Although the paper addresses an important problem and contributes new models to multilingual multimodal research, it offers limited architectural and methodological novelty.

W3: The clarity of writing could be improved, as some important details are missing. For example:
 - Table 4: The Gemma2-2B/9B-IT models are not included in the comparison. How do the results compare to fine-tuning the IT model?
 - Table 5: Details about the dataset used for evaluation are missing. What data was used for these evaluations?
 - Line 398: The term “low-data regimes” should be clarified. What exactly does it refer to in this context?
 - Table 7: How does multilingual data affect video fine-tuning for the 9B model?


Decision: Reject