Title: Aya Vision: Advancing the Frontier of Multilingual Multimodality
Primary area: foundation or frontier models, including LLMs
Keywords: deep learning, multimodal learning, vision-language models

Abstract: Building multimodal language models is fundamentally challenging: requiring alignment of vision and language modalities, curating high-quality instruction data, and preserving existing text-only capabilities once vision is introduced. These difficulties are further magnified in multilingual settings, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address these issues, we propose: (1) a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data across many languages; (2) a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Together, these contributions yield \textbf{Aya Vision}, a family of open-weights multilingual multimodal models (8B and 32B) that achieve \textbf{leading performance across both multimodal and text-only tasks}, outperforming significantly larger models. Our work provides guidance and reusable components for scalable multilingual data curation, robust multimodal training, and advancing meaningful evaluation in multilingual multimodal AI.

TL;DR: We release the recipe to build state of the art multilingual multimodal models