All Languages Matter: Evaluating LMMs on Culturally Diverse 100
  Languages
Authors: name: Ashmal Vayani, name: Dinura Dissanayake, name: Hasindri Watawana, name: Noor Ahsan, name: Nevasini Sasikumar, name: Omkar Thawakar, name: Henok Biadglign Ademtew, name: Yahya Hmaiti, name: Amandeep Kumar, name: Kartik Kuckreja, name: Mykola Maslych, name: Wafa Al Ghallabi, name: Mihail Mihaylov, name: Chao Qin, name: Abdelrahman M Shaker, name: Mike Zhang, name: Mahardika Krisna Ihsani, name: Amiel Esplana, name: Monil Gokani, name: Shachar Mirkin, name: Harsh Singh, name: Ashay Srivastava, name: Endre Hamerlik, name: Fathinah Asma Izzati, name: Fadillah Adamsyah Maani, name: Sebastian Cavada, name: Jenny Chim, name: Rohit Gupta, name: Sanjay Manjunath, name: Kamila Zhumakhanova, name: Feno Heriniaina Rabevohitra, name: Azril Amirudin, name: Muhammad Ridzuan, name: Daniya Kareem, name: Ketan More, name: Kunyang Li, name: Pramesh Shakya, name: Muhammad Saad, name: Amirpouya Ghasemaghaei, name: Amirbek Djanibekov, name: Dilshod Azizov, name: Branislava Jankovic, name: Naman Bhatia, name: Alvaro Cabrera, name: Johan Obando-Ceron, name: Olympiah Otieno, name: Fabian Farestam, name: Muztoba Rabbani, name: Sanoojan Baliah, name: Santosh Sanjeev, name: Abduragim Shtanchaev, name: Maheen Fatima, name: Thao Nguyen, name: Amrin Kareem, name: Toluwani Aremu, name: Nathan Xavier, name: Amit Bhatkal, name: Hawau Toyin, name: Aman Chadha, name: Hisham Cholakkal, name: Rao Muhammad Anwer, name: Michael Felsberg, name: Jorma Laaksonen, name: Thamar Solorio, name: Monojit Choudhury, name: Ivan Laptev, name: Mubarak Shah, name: Salman Khan, name: Fahad Khan
Published: 2024-11-25 15:44:42

Existing Large Multimodal Models (LMMs) generally focus on only a few regions
and languages. As LMMs continue to improve, it is increasingly important to
ensure they understand cultural contexts, respect local sensitivities, and
support low-resource languages, all while effectively integrating corresponding
visual cues. In pursuit of culturally diverse global multimodal models, our
proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
most comprehensive effort to date for evaluating LMMs across 100 languages.
ALM-bench challenges existing models by testing their ability to understand and
reason about culturally diverse images paired with text in various languages,
including many low-resource languages traditionally underrepresented in LMM
research. The benchmark offers a robust and nuanced evaluation framework
featuring various question formats, including true/false, multiple choice, and
open-ended questions, which are further divided into short and long-answer
categories. ALM-bench design ensures a comprehensive assessment of a model's
ability to handle varied levels of difficulty in visual and linguistic
reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
curates content from 13 distinct cultural aspects, ranging from traditions and
rituals to famous personalities and celebrations. Through this, ALM-bench not
only provides a rigorous testing ground for state-of-the-art open and
closed-source LMMs but also highlights the importance of cultural and
linguistic inclusivity, encouraging the development of models that can serve
diverse global populations effectively. Our benchmark is publicly available.