All Languages Matter: Evaluating LMMs on Culturally Diverse 100
  Languages
Authors: name: Ashmal Vayani; name: Dinura Dissanayake; name: Hasindri Watawana; name: Noor Ahsan; name: Nevasini Sasikumar; name: Omkar Thawakar; name: Henok Biadglign Ademtew; name: Yahya Hmaiti; name: Amandeep Kumar; name: Kartik Kuckreja; name: Mykola Maslych; name: Wafa Al Ghallabi; name: Mihail Mihaylov; name: Chao Qin; name: Abdelrahman M Shaker; name: Mike Zhang; name: Mahardika Krisna Ihsani; name: Amiel Esplana; name: Monil Gokani; name: Shachar Mirkin; name: Harsh Singh; name: Ashay Srivastava; name: Endre Hamerlik; name: Fathinah Asma Izzati; name: Fadillah Adamsyah Maani; name: Sebastian Cavada; name: Jenny Chim; name: Rohit Gupta; name: Sanjay Manjunath; name: Kamila Zhumakhanova; name: Feno Heriniaina Rabevohitra; name: Azril Amirudin; name: Muhammad Ridzuan; name: Daniya Kareem; name: Ketan More; name: Kunyang Li; name: Pramesh Shakya; name: Muhammad Saad; name: Amirpouya Ghasemaghaei; name: Amirbek Djanibekov; name: Dilshod Azizov; name: Branislava Jankovic; name: Naman Bhatia; name: Alvaro Cabrera; name: Johan Obando-Ceron; name: Olympiah Otieno; name: Fabian Farestam; name: Muztoba Rabbani; name: Sanoojan Baliah; name: Santosh Sanjeev; name: Abduragim Shtanchaev; name: Maheen Fatima; name: Thao Nguyen; name: Amrin Kareem; name: Toluwani Aremu; name: Nathan Xavier; name: Amit Bhatkal; name: Hawau Toyin; name: Aman Chadha; name: Hisham Cholakkal; name: Rao Muhammad Anwer; name: Michael Felsberg; name: Jorma Laaksonen; name: Thamar Solorio; name: Monojit Choudhury; name: Ivan Laptev; name: Mubarak Shah; name: Salman Khan; name: Fahad Khan
Published: 2024-11-25 15:44:42

Abstract: Existing Large Multimodal Models (LMMs) generally focus on only a few regions
and languages. As LMMs continue to improve, it is increasingly important to
ensure they understand cultural contexts, respect local sensitivities, and
support low-resource languages, all while effectively integrating corresponding
visual cues. In pursuit of culturally diverse global multimodal models, our
proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
most comprehensive effort to date for evaluating LMMs across 100 languages.
ALM-bench challenges existing models by testing their ability to understand and
reason about culturally diverse images paired with text in various languages,
including many low-resource languages traditionally underrepresented in LMM
research. The benchmark offers a robust and nuanced evaluation framework
featuring various question formats, including true/false, multiple choice, and
open-ended questions, which are further divided into short and long-answer
categories. ALM-bench design ensures a comprehensive assessment of a model's
ability to handle varied levels of difficulty in visual and linguistic
reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
curates content from 13 distinct cultural aspects, ranging from traditions and
rituals to famous personalities and celebrations. Through this, ALM-bench not
only provides a rigorous testing ground for state-of-the-art open and
closed-source LMMs but also highlights the importance of cultural and
linguistic inclusivity, encouraging the development of models that can serve
diverse global populations effectively. Our benchmark is publicly available.