IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
Authors: Ali Faraz
Akash
Shaharukh Khan
Raja Kolla
Akshat Patidar
Suranjan Goswami
Abhinav Ravi
Chandra Khatri
Shubham Agarwal
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models
VLMs
Multimodal models
Cultural VLMs
Mutlimodal Evaluation
OCR
Cultural VQA
Mutlimodal Machine Translation
MMT
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce IndicVisionBench, a large-scale cultural benchmark covering English and 10 Indian languages across 3 multimodal tasks including OCR, VQA and Multimodal Machine Translation (MMT).

Abstract: Vision-language models (VLMs) have demonstrated impressive generalization across multimodal tasks, yet most evaluation benchmarks remain Western-centric, leaving open questions about their performance in culturally diverse and multilingual settings. To address this gap, we introduce IndicVisionBench, the first large-scale benchmark centered on the Indian subcontinent. Covering English and 10 Indian languages, our benchmark spans 3 multimodal tasks, including Optical Character Recognition (OCR), Multimodal Machine Translation (MMT), and Visual Question Answering (VQA), covering 6 kinds of question types. Our final benchmark consists of a total of ~5K images and 37K+ QA pairs across 13 culturally grounded topics. In addition, we release a paired parallel corpus of annotations across 10 Indic languages, creating a unique resource for analyzing cultural and linguistic biases in VLMs. We evaluate a broad spectrum of 8 models, from proprietary closed-source systems to open-weights medium and large-scale models. Our experiments reveal substantial performance gaps, underscoring the limitations of current VLMs in culturally diverse contexts. By centering cultural diversity and multilinguality, IndicVisionBench establishes a reproducible evaluation framework that paves the way for more inclusive multimodal research. Our benchmark is publicly available at https://huggingface.co/datasets/krutrim-ai-labs/IndicVisionBench.

Published as a conference paper at ICLR 2026
INDICVISIONBENCE:: BENCHMARKING CULTURAL AND MULTILINGUAL UNDERSTANDING IN VLMs
Ali Faraz', F Akash", Shabarukh Khan', Raja Kolla', Akshat Patidar',
Suranjan Goswami2. Abhinav Ravi', Chandra Khatri', Shubham Agarwal!
Krutrim AI, Bangalore, India
2 2OLA Electric, Bangalore, India
Contact: {all.fars rajakolla, shubbaam.garwalll } olaakuurimmcom
ABSTRACT
Vision-language models (VLMs) have demonstrated impressive generalization across multimodal tasks, yet most evaluation benchmarks remain Western-centric, leaving open questions about their performance in culturally diverse and multi- lingual settings. To address this gap, introduce large-scale benchmark centered on the Indian subcontinent. Covering English and 10 Indian languages, our benchmark spans multimodal tasks, including Opti- cal Character Recognition (OCR), Multimodal Machine Translation (MMT), and Visual Question Answering (VQA), covering 6 kinds of question types. Our fi- nal benchmark consists of a total of 5K images and 37K+ QA pairs across 13 culturally grounded topics. In addition, we release a paired parallel corpus of annotations across 10 Indic languages, creating a unique resource for analyzing cultural and linguistic biases in VLMs. We evaluate a broad spectrum of mod- els, from proprietary closed-source systems to open-weights medium and large- scale models. Our experiments reveal substantial performance gaps, underscor- ing the limitations of current VLMs in culturally diverse contexts. By center- ing cultural diversity and multilinguality, Indic VisionBench establishes a repro- ducible evaluation framework that paves the way for more inclusive multimodal research. Our benchmark is publicly available at /tatp:::///hgginngfaceeee co/dataasts//krutt rim ai labs http//nndiciiissiBBeooo.o
the first
INTRODUCTION
languala models (VLMs) (Bai et al. 2023 Chen et al. 2024 Lu et al. 2024 Wang et al.
2024b Laurençon et al. 2024 Tong et al. 2024 Xue et al. (024) have demonstrated strong perfor-
mance across a variety of multim multimodal tasks. However, existing benchmarks (Antol et al. 2015 Fu
al. 2023 Goyal al. 2017) remain heavily Western-centric, limiting our understanding of how these models generalize to culturally diverse and multilingual settings. India, in particular, represents one of the most culturally and linguistically diverse regions globally. with 22 official languages and 28 states plus 8 Union Territories". each with distinct ethnic, visual, and cultural identities. While some recent efforts partially cover this diversity (Romero et al. 2024; Nayak et al. 2024] Vayani ett al. 2025), systematic, large-scale benchmark capturing India- specific cultural concepts across multiple languages is still lacking.
To address this gap, we introduce IndieVisionBench. culturally grounded evaluation benchmark tailored for the Indian subcontinent. To the best our knowledge, this is the first large-scale bench- mark explicitly designed to assess VLMM in the context of Indian culture and languages. We use states as a proxy for cultural groups following prior works (Adilazuarda et al., 2024 Nayak et al. 2024). IndicVisionBeee
comprises 5K unique images and 37K+ question-anseet answer pairs spanning 13 cultural topics, covering English and 10 medium-to
resource Indic languages supporting three multimodal tracks: Visual Question Answering (VQA), Optical Character Recognition (OCR), and
India

---

Published as a conference paper at ICLR 2026
Multimodal Machine Translation (MMT). Figure 1 illustrates examples reflecting diverse cultural nuances, including monuments, food, and digitized text. Rigorous human verification and correc- tion at every stage of data collection ensure the reliability and cultural fidelity of the benchmark, covering medium-to-low resource languages including Hindi, Bengali, Tamil, Malayalam, Telugu, Marathi, Kannada, Gujarati, Punjabi, and Oriya.
In this study, we evaluate state-of-the-ar (SOTA) VLMs on Indic VisconBench and find that per- formance drops considerably for low-resource languages and culturally specific content. We also observe a clear gap between proprietary and open-source models in their ability to capture linguistic and cultural nuances across multimodal tasks. Analysis across scripts and language groups further highlight the need for better support and representation of underrepresented regions and cultures.
Crowd Sourting
1110
100
IS-VQL Farallol
1502
Gharodte
LLM borsiion
Mamun Corsobar
MS-VDA-EN
Inages
87G
IvB OCR
OCR Cuulity
Image Sources
Quality
IndieWiioonench
IndiVisionBenchhhe
IndicVhiion Bench--Onnn
Log len
indernt rng
wil Acrn theea Hi fultu fox wi Ro hew Solgga
dem
ryfor mug tyttt sunc dudlut Mooul of
Figure 1: Becc (IVB) pipeline and 3 tracks. Top panel illustrates our image col- lection pipeline for 10 Indian languages, showing the number of images at each step, with human quality checks applied throughout. We also present sample outputs for the three tracks: VQA (Vi- sual Question Answering) in English, MMT (Multimodal Machine Translation) in Telugu, and OCR (Optical Character Recognition) in Punjabi. Further details are provided in Section
Our contributions could thus be summarized as follows:
We propose Indic Bench as the first large-scale, Indian-centric benchmark for evalu- ating VI.Ms on culture-specifi understanding, involving OCR, recognition, cultural identi- fication, multi modal translation and semantic understanding involving 5K unique images.
We conduct comprehensive evaluation of 8 prominent closed-source as well open- weight models supporting Indian languages and contrast their performance across the tracks. We highlight systematic performance gaps that underscore the limitations of current general-purpose VLMs in culturally diverse settings.
We systematically study the lingual variation in performance. Our benchmark and relevant code are publicly available for future research in this direction.
biases, performance across topics and cross-
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Vision Language Models and Benchmarks. Cross-ateention models (Alayrac et al. 2022 Singh et al., 2022) and later visual instruction luning based auto-regressive models like the LLaVA fam-
ily (Liu al. 2023a 2024). have advanced multimodal learning, where vision encoders (Radford
et al. 2021] Zhai et al. 2023: Tschannen et al., (225)) are aligned with large language models. This
approach has since influenced range of VLMs (Lu et al. 2024 [aurençon al. [024] Tong al.
2024 Xue et al. 2024 Team et al. 2024). which follow similar design principles and achieve strong
results on translation, captioning. and multi-tuma vision language benchmarks (Hudson & Manning. 2019 Fu et al.) [023] Yu et al., 2023). In contrast, multimodal models that handle Indic languages remain relatively underexplored. Most open- systems provide support only for 2 to 4 medium-
resource Indian languages (Maaz et al. [2224] Alam et al.) 2025 Yue et al., 2025). with the notable exception of Chitrarth (Khan et al. 2025b), which extends coverage to all ten languages, considered in this work. We include all these models in our benchmark to assess their relative strengths.
Optical Character Recognition (OCR), OCR has progressed from early rulc-based engines such as Tesseract (Smith 2007) to modern transformer-based approaches like TrOCR (Li al., 202T and docTR (iao et al. 2023). Recent efforts in document understanding further leverage multimodal
architectures, including the DocOwl series (Hu et al.) 2024a b), DocLLM (Wang al. 2023), and
Donut (Kim et al.] 2022). These systems are typically evaluated on benchmarks such as RVL-
CDIP (Harley et al. 2013), FUNSD (Jaume et al. 2019), and Doc (Mathew al. 202T). to
name a few. Despite this progress, existing OCR benchmarks are largely English-centric offering
minimal coverage of Indic scripts and multilingual contexts.
Multimodal Machine Translation (MMT). Recently, Multimodal Machine Translation
(MMT) (Calixto Liu 2017] Elliott Kádár 2017 Delbrouck & Dupont 2017 Yao & Wan
2020) has gained traction, where the translation leverages auxiliary modalities (e.g.. images). Prior
works have largely centered on
language pairs (Elliott et al., 2016 [Specia al.)
2016), with subset of medium-resouree Indian languages (particularly Hindi, Bengali, Malayalam)
also explored in the shared task series (Nakazawa et al. 2019 2020 2021 2022 2023) based on
Visual Genome images (Krishna et al. 2017). We support similar task based on diverse set of cultural images avoiding potential data contamination issues (Balloccu et al. 2024
Cultural VQA. Several benchmarks have begun probing cultural and multilingual reasoning in VLMs. ED-VCR (Yin et al. 202I) and Henna (Alwajih et al. (2024) emphasize culturally specific content but are largely limited to English Arabic, while WorldCuisines (Winata et al. 2025) focuses on food and cuisines. Multilingual benchmarks (Liu et al. 2023b] Zhang et al. 2023 Sun
et 2024 2022 Das et al.] Wang al. al. 2024aa) Fu al. (2024)) expand language coverage but often lack cultural and task diversity. Datasets like MaRVL (Liu et al. 202I) and xGQA (Pfeiffer et al.)
2 broaden multilingual reasoning but do not incorporate Indic cultural grounding. Closest to our 2024). and ALM-Bench (Vayani al. 2025), which partially touch on India- aedia-specific contexts, yet none offers a unified framework capturing both Indic cultural diversity and multilingual multimodal evaluation.
work are CVQA (Romero et al. 2024). CulturalVQA (Nayak et al.
3 BENCHMARK CREATION
Figure illustrates curation pipeline across all tracks; additional details are provided below.
3.1 INDICVISIONBENCH VQA
We constructed the VQA split using two approaches: (i) controlled crowd-sourcing and (ii) large- scale web crawling. In the first phase, we recruited volunteers (including authors) who contributed images captured on their personal devices along with corresponding annotations. These images were further reviewed to determine whether they were culturally specific to Indía and, if so, mapped to one of 13 predefined topics and to the relevant State/Union Territory (UT). Irrelevant images were discarded, resulting in 615 valid samples. As several categories and regions were underrepresented, expanded coverage in the second phase, where cultural experts systematically collected Creative
3

---

Published as a conference paper at ICLR 2026
Sar the over fo uod Becrmeth-e
Figure 2: Examples from Indic Vision VQA. Illustrative samples from different regions are shown on the left. The map the right depicts the regional distribution of images across India, with counts per State/UT. Further details are provided in Section of the Appendix.
Commons-licensed images" from Google Search, targeting roughly 100 per State/UT across the same categories. This yielded 3,502 additional images, bringing the total corpus to 4,117 (3,797 region-specific and 320 pan-India).
Each image was first annotated with concise keywords by humans, expanded into intermediary synthetic detailed captions using VL.Ms English, and then used to generate six QAs per image: two short-answer, one long-answer, one multiple-choice (single-correct), one True/False, and one adversarial question. Notably, adversarial questions incorporate false assumptions, requiring models to explicitly reject them, enabling systematic probe of cultural knowledge beyond surface-level recognition. We employed Gemini-1.3 1.5-iih and Gemini-2.5 -la-h (Gemini Team, Google 2025) for QA generation, informed by small pilot study and cost considerations (see Appendix; Table I7 Human reviewers then refined all outputs for factual accuracy and cultural alignment, resulting balanced set of open-ended queries that jointly test recognition, reasoning. and robustness in
VLMs. Guidelines provided to annotators are detailed in Appendix E while Figure 22 shows the
annotation interface.
From this pool of 4K+ images and their corresponding 6 QAs, we translated subset into the dom- inant regional language using text-only Gemini call, followed by human correction, resulting in an VQA-Indic version. Additionally, we sampled disjoint set of 106 images and translated them into all 10 Indian languages, creating a VQA-Parallel corpus to systematically study cross-lingun variation in VLMs' cultural understanding and robustness.
3.2 INDICVISION MMT
The Multimodal Machine Translation (MMT) track extends the 106 images from the VQA-Parallel corpus, where each English caption was translated into 10 Indic languages with access the image context. All translations were manually annotated to preserve meaning and align with cultural nu- ances, resulting a multimodal parallel dataset tailored for evaluating vision-grounded multimodal translation in medium-to-low resource Indic languages.
3.3 INDICVISIONBENCH.OOO
For benchmarking OCR performance. we construct a multilingual corpus from Wikisource (Foun- dation 2025). public-domain repository of digitized literary works. The corpus spans 10 Indic languages and includes both printed and handwritten styles. To ensure reliability, we restrict collec-
icnnsss/

---

Published as a conference paper at ICLR 2026
tion to Level-4 verified pages, which have been human-reviewee the platform. For each page, pair high-resolution seans (prppageimage) with their corresponding verified text (pagetext). Further implementation details are provided in Appendix B
(h)
NAN
64)
(4)
Captlcy Outriatan
V0K. beg knmeer Coure
Heahi 131 548-4332
0C0 Prence NoT CorfEe Le0g
0.20
Coaden
Figure 3: Data analysis on IndicVision Distribution of VQA questions by category (a) and by language excluding English (b); average word counts for questions (c) and answers (d). For MMT (c) shows caption word counts in Hindi; and for OCR average words per language (f).
4 INDICVISIONBENCEE (IVB)
Indic VisionBench provides a diverse evaluation suite across 13 India-centric topics in English and 10 regional languages. Among Indic languages, Hindi dominates with 26.8% of QA pairs (Figure 3). For MMT, Hindi captions average 131 words, while OCR track word counts vary more widely, with Hindi (329) and Gujarati (247) highest. F that the dataset spans diverse cultural categories, with largest shares in Heritage (12.4%), Religion (11.2%), Architecture (11.1%) and Food (8 6%). More details Appendix D
Benchmark Tracks: Indic VisconBench consists of three evaluation subsets: i). OCR: 876 docu- ment images across 10 Indic languages. ii). VQA: 4,011 English and 1,007 multilingual culturally disjoint
grounded images with 6 QA types each. We also benchmark cross- lingull performance set of 106 images with 6 paired questions across English and 10 Indic languages. iii). MMT: 106 image-caption pairs translated into 10 Indic languages, enabling multimodal translation.
Models Evaluated. We evaluate three families of VLMs with varying degrees of Indic language support: i). Proprietary models: Gemini-2.5 Flash (Gemini Team, Google 2025), GPT-4o (OenAAl
2023). ii). Large open-weight VLMs: Gemma-3-27B (Team et al.) 2025). Maae----avvrrick weight VLMs (7B): Maya Nahid Alam et al. 2024). PALO (Mazz al., 2024), Pangea (Yue et al.) 2025). and Chitrarth-1
17B (LLaMA-4 for brevity) (Metal 2025). iii). Medium-scale
Khan et al. 20255). For the OCR subset, we additionally include closed-source Chitrapathak de-
signed specifically for Indian languages as well as open-sourced Surya (Paruchuri & Team (025)
model. For MMT, we additionally include Chitranuvad (Khan et al. 2025a). winning entry of the
English-to-lowre MMT 24 (3 Indian languages) shared task (Parida et al. 2024).
https: http:://wwww.e0 statmt. .rg. 24 imoda 1owresmt- taatttttahhhhth..
5

---

Published as a conference paper at ICLR 2026
Evaluation Metries We assess model performance using combination of deterministic and judgment-buud based metrics, tailored to each task. In the VQA track, Exact Match (refer Table 2T] used for multiple-choice and True/False questions, while short/long-anse answer and adversarial ques- tions are evaluated using A.M--s-a-Judde (GPT-40, 0-10 scale) following prior works (Vayani et al. 2025) capture contextual and cultural appropriateness (prompts in Appendix [F). For the MMT lask, we evaluate performance using BLEU (Papineni et al. 2002) and RIBES (Isozaki et al.) 2010) scores across ten Indic languages, following the setup of prior shared tasks (Parida et al. 2024 For OCR evaluation, we follow OCRBenchv2 (Fu et al. 2024b) and report Average Normalized Lev- enshtein Similarity (ANLS) (Biten et al. [2019) along with Word Error Rate (WER) and Character Error Rate (CER) standard 'metrics (Smith, 2007; Neudecker et al.) 2021). We report ANLS as main metric which is more robust to outliers and also provide detailed discussion Section
Table I: Model performances on English QAs in IndicWisionBeech--Ok. Average scores of different models for the six question-types. MCQ and True/False are binary (0-1), while Long Answer, Short Answer-1, Short Answer-2, and Adversarial descriptive questions use a 0-10 scale. The best score is shown in bold, and the second-best is underlined.
Model
MCQ f
True/False
Long-answer
Short- I f
Short-2 f
Adversarial 1
Maya
0.69
0.71
6.98
5.00
5.50
0.16
PALO
0.72
0.43
7.12
5.51
5,81
0.19
Pangea
0.85
0.37
7.01
6.72
6.95
0.67
Chitrarth-1
0.81
0.68
7.53
6.22
6.33
0.03
LLaMA-4
0.87
0.92
8.55
7.98
7.91
2.62
Gemma-3
0.87
0.88
8.56
7.68
7.61
1.50
GPT-4o
0.90
0.91
8.75
8.19
8.02
2.95
Gemini-2..
0.94
0.95
9.30
8.58
8.49
5.79
RESULTS
VQA: Table [ reports results on English subset of cultural VQA task. Gemini-2.5 achieves the highest scores across all 6 question types, with GPT-4o and LLaMA-4 as the strongest challengers. Binary-style questions (True/False, MCQ) yield the highest accuracy, while long- answer questions also show robust performance. Short-answer types remains harder, reflecting the difficulty of con- cise factual recall. This pattern highlights how answer format modulates model performance. In multilingual settings. Gemini-2.5 continues to lead overall, while LLLMA-4 and Gemma-3 exhibit comparable performance with language-specific strengths. GPT-4o consistently lags behind these models, followed by the 7B variants. Among the 7B models, Chitrarth-1 generally outperforms Pangea for short and long answer questions, with the latter holding an edge for MCQ and True/- False questions. (Figure 4 Table g in Appendix VQA-Indic). Adversarial questions, which embed false assumptions, remains the most challenging both in English and Indic (Tables [nd 2 Though Gemini-2.5 consistently outperforms all models, even its scores notably lower compared to other QA types, reflecting the increased difficulty. On these select questions, GPT-4o is distant second, while both Gemma-3 and LLaMA-4 struggle across the board. We also conducted paired statistical significance analyses to verify the stability of the reported model comparisons (refer to Section the Appendix).
MMT: Gemini-2.5 also dominates the MMT track, with LLaMA-4 and Gemma-3 performing com- parable across most languages in Table based both BLEU and RIBES metrics. LLaMA-4 attains second-best results in Bengali, Kannada, Malayalam, Odia, and Punjabi, while Gemma-3 ranks second in the remaining languages. Malayalam proves most challenging, with the sub-par per- formance across all models. Chitranuvad, finetuned version of Chitrarth-1 on Visual Genome (Kr- ishna et al. 2017) for image grounded translation of English into 3 languages (Hindi, Bengali and Malayalam), outperforms the base model Chitrarth-1 in Hindi, Kannada, Malayalam, and Telugu but lags in Bengali despite being specifically fine-tuned for Nevertheless, both Chitranuvad and Chitrarth-1 substantially outperform other 7B baselines (Maya, PALO, Pangea).
OCR: We report ANLS scores in Table median WER/CER in I0 Gemini-2.5 leads across all languages and metrics, achieving SOTA performance both the word and character-level. OCR

---

Published as a conference paper at ICLR 2026
Figure 4: Model performances on Bisionneennnnhh- VQA-Paralled Average scores across lan- guages for the three open-ended (long and short) questions (on left) and scores across languages for the structured tasks (True/False and MCQ) on the right.
difficulty remains language-dependent, with higher scores for Malayalam (59.64), Odia (41.7), Tel- ugu (33.32), and Gujarati (24.09), underscoring persistent challenges Indic scripts. Surya ranks second in Gujarati, Hindi, Kannada and Marathi, while Chitrapathak ranks second in Bengali, Tamil and Telugu. For the other languages, the two models show complementary strengths, with shifts whether word or character level ANLS is considered. Surprisingly, GPT-4o performs poorly in OCR with word-level ANLS scores (e.g.. 94.67 in Malayalam, 90.54 in Gujarati) significantly below expectations while 7B open-source models fall further behind.
in rank depending
Across all evaluation tracks, the closed-source Gemini-2.5 demonstrates clear superiority, while Gemma-3 and 1.aMA-4 show notable strengths with observed disparities across languages and question types. We show qualitative results and more details in Appendix C
Table 2:
Model performances for Adversarial Questions in Indic Visioneenen---eee-eeee VQA. We report the average scores for only top 4 models since scores of other 7B models approached to O. Even proprietary models perform poorly on these kinds of hard and challenging questions.
Model
Bengali
English 1
Gujarali 1
Hindi 1
Kannada
Malayalam
Marahhi 1
Odia
Punjabi 1
Tamil
Telugu
LLMA-I
0.38
2,02
0.52
LT8
0.14
0.33
0.3]
0.33
1.03
1.11
0,07
3
T07
135
0037
T.60
100
0.77
0.68
0.50
256"
"E13
GPT-40
2.23
2.95
3.10
2.25
0.67
2.28
2.89
188
4.00
1.70
2.04
2.5ini-2.5
5.17
3.79
2.94
4.46
3.17
3,32
4.84
3.92
831
5.15
2.73
Table 3: Model performances on Indic VishniiiiBB......M..... RIBES (R) and BLEU (B) scores across ten Indic languages, with Gemini-2.5 achieving the highest performance consistently.
Model
Benzuli
Gujarati
Ilindi
Kanaada
Malayalam
Marnthl
Odia
Punjabi
Tamil
Teloga
R
B
R
B-
R
B
R
B
RT
B1
R
B.
R
B
R
B
R
B
R
B
Maya
0.45
5:48
0.00
18:00)
PALO
0.41
4.58
0.58
11:19
Pangea
0.00
10.51
0.75
25.29
0.15
5.40
0.62
12.59
Chinaetb-1
0.76
21.80
0.72
21.07
0.71
21.00
0.65
12.85
0.59
7.49
0.70
16.25
0.62
11.10
0.30
10.39
0.71
17.59
0.7
15.60
Chimmnnad
0.74
18.13
0.68
18.66
0.74
21.36
0.60
12.95
0.60
7.36
0.00
14.74
3.6
0.66
0.37
1.61
0.67
15.85
0.71
16.56
LLaMX-4
0.82
10.20
0.80
29.84
0.51
39.35
0.26
20.01
0.22
13.96
0.76
20:9
0.22
15:35
0.55
41.1
0.80
25:22
0.78
2233
Gemma-3
0.51
20.75
0.8%
35.30 26
0.62
36.40
0.77
16.25
0.08
10:29
0.80
26.96
0.6
836
0.51
32.05
0.67
20.97
0.82
11:35
CPP--4
0.50
78.00
06
0.71
21.00
0.70
35.55 30
0.07
11.25
0.5g
816
0.75
25.19
0.65
9.12
0.75
55.75
0175
16.77
0.71
T7.5
Clmmini-2.5
0.87
44.51
8.90
53.27
0.53
38.91
0,80
30.05
0.91
28.65
0.88
45.00
0.65
39.08
0.89
$2.39
0.88
46.32
0,87
44,85
DISCUSSION
VLMs without vision: Are images necessary? We evaluate models on the paired VQA-Parallel corpus spanning 10 Indic languages plus English, comparing performance with and without visual input. Removing images leads to a substantial drop in accuracy, most pronounced for short-answer tasks (see Table 5) where precise, detail-oriented responses are required. Long-answer questions comparatively more resilient, though still affected. Across the representative models of each category: Chitrarth- 1, Gemma-3, and Gemini-2.5, the trend is consistent (Table D in Appendix), showcasing that visual grounding is necessary for answering questions in VQA benchmark.
7

---

Published as a conference paper at ICLR 2026
Table 4: Model performances on IndicVisionBench-OCR: ANLS (Average Normalized Leven- shteii Similarity) across 10 Indic languages for different models. ANLS-W and ANLS-C denote word- and character-level scores, respectively. For each language, the highest score is marked in bold, while the second-highest is underlined. Gemini-2.5 performs the best followed by Chitrap- athak most languages.
Model
Deagsl
Gejoon
Iildi
Kinada
Mauysan
Marahi
G4
Paighi
Tavil
Thllgs
Werl
Wes
Werl
Clar
Werl
Clerl
Weoll
14.51
01:15
04:36
Paggs
14.05
30.33
73:33
91.35
19 64
96.3
93.05
9151
LG3054
51.2
-3255
00.3
15733
-0.37
3330
-033
5530
-6.3
2.8
1338
9:31
-300
1235
31558
0.5
55Y0
(irera
21.41
40.00
85.17
51.14
07.06
73.6
40.01
1100
00.87
70.7
70.8
47.65
16.51
66.76
1923
Surye
09.36
1751
11.11
13:46
11.11
8.14
21.4
4.13
17:8
16.32
13.01
4.3
16.00
13
2.31
771
17
32.05
02.37
3862
1334
6718
94.07
06.83
13.90
08.88
4070
74.35
2158
1.79
381
31,03
Table 5: Are images necessary for in short-answer questions across languages for Chitrarth-1, Gemma-3, and Gemini-2.5, comparing with vs. without image input.
Average performance drop
Model
Type
Bengali
Bnglish
Cujand
Hind
Kantada
Malayalam f
Marati f
Odia
PunjaN
Tamil
Telego
ing
3.99
4.50
4.06
ing
5.76
5:97
5,01
5.11
5.50
G
"626""
"470"
-725
7.31"
5.0
447
"4765"
ing
6.08
7.08
6.87
6.29
6.41
6.68
5.94
6,80
0.92
6.03
GumaieE5
willfnn fing
7.57
7.60
"4.0
withing
8.09
7.90
8.33
7.57
7.89
7.00
7.72
8.15
7.56
7.76
Do VLMS exhibit cross-lingual variations in performance? We systematically conducted study on the VQA-Parallel corpus to measure the cross-lingual performance across 11 languages including English. For long-answer questions, Gemini-2.5 attains the highest overall performance, followed by 3emma-3 and LLaMA-4. On the MCQ type questions, GPT-44 and Gemma-3 perform comparable across all languages (next to Gemini-2.. 2.5)) as in Table 7 In the adversarial question, Gemini remains the best-performing model but shows considerable drop in performance compared to its long-form results. Excluding English, Gemma-3 consistently outperforms both GPT-4o and LLaMA-4 across all Indian languages, while LLaMA-4 maintains slight advantage over Gemma- 3 in English. For short-answer questions. LLaMA-4 generally secures the second rank, but falls behind Gemma-3 in Tamil and Telugu. Nonetheless, both LLaMA-4 and Gemma-3 outperform GPT-40 across all languages. In the True/False setting, GPT-4o ranks second in Punjabi, Telugu, and English. By contrast, Gemma-3 shows notable weaknesses in Bengali, Punjabi, and Telugu, even trailing behind Chitrarth-1 in Punjabi and Bengali.
Do VLMs perform better in some cultural topics in English? Model performance also varies by cultural category. As shown in Figure 5 Gemini-2.5 consistently achieves the strongest results across topics with slight variations, establishing itself as the most reliable model. LLaMA-4 and Gemma-3 show advantages on certain topics, while GPT-4o retains slight edge in others over both. Among 7B models, Chitrarth-1 and Pangea demonstrate moderate and roughly comparable capabilities, whereas Maya and PALO cluster together at the lower end. These topics-level patterns suggest that stronger models generalize more evenly across cultural domains, while weaker ones exhibit sharper inconsistencies.
Do VLMS know more about certain regions show regional-language biases? We investigate whether VLMs exhibit region-specific strengths or biases by comparing performance across cultural images from different Indian regions (states) and the corresponding multilingual queries. Gemini- 2.5 (Figure 6) generally performs well across regions but consistently struggles with Odia cultural content, regardless of query language. Across models, English questions yield the best results, followed by Hindi and Bengali, with no clear alignment between the region depicted in the image and the language of the query. Among open-weight models (Figure 7 in Appendix), Gemma-3 favors Punjabi, Marathi, and Bengali, while LLaMA-4 performs best Punjabi and Hindi content. 7B Chitrarth-1 records its lowest scores on Odia and Punjabi and often performs better in English than in native Indic languages for Hindi-speaking states. Pangea performs strongest in English and weakest in Tamil. Maya and PALO remain relatively stable in English but show weaknesses in Hindi and Bengali. These results suggest that while certain region-language preferences exist, systematic
8

---

Published as a conference paper at ICLR 2026
Food
Horiago
7.8%
CARAYS
82%
21.3%
8%
Uterarure
Religion
3,9%
35%
4.6%
4.5%
Doonemy
Vedia
Peggon
Sports
Figure Performance across topics IndicVisionBench VQA. Distribution of categories of questions (on left) and model performances averaged over the two short and long answer open- ended questions (on right). Gemini-2.5 shows comparable performance across all topics.
6/1
Figure Average performance on open-ended question (Long and Short answer types). Gemini-2.5 (on left), GPT-4o (right) and other models in Figure 7 in Appendix. X-axis displays query languages and Y-axis displays Indian states grouped by dominant language. Numbers in parentheses indicate the count of states for dominant language.
9

---

Published as a conference paper at ICLR 2026
region-level cultural alignment is largely absent, with Odia emerging consistently difficult case across all models.
How do we evaluate OCR outputs? Apart from the ANLS metric, we also report average and median WER/CER metrics (Tables [T and 12 in the Appendix), based on Levenshtein Distance [Levenshtein 1966). Mathematically, this metric is unbounded and can over-penalize models for few extreme cases (e.g., inflated scores for LLaMA-4 repetition upto maximum length; see Figure 8). To quantify this effect, we report in Table 13 the proportion of instances exceeding value of 1. Notably, L4 accounts for only 7% of such cases, yet includes strong outliers with an average worst-case WER of 25 in Malayalam, while still ranking third best under ANLS which remains relatively robust to these anomalies. While underexplored for LLMs and VLMs, that often produce long repetitive outputs (Hiraoka & Inui] 2024), even median-based reporting (Patel al., 2025) fails to capture such edge cases. Other statistical alternatives like Word Recognition Rate (WRR) / Character Recognition Rate (CRR) (Bhattacharyya et al. 2025) ignore ordering in the outputs, so adopt ANLS as the most interpretable metric in our setting.
7 CONCLUSION
We present Indic Besionnesnn, large-scale benchmark consisting of 5K unique images, 37K+ questions spanning 13 culturally grounded topics across English and 10 Indic languages. Covering VQA (6 kind of questions), OCR, and MMT tasks, it combines curated images with linguistically diverse queries to probe recognition, reasoning, and translation. Experiments with proprietary and open-weight models reveal substantial performance gaps, especially in low-resource languages and culturally nuanced settings. By centering cultural and linguistic diversity, our work provides producible foundation for building more inclusive and globally robust multimodal systems.
ETHICS AND REPRODUCIBILITY STATEMENT
Ethics Statement This work focuses on the responsible development of an evaluation benchmark for multimodal cultural understanding in regional Indian contexts and languages, spanning diverse tasks. We applied careful filtering reduce harmful or unsafe content, though model outputs remain beyond our full control. All external datasets and tools are properly cited. Human involvement was limited to annotation and quality control; no sensitive or personally identifiable information (PII) was collected. Participants were informed that their contributed images and annotations would be used in a VLM benchmark and provided prior consent, with instructions to obscure any identifiable information. Dataset curation performed by team of in-house annotators who were fairly com- pensated according to local market standards. As the study did not involve personal or medical data, formal IRB approval was required. Throughout the process, we prioritized preserving cultural nuance while minimizing bias and harm. Despite careful filtering, dataset bias may remain, reflect- ing regional, socio-cconomic, or cultural imbalances. The resulting benchmark aims to support the development of multilingual and culturally inclusive vision-language models.
Reproducibility Statement To support reproducibility, have released all benchmark-relatee artifacts publicly, along with detailed documentation. Our experimental setups and evaluation pro- tocols are thoroughly recorded to facilitate precise replication of results. For components involving human annotation or judgment, we include the instructions and guidelines followed, ensuring trans- parency and consistency throughout the process.
ACKNOWLEDGMENTS
We express our sincere gratitude to the leadership at Krutrim for their unwavering support through- out the course of this research. We would also like to thank the Al Research team at Krutrim for their valuable feedback and insightful discussions during various stages of the project, well the Krutrim team members who contributed to data collection. We further acknowledge the dedicated efforts of the data collection and annotation teams, including Sanmathi and Aravind, for their work in building this benchmark. Our experiments were conducted with generous computational support from Krutrim Cloud using Krutrim credits.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Al- ham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling" culture" in llms: A survey. arXiv preprint arXiv:2403. 154/2, 2024.
Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze- Yin Chan, SM Uddin, Shayekh Bin Islam, et al. Behind maya: Building multilingual vision language model. arXiv preprint arXiv:2505.089 08 2025.
Jean-Baye SannBBptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Bart, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.0103.1 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1425-2233, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-Ve VL: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušck. Leak, cheat, re- peat: Data contamination and evaluation malpractices in closed-source 1lms. arXiv preprint arXiv:2402.03997. 2024.
Shaon Bhattacharyya, Souvik Ghosh, Prantik Deb, Ajoy Mondal, and CV Jawahar. Adapting vision- language models for hindi ocr. In International Conference Document Analysis and Recogni- tion, pp. 577-594. Springer, 2025.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 4291 13301, 2019.
lacer Calixto and Qun Liu, Incorporating global visual features into attention-based neural ma- chine translation. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceed- ings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 992- 1003, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1100 URI. htRlhhhts::/eeet antho ogy. oggy.org//D1711100
Soravit Changpinyo, Linting Xue. Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. MaXM: Towards multilingual visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2667-2682, 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXhv:2404.1682) 2024.
Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. arXiv preprint 1rXiv:2403.10372 2024.
Jean-Benoit Delbrouck and Stéphane Dupont. An empirical study on the effectiveness of images in multimodal neural machine translation. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Process- ing. pp. 910-919, Copenhagen, Denmark, September 2017. Association for Computational Lin- guisties. doi: 1.. 1111186/11/D1111100 URLhtttp.. ceteeaannthllo wtw.olatoooooooooooooooooo hagehholooyyorrg/111119999

---

Published as a conference paper at ICLR 2026
Desmond Elliott and Åkos Kádár. Imagination improves multimodal translation. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 130-141, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL /tap:::://aaaathoooo www.ohooolyyoor.............
T17-1014
Desmond Elliott, Stella Frank, Khalil Sima'au an, and Lucia Specia, Multi30k: Multilingual english- german image descriptions. arXiv preprint (oXXiv:665.00450 2016..
Wikimedia Foundation. Wikisource: The free library.
https://en.wikissureee
wiki/Main_Parte 2025, Accessed: 2025-07-09.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models, 2024a. URL httpthtss//asiii.r/////////
2306. 13394
Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.0032/1 2024b.
Gemini Team, Google. Gemini 2.5: long context, and next generation agentic capabilities, 2025. URL https://arxiv.or
Pushing the frontier with advanced reasoning, multimodality,
abs/2507. 06261
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913 2017.
Stig-Are Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, et al. The memad submission to the wmt18 multimodal translation task. arXiv preprint arXiv: 0008./0802. 10802, 2018.
Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference Document Analysis and Recognition (ICDAR), 2015.
Tatsuya Hiraoka and Kentaro Inui. Repetition neurons: How do language models produce repeti- tions? arXiv preprint arXiv:2410. 13497, 2024.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403. /2895, 2024a.
Anwen Hu, Jiabo Yc, Qinghao Ye, Haiyang Xu, Ming Yan, Yuhao Dan, Qian Qi, Junfeng Tian, Ji Zhang, and Fei Huang. mplug-docowls High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03440, 2024b. 2025.
Drew A Hudson and Christopher D Manning. GQA: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In Proceedings the 2010 Conference on Empirical Methods Natural Language Processing, pp. 944-952. ACL, 2010.
Guillaume Jaume et al. Funsd: A dataset for form understanding in noisy scanned documents. arXiv preprint arXiv:1905 13538, 2019.
12

---

Published as a conference paper at ICLR 2026
Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala, Ashish Kulkarni, Chandra Khatri, Abhinav Ravi, and Shubham Agarwal. Chitranuvad: Adapting multi- lingual Ilms for multimodal translation. arXiv preprint arXfv:2502.20420, 2025a.
Shaharukh Khan, Ayush Tarun, Abhinav Ravi, Ali Faraz, Akshat Patidar, Praveen Kumar Pokala, Anagha Bhangare, Raja Kolla, Chandra Khatri, and Shubham Agarwal. Chitrarth: Bridging vision and language for a billion people. bttrg//bb/25552 15392
com/c ol
2025b. URL httts:://aaxttt wwtparwwwwwvvoaaa/oa/a.aaaa
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park. Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In European Conference on Computer Vision (ECCV), 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International jourrna of computer vision, 123(1):32-73, 2017.
Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia. Sheffield submis- sions for wmm18 8 multimodal translation shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 624-631, 2018.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building Vision-language models? arXiv preprint arXiv:2405.02246, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024. httlhhttt:///a/t wwwpiii.orr//baae wtt.orgggaaa.......///// 02246)
VI Levenshtcin. Binary coors capable or 'correcting deletions, insertions, and reversals. In Soviet physics-doklady, 1966.
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhou- jun Li, and Furu Wei. Troer: Transformer-basee optical character recognition with pre-trained models. arXiv preprint arXiv:2109. 2021.
Haofu Liao, Aruni RoyChowdhury, Weijian Li, Ankan Bansal, Yuting Zhang, Zhuowen Tu, Ravi Kumar Satzoda, R. Manmatha, and Vijay Mahadevan. Doctr: Document transformer for structured information extraction in documents. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19584-19594, 2023.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. arXiv preprint UrXiv:2109.13255 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 26296-26306. 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 230.02307..6681, 2023b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vi. towards real-world vision-language understanding. arXiv preprini 240:22403005522 2024.
Muhammad Maaz, Hanoona Rasheed, AbdeIrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad Khan. Palo: polyglot large multimodal model for 5b people. arXiv preprint arXiv:2402. 14818, 2024.
13

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209 2021.
AI Meta. The Ilama 4 herd: The beginning of new era of natively multimodal ai innovation. https:/aii. meta.
model, arXiv preprini mati://mmm---immmmmmmamaaenlllie
checked on, 4(7):2025, 2025.
Karthik Reddy Kanjula Nahid Alam et al. Maya: An instruction finetuned multilingual multimodal
2024. doi: 10448550/rX..2212.0011
httppt::
Toshiaki Nakazawa, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Nobushige Doi, Yusuke Oda, Ondřej Bojar, Shantipriya Parida, Isao Goto, and Hidaya Mino. Proceedings of the 6th workshop on asian translation. Proceedings of the 6th Workshop on Asian Translation, 2019.
Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Win Pa Pa, Ondřej Bojar, Shantipriya Parida, Isao Goto, Hidaya Mino, Hiroshi Manabe, Katsuhito Sudoh, Sadao Kurohashi, and Pushpak Bhattacharyya (eds.). Proceedings of the 7th Workshop on Asian Translation, Suzhou, China, December 2020. Association for Computational Linguistics. URL https://acllan www.thhooo.....oo ogg. wttp:/00.wttt1... wat-1.0
Toshiaki Nakazawa, Hideki Nakayama, Isao Goto, Hideya Mino, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Shohei Higashiyama, Hiroshi Manabe, Win Pa Pa, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, Yusuke Oda, Katsuhito Sudoh, Sadao Kuro- hashi, and Pushpak Bhattacharyya (eds.). Proceedings of the 8th Workshop Asian Trans- lation (WAT2021). Online, August 2021. Association for Computational Linguistics. URL attps::/accaanthlo.ss org/ 2021. wat-1.0
Toshiaki Nakazawa, Kazutaka Kinugawa, Hideya Mino, Isao Goto, Raj Dabre, Shohei Higashiyama, Shantipriya Parida, Makoto Morishita, Ondrej Bojar, Akiko Eriguchi, Yusuke Oda, Akiko Eriguchi, Chenhui Chu, et al. (eds.). Proceedings of the 9th Workshop Asian Translation, Gyeongju, Republic of Korea, October 2022. International Conference on Computational Lin- wwt/oo/2/2222... wat-1.0)
guistics. URL https:
/tap::://ccclac http:///aaaaathooooo
Toshiaki Nakazawa, Kazutaka Kinugawa, Hideya Mino, Isao Goto, Raj Dabre, Shohei Higashiyama, Shantipriya Parida, Makoto Morishita, Ondrej Bojar, Akiko Eriguchi, Yusuke Oda, Akiko Eriguchi, Chenhui Chu, and Sadao Kurohashi (eds.). Proceedings of the 10th Workshop Asian Translation, Macau SAR, China, September 2023. Asia-Pacific Association for Machine Trans- lation. htRtsttts:
anthe wrg. /2023.2223...2 wat wat-1.0
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hen- dricks, Karolina Stańczak, and Aishwarya Agrawal. Benchmarking vision language models for cultural understanding, 2024. URL https: /ttp/aaxvvv.../////v//eaaeaaaava org/ /000//2407 10920
Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopou- los, and Stefan Pletschacher. A survey ocr evaluation tools and metrics. In Proceedings of the 6th International Workshop Historical Document Imaging and Processing, pp. 13-18, 2021.
OpenAl. Gpt-4v, 2023. Retrieved from https: //tp::///cdnccoo openai
coompppers//PPP http//hhh..ppeeeaccccccppppeerrp
DysstemCardd.pad Card.pdf]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pp. 311-318. ACI., 2002.
Shantipriya Parida, Ondřej Bojar, Idris Abdulmumin, Shamsuddeen Hassan Muhammad, and Ibrahim Sa'id Ahmad. Findings of wmt2024 english-to-low resource multimodal translation task. In Proceedings of the Ninth Conference on Machine Translation, pp. 677-683, 2024.
Vikas Paruchuri and Datalab Team. Surya: A lightweight document ocr and analysis toolkit, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2026
Tanvina Patel, Wiebke Hutiri, Aaron Yi Ding, and Odette Scharenborg. How evaluate auto- matic speech recognition: Comparing different performance and bias measures. arXiv preprint arXiv:2507.05885, 2025.
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xgqa: Cross-lingual visual question answering. arXiv preprint arXiv:2109.06082, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763 8763. PmLR, 2021.
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, al. CVQA: Culturally-diveree multilingual visual question answering benchmark. arXiv preprint 1xXiv:2406.05967 2024.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 2022.
Ray Smith. An overview the tesseract ocr engine. Proceedings of the Ninth International Con- ference on Document Analysis and Recognition (ICDAR), pp. 633. IEEE, 2007. doi: 10.1 109/ ICDAR. 4376991... BeLLattts:///eeeppppep
Leee. org http///ctttt/3////3c
Lucia Specia, Stella Frank, Khalil Sima'an an, and Desmond Elliott. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Trunslation; Volume 2, Shared Task Pupers, pp. 543-553, 2016. URL https:
77 7/www. ac web. org/ antho cochooooyyy wggy//ll. W1 6 2346/
Hai-Long Sun, Da- Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, et al. Parrot: Multilingual visual instruction tuning. arXiv preprint
arXiv:2406.02539 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08299 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova. Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer. Xichen Pan, et al. Cambrian-1: fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.
Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, and al. All languages matter: Eval- uating imms on culturally diverse 100 languages, 2025, URL
2411. 16508
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. Doclim: layout-aware generative language model for multimodal document understanding. arXiv preprint arXiv: 2401.00908. 2023.
Hongyu Wang, Jiayu Xu, Senwei Xie, Ruiping Wang, Jialin Li, Zhaojie Xie. Bin Zhang, Chuyan Xiong, and Xilin Chen. M4u: Evaluating multilingual understanding and reasoning for large multimodal models. arXiv preprint arXiv:2405.1 200.:25355..38. 2024a.
15

---

Published as a conference paper at ICLR 2026
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song. Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong. Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2024b. CtLhtts::// http:://aaxivvvroooee
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80-33, 1945.
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, and Worldcuisines: massive- scale benchmark for multilingual and multicultural visual question answering on global cuisines, 2025. URLlttpps:// www.aawwwwv.gaaaaeeaeeea htw.aa///211..20000
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), ACL, pp. 6153-6166, Online, August 2021. Association for Computational Linguistics. doi: 10. 106133/111/2211 www.///ooo......oonoo.o.ooooo aacl-long.48). URL
ogy. org 2021 secl-llone 480
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408. 08872, 2024.
Shaowei Yao and Xiaojun Wan. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp- 4346 4350, 2020.
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. arXiv preprint arXiv: 200.:006000000000 2021.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. MM-VET: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0m 02490, 2023.
Xiang Yue, Yueqi Song. Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A fully open multilingual multimodal L.M for 39 languages. The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL
org.i/.orrgaaao abs/
22410. 16153
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 11975-11986, 2023.
Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. MBexam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484-555 5505,, 2023.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A LIMITATIONS
Indie covers English and len medium- to low-resource Indic languages across 13 cul- turally grounded topics, but some limitations remain. Language coverage is still limited for some of the lowest-resource languages, and topic diversity could be further expanded to cover additional cul- tural contexts. Human annotations and translations are usually subject to interpretation, especially for cultural nuances, which may introduce inconsistencies or simplifications. Additionally, since parts of the dataset were seeded using LLM-generatee drafts, even though these were extensively rewritten by annotators, a small degree of induced self-enhancement preferential bias may still persist. We also acknowledge that existing evaluation metrics and LLM-based scoring may not fully capture cultural grounding or multimodal reasoning, highlighting opportunities for more nu- anced evaluation approaches. Although we systematically study the impact of visual modality for VQA, we have not yet explored this effect for the MMT track with prior works (Grönroos et al.) 2018 Lala al. 2018 Wu al. 202T showcasing minimal impact of visual modality, We plan to cover this as part of future work.
B IMPLEMENTATION
Most of our implementation is based in Python and we use the HuggingFace library in Py PyToc (aszkee 2019 Wolf al. 2019). We evaluate broad range of frontier and open-source mod- els on the Indic
For the IndicVisonneema VischnBench benchmark using their respective APIS as well as HuggingFace pages. subset of our evaluation, we used the implementation
OCRBench V to compute the ANLS metrics. We adapted this script to report both word-level and character-level ANLS metrics. Beyond OCR, our full benchmark includes additional evaluation dimensions. We compute metrics such as BLEU where we do not have an additional pre-processing step of tok- enization, RIBES Exact Match (EM), and employ LLLM-as-a-Judae evaluation to capture semantic and lask-level correctness. We official Batch APIs to get the results from GPT-4g for LLM-as- a-ju a-judge results and Gemini-2.5 for annotations.
OCR benchmark. We began with dumps from Wikisouree from the official Wikimedia snapshots [T] Each compressed XML dump was parsed to extract page titles, which were then converted into canonical Wikisource URLs. From the harvested URLs, fetched the corresponding Wikisource pages, retained only pages with images and applied filtering retain only those marked as proofread by the community (quality level 4), ensuring high-fidelity ground truth. For every verified page, we used the porp-page-image tag to collect the page scans and the <pagetext> tag extract the corresponding OCR text, which reflects the latest human- edited annotation. This pipeline, implemented with custom parsing code and filtering, yielded linguistically diverse dataset of high-quality scanned documents paired with verified text, which forms the foundation of the OCR evaluation track in IndicVisionBench
10 for Indic languages, obtained
eval
sacrebleu.py
https://platfomm www.oo.enc....o. copeaai.cmm/ae http://cs//g/geee/e/eot/eoee
"GP440-0024008006 is the model used as LLM-as-a-Judge.
16
hettas:/hi.wikissee
http::/
17

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
We would like to highlight that IVB-VQA consists of both web scraped as well as crowdsourced images. These user-contributed images are not indexed the web and thus do not appear any public pretraining corpus directly. On this fully unseen subset, Gemini-2.5 again achieves the highest performance, mirroring the overall trend (see T).
Table Model performances on English QAs on the subset containing crowd-sourced images of Indic Visieiiininnnc-ve VQA.. Average scores of different models for the four open-ended question- types: Long Answer, Short A Short Answer-2, and Adversarial. The scores are on scale of 0-10. The best score is shown in bold, and the second-best is underlined. We find similar trend as the whole VQA dataset.
Model
Long-answer
Short-
Short-2 t
Adversarial
Maya
6.80
5.28
6.45
0.06
PALO
6.98
5.35
6,51
Pangea
6,42
6.76
7.48
0.41
Chitrarth-1
7.54
6.79
7.28
0.04
LLaMA-4
8.64
8.43
8.45
1.56
Gemma-3
8.74
8.47
8.45
0.99
GPT-4o
8.78
8.40
8.31
0.28
Gemini-2.5
9.50
9.27
9.09
4.84
Table 7: Model performances Indic mndieeisioonBecc--OO----P
Comprehensive scores
across languages, question types, and models for Indic VidicVissioBennch www.eeenncc-A-aa-alaaaalaaaaa
O Q-Pyp
Model
lieggali
English
Gujarati
Hindi
Kannada
Malayalam
Marathi
Odla
Funjabi
Tamil
Teluga
Maya
0,402
01632
0.375
PALO
0.604
0.800
0.660
Pangea
0.783
0,840
0.349
0.670
0,764
Chimmt--1
0.736
0.755
0.611
0.736
0.774
(Exact Match,
LLaMa
"5502
0
0.822
0.502
103
07744
0000
0302
10775
Gemmu:3
0358
10705
00735
0.933
0.972
PAL.O
0.310
Pangea
0.770
0.670
0.50
0.640
0.790
True/Faise
Chimanth-3
0.560
0.730
0.630
0.450
0470
0.421
0.566
0.632
0.004
0.349
01340
(Fxact Match, out of 1.
LLMMA-4
0.806
0.877
0.830
0.86
0 0.506
0.858
0.025
0.868
0.792
0.21
0. 0.702
Gemma-1
0.547
0.868
0.858
0.066
0.785
0.95
0.99
0.425
0.585
0.896
0.557
GPT
0.502
0.913
0.8:9
0366
0321
0.302
D387
00032
00377
USTF
0.79"
Gemiis
0.972
0.981
0.943
0.945
0.43
0.93
0.553
0.925
0.15
0.953
11935
Maya
3.538
6,915
6:177
PALO
2.557
7,057
52117
Pangea
6.585
7.058
7.009
5.06
5.887
Long (LIMaa-Julge,co
Chirauth
7.443
1.547
7.311
7222
7,406
1.472
7.511
7.443
LLMMA
8.217
3300
8.87
7894
8202
Gemmu:3
7,800
8.764
3.142
3,066
Pangea
4.557
6,170
5.445
3,066
4.93
Shot-ansee 1
Chiranth-1
5,806
755
5.02
5.575
467H
5613
5.114
5.43L
4.95
5,500
(LLM-as Judge, out 10)
LLMMO
7,387
73
608
238
Gemma-1
6.570
6.981
7.078
6668
6.503
6406
K373
6802
6.055
6.14
GPT
6.726
7.54
6.023
7075
55396
5583
6319
5333
Genais
8.694
8.217
7,8%6
8.330
7.566
7,877
7.991
7.317
8151
7.962
7355
Maya
3.462
5.094
4.472
PALO
2.774
5.472
4,28
Pangea
5.340
7.255
5.785
3.236
4.396
Shert-nnewer
Chimmoh-1
6.519
6.085
792
56604
5.449
5330
5.698
5.651
4.053
5.670
6066
(LLM as Judge, out 10
LLaM
7.611
3,238
7.019
7755
"T.n
11.22"
6335
"7332
6335
Gemma-3
7,170
7.179
7.125
6.423
6.785
8.994
8.236
8151
0.255
PALO
0.133
Pangea
Adversiiaal question (LLMaax-Julg cut 10,
0.000
0.094
0.000
0.844
0.000
0000
0.077
0.42
LLaMA-4
Gemmai
1.123
1,506
2,179
1,387
0.840
1915
2:349
1,500
0.55
2243
0953
2:226
1.547
0.21
-915
0.406
0991
2422
0.49
2,085
25006
0.660
GPT.6
0.532
1,109
0.708
07715
07726
0.05
10822
U0225
117266
Genini
4.745
6.142
4962
5.528
4441
4.91
4.575
4004
4660
4.670
5,019
18

---

Published as a conference paper at ICLR 2026
Guiee
6.00
230
A71
652
KC2
431
609
6:50
5:80
6.4
140
6.1
635
454
117
0.M
1:5
210
TM
747
7M
740
335
175
125
A47
754
611
211
3.
345
15A
175
T4
S4T
217
T:4
730
741
KM
745
K35
112
TM
740
611
250
235
340
440
125
117
A
Makyyten
1:20
271
740
144
612
1.53
1.25
125
Makyyllen
7:30
238
74T
353
100
1:00
Meral
6:44
2.50
840
112
622
F13
229
250
184
4.07
117
1.2
621
612
6:00
3.70
SCE
4:04
0.31
8:31
6:2
539
4:28
3.40
6M
017
X:2
504
142
1.0
6.4
AG
0.33
6.44
830
X11
6.11
109
6.5
1.54
6/5
1.7
F1
7
764
2.9
1:39
1:25
0.1
7.5
9:21
7.5
7.25
1.39
1:5
1.2
321
1:50
6.0
FM
7M
M41
4:1
4:4
1.58
6:25
144
741
749
152
6:2
1:4
5.4
of
KM
611
F14
AZ1
417
4.7
KM
471
405
425
475
4.0
140
A17
688
2M
402
100
4:33
347
330
600
K0
318
4.0
138
60
340
A2
342
4AC
30
447
1/2
280
631
Makyoten
CM
SW
6/A
342
38
20
10
Menat-D0 20-
6.54
5:M
SK1
640
6.5
50
6.29
Masderd
10
18
K37
6/1
SM
5:3
541
4.7
4:55
555
2
4.50
1.2
SM
181
18
6.2
6:
TM
6.7
6.1
411
43
25
412
120
142
13
18
331
624
KCB
401
AM
10
621
400
12
1:0
6.11
121
111
K3C
114
1.5
10
14
3.29
Sepaiit)
Spsorti
3:1
1:5
1.0
3:51
1.1
3:
1.3
135
1:33
4.20
Emelutt
313
121
1.5
1:
471
1.1
133
421
1:0
17
18
1M
Moeki))
KM
Cdutt
18
454
Sdlit
134
1M
Nyaint
180
1.0
671
Nepant)
1/4
1M
113
14
1M
1.1
1M
Megr
171
Average scores on the three open-ended questions (Long and two Short) for six models across different languages (X-axis) and images corresponding to Indian states grouped by primary language (Y-axis). Left to right, top row: Gemma-3-27B (left) and LLaMA-4 (right); middle row: Chitrarth-1 (left) and Pangea (right); bottom row: Maya (left) and PALO (right).
Figure 7: Model performances on Indie BisiionBeecch-
19

---

Published as a conference paper at ICLR 2026
Table 8: Model performances on Indic VisiionBenc---QA--n-ne Comprehensive scores acrOss
languages, question types, and models.
O-Typ
Model
Bengali
Gajarati
Hindi
Kannada
Malayalam
Marathi
Odia
Punjahi
Tamil
Telagu
Mayn
0.435
0.638
PALO
0.467
0.576
Pangea
0.733
0.822
0.670
0.768
MCQ
(Exact Match, out of
LEaMie
0.700
0.87
0.502
D335
0.36
DX3y
1039%
Gemma:3
0.817
0.839
0.44
0.78
0.783
0.19
0368
0.710
0.296
0.570
GPT-lo
0.750
0.613
0.839
0.19
0733
0.35
0.68
0.306
0.715
'03122
Gemini-25
Maya
PALO
0.317
0,714
Pangea
0.717
0,746
0.585
0.725
True/False
Chitrath-
0.600
0,452
04115
0.374
0.500
0.514
0.558
0.548
0.277
0.377
Motch
LeaMA
0.830
0.80
0389
078T
0.06
0330
1036
'0370"
Gemma-3
0.581
GPT-66
0.833
0.732
0.842
1066
1250
D.22
0.71
DDGC
0.799
Gemini-25 2.5
0.917
0,871
0.924
0.788
0.867
0.973
0.868
1.000
0.915
0.925
Maya
3,867
6.504
PALOO
3.150
Pangea
Long answer
Chitrasth-
7,233
7.484
7.547
7.609
7.433
7.351
7,553
7,200
7.298
7,290
(LLM-as- Judge, of 10,
LLaMA
8.050
8.290
8,482
8.489
8.267
8.456
7,763
8.452
8.245
$.000
Gemma:
5:400
8.613
8.498
8,576
8.317
8.511
7,337
8.613
$,189
8:217
GPT-fo
K.283
W.487
8.181
834
7366
7.755
7311
Gemini-2.5
8,883
9.226
8,988
8.944
Maya
2.117
4.199
PALO
2.733
1,984
Pangea
4.617
5.772
4.160
4.768
Shoet (.--Mas-Jadge, out
Chitratth-1
5.550
TraMAc
7.71
Gemma-3
6.783
8.032
7.578
7.158
6.417
6.666
6,526
7,516
7.266
7.101
GPT-4o
7,500
7.355
8. 6.000
77309
6300
63665
6.1.8
7105
7222
57580
Gemini 2.5
8.117
8.355
8,540
8.734
7,000
8.486
8,263
8.355
8.245
8.116
Panges
4.983
6.188
4.213
4.622
Shoet answer
Chitrarth-
5.883
6,613
6,212
6.009
5,567
6.358
6.132
6.002
6.468
6.087
(LLM-as Judges out 10,
LLaMA 1
6.917
7,581
7,663
1635
6917
7338
6.47
7.742
6396
7217
Gemmuu3
T350
Gemini 2.5
7,950
8.484
8,268
8.691
8,083
8.730
8,289
8.452
8.117
8.188
Maya
0
0,188
PALO
0.017
0.111
Pangea
Advenarial
Chiratts11
LL-M-ss-Judge, out 10,1
LEMMAT 0.583
0.516
1.179
W.333
030
UU366"
110032"
"1.13
"0072"
Gemmu-3
1.667
0.968
16666
1,002
0.767
0.676
0.895
2.935
1881
1.130
GPT-4o
2,233
3,097
2.248
0.600
2.781
2,892
18816
4.000
1,702
2.033
Gemini-2.5
2.935
2.725
Table 9: VQA with and without image. Chitrarth-1, Gemma-3, and Gemini-2.5 on IVB-VQA-Paralle across languages, evaluated with and without image input.
Average scores on long-answer type questions of
Model
Type
Bengali
English
Gujane
Hind
Kintada 1 Malayulun
Maratli
Odia
Pujjabi
Tumil
Telugu
Chitaahh
0.50
6.37
6.79
6.62
6.69
6.67
6.52
0.59
6.00
0.78
6.05
wath mmp
7.44
7.49
7.55
7.31
7:29
7.11
7.47
7.51
6.87
7:14
7.56
Geñoma
ing
7.30"
16335
77.30"
7,33
7.53
7:4
7.30
6.33
7700
7.70
"T220"
mng
6.08
6:44
8.10
8:00
8.24
Demum 25-
'6.66"
"6.55"
3.2
5.D
3.27
5.00
3..1
"62r
9.00
8.88
8.76
Table 10: Model performances on IndieeisionBencc-OCCC Median WER and CER scores across Indic languages for various models.
Modd
Kamola
Martii
CER
WERL
CEALL
WERL
Maya
1101
1:
1.00
1381
038
1.00
(153
100
0.56
100
0193
500
005
0.9
100
0.55
1.0
100
1.20
500
1000
009
100
0355
033
032
671
0123
376
011
100
034
002
017
0.53
613
0333
20

---

Published as a conference paper at ICLR 2026
pruteeta anticia ond revine our cre of oon fur cant rapor mes odlec awarrd
suggin wbcr comtt tuak /laerssoerooo
Ground Truih
mituuaa 1ih crrinnaci soe sitryrhhe noth Mechond to mrune nndigh sursing
syetial
Prediction
Figure 8: Observed repetition in OCR outputs. We show an example where LLaMA-4 provided repetitions upto maximum sequence length in the prediction for an OCR example in Malayalam.
Table 1I: Model performances on Indic BndtevisionBeechOOCC Average WER and CER (+ standard deviation) for Bengali, Gujarati, Hindi, Kannada, and Malayalam.
Model
Bengali
Gujanti
Hindi
Kannada
Malayalan
WER
CER
WER
CBR
WER
CER
WER
CER 1
WER
CER
Maya
1,15 a: 0.56
0.99 0.07
2.32 8.94
1.90
2.78
PALO
2.77 + 3.45
2.06 = 2.02
1.58 2.43
1.06
+ 0.50
Panges
1.25 0.0.0
0.99 = 0.59
1.22 1.31
1.07
1.07
Chimrrth-1
1.34 1:0.9
1.07 = 0.64
1.38 1.05
1.00 0:48
1.09 0.41
0.96 0:22
1,37 0.49
0.95 0.133
2,45 7:5:::888 7.82
1.16 1: 0.61
Chrraputtak
0.33
0.08
0.37 0.37
0.22
0.34
0.09
0.38
Ganimas3
0.53.+0.11
1.26-0.1
0.71 9003
0.99
0:44
6.35
034
0.58
1,05 5.4
0776 + 0.15
0.71
0.9
7.62
1.12
Gamini-2.5
0.09
0.33
0.11
9.16
Table 12: Model performances BeCcchiss Average WER and CER (+ standard deviation) for Marathi, Odia, Punjabi, Tamil, and Telugu.
Model
Punjata
Toluge
WER
WFR
WFR
Maya
PAL.O
Pangea
1.37 1
1.02
0.97
0.43
1.29
0.67
0.98 0.35
Chitrarth-1
1.152
0.93 0:21
189
1.50 0.93
1.59 1.20
1.41 1.17
135
1.58
1,09
109-0:56
0.56
1,56
1.27
1.05
0.50
Chirrpathak
0.11
0.7 0.09
0.27
0.16
0.15
0.12
0.54
0.121.11
Gemma:
0.59
0.32 0.13
0.12
1014
0.1
0.6
LLaMA-
0.77
GFT-4
0.76
0.42 2: 0.23
126
1.18
0.4
0.75
0.88
a 0.42
0.7
0.50
122
1.97
0.09
1.63
1,41
098
0.85
Gemini-2.5 5
0.25 0.11
0.03 + 0.02
0.55
0.25
0.21
0.14
0.25
0.15
0.06 + 0.11
0.42 + 0.18
0,85
0.04
0.51
0.29
0.08
0.10
Table 13: Indichisson WER and CER statisties.
Model-wise WER and CER statis-
tics where the scores more than 1. We present the count well as percentage of the examples for each model.
WER >
CER
Model
Count
%
Count
%
Maya
22
2.51
15
1.71
PALO
51
5.82
45
5.14
Pangea
77
8.79
34
3.88
Chitrarth-1
302
34.47
169
19.29
Chirapatthak
8
0.91
0
0.00
Gemma-3
79
9.01
10
1.14
LLaMA-4
68
7.76
28
3.19
GPT-4o
286
32.64
115
13.12
Gemini-2.5
15
1.71
0
0.00
21

---

Published as a conference paper at ICLR 2026
GPT
The degiced this shanl is diancctiisde tecile
image ludit, spesifeally the lannd dtle. liod traliinnal firm of perntesa pruchied by the creed trine, one beged
traditioral degiets practied by the Gond other mdlate tribes incentei Tedlo, pantessarly io the stites Madiya Prudesh, Cheattigurh, and
desigs prutioaa Good people, ene the largen infigeccs communtties Tedia This vinma colen,
india, i'nalsh, (Ssbuttaggrh, parts Maturadlera and Andher Pradeeh...
animale....
Adversarial Question
Chitrarth
The shood in the image Senuree a traditionna femn knmwn paintrng charocterized amumdl, human figanes Gend people. who indigmoua to cestral india, am Iennen beritage
Besod uz the imaagey and stylo visible the insas, the ahant depicis Pilhoes painting, a traditiora
frrm originsting but doe found and pructood other tallea contral India. Waile the Ruthis ane the peiroory croturs,
MI
Madhyn neighoonig
Adversarial Answer Neguland, state Northeal ludia cantrll Indis.
Pangea
Maya
Palo
accciicall region of seocivted with the local rebaal communitvve each ar Gand, who zne known their gybbol
The elosel to the Imege fentree a de-ign than frrm
rentniccent trxditirrll
The-hhend festune traditional Irbbal from central Indas which incudde designs degicting various arinels such os othnn depicted in black and white patterta, showooding antiate style the region. The sheut dhowesse sos the coltural herllage and contomanchip
uial ant from central Iodia spedficlll from suimie inclading skeplants lenda, and people, which common modis in tibell zrt of this region.
surd the image fertone traditional
Prudell
Figure 9: Model outputs on VidiivissinBench-OOOo VQA. We show an example an adversarial question along with the corresponding model outputs.
22

---

Published as a conference paper at ICLR 2026
Srghis
coaleie de tree
the voc cou snitic
antrerct
youn
ooe tolabbe
venur sen mara Stam
welly
yeapee Treps surni
and wall ovens ner Jtfft
oor suraa
one resercttin
de cew
en
cefieeee Pnceetae mver prverit
Figure 10: Model outputs on Indic BinneVisio Example an MMT question and the corresponding responses from multiple models.
Question
Wee GDAd atab åd @DAr sbteet otredd Aestred dootee ckr3d exga dattoo cleerceeaaacceehh
Answer
Bovg eag wct adoo aarddoowss ef dd wase eng, Aodsos slsodd, sct rasteedy car expeze rasereeaabb Nowco eootii rech bocte sjoarrrie)) alssooo wouetàd opsida, sisootie acdedd, edtss,n dus engleze den oi Rotooadd
GPT
Chitrarth
rovad dh 18 alglcee) ao reaaez espifracc abtel wedd codo raco&vico Xa
ol &yot Apt ajdhas aetart abredd diddddc coodlo woresoessch).
Gemini
Gemma
el, goo 3d (0At abre wod atd
sycedyytost aduateresny mastras,coo Bas cttideddd stendstsged racto Bates
(aarl) cosdo dietcceessieea raco boocue globe clO), ejestwes,ppoo
sittddateees exglaae rordineaads
audrettio zs Brawarhd
harteuu racts secco cleatare ajadon wwtceeeeegg.
LJama
ead addod alateletaat 1336056) masy martomma eapiee awrorrt didrdtt dl caatoonns,, stetztevattri tacto Bate Rodraarnnd
Figure 11: Model outputs on Indie VisionBench--OO.. We show an example of an VQA output for corresponding models.
23

---

Published as a conference paper at ICLR 2026
Chitrapathak
Gemini
Bonn Buod hes oot it ceéfiit
vioue was oper wiil it one formy move fireps fix fist an él worn ow qérin ofeer wolk feore will uni usselebve ii it, Servey
fire qen and i-tr in gellee when whe wfeu -2r uff op foon fiflyy fins fen
expl
uys tom gang vori armett tre maenr
eneervg fore unl fait wr fer warr on, ownn go livar aed sevil it wid soure off it rotl refee "stfien
crpee are they arw mark wery apschvire wich en birz edter was off sezeet th -itt thena that mnarerieng thre and the tea trar en zear frest enei ria sally umys drnn yrs ara turt
efft
uhitk ofee
wide on wnd feaiienn alll Bect ef wel stupel fiem and rich
flwg fire
ofjen
iiiht thyeer a gran mnmrwwygg
ang ztzwe gra
aver wflll own
newn sot twrr rng seae oif 2e8 ey sarta un amer atry nystae any -ten watte ant gea gro tem eie gobe fiver ou wel it eu er è wiile of ovu foaa wre you inl waper armm np
ou toly withe offt i werer tver of that the evers ez order it"
gn vre sure
geron spenss say <ith grbbe oun
sga
rift deser "eft Reee, unte "Nuc
by
yart loftr it git Ries, speaa thet fore re anthy-a wrii en affeen el neen if rejmn
h borm
thear
form
the
writ ez refurm wae obe
spPiil en ap-u
thrm &t thai fire spth uuf anit amoot zave bur erra gar rorard ond soma of gatter eit and cirre vae alt ug aftua vigmi unllae una fore more lifiss stnq an soem ten yirtr iree crele are free andk grel und upventt west thifrr eftrr whtch it. vored a el, fiées il
fowm ls Fie yorr ued epyy ar8 agral ier vmpon it samvet yaar rdf wiilist aut "2t5e if sit eay attrae urea 1879 catt and frue que filios finq fivs as we di eove (ou
Htra frord ie thre it arit sprah rerent searet right nortin om tou ag Wiern ureg cpm affil ard flvee. avans miria foy ar am evrn shita therar geu ouf
girin orjer saate are suseirrn wor frae by eflen wrz iifil smrer it she terr it crvmmmaag and the be
arth rath woi
geo it
yuur
fives evnét wid witt
met
angiire wotton
vifiee
it -Httr wrfite ipen
ont yrum thee reyh gream wall au fru it swou star rof the wen aptor, mg areie
was
wiille ofil filen ror ansmi cot they foom arstrs amtter By
saau
spvr
cnt thaee arat. ritt grant of are th wose feem, it cye bayel Hift fuft wath when firs it right uppisstio sypal ottmm oft of
sxgther thmn vtte on ngg aighte whrrr of are di efre
Seer bersf lefin
grtr
dern 8ish ze Bay wif left sprthe stee, Reer of sopien sitea setroeeth th scon if vipn
coff fboeee went wiih yagit mp pree yast
Maya
Palo
Panges
The set keage is appeas to and
à argan aeyer sagen argent urpert argann tird argenz (ht arger wi....
memer withou
sorry to
imuig
ligtte enegs quiley prcoue dupendin clacity Karg--ta
Figure 12: Model outputs on Indic OicC We show an example of an OCR output for corresponding models.
24

---

Published as a conference paper at ICLR 2026
C.I STATISTICAL ANALYSIS ON THE RESULTS
To assess whether Gemini provides statistically significant improvements over baseline models (GPT-4o and Gemma-3) across languages, question types, and corpora, we conduct paired statistical tests on model outputs. For each combination of corpus (VQA-Indic and VQA-Parallel), language, and QA category (Long-QA, Short-QA, and Adversarial-QAl we compute paired score differences between Gemini and each comparison model.
Statistical Tests, We use the Wilcoxon signed-rank test (Wilcoxon 1945) determine whether the distribution of paired differences differs significantly from zero.
Confidence Intervals. To estimate the magnitude and robustness of these differences, we also construct bootstrapped confidence intervals. Specifically, we perform 10,000 bootstrap resamples over the paired differences and compute the mean for each resample. The empirical 2.5th and 97.5th percentiles form the 95% confidence interval for the mean paired improvement.
Findings. Overall, Gemini performss consistently better than both GPT-4o and Gemma-3 across every language and question type we evaluated. For VQA-Indic, the statistical tests produce reli- ably small p-values, and for VQA-Parallel the values are even smaller showing that Gemini's im- provements are not due to chance. The bootstrapped confidence intervals are also entirely positive, reinforcing that Gemini's 's advantage both meaningful and robust (Tables 14 and 15).
The differences in statistical strength stem primarily from dataset sampling. VQA- includes a uniform 106 items per language, whereas VQA-Indic varies widely in sample size (c.g.. Punjabi 31, Odia 38, Hindi 448), which naturally produces wider confidence intervals and weaker p-values for smaller languages. The only instance where significance is not observed is the Punjabi Short-QA comparison between Gemini-2.5 and Gemma-3, where the results do not provide sufficient evidence to establish a clear advantage.
Despite this expected variability, the overall results consistently show that Gemini-22 2.5 outperforms baseline models across languages and QA types, with improvements that are statistically significant, meaningful, and robust across both balanced and imbalanced multilingual evaluation settings.
Table 14: Statistical comparison of models on the VQA-Indic set across languages with p-values and confidence intervals for Long-QA, Short-QA, and Adversarial-Q..
Model A
Model B
Language
Long-QA
Short-QA
Adversarial-QA
n-items
p-value
CI
p-value
CI
p-value
CI
Gemini-2.5
GPT-4o
Punjabi
5e-4
(0.26. 0.71)
20-2
(0.29, 2.35)
2c-2
(0.48, 3.0)
Gemini-2.5
Gemma-3
Punjahi
le-2
(0.1, 0.65)
0.06
(0.0, 1.68)
3e.3
(1.29, 4.26)
Gemini-2.5
GPT-4o
Hindi
1e-31
(0.49, 0.72)
9c-4
(0.26,
9c-21
(1.81,
448
Gemnnn--25
Gemmu-3
Hindi
(0.5,
(0.72, 1.22)
3e-26
3.25)
Gemini-2.5
GPT-4o
Odia
1o-4
(0.53, 1.24)
36-5
(1.26, 3.03)
le-2
(0.63, 3.666
Gomini-2.5
Gemma-3
Odia
1o-4
(0.58, 1.47)
1o4
(0.95, 2.61)
8c-4
(1.55,4.61)
Gemini
GPT-4o
Marathi
le.4
(0.51, 1.62)
3e-3
2.57)
2e-2
(0.32, 3.544
Gemini-255
Gemma-3
Marathi
2e-5
(0.46, 0.97)
le.3
(0.89, 2.76)
le-4
(2.73, 5.7)
Table 15: Statistical comparison of models the VQA-Parallel set across languages with p- values and confidence intervals for Long-QA, Short-QA, and Adversarial-QA.
Model B
Langnage
Long-QA
Short-QA
Adversarial-QA
n-items
p-value
p-value
p-value
Gemini-2.5
GPT-4o
Malayalam
2c-14
(1.07, 1.6)
Se-9
(1.33, 2.53)
6c-12
(3:72, 5.43)
106
Gemini-2.5
Gemma-3
Malayalam
1e-8
(0.51, 1.07)
2e-7
(0.94, 2.05)
4e-7
(187,3.66)
106
Gemnin--2 2.5
GPT-4o
Gujarati
2c-15
(1.0, 1.52)
8e-9
(1.33, 2.41)
2e-11
(3.43,
106
Gemini-2.5
Gemma-3
Gujarati
Ic-11
(0.56, 0.98)
5c-4
(0.33, 1.33)
lo-7
3.95)
106
Gemini-2.5
GPT-4o
Telugu
Ic-13
(0.97, 1.49)
Ic-9
(1.13.2.13
Io-11
(3.42,5.16)
106
Gemini-2 2.5
Gemma-3
Telugu
2e-8
(0.47,
le.3
(0.36, 1.3)
le:5
3.03)
106
Gemini-2.5
GPT-4o
English
Se-12
(0.59, 1.02)
2e-2
(0.12,
Sc-13
(4.2, 5.88)
106
Gemini-2.5
Gemma-3
English
7e-11
(0.58,
1.85)
(3.15, 4.79)
25

---

Published as a conference paper at ICLR 2026
C.2 RELIABILITY OF LLM-AS-A-JUDGE
To remain consistent with prior work Vayani et al. (025) and avoid dependence the Gemini family (which was used earlier for draft generation), we adopt GPT-4o as our L.L.-as-a-judge. To evaluate its reliability and potential bias, we conduct human study on two question types includ- ing Short-Answer (QI) and Adversarial, using the English VQA-Parallel subset (two models; 106 samples), with two independent human annotators.
Across both question types, GPT-4o matches or exceeds human -uman agreement (see Table 16 For Short-Answer QI,
Pearson correlations range from r = 0.66 to r 0.68, while GPT-4o vs. human- mean correlations range from 0.74 to 0.79. For Adversarial questions, human-human agreement ranges from = 0.64 to r = 0.74, and GPT-4o correlates with the human-mean
0.80. These results indicate that GPT-4o achieves human-level consistency, supporting its use as reliable and scalable judge. However, we also acknowledge that LLM-as-a-judge might still reflect some bias where the judge may not fully capture all cultural nuances, introducing a layer of uncertainty.
0.73
Table 16: Correlation results for Human judges and LLM judge (GPT-4o) for Short-Answer (QI) and Adversarial questions. We used Pearson correlation to report results.
Model
Short-Answer (Q1)
Adversarial
Human-Human
GPT-4o Human-Mean
Human- Human
GPT-4o vs Human-Mean
Gemini-2.5
0.66
0.74
0.74
0.80
GPT-4o
0.68
0.79
0.64
0.73
D DATASET ANALYSIS AND BENCHMARK DETAILS
We provide more details about our dataset here. Figure 5 shows that the dataset spans diverse cul- tural categories, with the largest shares in Heritage (12.4%,, Religion (11.2%)..rcciteeenn)) Architecture (1 1.1.1 1%), Food (8.6%), and Lifestyle (8).... The numbers of these categories are provided in Figure 3 with 4698 and 4212 questions for Heritage and Religion respectively. Also, Hindi has the largest share of QA pairs (26.8%) followed by Kannada (11.9% ) and Tamil (9.7%) among Indic languages. In- terestingly, Adversarial questions have the shortest length (15) while their answers have greater length (44) than Short-QAs (18 and 21). Meanwhile in IVB-MMT, the caption length in number of words follows a distribution with mean 131.30 and std 43.52 for Hindi. We expect that the distribution for other languages will also be the same. On the other hand, in the OCR track, the average number of words for different languages vary significantly with Hindi (329) and Gujarati (247) topping the table and Tamil (121) and Malayalam (127) being the least. Please refer to Table
18 for details of the dataset. Moreover, T9 shows the State/UT-wise image distribution of the data set. Figure 13 presents word clouds by category, where the word 'traditional' emerges dominant term across domains, underscoring the cultural grounding of the benchmark. Alongside, category-specific concepts appear prominently e.g., sweet and dish in Food, palace and temple in Heritage, dance and instrument in Music, and buddhist and church in Religion. This distribution confirms that IVB-VQA emphasizes India's traditional practices while maintaining diversity across food, heritage, festivals, lifestyle etc. We also included the adversarial questions in our dataset to probe for the ability reject false cultural assumptions. For example: "What kind of rituals are performed for the Hindu deity Krishna in this temple?" where the image depicts Jain temple, not a Hindu temple. All adversarial prompts were manually validated to ensure the false presuppositions are realistic, culturally plausible, and not trivially detectable. These questions specifically test chal- lenging multimodal reasoning skills: detecting presupposition errors, handling culturally specific counterfactuals (see Figure 9). and rejecting misleading cues.
Following prior work (Vayani et al., 2025), we use LLM-generated seed questions to accelerate the creation of six QA types per image, while explicitly instructing annotators to substantially revise or rewrite any questions lacking cultural grounding. To validate our choice of the Gemini family for this initial synthesis step, conducted small human preference pilot study comparing Gemini-2.5. Flash and GPT-4o across English QA generation and translations (Hindi/Telugu). Gemini provides comparable quality for English QA (48 34 votes) and clear advantages for Hindi and Telugu
26

---

Published as a conference paper at ICLR 2026
translations (40 vs. 25; 55 vs. 23). Given this quality profile and its substantially lower generation cost (see Table 17p. we select Gemini-2.5-Fi as our synthesis model.
Table 17: Cost comparison for Gemini and OpenAl models in OCR and Captioning tasks (rounded to 2 decimals in S). We provide an approximate cost at the time of submission for sample of 1000 images based on the assumptions of input and outputs tokens. Batch APIs are half the price of Single calls. Here Gemini- 2.5-F denotes Gemini-2.5 2.5-hlash and Gemini-2.55 P denotes Gemini-2.5-Pro Our benchmark further involved a multiple for number of languages and questions.
Took
Pimages
Inpot IM)
Uurpet (M)
Baich (Gemini
Singl (Geminl)
Barch (Upen.A):
Single (OpenAl
Gemaazz5
Gemmezsny
GFT-40
GPT-lomm
GFT-36
GPT-oomin
DCR
1,000
008
0.30
0.38
2.34
1:15
4.68
252
2.01
3.4
6.01
Captoning
1,000
008
0.15
0.78
3.54
We emphasize that Indic Vidioniiio uses Indian states as a proxy for cultural groups. The language distribution in the benchmark is therefore direct outcome of our state-wise, culture-firyt sampling strategy, in which each state is mapped its most widely spoken language. Several India's largest states like Uttar Pradesh, Madhya Pradesh, Rajasthan, Haryana, Bihar, Uttarakhand, and Delhi are predominantly Hindi-speaking. As result, when aiming for balanced cultural rather than strictly linguistic coverage, a higher proportion of Hindi QA pairs naturally emerges. All images were annotated and translated according to the primary language associated with their corresponding state. Our design goal was reflect India's geographic and cultural diversity rather than enforce artificial uniform distribution across languages.
Our subset selection follows directly from this state-wise design: each collected image is associated with a specific Indian state or union territory, either through metadata or through culturally identi- fiable attributes. This mapping guided the creation of the VQA-Indic subset, where translated QA pairs were chosen to maintain region-level balance. Many crowdsourced images already contained explicit cultural or regional cues, making them natural candidates for creating languaae QA in the corresponding state language.
The translated subset was constructed to preserve geographic and cultural diversity across regions. Rather than random or convenience sampling, we selected images to ensure broad state-level representation, resulting in culturally grounded, geographically balanced translated set.
VQA-Indic.
VQA-Parallel (106 images). The VQA-Parallel subset was intentionally designed as a controlled
multilingual evaluation slice rather than large-scale one. We selected 106 images such that: (i) all 10 Indie languages are represented without over-emphasizing any specific region, and (ii) all 13 cultural topics are represented. This produced consistent and balanced set suitable for high-quality human translation into 1l languages for controlled cross-lingual comparison.
Table 18: Summary of Vadionisio datasets.
Task
#Images
Languages
Type
OCR
876
10
Image text pairs
VQA-EN
4117
English
6 QA types
VQA-Indic
1007
10
Indie langs QA
VQA-Parallel
106
English+10 English+10
Parallel QA
MMT
106
Parallel captions
27

---

Published as a conference paper at ICLR 2026
Table 19: State/UTT-wise image distribution in Indie VisionBecch--OOOA.
State/UT
Almages
State/UT
Almages
Andaman Nicobar
97
Madhya Pradesh
Andhra Pradesh
107
Mahurashira
128
Arunachal Pradesh
99
Manipur
100
Assam
101
Meghalaya
75
Bihar
Mizoram
Chandigarh
100
Nagaland
94
Chhattisgarh
90
Odisha
116
Duira & Nagar Haveli, Daman & Diu
54
Puducherry
106
Delhi
141
Punjab
108
Goa
101
Rajasthan
131
Gujarat
110
Sikkim
97
Haryana
Himachal Pradesh
99
Telangana
111
Jamnnu & Kashmir
105
Tripura
Jharkhand
94
Utaar Pradesh
129
Karnaank
242
Ultarakhand
112
Kerala
West Bongal
109
Ladakh
99
Pan-India
320
Lakshadweep
D.1 TOPICS COVERED
Topies covered using Indic VisionBench
The categories that have for the crawled images follows:
Food: Iconic regional cuisines and dishes
Lifestyle: Traditional attire, daily routines, and modern practices
Literature: Renowned works, authors, and poets
Music and Dance: Classical, folk, and traditional performance arts
Religion: Major faiths, rituals, and festivals
Customs: Cultural etiquette and greeting practices
Festivals: National and regional celebrations
Heritage: Monuments, sites, and landmarks of historical importance
Economy: Key industries and occupations
Media: Popular entertainment figures, cinema, and television
Architecture: Traditional Art and Architecture
Sports: Indigenous games and traditional sports
Notable Figures: Influential leaders and historical personalities
Table 20: Comparison of existing VQA evaluation datasets with Indie VisconBench. Indic Vi-
sionBench supports multi-lingual tasks compared existing benchmarks.
Dataset
No. Questions
No. Images
Multilinuual?
Task Format
Culturally Diverse Images?
MaXM CCangninyo al 2023
2.142
335
/
VQA
No
GDVCK Tim al. 2021
328
X
VQA
Yes
MaRVL La
2021
5,670
4,914
/
VQA
Yes
CVQA (Ronnor dt 2024
4,560
VQA
Yes
Cultural ViluuratQqaannn Komero di
2024
2,378
2,328
X
VQA
Yes
Vayani
Yes
Visioniichhh
31,74/
-[99.
VQA, TR MAT
"Y6s"
28

---

Published as a conference paper at ICLR 2026
THE Flour
Swee
Sur
face
MilM
punjabn worndresse
Aiievenn Deventraa grad
dance e
Sostune dancer Sed
per Tor mers S yeena
dress t lifee musiC
musical aaaasscaa
sweets
served
darky re
me li sou
acto
hoster
white
tokeup
forh musi cianscinsz
Tumntt
noTe
Kela Inns irttruenn
S
tr tradi Colitiona Lighting
traditi tonal
B
factre ess type actresss
common ha Fo cinema
Indical
per
Performi mingg peer
goldinggss
tradttionaal
pest intr
Stage
Teare
MeR dry wnwggndeentspinnnn. texture
pert orman ce
aesors ee Lival
nodern
Woren
attire folk
for
e gge
(a) Food
(b) Media
(c)
Music
Blayersiva ter E
punjab wrestling Ling Whiite
S stadium: sport played
clothing context tnttetaattre andaman boar dhina
match feld hicobar islands
Raabaddi
event portt tha
apintsiine wereetgrennneee we award
traditiona
Crickettrrr
football game
local natural
proninent
temp ty
archi
type BOD @ paiceel
ceremoy Panny T
common
dark
speci fiee br bridecommon bn
dancer
woman
festivall Lce called
aadi t 1 Lnoo whi
of
te Lehenga dances
guni aattre woner
l'
reime as samese
tual re LELuU
wedd
red hindn
region 1on
curtural
bihu
white
PULLE
S
ecoooomy text
entrance
Tradi
bui Lding
type g an
indus
ecconon
common
F iitttionn ional
Commonadelte
o
oi commonty
activity
triaa
C
boats
e
Sign
witl
dst
(d) Sports
(e) Customs
(I)
Economy
oap Faace
indicati Ve haaaaraaaarrrr Soroem
Clouds saih whire marble
buddhi's
type
traditi
aattre
aglo ned dance cer
onal
religiousp innrrr
emp
statue
e
mahar a asbrran D
be
sultura
formere
I
Ladakh heature
stoneattom
Temple
Fed
piprominet t
surrdöünde
agates
T
tage
structure
pepraacc
water
cditnnly
shrine ma
b
cnu
ü'e
dailty
mill
C
ddo
upper
ch
This
the
s sikh S
cos tuffle Spean
tage
dedeent Ga
hindu
ti pentr per
whi
handu cultural
aarchttecture
(g) Heritage
(h) Religion
(i) Festivals
type black womenarel
mans tami
surt lef totnuummente com
ohial
ed
Materii
cene
Luee C common
speci fic fie
litecary your Saara
ECE
littrattre
uurrraan urbbntty pystanttpe
se lbaard
moder
i wwoaans mans
sc Lothang
shirt S sCr ipt
painting
t
Tonss
modern
grass
phitog at cant
any
diigg
aattire
Fraditional
des Teryelloo egiön
e Laaees
houses
locat
optaia
book
hunlopp
draditi
oun
dionttnnal
cove
fust
e
eee's Con't
trassinng
boo tistinn
prominent tm ur ban worn
saree woman cul fural
benga a whh1
kurta modern
attire
hair
maral Sheat
unchitecture
tria
architccture
(i) Lifestyle
(k) Literature
(I) Architecture
Figure 13: Word clouds of different categories in
We omit the words
"India" & "Indian" in the word clouds to show other important topics.
Table 21: Evaluation metrics used for different tasks in Indic IndicVisionBenhh highlighting deter- ministic and non-deterministic measures along with their rationale.
Task
Deterministic
Non-deterministic
Rationale
OCR
ANLS, WER, CER
Robustness to script
VQA
Exact Match
LLM--as-aJJude
QA accuracy reasoning quality
MMT
BLEU, RIBES
Translation quality
29

---

Published as a conference paper at ICLR 2026
D.2 EXAMPLES OF OUR DATASET
MCQ red Nrot GOEES BT and sndre 5928 wh
A nO
of errsrt
3 oaes Fottut uniit
Short G2 cou ofile Sourged
Short A2
Taxa ant (it covatto light wagn paa
Truse nD way ol usuut
sireralla Long gil
Long A
Hoe lrfty elke sxrad a then saam ale croed ande spper dows mare are sed chert dliit of al and aan an am utit soger and siggh andte was uife apin d8 ute col't snt var cas thlass It gi ant
tht
Adversarial (Rgg sngler 07
Adversartal Foard 1. E.lest likee voril epoe the 2.
Figure 14: Example from IndieVisionBench--OA. for Hindi
HABR MaNadaa und lukhey
A
NaBPPs or MATMAC vilrii
Short Lo0o0 Dio ondll Cgp
A1
"all sls vialo
Short Q2
Short A2
CUR UPNOL 190 soapsspe vid
OPF CANDOS go spee toara seruuooe wnd
TruelF alse
CRpo0 gprcdioals eei Loraua carong
Long
Long A
genglee
ugllgght ins serese pairesus thio pnd Cuandigge onie
200 wll oig boaee saro eotii wanchee 90g00 Ve mamt Wiii gloai del wee the aka muto wboe dilegg HRP dagg 200 indiggi oocnn 2000 trwe eo nas sdbry 18 o0ca ocsed coaroren coll es
Adversarial
Adversarial
Caa Sott coc eb regen Regen Barce
<B Panns saar "aadg pupen goll ets egoo arce couarall coco epo Pasing flcas sroc eyrit URue Cig ohuer Unnrgy Far some wiaal 90P4 IN s0 venue den dherto siivise
Figure 15: Example from Indic IndieVisionBBenc--OAA for Odia
30

---

Published as a conference paper at ICLR 2026
Ecafaan condit Atero purbess aeoia Meraiicaaae aile galae purtu sighliig
Sb bur of son LC aucaaaa aad Diping
TruaFalsa
Short OZ
Shart
Rigi vnl bit qumeated
Reaty
Long
dige Siea dey
Ohe moiea
AdvarantaaA
L.NA Le LLB Kog au ouree adiu
Figure 16: Example from IndicvisionBench--OO for Tamil
Ab Hogpeaeee del orpart Ba saha calea an ArAAnB
Roooii ferm Eni
Parc th
Short A1
Por Ehtee
202 ure siall godbo favo
IN
True/Falso 83d spci
we diloo ule al sprfio
Short A2
TWV
ad lard a
frred
Long
ent les yet bo the pham pa regee Bee ming une the bove end santer comrle antea bolter
Long A
dea
viiva cange
Adversarial
fraloan the
Chagnn beice tyt
sendo Sig tooe tount e forn dert the premy busil
Figure 17: Example from Indie IndieVisionBenchh-00 for Punjabi
eevell Siers wir spptte coo 2re maneti R0gt molee febert
TrusiFales
Short A2
afherma
her aick retrotee sitee Ware snfocaaee newe wo wer fen pndiocaae wivee enyher
rodies veterre soumeff surtia Riche
veturoo With nen
caur vowry smana memee
Figure 18: Example from IndieeissonnBechh-OO. for Bengali
31

---

Published as a conference paper at ICLR 2026
mH oir titere andt Bowert
Short Of
Shart Sitter cieers
mag prrer
Short G2
Short A2
wreee Sort aroe
TruniFalae
a waen Bing
Dowence
zin Brownde herrgt oo teree ache them line Jostng apheaatr mhatd and ofeo andiee antooni parmentt
wanno de pma Whan 803 vnate
gen't und iorss Hight medi lit leg wash ciew ini riginaaie bax With Sramnnn
Adverserial
merrate Pora (il ce 421 anowr inargigg lite
Figure 19: Example from Indic Indiceisionnench--OO for Marathi
Sergall
Hidd
ensor Plarze solre fajio
whe yanrm
Piacc
untecii tee maae dritia
cof rutt gOu gas ca thos tenant
or elll spen will lo surge
etf oot
worter regilli
ven thone onmes othe wart aoicn clfeel)) otherme eim sarves were whee flams caro relsharf The the speeccu
cun we archer ter ri oppicc welh
hapy
otor fot feow twis carnd cath
furn molk Roond Lnt clonk, oat wili reporie cill Fiati is Rira. precic Aayet
sonti gua
rerve eosicii noce sapur cae the sani sasa
Tame!
Magu
Gaanett unros
angoga dosieria enta fakous
cloodd plrsdou
eracoveiyy pendie far suttea corte cohir dktt ta the Stwn Fingdd payl bereen rayonsdods
ausiidd googl
lagins
saace antedl aig cemena) cadieted coeraaii asing
onge pi do dil ooooeetoed stoa
mnnueersi High spiig
urtcurgannc a.haa uovilo got pa
The daadee paddo glack pol.
Figure 20: Examples Indic VisceiiiinnR,,,..OCC. We show documents and the corresponding Ground Truth texts in Bengali, Hindi, Tamil and Telugu.
32

---

Published as a conference paper at ICLR 2026
English
Malayalam
a'uinti mos ragayee
audivardddl sonent ir In lo poge argi ogftri iesiig stanlevenlleggg yne infiiingg pagae crutigae aadibm
cott =fGaanann qare aut
stceedd beard, sky.
woll vaw
clecian
al assrrt stagr seuum, recav oxtion ae aresl leam eeeeeg
resting
direrity or curver Terefllcoo whoot stalle ane cicite greund tire, mone standing wheat tilling barkgrennd Noocher persor's shin partielly visitle aeka the frunse.
cars ocuaal dupara aroargm, sycila auseeve qupuddddd must
whygfare eame won
nee shear vritra ma wey Aierana (pprar)
dusyyou8 aucomeni auul akapamem nemsur to8eeyym Re to inleenn essnyueenn aeganaguns - -iarn and scu mom aocmoo
dee Briatu Garvitr, suarrd sumb patgpkise erdce
Autrea, sgoof, crboa do bass sird donk erjotn, atiiyy Arrdyy heecong doct dis
somd flduas troad Aus dirr
thec addd adaad atgaao
averaon darlyon 2058 fioudo sabiiarroo dbadzed audios stot avergon (ld an gig avess ayt coods abbl chey ard schilaads
Ragan
and toe aana deap venu undalligny
Gracuded borg dugiarli aist acceeftt adlido gomps aovee's gharggen bag Scbraark almerbe an cangani udd jain dhucd shree wiuu Bi ds chioeaiiee aprigna pur the Paa mine. praning and filmee
Iildd
Punjahi
OMla
tge sjo al audi stes Reemn,, vitill musili undi eellle,, a ehen oile oikig sin ael ungg are
will Bi farn, finn clud de yel trei wnd wiihi, vndid dee eu file de theer la sig) Del Et gnlioal wase le ware det
Ph tinig tngge tgant sttnt ciana (252-1p) us spon sale ahi uaal var oo
ban il crede oo tout jet oie ahe on far in vntt and
widh efou it
whut the Riu ltus ald esal whiwe orew o, ehrawl foo on. the the feunt de tife frweent
one ose sonele nes
eut qagPhoo
life
urther
Figure 21: Indic IndicVision WisionBench--MMF benchmark. We show example image and corresponding translations the caption in languages.
33

---

Published as a conference paper at ICLR 2026
sands
Upliaed
bet.teatnnnneeee gaele andhou wh serrccie sudiggack
bortan arguallcac whis nacisoonll stegaca
Upttad doo
Oeed roine endocceer te enfreedatoo the dir and holg
Dend cheter and comoo aberos rnfernt therthe dinate helgnne
Updlent Pherctl
Upileed lnc
whrr planao rssiall nebocardd the fast piefow Stattart
Wiscirdee by mate und frow benand fooing didiore valani dtreiian
Wedderop
updetd
Figure 22: Image QA pairs' correction tool. Interface of the QA pairs' correction tool provided to the human annotators.
34

---

Published as a conference paper at ICLR 2026
E HUMAN ANNOTATIONS
To quantify the extent of human correction our benchmark, we compute the Edit Rate and Average Relative Edit, defined as:
Edit Rate E
Levenshtein Distance (Edited instance, Original instance)
Avg. Relative Edit
Length (Original instance)
where, N is the total number of instances and E the number of edited instances.
Across all stages of dataset construction, human annotators applied extensive revisions to the LLM- generated drafts. Table 22 reports the edit rates and average relative edit scores across the three corpora. VQA-En (English QAs of IVB-VQA) exhibits an edit rate of 18.61% and a high aver- age relative edit of 77.2%, indicating that the edits themselves were often substantial. VQA-Indic shows a markedly higher proportion of edited items (62.5%), reflecting its linguistic diversity and greater variability in the initial drafts, with particularly large edits for MCQs (86. 6)) and Short- QAs (58.18 1%). The MMT corpus exhibits an edit rate of 79.1% and an average relative edit of 21. 21.6% per caption, underscoring the extensive human refinement involved the translation process. A relative edit value of 128% for Levenshtein distance shows that the Short-QA corrections were sub- stantial, often exceeding the length of the original text and reflects major rewrites rather than minor adjustments.
Table 22: Corpus-level human annotation statistics: overall edit rates, overall average relative edit, and per-question-tyyp average relative edit.
Edit Rate (Overall)
Avg. Rel. Edit (Overall)
Corpus
Avg. Relative Edit (per QA type)
Short-QA 128.9%
Long-QA
Adv-QA MCQ
TVF
VQA-EN
18.6%
77.2%
66.5% 37.2%
54.4% 22.9%
53.5%
31.1%
VQA-Indic
62.5%
48.9%
58.1%
86.1% 1
25.2%
Across languages, Hindi shows the highest edit rate (Edit rate 73.7%, 33.2% Avg. relative edit), followed by Kannada (69.7%,30.9%), Bengali (57.6%, 37.2%), and Punjabi (55.0%, 29.8%). No- tably, Bengali yields the highest average relative edit, while Hindi has the largest edit rate, with the remaining languages showing moderately high but consistent correction levels. All these statis- ties highlight the extent of human intervention and confirm that the final benchmark content differs substantially from the initial -enerrtte drafts.
We also report the demographics of our annotation team in Figure 23 For the VQA subset, crowd- sourced images were collected from 18 contributors, including some of the authors, who provided photographs spanning diverse cultural contexts. This group comprised 13 male and 5 female par- ticipants (ages 20-50). The QA correction and translation phase was carried out by a team of 25 annotators responsible for refining, validating, and translating all QAs across languages. This team included 13 female and 12 male annotators (ages 20-40), with 8 annotators for English, 3 for Hindi, and 1-2 annotators for each of the remaining Indic languages represented in the benchmark.
35

---

Published as a conference paper at ICLR 2026
(a)
Distribution image Proviciers
Gender Distribution Image Providers
10-
0-
Court
Court
6-
2-
20-30
3D-40
40-50
Female
Gender
(c)
Age Distribution Annotators
Gender Distribution Annotators
14
COunt
10
Count
2-
2-
20-30
30-40
Male
Temmall
Garnder
Figure 23: Demographic distribution. We shows the demographics of crowd-sourced image contributors and QA annotators: (a) Age distribution of image providers, (b) Gender distribution of image providers, (c) Age distribution annotators, and (d) Gender distribution of annotators.
36

---

Published as a conference paper at ICLR 2026
Annotation guidelines for the Indic VisionBench Cultural QA Dataset
The task involves enriching image-caption pairs with culturally grounded question-anss answer (QA) annotations. The focus on determining whether an image depicts Indian culture and, if so, generating diverse QA pairs that capture cultural specificity.
Scope
Annotators assess cultural relevance (e.g., attire, festivals, rituals, food, architecture) and generate QA pairs to support training and evaluation of multimodal models.
Captions
Each image is provided with human-annotated keywords and an auto-generated caption. Keywords are fixed; captions may edited for accuracy. Both should be used as context when forming QA pairs
Annotation Process
an image does not depict Indian culture:
Set depicts indian culture = False
no Leave other fields unchanged.
Set wtt__seeecfccccuuuuree
an image does depict Indian culture:
Update caption if necessary.
Provide the following QA annotations:
Short Answer QAs: Two concise cultural QA pairs.
* MCQ: One multiple-choice question with one correct option and three plausible distractors (exact string match required between answer and option).
True/False: One fact-based cultural question with answer True False.
Long Answer: One descriptive QA pair (4-6 lines).
State-Specific Culture
the image reflects a specific Indian state, record the state name; otherwise enter no.
Adversarial Question
Each culturally relevant image requires one adversarial question designed to include plausible but incorrect cultural assumption.
Dos:
Specific, confident, culturally relevant, and misleading but realistic.
Speculative ("Is this...?" ), vague, hedged, or trivial. Examples: Mistaking Aipan art for Bikaneri art, Pongal for Eid, etc.
Don'ts:
Bad Irrelevant Images
I Low-quality, generic, or culturally irrelevant images are marked as:
depicts_inde indianndiiiic -eadiurne Falsc
state specific cealffrcccuuureeeee = no
Checklist
Verify cultural relevance.
Mark depicts_indian iatiss_iddin__nnee _euture and caatesppccffcccuttrre appropriately.
relevant, complete all QA fields.
Do not leave fields empty.
Note. Annotations remain brief, factual, and faithful Indian culture. Careful, consistent labeling ensures the dataset's reliability for benchmarking multimodal models.
37

---

Published as a conference paper at ICLR 2026
Indic Beec Image Collection Guidelines
The goal is to curate culturally relevant, open-license images across Indian states for the In- Bisiinnnnee vision-language benchmark. Each state-category pair must contain at least 10 high-quality images that satisfy the following requirements:
1. Cultural Authenticity
Images should accurately reflect Indian cultural elements tied to the specified state.
2. Category Relevance Each image must belong to one of the predefined categories (e.g., food, literature, festivals).
3. Geographic Specificity Content should clearly associated with a specific Indian state.
Open Licensing
Only images under Creative Commons licenses permitting commercial use are eligible.
Collection Process
1. Formulate search queries in the format: "<category>in <state>> (e.g., Music in Nagaland, Traditional sports in Kerala).
2. Apply Creative Commons usage rights filters.
Manually inspect results for authenticity and verify license details.
4. Select as many images category-state pair as possible, organize them into subfolders. and record metadata (filename, source URI., category, license). Exclusions
Low-quality, blurry, watermarked, stereotypical content.
Images not clearly tied to culture or state.
Content with unclear or invalid licensing.
Target Categories
Food, Lifestyle, Literature, Music & Dance, Religion, Customs, Festivals, Heritage, Economy, Media & Entertainment, Architecture, Sports, Notable Figures. Submission
Each state folder is expected to contain least 100 images across categories, with metadata file. For states with limited available material, collect as many culturally relevant images as possible. All submissions uploaded to the shared repository.
38

---

Published as a conference paper at ICLR 2026
F
PROMPTS USED
Instructional Prompts for Each Question Category
Long Question:
{Question} Answer the question in detail in (target language.
Short Question:
{Question} Please provide brief, clear responses in (target_languag) language.
Adversarial:
{Question} Answer the question in detail in (target language language.
Multiple Choice (MCQ):
Striet Instruction: Respond with only one choice in the format <A>, <B>, <C>, or <D>, Do include any explanation, reasoning, or extra text.
{Question} Your question here Choices:
A. Option 1
B. Option 2
C. Option 3
D. Option 4
True/False:
Striet Instruction: Respond with only [lang.true} or false). {Question} Your question here {lang true) or Hlan-falsss
Choices:
OCR:
Extract the exact text from this image using OCR. Respond with only the text.
Multimodal Translation:
(Question} Answer the question in detail in (target language) language.
We release all prompts used in our study. These include one prompt for generating four QA types (Long, Short, MCQ, and True/False), and separate dedicated prompt for adversarial QAs. We design adversarial prompts independently because these questions are more challenging and require detailed instructions. We also include prompts used for evaluation via the LLM-as-a-judge frame- work, where responses are scored on a 0-10 scale. Furthermore, we provide the prompts we use for each type of question during response generation from different models being evaluated.
Prompt for judging short answer questions via LLM-as-aajudge:
You are a judge evaluating how well a Vision-Language Model answers short-answer type questions.
Evaluate the model's response based on accuracy and correctness with respect to Ground Truth answer. Assign high score when the model's response matches closely with the ground truth and low score when the response lacks knowledge or is unrelated to the ground truth.
Question: {question]
Ground Truth: (ground_trutte
Model response: {predicted answer)
Provide a single overall score from 0 to 10 based on the given criteria. Strictly return only the numeric score, without any additional commentary.
39

---

Published as a conference paper at ICLR 2026
Category classification prompt for crowdsourced images:
You are cultural content classifier. Given the image and the caption, classify it into one or more the following #*top-level categories** Use the definitions to guide your classification, but **only return one category name** (e.g.. "Food", "Lifestyle") that the image can be best
classified into in your answer no subcategories or descriptions.
Food: Iconic regional cuisines and dishes
Lifestyle: Traditional attire, daily routines, and modern practices
Literature: Renowned works, authors, and poets
Music and Dance: Classical, folk, and traditional performance arts
Religion: Major faiths, rituals, and festivals
Customs: Cultural etiquette and greeting practices
Festivals: National and regional celebrations
Heritage: Monuments, sites, and landmarks of historical importance
Economy: Key industries and occupations
Media: Popular entertainment figures, cinema, and television
Architecture: Traditional Art and Architecture
Sports: Indigenous games and traditional sports
Notable Figures: Influential leaders and historical personalities
Take help from the detailed caption of this image in this task of categorization. Caption: {caption]
**Rsponnd only with the of the most relevant category from the list above i.e., Food, Lifestyle, Literature, Music, Religion, Customs, Festivals, Heritage, Economy, Media, Archi- tecture, Sports, Notable Figures** **Do not give any other response and do not provide any explanation or any unnecessary **x
40

---

Published as a conference paper at ICLR 2026
QA pairs creation prompt:
Here is India-specific image and the image filename, caption, and category of the image I have on hand.
The image filename is this: (image filename)
The caption is this: (caption)
The category is this: {category)
I'd like you generate two short questions and answers, one multiple-choice question and answer, one true/false question and answer, and one long question and answer. Refer to the image filename, category, and caption for context and hints. Take into account the cultural diversity of the category that this image falls under with respect to India
Follow these rules while designing questions and answers:
1. The question must be answerable only by looking at the image.
Ensure that the questions are culturally relevant India and specific to the image.
3. Make the questions in such way that someone who is not well aware of Indian culture will find them difficult to answer.
4. Provide answers that are concise, accurate, and directly related to the question.
For MCQs, provide correct option and incorrect (but relevant) distractors.
6. For MCQs, the question must be answerable even without the choices.
Example of an invalid question: "What song is not performed by this musician" not answerable if you don't know the choices.
7. Write all questions fluently in English.
8. Be mindful of cultural sensitivities and avoid stereotyping misrepresentation.
9. Ensure variety: include identity questions ("What is this?", "Where is this?"), reasoning,
referencing, and commonsense knowledge.
10. Generate only in English.
11. For ahort-ansswr questions, keep answers brief (1-2 sentences).
12. Make all questions distinct and unique.
Give the answers the following JSON format and output only valid JSON:
{{
"short questions": [ "question": <question>> "answer": <answer>>]] J.
"multiple.chon choice "uwioiieeuuustioss.::: [{ "question"" <question>, "answer": <answer>, "options". [<opiion>>, <option2>, <option3>>, <option4>11] "true false fawle question": { "question"": <question>, "answer": <answer>>.
"long question"; { "question"" <question>> "answer": <answer>>
}}
41

---

Published as a conference paper at ICLR 2026
Adversarial QA pairs creation prompt:
You are given an image from India along with its caption and the category belongs to. that makes confident but
Your task is to create adversarial question for the image subtly incorrect cultural assumption about what is shown, especially with respect to India's di- verse regional traditions, foods, festivals, clothing, rituals, and art forms.
Here the image's caption: {caption} The category it belongs to is: (category)
Generate:
Adversarial Question: misleading or culturally incorrect question that confidently makes specific, wrong assumption which is plausibly close the truth (e.g., mixing up similar Indian art styles, dishes, festivals, or traditions). The question must not reveal that it is making an assumption - just ask the question normally, as the incorrect assumption is true.
Avoid yes/no or speculative questions.
Examples:
How this Bikaneri folk painting prepared canvas? (image shows Kumaoni Aipan)
What are the main ingredients in this chicken tandoori dish? (image shows paneer tikka)
Which Sikh gurdwara is being shown here? (image shows Hindu temple)
How is this Eid offering typically presented in Tamil Nadu? (image shows Pongal celebra-
tion)
Non-examples:
-Is Is this Bikaneri art piece?
Are these Modaks made coconut?
Connidering this is South Indian Onam celebration...
The adversarial question should be close enough the actual content that it forces model to distinguish finely between culturally similar options and reject the incorrect assumption. You also need generate the answer for the question that you generate.
include introductions or explanations. Output only the following two fields in exactly this format:
Adversarial Question: <your question here>>
Answer: <your answer here>>
42

---

Published as a conference paper at ICLR 2026
Prompt for judging adversarial questions via LLM-as-a-judge:
You are judge evaluating how well Vision-Language Model (VLM) answers adversarial questions. In adversarial questions, the question contains a false assumption about the image.
You are given:
Question: {question]
Correct Answer (Gold reference): {ground truth}
VLM Answer: (predicted answer)
Your task:
Evaluate the VLM's answer according to this rubric:
1. Identifying the false assumption (0 or points);
Award 5 points if the VLM explicitly identifies that the question contains wrong assump- tion OR expresses uncertainty/dificully in confirming the assumption.
Award points if the VLM ignores the wrong assumption and answers if the question were correct.
Identifying what the image is actually about (0-5 points):
Award 0-5 points depending how well the VLM correctly identifies the real content of the image.
0 completely wrong attempt.
1-2 = vague or partially correct.
3-4 mostly correct but incomplete.
5= fully correct identification.
Final Score Assumption Score (0 5) Identification Score (0-5) 0 to 10.
Instructions:
-Only output the final score number between 0 and 10.
Do not explain reasoning or repeat answers.
Always respond with the final score. Do not return blank response.
Be fair but consistent: partial credit encouraged for partial identification.
Now, provide the score.
43


Reviews:
[{"review_id": "c08PsOp9GT", "reviewer": "Reviewer_HKVC", "summary": "This paper addresses the Western-centric bias in existing vision-language model evaluation benchmarks by introducing IndicVisionBench. This new benchmark is the first of its kind to focus on the cultural and linguistic diversity of the Indian subcontinent, covering English and 10 Indic languages. The benchmark is composed of 5K images and over 37K question-answer pairs, structured into three distinct tasks: VQA, OCR, and multimodal MT. The data curation process involved a combination of web crawling, crowdsourcing, and rigorous human verification to ensure cultural and linguistic fidelity. The authors evaluate eight prominent VLMs, from proprietary systems to open-weight models, and reveal significant performance deficits, particularly for low-resource languages and culturally nuanced queries. The analysis delves into regional biases, cross-lingual performance variations, and topic-specific capabilities, demonstrating the benchmark's utility in uncovering the limitations of current state-of-the-art multimodal systems.", "questions": "No", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors' motivation for this work isintuitive They address a well-known but often-neglected limitation in our field: the Western-centric nature of most vision-language evaluation benchmarks. The paper makes a very convincing argument for the necessity of developing resources that capture greater cultural and linguistic diversity, a direction of research that is becoming increasingly critical.\n\n- A major strength of this paper is the comprehensive design of IndicVisionBench. It is not limited to a single task but integrates three distinct multimodal evaluations (VQA, MMT, and OCR) especially the MT part, which makes it a very comprehensive resource. The VQA component is especially well-designed, featuring six different question formats. The use of \"adversarial questions\" to probe for the ability to reject false cultural assumptions is a particularly novel and insightful method for assessing model robustness.\n\n- The experimental evaluation presented is comprehensive, covering a wide range of current vision-language models. Importantly, the analysis goes beyond a simple ranking of models. The authors provide valuable insights through detailed breakdowns of performance, for example by investigating regional-language biases and performance variations across different cultural topics.", "weaknesses": "- There is a potential for bias in the data generation process. The VQA pairs were first generated using Gemini models and then corrected by human annotators. While this is a practical and common approach, it introduces a risk of \"self-enhancement\" bias, where the models used for generation might produce content that is easier for them to evaluate later. The paper would be strengthened by acknowledging and discussing this potential limitation.\n\n- The evaluation of open-ended questions relies on GPT-4o as a judge. Such judges can have their own biases and may not fully capture all cultural nuances, introducing a layer of uncertainty. It would be beneficial to include a brief discussion on these known limitations. Reporting inter-annotator agreement from a small subset of human judges, to calibrate the LLM judge's performance, would also add significant value.\n\n- The benchmark is presented as \"large-scale,\" but the amount of data for some specific languages and tasks is quite small. For instance, the VQA-Parallel and MMT tracks are built upon only 106 images. For some of the lowest-resource Indic languages, the number of samples is also limited. This can affect the statistical significance of the conclusions for these specific languages. The authors should be cautious with making very strong claims about performance on languages with a small number of samples.\n\n- A very interesting point is made in Section 6 regarding the limitations of WER/CER metrics for OCR evaluation, and the authors make a good case for using ANLS instead. This discussion is valuable. However, it appears quite late in the paper. The impact would be greater if this challenge were introduced earlier, in the experimental setup (Section 4), to provide better context for the results presented in Table", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YQLuDoPMr8", "reviewer": "Reviewer_f28F", "summary": "This paper proposes a human-annotated multimodal benchmark for Indic languages. The benchmark includes tasks such as OCR, MT, and visual QA. The authors evaluated several vision-language models, including both open-source and commercial ones, and identified a notable performance gap.", "questions": "-", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work addresses an underrepresented language and provides a valuable resource for the multilingual AI community, particularly for the Indic community. In the current era of LLM-generated and synthetic data, I appreciate the thorough effort to involve humans in data creation. The resulting dataset covers multiple tasks and languages. The annotation guidelines and interface are also clearly described, further demonstrating the authors’ commitment to transparency.", "weaknesses": "Annotator demography is missing, which will be useful information to add in this type of dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbyxannz6g", "reviewer": "Reviewer_iVmL", "summary": "This work presents IndicVisionBench, a large-scale benchmark designed to evaluate Vision-Language Models (VLMs) on culturally grounded and multilingual tasks focused on the Indian subcontinent. The benchmark comprises 5,000 unique images and over 37,000 question-answer pairs spanning 13 cultural topics across English and 10 Indian languages. The evaluation framework encompasses three primary tasks: Visual Question Answering (VQA) with six question types, Optical Character Recognition (OCR), and Multimodal Machine Translation (MMT). The authors evaluate models, including both proprietary systems (Gemini-2.5, GPT-4o) and open-source variants, revealing substantial performance gaps particularly for low-resource languages and culturally specific content.​", "questions": "- Why was **Surya OCR** excluded from the OCR evaluation? Given that Surya has demonstrated superior performance on Indic languages, its absence represents a significant gap in baseline coverage\n\n- Authors mention adversarial questions \"incorporate false assumptions\". What methodology ensured these questions were appropriately challenging? Even Gemini-2.5 scores only 5.79/10 on adversarial questions - does this indicate poor question design or fundamental model limitations?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- **Comprehensive Multi-task Framework**: The benchmark's tri-modal evaluation approach (VQA, OCR, MMT) provides a holistic assessment of VLM capabilities beyond simple question-answering. The inclusion of adversarial questions represents a good approach to probing deeper cultural knowledge beyond surface-level recognition.\n\n-  **Linguistic Coverage** : The benchmark covers 10 Indic languages with diverse scripts.", "weaknesses": "- **Limited Methodological Novelty**:    \nWhile the cultural focus is valuable, the benchmark construction methodology largely follows established paradigms without introducing novel evaluation frameworks. The reliance on synthetic question generation using commercial models (Gemini-1.5-Flash, Gemini-2.5-Flash) raises concerns about potential biases inherited from these models.\n\n- **Self-Preferential Bias and Evaluation Circularity**:   \nThe most severe methodological flaw is the systematic use of Gemini models throughout the benchmark construction pipeline, followed by its evaluation on the same benchmark. Gemini-1.5-Flash and Gemini-2.5-Flash generate synthetic captions and QA pairs​ and translations across languages​. This creates a circular evaluation where Gemini is assessed on data it helped generate, violating fundamental evaluation principles established in contamination literature. The consistently superior performance of Gemini-2.5 across all tasks becomes suspect given this methodological flaw.​  \n\n- **Absence of Design Choice Validation**:   \nThe paper provides no systematic benchmarking for the choice of Gemini variants. While mentioning a \"small pilot study and cost considerations,\" no details, quality assessments, or comparative analysis with alternative generation models are provided. Table 6 shows only cost comparisons without quality metrics.  The benchmark's coverage appears skewed toward certain linguistic groups (Hindi at 26.8% of QA pairs) without adequate justification for this distribution.\n\n - **Complete Lack of Quality Control Statistics**:   \nCaption Quality: No validation of synthetic caption accuracy against image content using multimodal embeddings or human verification statistics​.  \nHuman Refinement Rates: The claim that \"Human reviewers refined all outputs for factual accuracy and cultural alignment\" provides zero statistics on edit rates, acceptance rates, or extent of modifications required​.  \nInter-Annotator Agreement: Despite being standard practice for evaluation benchmarks, no IAA scores or methodology details are provided.  \n\n- **Inadequately Documented Subset Selection**:  \nVQA-Indic: The selection of which subset from \"4K+ images\" was translated lacks stratification methodology, selection criteria, or coverage statistics​\nVQA-Parallel: The choice of 106 images for multilingual translation provides no justification for representativeness or selection rationale​\nStatistical Significance: With Hindi dominating 26.8% of QA pairs and other languages having minimal representation, the paper lacks analysis of whether minority language subsets support reliable conclusions\n \n- **Statistical Significance Testing**  \nThe paper lacks statistical significance testing for reported performance differences, provides no confidence intervals, and offers insufficient error analysis across the diverse evaluation scenarios. With Hindi dominating 26.8% of QA pairs and other languages having minimal representation, the paper lacks analysis of whether minority language subsets support reliable conclusions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fns1yZgH9G", "reviewer": "Reviewer_w7Rf", "summary": "The paper introduces IndicVisionBench which is a culturally grounded multimodal benchmark focused on Indian languages. The benchmark covers three tasks-- VQA, multimodal MT, and OCR. It has ~5k images and 37k+ QA pairs in English and 10 Indic languages. The questions include six types including MCQs, True/False, Short Answers (two versions), long answers and adversarial questions with opposite assumptions. Several multimodal LLMs have been evaluated against the developed benchmark and a comparison has been provided.", "questions": "As most of the images in the benchmark were collected from Google search under creative common license -- is there any correlation with collecting images from Google search and achieving highest performance from Gemini-2.5 (from Google's DeepMind)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses three tasks by presenting a benchmark for 10 low-resource Indic languages. Such benchmarks are crucial to evaluate capabilities of multimodal LLMs on less represented languages.\n- The benchmark covers the annotation of 13 cultural topics and three tasks--VQA, multimodal MT and OCR. Question have further six categories showing a comparative aspect and abilities of LLMs. \n- Eight LLMs have been evaluated covering open and closed source models that provides an inside of performance compared to different Indic cultures and languages.\n- Parallel slice for controlled analysis, the 106-image VQA-Parallel + MMT subset is well designed for cross-lingual/model-vs-no-image comparisons.\n- Adding adversarial questions with false assumptions provides realistic stress test and it really exposes even strong models.", "weaknesses": "- Synthetic data creation: many QAs are first generated by the LLM and then corrected by human. What was the error rate and it should be quantified.\n- Many images are Commons Google-search cultural images which are highly likely to have been seen by pretraining corpora.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)