RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding
Authors: Jiaang Li
Yifei Yuan
Wenyan Li
Mohammad Aliannejadi
Daniel Hershcovich
Anders Søgaard
Ivan Vulić
Wenxuan Zhang
Paul Pu Liang
Yang Deng
Serge Belongie
Venue: ICLR 2026 Poster (2026)
Keywords: Visual culture understanding
Cultural benchmark
Multimodal retrieval-augmented generation
Primary area: datasets and benchmarks
TL;DR: We present RAVENEA, a large-scale benchmark with 11,396 human-ranked Wikipedia docs for culture-aware vision-language tasks. We find retrieval boosts lightweight VLMs by 6% on cVQA and 11% on cIC, showing the power of cultural augmentation.

Abstract: As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiveness of retrieval-augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application
in multimodal scenarios remains underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a new benchmark designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). RAVENEA extends existing datasets by integrating over 11,396 unique Wikipedia documents curated and ranked by human annotators. Through the extensive evaluation on seven multimodal retrievers and fifteen VLMs, RAVENEA reveals some undiscovered findings: (i) In general, cultural grounding annotations can enhance multimodal retrieval and corresponding downstream tasks. (ii) VLMs, when augmented with culture-aware retrieval, generally outperform their non-augmented counterparts (by averaging +6% on cVQA and +11% on cIC). (iii) Performance of culture-aware retrieval augmented varies widely across countries. These findings highlight the limitations of current multimodal retrievers and VLMs, underscoring the need to enhance visual culture understanding within RAG systems. We believe RAVENEA offers a valuable resource for advancing research on retrieval-augmented visual culture understanding.

Published as a conference paper at ICLR 2026
RAVENEA: A BENCHMARK FOR MULTIMODAL RETRIEVAL-AUGMENTED VISUAL CULTURE UNDERSTANDING
Jiaang Li
Yifei Yuan
Wenyan Li
Mohammad Aliannejadi
Daniel Hershcovich
Anders Søgaard!
Ivan Vulié'
Wenxuan Zhang
Paul Pu Liang"
Yang Deng"t Serge Belongie
University of Copenhagen *University of Cambridge "Massachusetts Institute of Technology
ETH Zürich University of Amsterdam
"Singapore University of Technology and Design
'Singapore Management University
ABSTRACT
As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiivnesss of retrieval- augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application in mul- timodal scenarios remains underexplored. To bridge this gap, we introduce RAVE- NEA Reetrieall-Augmennn designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-centric visual question answering (cVQA) and culture-informet image captioning (cIC). RAVENEA extends existing datasets by integrating 11,396 unique Wikipedia documents curated and ranked by human annotators. Through the extensive evaluation on seven multimodal retrievers and seventeen VLMs, R RAVENEA reveals some undiscovered findings: (i) In general, cultural grounding annotations can enhance multimodal retrieval and corresponding downstream tasks. (ii) VLMs. when augmented with culture- retrieval, generally outperform their non-augmented counterparts (by averaging +6% on cVQA and +11% on clC). (iii) Performance of culture-aware retrieval augmented varies widely across countries. These findings highlight the limitations of current multimodal retrievers and VLMs, underscoring the need to enhance visual culture understanding within RAG systems. We believe RAVENEA offers a valuable resource for advancing research on serreval-auummnneed visual culture understanding.
Visual culturE uNdErstAnding), a new benchmark
Project page: jiaangli.g gitaeeaiorrvveeaa
INTRODUCTION
llnguael models (VLMs) are increasingly deployed in real-world applications, from educa- tion to assistive technologies (Baker al., 2021; Di Nuovo et al., 2024; De Marsico et al., 2024; Karamolegkou et al., 2025), where understanding not only visual content but also the surrounding cultural context is crucial. Despite achieving impressive performance on general tasks (Lin al., 2014; Yue et al., 2024; Bai et al., 2025a), VLMs often struggle to capture cultural nuances, such as traditions, symbols, and region-specite specific practices that require external, culturally grounded knowl- edge (Nayak et al., 2024; Khanuja et al., 2024; Liu et al., 2025; Winata et al.. 2025). For example, when analyzing images of cultural events, VLMS may misinterpret region-specific attire as generic clothing, thereby overlooking its ceremonial significance (Nayak et al., 2024). What's 's more, they also risk reinforcing existing cultural biases, as biased collections or metadata frequently lead VLMs to perform much better in mainstream culture contexts (e.g. western culture) while marginalizing
"quual contribution.
I Princippl senior advisor.

---

Published as a conference paper at ICLR 2026
cVQA
CIC
Spain Sports
Korea Daily Life
Rolevance
Relevance
High Relevance
OPEN
Relevance
Very High
based in Burcelone, [atalonni, Spašn.
Very High
Scouz, the espical and South Korea
High
BarceJona
Human: Many Korean people
High
Sinchoo-dong Swoul, South Korea
which city does this
cecropoliten Lon area
getting off bus the
have its sports headquarters?
Medium
Barselona
pras.deet FD
station in Seoul
Low
Urban rat1 transst
A. Gerona
Barcalona
B. Tarragona
Lerida
Low
Spanzsh
Wagen
Reference region: Seoul. South Korea
Low
States public transportation
China History
India
Nature
Low Relevance
Low Relevance
High Relevinnee
High
Very High
BIue and whltte
Very High
- Shielo
Jängdezhen...
Indsan
High
Ontnene Biue
High
Kanak Windaven
artwork?
Which famous for such
Low
Whito poreslein,
deaign
Vase with nine peach
houses on it
Human: Indian hillidde with
High
Indian alste
En Jadpur Indza
Southadka in the
A. Yiwu
Dalong
B. Jingdezhen
Zibo
Very
English
35
Reference region: India
Very Low
Figure 1: Samples from RAVENEA. RAVENEA evaluates the multimodal culture- RAG with VLMs across eight countries spanning four continents: Asia, Africa, Europe, and Latin America. comprises two tasks designed assess visual culture understanding abilities and the effect of extra cultural knowledge across different countries. With 1,868 instances and 11,369 human-ranked documents, RAVENEA covers eleven real-world scenarios, including architecture, cuisine, art, etc.
integration of external knowledge through promising approach to mitigate the limitations is the generation (RAG) (Lewis et al., 2020), which has shown success in improving cultural awareness in language models (Sco et al., 2025; Lertvitayakumjor et al., 2025). However, prior work culture has largely remained in text-only settings, while existing multimodal culture-related datasets mostly probe VLMM' memorized cultural knowledge instead of testing their understanding of culture in real-world contexts. Consequently, it remains unclear to what extent current mutlimodal retrievers support reliable cultural retrieval, and whether RAG can effectively advance multimodal cultural understanding for VLMs. This shortfall underscores the need for a dedicated benchmark that probes various facets of visual culture comprehension for multimodal retrievers and VLMs.
minority traditions (Nayak et al., 2024).
To bridge the gap. we introduce RAVENEA (Retrieval- Seriival-Augmmenne Augmented Visual culturE uNdErstAnding), a manually curated dataset designed to evaluate multicultural understanding VLMs with retrieval support. We construct R AVENNA based on two existing datasets: CVQA (Romero et al., 2025), which includes culturally relevant visual questions and corresponding answers, and CCUB (Liu et al., 2023), offering culturally contextualized captions to foster inclusivity in text-to-image generation.' For each instance drawn from the source datasets, we append a set of Wikipedia documents that have been human-ranked based their cultural relevance to the associated image. This curation effort, designed to ensure broad cultural representation, contains data related to eight countries and spans eleven diverse categories (e.g. architecture, art, cuisine, etc), comprising 18,680 human-labeled image-document pairs. RAVENEA thus provides a multimodal retrieval-aummented benchmark for evaluating multicultural sensitivity of various retrievers, and further allows for assessing how well VLMs utilize retrieved cultural context in their reasoming process. Specifically, as shown in Figure I, we focus on two culturally grounded tasks: (i) culture-centric visual question answering (cVQA) and (ii) multure-informet image captioning (cIC).
With RAVENEA, we first train and evaluate seven multimodal retrievers that use both visual and textual inputs to retrieve Wikipedia documents for given image query based on their cultural relevance.
We reuse the cultural captions from CCUB as ground-truth references for the inverse task, image-to-text generation, specifically for culture-aware image captioning.
2

---

Published as a conference paper at ICLR 2026
We then evaluate 17 widely used VLMs, both with and without multimodal retrieval, spanning open-source and proprietary models as well as sizes from 2B to 78B parameters, to assess the effect of retrieval augmentation
cultural understanding in the cVQA and cIC tasks. Our benchmark provides a testbed for evaluating the cultural relevance capabilities of multimodal retrievers and the effectiveness of VLMMs in consuming and using such retrieved cultural context.
Our contributions and key findings include:
RAVENEA benchmark: We introduce RAVENEA, the first benchmark for evaluating VLMs in using external knowledge for visual culture understanding. It covers eight countries across eleven categories, linking images to human-ranked Wikipedia documents on two tasks: cVQA and cIC.
Cultural grounding annotations enhance multi-modal retrieval: We evaluate seven retrievers that integrate visual and textual cues to retrieve culturally relevant documents. We find fine-tuning retrievers on calture-targete annotations leads to marked gains retrieval accuracy, highlighting the value of explicit cultural supervision.
Benefits of culture-aware retrieval: Culture Culture-aware aware retrieval boosts task performance across VLMs, with lightweight models showing the better improvement. This suggests that such retrieval can seamlessly integrate into downstream VLM tasks, enhancing their performance.
Cross-cultural variation: Evaluation across eight countries reveals that VLMs, both with and without cultural RAG exhibit distinct cultural preferences, with each model favoring different regional contexts, suggesting model-specifii cultural biases.
Histor
Histors
2 RAVENEA CONSTRUCTION
theie Life
Mexico
India
Arr
We present RAVENEA, the first benchmark explicitly de- signed to assess how retrieval augmentation influences the visual cultural understanding capabilities of VLMs. It is derived originally from 30 countries; see Table 4. Each im- age is paired with 10 algorithm-ranked documents based on cultural relevance. To ensure high data quality while maximizing linguistic diversity across major language fam- ilies, we sample data from eight countries spanning mul- tiple continents: China, Nigeria, Russia, Spain, Mexico, India, Indonesia, and 2orea. We then apply human re- ranking- based annotation to the collected documents to construct RAVENEA. Consequently, RAVENEA comprises 1,868 instances, including 2,331 image-text questions, 655 image-caption pairs, and 18,680 image-docund pairs, supporting two tasks: cVQA and cIC, respectively.
mstory
CVQA
Indonesia
Wistory
Spain
Cuisane
History
GUILI
China
Nigerta Moxica
Sigeria
CIC
History Russia
Art
India Korea
Art
Art
Av
Figure RAVENEa: Multimodal Retrieval-Augmentt Visual culturE uNdErstAnding dataset. Geographic and categorical distribution of cultural refer- ences in RAVENEA.
2.1 TAXONOMY
Motivated by the increasing need for VLMs to understand cultural context (Yadav et al., 2025a), we categorize the cultural understanding capability into two core tasks and several detailed axes, as shown in Figure 2. Additional dataset statistics are provided in Appendix D.
This task evaluates a model's ability perceive and reason over culturally grounded visual content, often requiring external, culturally specific knowledge beyond the image itself. Building on the CVQA dataset (Romero et al., 2025), which contains culture-related QA pairs, we augment with culturally relevant documents that supply the additional knowledge required for RAG evaluation. Therefore, each instance contains an image, a question, top 10 human-ranked documents based on culture relevance, and multiple-choice options, with only one correct answer.
cVQA.
cIC. This task assesses the VLM's ability to generate captions that are sensitive to and incorporate cultural nuances. It evaluates the model's generation sensitivity by requiring it to produce culturally contextualized captions for given image, leveraging information from human-curated, culturally related documents. The task reuses cultural captions from the CCUB dataset (Liu et al., 2023) as
21 List of languages by total number of speakers.
3

---

Published as a conference paper at ICLR 2026
Data Collection
Relevance Annotation
Quality Control
Culture-relltted Captiening
Data Combination
Postpraceesing Tools
Caption: The image
depacta the cust
Caption
GPT-4o
2. Docusent Retrieval
Caption
W
JUN Orappry
Iropection
Consencus
Filter
Figure 3: RAVENEA construction pipeline. Left: A two-stage retrieval process to match each image with relevant documents. Middle: Decomposition cultural relevance into three interpretable dimensions to improve human annotation. Right: Postprocessing methods for quality control.
ground-truth references for this task. As result, for clC. each instance consists of an image, human-written caption, and top 10 culturally-reatted documents.
2.2 DATASET CURATION
Data collection. As shown in Figure 3, the data collection process consists of two main steps: (i) culture-related captioning and (ii) document retrieval. For culture-related captioning, generate culturally grounded captions for each image to facilitate more effective attachment of relevant documents. Since the CVQA lacks captions and the CCUB provides only brief descriptions, we employ GPT-4o² (Achiam et al., 2023) to generate richer, culturally informative captions (see the prompt example in Table 13). For document retrieval, begin with carse--raaied filtering stage in which the generated cultural captions serve as queries to BM25 retriever (Robertson & Zaragoza, 2009). This stage retrieves pool of semantically relevant candidates from large-scale corpus of over six million English Wikipedia articles. To mitigate the impact of inaccurate captions and ensure precise document relevance, we then perform human annotation on the retrieved documents.
Cultural relevance annotation. Based on the initial BM25 retrieval results, we refine the cultural relevance label of retrieved documents via human annotation. For each image-caption pair, annotators are presented with the top 10 Wikipedia documents retrieved by BM25. They are asked to assess whether each document provides background or information that relevant to the culture described in the caption or the image (see interfaces in Appendix L). Specifically, we decomposed cultural relevance into three interpretable and independently verifiable dimensions:
the Wikipedia article associated with the same country Country association: Is the topic the the image and its caption?
Topic alignment: Does the topic of the Wikipedia article align with the semantic category of the image and its caption?
Explicit visual representation: Is the topic of the Wikipedia article explicitly mentioned or visually represented in the image and its caption?
Each dimension is framed binary (True / False) question to reduce ambiguity and improve annotation consistency. However, for the country association dimension (the first listed), we introduce an additional label, "Cannot be determined", to handle cases where this association is unclear from the annotator's perspective. Additionally, annotators are also instructed to include the title and URL of any relevant Wikipedia article they believe is missing from the top- retrieved results. These manually suggested articles are treated as the most culturally relevant references for the given image. Detailed annotation statistics, alongside positive, negative and missing ratios by countries, are presented in Appendix E.
Quality control. For caption quality, we employ two independent local annotators assessed both objective accuracy and subjective quality (naturalness and completeness) in subset, achieving 92%
"nowledgge cutoff: Oct 01, 2023.
'We used I.I.Ms generate calture-informer captions solely as auxiliary references for annotators; these LLM outputs were not directly included in the final datasct. Wikimedia/wikipedii
4

---

Published as a conference paper at ICLR 2026
and 94% accuracy with an inter-annotaton agreement (IAA) Cohen's Kappa (x) (Artstein, 2017) of 0.85, and yielding high subjective ratings (naturalness: 4.98/5, completeness: 4.66/5). The results demonstrate that the captions annotation quality, a separate group of annotators were trained with detailed guidelines and a mock test before performing the actual annotation. A meta quality checker further validated random subset, achieving 98.2% acceptance rate. The IAA Cohen's 's Kappa (k) between the meta checker and annotator on the sampled annotations is 0.83, More details are presented in Appendix F.
reliable for downstream retrieval of Wikipedia documents. For
3 EXPERIMENTAL SETUP
With RAVENEA, we train and evaluate seven multimodal retrievers to retrieve culturally rele- vant Wikipedia documents using multimodal inputs, spanning both generative and discriminative paradigms. To enhance cultural awareness the contrastive retrieval, we introduce Culture-Awar Contrastive (CAC) learning (details in Section 3.2), a supervised learning framework compatible with both CLIP (Radford et al., 2021) and SigLIP (Tschaaneen et al., 2025) architectures. Then, evaluate the effectiveness of these retrievers with 17 widely used VLMs across the cVQA and cIC downstream tasks. More details are in Appendix
3.1 EVALUATION METRICS
For multimodal retrieval, the performance is evaluated using standard retrieval metrics, including Mean Reciprocal Rank (MRR) (Shi al., 2012), Precision@l (P@k) (Järvelin & Kekäläinen, 2017), and Normalized Discounted Cumulative Gain (nDCG@K) (Wang et al., 2013), where & E 31,3,5).
For cVQA, we accuracy the evaluation metric, as all tasks fol- low multiple-choice format. Ac- curacy is computed per VLM and per retriever, representing the pro- portion of correctly answered ques- tions. For cTC, we employ several metrics for evaluating the generated caption, including ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), BERTScore (Zhang* al., 2020), and CLIPScore (Hessel al., 2021).
Table 1: Kendall's т rank correlation (Kendall, 1938)
between automatic metries and human judgments for the cTC' task. Statistically significant correlations (p < 0.05)
are marked with v. Our proposed metrics correlate stronger with human evaluation than the others.
Rouge-L. CIDER BERTScore
CLIPScore
RegionScore (ours)
-0.172
-03116
-0.011
0.139
0.442 /
To further evaluate the cultural relevance of the generated captions, we conducted a human study in which four researchers selected the most accurate captions from 14 VLMs. We find a significant mismatch between automatic metrics and human judgments of cultural appropriateness (Table 1). To bridge this gap, we further introduce RegionScore, a novel evaluation metric designed to quantify the extent to which captions reference specific geopolitical regions. See details in Appendix and H.
RegionScore. Let = (h, 1,, denote the set of images. For each image 7;, let ( denote its generated caption, and let c!1) represent the country name associated with 7,. We define A as the set of adjectives or demonyms corresponding to c For convenience, we introduce the set of region-relatee terms associated with 1;: T,, = {c( U A.. The instance-level R(g(t I.) metric for sample 1; and g (6) is defined as:
if w g ( such that E Tr. otherwise.
R(g (()))
(I)
The overall RegionScore for predicted captions are then computed as:
RegionScoret
1
Σ R It),
RegionScorer E 00,
(2)
Ne
'The Swen33-VI. models were omitted from the human evaluation for the cTC task hecause the used model variants were released that time.
5

---

Published as a conference paper at ICLR 2026
Table Performance with different retriever models. Fine-tuned contrastive models consistently outperform their frozen counterparts (shown in gray). Ravenea-Sig and Ravenea-CLL are SigLIP2-SO/14G 384px and CLIP-L/14@R 224px models fine-tuned on RAVENEA, respectively.
Method
MRR
P@1
P@31 f
P@51
aDCG@11
aDCG@31
nDCG@5
SigLIP2-SO/140384p
68.62
54.66
37.47
32,92
61.22
63.82
71.44
2LIP-L/14@224pX
75.44
60.87
41.41
34.41
67.75
72.31
78.09
VisualBERT
61.33
46.71
33.14
29.42
55.76
59.42
VL-TS
58.33
39.86
32.58
29.35
47.74
57.29
62.12
LLaVA-OV-7B
37.48
30.68
28.21
Ravenea-Sige SigLIF (ours)
70.95
57.14
40.99
33.29
63.86
68.31
73.92
Ravenea- CLII (ours)
82.17
72.05
45.76
36.77
77.08
78.96
84.09
RegionScore is designed to complement, rather than replace, existing general- pureroe captioning metrics (e.g., CLIPScore), which primarily assess semantic alignment but do not explicitly capture geographic specificity. Therefore, RegionScore serves as a proxy for regional awareness by quantify- ing the proportion of captions that include explicit references of a country its common demonyms, with higher values indicating stronger region awareness. In CCUB, RegionScore ground truth is 99%.
3.2 CULTURE- AWARE AWARE CONTRASTIVE LEARNING
each document DJ is annotated with a binary label yf E {0, where 3 1 indicates cultural relevance and y 0 indicates irrelevance. For each image-document pair (I, D.), we employ a shared langunlanguage encoder, such as CLIP, to obtain modality-specific representations: E,, &v (I,) for the visual input and E (D() for the textual input. We then compute the cosine similarity score of between E,, and each corresponding E D resulting in similarity vector S; = [s] s....,.. Culture-awarenee awareness classification amounts to:
Given an image I, associated with set of textual documents D = [D], D ).
Ceultie Classify
v log logo(())) + (1 u/) log(l (og((1)()))]
a(s())
LEEDDED
where (-) denotes the sigmoid function.
To prioritize culturally relevant descriptions in the ranking, apply margin ranking loss between all pairs of descriptions with differing cultural relevance. For each image 4,, we compare pairs (D). D..) such that y/ and y and encourage the model to assign higher similarity score to the relevant description. The ranking loss is defined as:
CREaK
max (0,6 (s) - -t:))
(4)
J.=
ME =1=10===0
To mitigate the risk of overly similar positive text embeddings for the same image, we introduce a penalty that encourages intra intra-modal diversity among textual representations. We apply a diversity- promoting loss that forces the similarity between different text embeddings to be reduced while keeping each embedding highly similar to itself. Specifically, the penalty formulated using an exponential function to emphasize the dissimilarity between embeddings:
[D]
exp(s!) 1-1 exp(x?)
Luiversity (S.)
Σ log
(5)
Then get the celture-awarr contrastive loss:
Lesc 3 (Leatuic Classity + CRank + Luiversat)
(6)

---

Published as a conference paper at ICLR 2026
Table cVQA and cIC Performance w/ and w/o RAG. Models in gray are frozen retrievers. The
best results are in bold. VLMs augmented with finetuned retriever generally perform better.
Open
Propridary
Diep PoppSee--V..--iin VL.2-Ting
Owen25. VL-771
Phi Multimodal
DeepSeek VL.2
Qwen2.5. 1-..-1--33
Owen25-11-772
InteraV 13-22
Intern
InternVI
Genann--15
Gomno3-271
Fistrab 12B
MaVA-ov-m
Owen
Qwen3-VI VL-88--1--8 VL-8B
Quen3-V1
GPT-44
Retriever
Average
d'QAtemees
W/O RAG
67.1
49.8
68.9
62.7
52.6
83.3
64.1
71.3
81.2
66.5
79.4
42.6
73.2
62.7
37.8
73.7
81.3
86.6
SigLiF2-SO/118 10
71.0
18.8
76.1
63.2
67.5
82.3
71.3
73.7
86.1
67.5
77.5
69.5
75.1
01.5
67.9
75.6
83.3
82.8
CLIP 214
71.9
50.2
77.5
63.6
06.0
82.8
73.7
78.5
84.2
69.4
76.6
67.0
76.6
44.5
68.9
76.6
83.3
83.3
Finetuned Models
VisualBEKT
67.6
48.8
78.5
66.5
67.5
82.
63.6
75.1
61.2
68.4
37.3
60.8
73.2
83.7
85.2
VL-T5
66.1
16.6
72.7
584
65.1
78.5
65.1
67.9
S0.
98.4
75.6
58.9
62.7
35.4
60.3
75.6
78.9
83.7
LLaVA-OV-7B
67.9
50.1
66.0
60.3
67.0
81.8
66.5
66.0
84.2
65.1
75.1
60.8
68.9
37.3
59.8
77.0
83.7
84.7
Ravenea-SigLiz (ours)
Ravenea-CLIP
73.3
50.7
75.1
69.9
82.3
76.1
81.3
69.9
79.9
69.4
75.1
50.2
69.4
77.5
84,2
86.1
W/O RAG
57.5
26.5
63.3
53.1
09.4
71.4
38.8
49.0
65.3
73.5
75.5
28.6
53.1
49.0
59.2
61.2
73.5
67.3
Sigl JP2-SO0148 354px
54.8
65.3
44.9
69.4
46.9
49.0
59.2
59.2
CLI--L144624pps
65.1
38.8
69.4
71.4
69.4
65.3
65.3
73.5
73.5
61.2
09.4
63.3
67.3
71.4
68.3
Finetaned Models
VisuaBBERT
57.4
36.7
57.1
53.1
61.2
69.4
44,9
59.2
61.2
63.3
69.4
429
57.1
44.9
55.1
65.3
694
65.3
V1-T5
57.0
73.5
46.9
67.3
44.
67.3
LLaVA-OV-7B
56.8
38.8
63.3
63.3
69.4
46,9
57.1
65.3
71.4
40.8
55.1
49.0
61.2
673
Ravanea-SigLEz SiigLD (ours)
60.6
30.6
73.5
57.1
67.3
71.4
44,9
59.2
694
67.3
69.4
57.1
61.2
40.8
53.1
65.3
714
71.4
Ravenea-CLip (ours)
68.8
46.9
73.5
61.2
77.6
75.5
69.4
67.3
73.
75.5
75.5
63.3
71.4
55.1
67.3
73.5
73.5
69.4
EXPERIMENTAL RESULTS
4.1
MAIN RESULTS
Multimodal retrieval results. We perform a comprehensive evaluation of both frozen and fine- tuned retrievers, and present the results in Table 2. We find that fine-tuned models, particularly those based on contrastive learning, consistently outperform their frozen counterparts. For instance, Ravenea- - achieves a substantial improvement P@ rising from 60.87% to 72.05%, and sets a new SOTA across all evaluation metrics. Although SigL.P2-SS0/440 384px (Tschannen et al., 2025) also benefits from fine-tuning, the performance gains are comparatively modest. contrast, models such as LLa LLaVA-OV-7B (Li et al., 2025), VL-T5 (Cho et al., 2021), and VisualBERT (Li et al., 2019; 2020) lag behind after fine-tuning, even underperforming relative to frozen baselines. This un- derperformance likely stems from the fact that models such as LLaVA-OV-7B and VisualBERT were originally pretrained for generative tasks with different objectives, whereas CLIP-L/I4@ 224px (Rad- ford et al., 2021) and PP..IPP-///46 384px providing them with a structural advantage in retrieval settings.
explicitly trained for similarity- based alignment,
Downstream tasks. We present downstream performance both eVQA and cIC tasks in Table 3. The results demonstrate the efficacy of incorporating culture-aware retrieval augmentation. Our proposed fine-tuned retrievers, Ravenea Srrena-SSgLLL SigLIP and Ravenea- CLIP, consistently achieve the highest scores across both cVQA and cIC tasks, frequently outperforming both the non-RAG base- lines and frozen retrievers (SigLIP2 CLIP). Specifically, averaged over all VLMs, Ravenea-CI. achieves the highest performance for both tasks, improving +6.2% in cVQA accuracy (67.1% - 73.3%), and 1.33% in cIC RegionScore (57.5% - 68.8%). While CLIP-L/14 224px also offers improvements, culture-aware contrastive learning consistently unlocks further potential. Further- more, without retrieval, larger VLMs still achieve strong absolute performance on both tasks (e.g., IntemnVL3-78B at 84.2% on cVQA, and Gemma3-27B 75.5% on cIC), Retrieval contents, however, disproportionately benefits small and mid size models. For lightweight models (< 8B parameters). Ravenea-CLIP narrows the gap the larger models. For example, Qwen3-VL-SS improves to 77.5% on cVQA, approaching the performance of the models an order of magnitude larger, highlighting the potential of culture-aware retrieval.
7

---

Published as a conference paper at ICLR 2026
4.2 FURTHER FINDINGS
Culture-aware retrieval augmentation substantially benefits VLMs across both cVQA and cIC tasks, compared their non-RAG counterparts, In this section, we explore the margin improvement, cultural preference, the effectiveness of cultural annotation, and the impact of context length.
While larger VLMS see limited gains from different retrievers, lightweight VLMS benefit substantially from culturally grounded retrieval contexts.
Finding 1.
To evaluate the scaling behavior of VLMs incorporating with Ravenea-CLIP, we se- lect four model families spanning from 2B to 78B parameters and quantify the perfor- mance improvement A relative to non-RAG baselines. As shown in Figure 4, the impact of Lavenea-CLIP does not scale monotoni- cally with model size: smaller models exhibit the most substantial gains, while larger mod- els (27B to 78B) exhibit only modest improve- ments. For instance, Qwen3-VL-2B achieves +31.6% absolute improvement on cVQA and +8.1% on cIC, whereas Owen3-VL-32B shows only +2.9% better on cVQA and no gain on cIC. This trend indicates that larger models may already internalize much of the relevant knowl- edge, especially the Wikipedia content retrieved by Ravenea CLIP, resulting in redundancy and plateau performance. Overall, these re- sults suggest that Gavenea-CLi serves effective plug-an and -lay enhancement for lightweight models on both cVQA and cIC tasks, providing diminishing yet consistent gains as model size increases.
Qwen3VL
Qwen2.5VL
30
RagionS.ooee &
20
-0
onc
Figure 4: Performance improvements per model family with Ravenea-CLIP Scaling VLMs yields marginal gains on both cVQA and cIC tasks. "A" represents the change with Ravenea-Cri relative to the non-RAG baseline.
With RAVENEA. VLMs show cultural disparities with and without RAG. Moreover, large inter-model variance highlights model-specific biases.
Finding 2.
For geographic variability, we evaluate all mod- els using Ravenea-CI. across a range of countries for both tasks, shown in Figure 5.
Without RAG
Ravenea-CLIP
ReglonSeare
For cVQA, most VLMs exhibit substantially di-
minished performance culture-specifi ques- tions regarding Nigeria and Indonesia, con- trast their performance questions under other national contexts. Interestingly, ques- tions related Spanish culture reveal high inter-model variance, with accuracy differentials reaching up 50%, underscoring significant discrepancies in cultural representation across models. For cIC, VLMs consistently underper- form on images and documents associated with Mexico cultural contexts, while achieving the highest RegionScores on Indian culture-related inputs. Model performance on Mexico culture particularly volatile, indicating inconsistent cultural grounding across architectures. By compari- son, Korean and Chinese cultural inputs yield more stable performance across models, suggesting entrenched model-specific preferences in cultural alignment (see more results in Appendix J).
Nigeria
Accuray
60 80
20
Spain
Nigeria
Figure Geographic variability in VLM perfor- mance. While Ravenea-Cii yields gains in general, there's still variance both baseline and relative improvement across geographic regions.
8

---

Published as a conference paper at ICLR 2026
DeepSnokVVL2-Ting
eVQA
(AVG)
SigLIP2
LLeva-OV-7B.
@wen2.5VL-78
70
Accuracy
72
RegiunScore
%
Pixtral
InternVL5-88
02:0111
J0h
82
05
30% R2- OLOGT
Phi4-MM
Gemma-48
FT
w/O BAG
Unrelated
Rendom
Ravenea-CL
(a) Finetuning un three questions consistently uchieves the hetler performance (averages shown across 16 open source VLMs). @ denotes frozen models, FT indicates linetuning on the corresponding annotation questions.
(b) Ravenes-CLiP generally outperform the other
cVQA
Figure 6: Ablations for annotations and retrieval content. Combining all three culture-relevant annotation questions consistently achieves the highest performance on both downstream tasks, validating the effectiveness of RAVENEA setup.
Finding 3. Generally, models benefit most when trained with the full set of culture-relevant annotations. Culturally grounded retrieval substantially outperforms random or irrelevant contexts, underscoring both the necessity of culture-aware contexts. Furthermore, more or longer retrieved context length yield diminishing gains.
Annotation questions ablations. We further perform ablation studies across diverse combinations of annotation questions to assess their impact on downstream performance. Specifically, we evaluate 16 open- weight VLMs equipped with either 10gLIP2-SO/144 384px ELIP-L/14@224 each fine-tuned on datasets constructed using varying subsets of culture-relevant annotations. From Figure 6a, Q2 for topic alignment; Q3 for explicit visual representation) yields the strongest performance on both cVQA and cIC tasks. For the cVQA task, find Q1 provides the most significant benefit to Ravenea-SigLiE combinations, the joint supervision from Q1 and Q2 proves slightly more effective than other pairs. In the cIC task, both Saaeena--Sihhn improvements when trained with data derived from Q1, compared to other single-question sets. For pairwise combinations, Ravenea-CL.P benefits most from the Q1+Q3 combination, while Ravenea-Sion
observe that leveraging all three questions (Q1 regarding country association;
whereas Ravenea-CLIP gains more from Q2. Among all pairwise
and Revenea-CLLT achieve better performance
shows a clear preference for the Q2+Q3 setup.
Context relevance ablations. Recent work (Roth et al., 2023; Ma et al., 2025) suggests that randomly augmented prompts may yield performance gains even the absence of meaningful knowledge, raising concerns that improvements attributed to retrieval could in fact stem from other factors, such as superficial prompt lengthening and so on. To rigorously assess the effectiveness of culturally grounded retrieval, design control conditions: random character sequences, culturally unrelated documents. For random character sequences, we randomly generate synthetic strings in which each string consists of 100 to 300 substrings, and each substring contains randomly sampled alphabetic characters, ensuring comparable token budgets without introducing interpretable content. For culturally unrelated documents, randomly sample hundreds of Wikipedia articles from countries not represented in RAVENEA, thereby maintaining language structure and topical coherence while explicitly breaking cultural relevance. Together, these conditions isolate whether gains arise from the related cultural context or merely from the presence of additional text. As shown Figure 6b, models augmented with Ravenea I consistently outperform other configurations, highlighting both the effectiveness of cultural retrieval and the robustness of the RAVENEA evaluation protocol. More results are provided Appendix Table 11.
Context length ablations. We also extend our study with controlled analysis of retrieval length, comparing three additional context granularities: paragraph-level, full top-1 document, and full top-3 documents. As shown in Figure 7, providing compact, high-salienee context (Top-1 1 256 tokens) consistently achieves the best or tied-best performance across most model scales. In contrast, ex expanding the context to full documents multiple documents often leads to marginal gains or

---

Published as a conference paper at ICLR 2026
Qwen3VL-2B
Qwen3VL--BB
Gemma3-4B
Gemma3-278
Ravenea-SigLip
Ravenea-SigLip
Ravenea-SigLiP
Ravenea-SigLi
CLIP
CLIP
CLIP
CLIP
DOC
86 2
SigLIP2
SigLIP2
SigLIPZ
SigLIP2
Top Top Docs
Top Paragraph
Top 256 Tokens
Figure 7: Context length ablations for cVQA task. With Ravenea- adding longer or
multiple documents yields diminishing returns, highlighting the importance of the context length.
degradation. This trend is particularly evident for lightweight models, which appear more sensitive to the context length. Even for larger models, additional context yields only marginal improvements despite substantially increasing input length. These results indicate that simply retrieving more content does necessarily translate into better performance, especially for the lightweight models.
5 RELATED WORK
Retrieval augmentation for cultural understanding. Retrieval augmentation has demonstrated significant efficacy for culture-related NLP tasks (Conia et al., 2024; Hu et al., 2024; Fung et al., 2023). Some works have explored retrieval- -uggeentd approaches to cultural question answer- ing. One line of work implemented retrieval using the World Values Survey to enable in-context learning for culturally contextualized questions (Seo et al., 2025; Yadav et al., 2025b), while others enhanced cultural question answering by retrieving information from web searches and knowledge bases vertvittayakumjonn et al., 2025). In cross-cultural applications, retrieval has also been shown effective for adapting recipes across cultural contexts (Hu et al., 2024). For multimodal understanding, however, the direction of culture-relevant retrieval remains underexplored.
Vision-language culture datasets. Recent studies examine cultural understanding in VLMs through multicultural VQA (Liu et al., 2021; Nayak et al., 2024; Liu et al., 2025). Some researchers focus on culturally grounded visual categories such cuisine, where models must distinguish dishes that are visually similar yet culturally distinct, highlighting fine-grained, geographically situated knowledge gaps (Winata al., 2025; Li al., 2024). Socioeconomic signals have also been explored, for example by studying how models infer or misinfer income level and living standards from visual cues (Nwatu al., 2025). Others benchmark cultural entity recognition using Wikipedia- based prompts (Nikandrou et al., 2025), curate culture-specifie image sets for value-bases tasks (Yadav et al., 2025b) or concept-basse based image retrieval without explicitly considering cultural relevance (Bhatia et al., 2024). While prior efforts highlight the limitations of current VLM, they mainly assess the performance on retrieval and downstream tasks respectively, which are not suitable for probing how RAG shapes visual-cultural understanding. Our work specifically aims fill this research gap.
6 CONCLUSION
We presented RAVENEA, new benchmark for multimodal retrieval-angmented visual culture understanding that enables the evaluation of how external cultural knowledge improves VLMs. By pairing culture-centric visual question answering and culture-informed captioning with 11,396 human-ranked Wikipedia documents across eight countries and eleven categories, RAVENEA provides the first large-scale testbed for studying multimodal retrieval culturally grounded settings. Our experiments across seven multimodal retrievers and seventeen VL.Ms show that incorporating cultural grounding consistently improves both retrieval quality and downstream task performance. Notably, lightweight VLMs benefit significantly from culture-aware retrieval, highlighting multimodal retrieval augmentation as an efficient way to enhance culturally situated understanding and reasoning. Overall, RAVENEA offers standardized framework for diagnosing limitations and advancing research in multimodal retrieval, grounding, and generation for VLMs in culture domain. We hope this benchmark will drive future work toward more culture-sensitive and globally inclusive multimodal intelligence.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on improving the cultural awareness of VLMs through aurieval-augmenter methods. All data used in the construction of the RAVENEA benchmark were sourced from publicly available datasets and Wikipedia, community-curatee open-access knowledge base. To protect individual privacy, we apply automated face detection and blur all identifiable faces in images prior to release. To mitigate cultural bias and ensure broad representation, the benchmark includes images and documents spanning eight countries and eleven cultural domains, curated and annotated by a diverse group of annotators. We provide detailed documentation of the annotation process and guidelines to support transparency and reproducibility. While enhancing cultural understanding is a central goal of this work, acknowledge that culture is inherently complex, dynamic, and context-dependent. Consequently, the benchmark cannot capture the full richness of any cultural context.
Unavoidably, when models fail, errors may go beyond reduced accuracy, leading to misattributed traditions or distorted cultural practices. In heritage contexts, such failures may undermine trust in digital tools or inadvertently disrespect the communities represented. Reliance broad but uneven sources like Wikipedia can further compound these risks by amplifying existing biases and privileging cultures with richer digital footprints, echoing broader challenges in Al for cultural heritage (Münster et al., 2024). Such distortions could affect heritage management by marginalizing underrepresented voices, eroding local agency, or encouraging overreliance on automated outputs. To mitigate these risks, we emphasize safeguards such as human-in-the-lopp validation with cultural experts, transparent documentation of dataset limitations, and fairness-orientee evaluation protocols. Looking forward, retrieval grounded in domain-specifie knowledge bases curated by GLAM (Galleries, Libraries, Archives, and Museums) institutions offers path toward higher- quality cultural data. We also highlight the importance of systematically incorporating expert human evaluation to validate both cultural relevance and interpretive accuracy.
Finally, this work does not involve any personally identifiable information (PII), biometric data, sensitive attributes. All human annotators were compensated fairly, and data annotation adhered to ethical guidelines for responsible research.
ACKNOWLEDGMENT
We would like to thank all the annotators for their work. Special thanks to Nico Lang, Peter Ebert Christensen, Zhaochong An, Stella Frank, Srishti Yadav, for providing helpful research advice. This project is mainly supported by the National Research Foundation Singapore under the Al Singapore Programme (AISG Award No: AISG3-RPPGV22255- by the Pioneer Centre for AI, DNRF grant number P1. Ivan Vulié is supported by personal Royal Society University Research Fellowship 'Inclusive and Sustainable Language Technology for Truly Multilingual World' (no 221137) 137). Wenyan Li is supported by the Lundbeck Foundation (BrainDrugs grant: R279-2018- 145). Yang Deng support by the Lee Kong Chian Fellowship awarded by Singapore Management University.
Jiaang Li and Serge Belongie supported
REPRODUCIBILITY STATEMENT
Section 2 and Appendix E, F and L details the data collection and processing steps. Experimental setups are provided Appendix I. Code and dataset are available here.
REFERENCES
AbdeIrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loral arXiv preprint arXiv:2503.01743. 2025. 21
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 233.003033 2023. 17,
11

---

Published as a conference paper at ICLR 2026
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. 21
AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024. 17
Ron Artstein. Inter-annotator agreement. Handbook of linguistic annotation, pp. 297-313, 2017. 5
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge. Wenbin Ge. Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng. Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang. Mingkun Yang, Jianxin Yang. An Yang. Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025a. URL https: / aariv.iiivv////..///a.////aavtvv org/ abs/ 2511. 21 231.1. 1,
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025b. 21
Katie Baker, Amit Parekh, Adrien Fabre, Angus Addlesee, Ruben Kruiper, and Oliver Lemon. The spoon is in the sink: Assisting visually impaired people in the kitchen. In Proceedings of the Reasoning and Interaction Conference (ReInAct 2021). pp. 32 32-39, 2021.
Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, EunJeong Hwang, and Vered Shwartz. From local concepts to universals: Evaluating the multicultural understanding of vision-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024, doi: 10. 1..1886//1/22224444 122024.mmllmmmiin.3e.... URL logy org/ 2024 emn ma in.innn.. 385/. 10
https:
Jiangui Chen, Ruging Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. Corpusbrain: Pre-train generative retrieval model for knowledge-intensive language tasks. In Proceedings of the 3 ACM International Conference on Information & Knowledge Management, pp. 191-200, 2022. 20,21 21
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Cawrrnnce Zitnick. Microsoft coco captions: Data collection and evaluation server, arXiv preprint arXiv:1504.00323 2015. 18
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021. 7, 20
Simone Conia, Daniel Lee, Min Li, Umar Farooq Minhas, Saloni Potdar, and Yunyao Li. Towards cross-cultural machine translation with augmenel generation from multilingual knowl- edge graphs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024. doi: 10. 100.18653vv 12222.emmlppmmm.e.... 914. URL https: / aaccccccll anthologyy org/ / 024 em ma ma.in. 914/. 10
Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. balaneb loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 20
Maria De Marsico, Chiara Giacanelli, Clizia Giorgia Manganaro, Alessio Palma, and Davide Santoro. Vqask: a multimodal android gpt-based application to help blind users visualize pictures. In Proceedings of the 2024 International Conference on Advanced Visual Interfaces, pp. 1-5, 2024. 1
Elisa Di Nuovo, Manuela Sanguinetti, PIER Balestrucci. Luca Anselma, Cristian Bernareggi, Alessan- dro Mazzei, et al. Educational dialogue systems for visually impaired students: Introducing a
task-oriented user-agent corpus. In Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 5507-5519. ELRA and ICCL. 2024.1
12

---

Published as a conference paper at ICLR 2026
Yi Fung, Tuhin Chakrabarty, Hao Guo. Owen Rambow, Smaranda Muresan, and Heng Ji. NORM- SAGE: Multi multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Singapore, 2023. Association for Computational Linguistics. doi: 10. 18853311/2023..col 102.223..mmmpmmmam..444 URL
https: //aa//aaatttttta logy- org/ 2023 amnnp-ma I. 941/ 10
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 emnlp-main.595. URL htpps: : / /t///aaaaanttt /aaaaaaannnooooooyooooooooooooll org//020//// 2021 emnlp ma in. 595/.
Tianyi Hu, Maria Maistro, and Daniel Hershcovich. Bridging cultures in the kitchen: A framework and benchmark for cross-cultural recipe retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, November 2024. doi: 10. 06653//12024.cmmlppp main... URL https: ac an /co.pppo///..tthhooooo... org/2024 emnlp-main 61/. 10
Kalervo Järvelin and Jaana Kekäläänen. evaluation methods for retrieving highly relevant documents. In ACM SIGIR Forum, volume 51, pp. 243-250. ACM New York, NY, USA, 2017. 5
Antonia Karamolegkou, Malvina Nikandrou, Georgios Pantazopoulos, Danae Sanchez Villegas, Phillip Rust, Ruchira Dhar, Daniel Hershcovich, and Anders Søgaard. Evaluating multimodal language models as visual assistants for visually impaired users. In Wanxiang Che, loyce Nahende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 25949-25982, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10.18653/ 000.665//11222255 (112025.acl-oopg 1260. URL httthhts///aat http://s///aaaaahoooooooo 002222 acl soll-log..o /.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-9) 1938. 5
Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig. An image speaks thousand words, but can everyone listen? on image transcreation for cultural relevance. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 19258-0077,, Miami, Florida, USA, wwt.:024...mmmmaaaaa.a 573. URL https: / acc antho logy ogy org / 2024. amnlp-ma ma i n. 573/.
November 2024. Association for Computational Linguistics. doi: 10, 111111//1//224....00
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023. 21
Piyawat Lertvittayakumjon. David Kinney, Vinodkumar Prabhakaran, Donald Martin, and Sunipa Dev. Towards geo-culturally grounded 11m generations. arXiv preprint arXiv:2502.13497, 2025.
10
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. tion for mowweedg-intensivee nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020. 2
genera-
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLa -nneviiinns Easy visual task trans- fer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https: //openreview.oo www.eneeuuutteeeee forummid=zkker
6n. 7, 20,
Liuniam Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: (90.:::0000055777 2019. 7, 20
13

---

Published as a conference paper at ICLR 2026
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does BERT with vision look at? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5265-5275, Online, July 2020. Association for Computational Linguistics. doi: 10. 10..88655122200 acl-main.469. URL htwp:::tpp........ /atc//acaaacaa wwwcclnnnhooogggoo org/ 2020 aac--maing memmii..449... 46% 7,
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott. FoodieQA: A multimodal dataset for fine-grained understanding of Chinese food culture. Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, 2024. URL
https: //aaaaaaantooooooooeeee org/ 4 emnlp main. 1063. 10
Chin-Yews Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL wttphttt///// / ac Lanthologtyyy... org W04 1013/.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740-755. Springer, 2014.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. Proceedings of the 202 I Conference on Empirical Methods in Natural Language Processing, pp. 10467-10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
URL https:://///
org/ 2021.02112222 emnlp main. 818. 10
Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. Culturevim: Characterizing and improving cultural understanding of vision-language models for over 100 countries. arXiv preprint urXiv:2501.01200 2025. 10
Zhixuan Liu, Youeun Shin, Beverley-Claire Okogwu, Youngsik Yun, Lia Coleman, Peter Schalden- brand, Jihie Kim, and Jean Oh. Towards equitable representation in text-to-image synthesis models with the cross-cultural understanding benchmark (ccub) dataset. arXiv preprint arXiv:2301.12073, 2023. 2,
Pingchuan Ma, Lennart Rietdorf, Dmytro Kotovenko, Vincent Tao Hu, and Bjöm Ommer. Does vlm classification benefit from Ilm description semantics? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5973-5981 2025. 9
Sander Münster, Ferdinand Maiwald, Isabella di Lenardo, Juha Henriksson, Antoine Isaac, Manuela Milica Graf, Clemens Beck, and Johan Oomen. Artificial intelligence for digital heritage innovation: Setting up a r&d agenda for europe. Heritage, 7(2):794-81) -16,, 2024. 11
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. Benchmarking vision language models for cultural understanding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5769-5766779 5790, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 18.18653/v1/ 2004.eemll-main. 2, 10
URL https: http:///laaaatthotoo.oooo org/ 22024.44. emnlp main. 329/.
Malvina Nikandrou, Georgios Pantazopoulos, Nikolas Vitsakis, Ioannis Konstas, and Alessandro Suglia. CROPE: Evaluating in-context adaptation of vision and language models to culture-specific concepts. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 7917-7936, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8 189-66. URL https: //acle //aclanthol logy- orgg 2025. naacl- long. 402.
Rodrigo Nogueira and Kyunghyun Cho.
Passage re-ranking with bert.
arXiv preprint
arXiv: 04085, 2019. 20
14

---

Published as a conference paper at ICLR 2026
Joan Nwatu, Oana Ignat, and Rada Mihalcea. Uplifting lower-income data: Strategies for socioe- conomic perspective shifts in large multi-modal models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 2127-2144, Albuquerque, New Mexico, April 2025. Association for Compu- tational Linguistics. ISBN 979-8-89176-1888 189-618966.6
doi: 10. 2..18653/v1/2022/ http:////2222...21oog.... aaaaalllonn.066666 URL
org/ 2025. www.anaac-aaaaaa -ong-long.... 106/. 10
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh. Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763 PmLR, 2021. 5, 20
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relvaacce framework: Bm25 and beyond. Found. Trends Inf. Retr. 3:333-389 2009. URL https: semanticscholat org/
CorpusID 207178707 4
David Romero, Chenyang Lyu, Haryo Wibowo, Santiago Góngora, Aishik Mandal, Sukannya Emilio Cueva, Jinheon Baek, Soyeong Jeong, et al. Cvqa: Culturally-diverse multilingual visual question answering benchmark. Advances in Neural Information Processing Systems, 37:11479-11500 2025. 2, 3
Purkayastha, Jesus-German
Karsten Roth, Jae Myung Kim, Andrew Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata, Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15746-15757, 2023. 9
Wonduk Seo, Zonghao Yuan, and Yi Bu. Valuesrag: Enhancing cultural alignment through retrieval- augmented contextual learning. arXiv preprint wrXiv:2501.0033/ 2025. 2, 10
Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Nuria Oliver, and Alan Han- jalic. Climf: learning to maximize reciprocal rank with collaborative less-is-more filtering. In Proceedings of the sixth ACM conference on Recommender systems, pp. 139-146, 2012. 5
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems, 66:46345-463611 2023. 20, 21
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 1rXiv:2312./ /805, 2023. 17
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 2503. 2025. 21
Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. 2025. 5,7,20
Ramakrishna Vedantam, C Lawrence Ziticc, and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015. 5
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie- Yan Liu. A theoretical analysis of ndeg type ranking measures. In Conference on learning theory, pp. 25-54. PMLR, 2013. 5
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das. Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Gany Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz,
15

---

Published as a conference paper at ICLR 2026
Jan Wira Gotama Putra, Junho Myung. Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, and Chong- Wah Ngo. WorldCuisines; A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 3242- 3264, Albuquerque, New Mexico, April 2025, Association for Computational Linguistics. ISBN 979-8-89176- 189-6. 10
URL https: accl antho hogttyoloy....... org/ 20055 naacl- 167/. 1,
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture-of Mixture-of-experts lssionlanguuae models for advanced multimodal understanding. arXiv preprint arXiv:24 10302, 2024. 21
Srishti Yadav, Lauren Tilton, Maria Antoniak, Taylor Arnold, Jiaang Li, Siddhesh Milind Pawar, Anto- nia Karamolegkou, Stella Frank, Zhaochong An, Negar Rostamzadeh, et al. Cultural evaluations of vision-language models have lot to learn from cultural theory. arXiv preprint arXiv:2505.22793, 2025a. 3
Srishti Yadav, Zhi Zhang, Daniel Hershcovich, and Ekaterina Shutova. Beyond words: Exploring cultural value sensitivity multimodal models. Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguisties: NAACL 2025, pp. 7592-7608, Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics. ISBN 979- 8-89176-195-7. URL httpt:tt////.. aaclcaccccaoooo anthol ogy...............o........ .og. /2025. fi nd monnnngsann 422// 10
Yifei Yuan, Clemencia Siro. Mohammad Aliannejadi, Maarten de Rijke, and Wai Lam. Asking multimodal clarifying questions in mixed-initiative conversational search. In Proceedings of the ACM Web Conference 2024, pp. 1485-1485, 2024. 20, 21
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi dilii-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Tianyi Zhang*, Varsha Kishore*, Felix Wu Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: /ttp:r/://///peeeeee/ee/ tett://orrmunmd--eeee
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.1047) 2025. 21
16

---

Published as a conference paper at ICLR 2026
A THE USAGE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMs as writing aids, limited to polishing grammar, improving clarity, and refining readability. Moreover, during data curation, we used LLMs to generate culture-informed captions solely as auxiliary references for annotators; these LLM outputs were not directly included in the final dataset. All technical content, results, and conclusions are original and authored entirely by the listed contributors. The responsibility for the manuscript's 's scientific validity and integrity rests fully with the authors.
B LIMITATIONS
While RAVENEA establishes solid foundation for advancing the study of visual culture understanding with retrieval augmentation, it has three limitations that warrant future attention. First, due to budgetary constraints, the dataset's cultural scope is currently limited eight countries and eleven categories, Although this selection introduces meaningful diversity, it does not comprehensively represent the global spectrum cultural perspectives, particularly those of underrepresented or marginalized communities. Second, introduces inherent biases and may lack the depth, plurality, and contextual richness necessary for nuanced cultural interpretation. Finally, due resource limitations, we were unable include certain proprietary VL.Ms that require paid APIs, such as Gemini 2.5 Pro (Team et al., 2023) and Claude Opus 3.7 (Anthropic, 2024). We hypothesize that their performance would be comparable to GPT-4.1 1 (Achiam et al., 2023), which was included in our evaluation, but this remains an open empirical question.
of Wikipedia the primary external knowledge source
C FUTURE DIRECTIONS
Our work opens several avenues for advancing visual culture understanding multimodal models. In constructing RAVENEA, we algorithmically generated a raw dataset spanning 30 countries, comprising over 3,900 images and 78,000 documents (Table 4). Due to budget constraints, annotations were limited to data from eight countries. Therefore, extending RAVENEA to include more countries, cultural categories, and diverse knowledge sources, beyond Wikipedia, would improve coverage and reduce institutional bias. Second, future benchmarks could include richer tasks beyond eVQA and cIC, such as culture-groundee object recognition, historical retrieval, and symbolic interpretation, to better capture cultural semantics. Third, our results suggest need for culturally evaluation metrics, particularly for text generation. The limited effectiveness of retrieval augmentation in larger models also warrants further study, especially regarding how cultural knowledge is integrated and utilized. Exploring retrieval pipelines grounded in domain-specific knowledge bases curated by GLAM (Galleries, Libraries, Archives, and Museums) institutions could provide higher-quality cultural data and more reliable grounding. Finally, systematic human evaluation by cultural experts remains indispensable for validating both relevance and interpretive accuracy. Together, these directions aim to support the development of more culturally-sense sensiiive and globally robust vision- lisgun-language models.
D DATA STATISTICS
Table 4: Statistics of the RAVENEA dataset. The dataset constructed by curating existing sources and augmenting them with wiki-derived documents to broaden cultural knowledge coverage and enhance content diversity.
Dataset
Countries
Images
Documents
Pairs
Questions
Captions
CVQA selected CCUBaclected
7
1,213
8,194
12,130
2,331
CCUB selected
5
655
4,354
6,550
655
RAVENEA
8
1,868
11.396
18,680
2,331
655
BM25 filtered
30
3,912
78,240
We then apply cryptographic hashing (SHA-256) to identify and remove duplicate images, resulting in cleaner and more distinct cultural image set. Consequently, R RAVENEA comprises images collected
17

---

Published as a conference paper at ICLR 2026
Table 5: Statistics of images in each country in RAVENEA.
India
Mexico
Indonesia
Nigeria
China
Korea
Spain
Russia
347
309
223
214
142
77
Table 6: Statistics of images in each category in RAVENEA.
Architecture
Daily
History
Art
Companies
Sports
Transportation
Religion
403
402
347
278
86
80
73
68
52
31
21
from eight countries across four continents, spanning eleven distinct categories. The distribution of images by country and by category is detailed in Tables and 6, respectively. During constructing RAVENEA, we have constructed comprehensive, algorithmically generated raw dataset covering 30 countries, comprising over 3,900 images and 78,000 documents (see Table 4). Owing budgetary limitations, for RAVENEA, we have only annotated data from eight of these countries. Overall, we believe that both the raw algorithmically generated data and our human-labeles RAVENEA could be valuable source for improving the cultural understanding in VLM tasks.
E ANNOTATION DETAILS
Based the initial BM25 retrieval results, we refine the cultural relevance label of retrieved documents via human annotation. We found that directly asking annotators to rate overall cultural relevance on continuous scale (e.g., 0-10) led unreliable and inconsistent labels. This difficulty arises from several factors: (1) the semantic meaning of intermediate scores is ambiguous. (2) annotators tended to overemphasize a few salient visual elements (Chen et al., 2015), and (3) small numerical differences (e.g., between 5 and 6) often fail to reflect meaningful distinctions, especially given the cognitive load of processing lengthy Wikipedia documents, resulting in intra- annotator variance we decomposed cultural relevance into three interpretable and independently verifiable dimensions: country association, topic alignment, and explicit visual representation.
repeated examples. Instead, given an ceaag--captoondoccmee triplet,
Yes
Unclear
Country association
Topic alignment
Explicit visual representation
0.47
0.80
0.12
0.20
0.53
0.36
0.45
0.55
0.24
0.76
Migeria
0:35
0.56
0.45
0.55
0.29
0.71
Russia
0.42
0.52
0.54
0.46
0.61
0.39
Spain
0.56
0.38
0.55
0.45
0.39
0.61
Figure 8: Statistics on the Yes No response distributions for the annotation questions.
Prior to the annotation process, all annotators are required to carefully review a detailed instruction file outlining the relevance criteria and annotation guidelines. To ensure proper understanding of the guidelines, annotators are required to complete a mock annotation test and correctly answer all questions before proceeding with the actual annotation tasks. For each image-caption pair, annotators are presented with the top 10 Wikipedia documents retrieved by BM25. They are asked to assess whether each article provides meaningful background contextual information that is directly
18

---

Published as a conference paper at ICLR 2026
relevant to the cultural elements described in the caption or depicted in the image. The statistics of the annotations are presented in Figure
Table 7: Statistics of missing documents, along with the positive and negative ratios for each country.
Country
Positive & Negative Ratio(%)
Missing Ratio(%)
India
54.42 45.58
9.56
Russia
54.42 / 45.58
0.0
Spain
48.17 51.83
0.0
Nigeria
34.47 65.53
0,45
Mexico
38.37 61.63
25.55
Indonesia
41.26 58.74
0.32
Korea
69.59 30.41
19.59
China
56.12 / 43.88
10.75
As shown in Table 7, regions such as Mexico (25.55%) and Korea (19.59%) exhibit notably high missing ratios, underscoring the indispensable role human expertise in identifying culturally specific documents that purely automated retrieval methods fail to capture. This missing ratio serves as an informative diagnostic metric: higher values directly reflect the effectiveness our human- in-the-loop pipeline in enriching the dataset with highly local and otherwise overlooked content, ultimately attesting to the superior cultural coverage and quality of RAVENEA. Furthermore, Table reports the distribution of positive versus negative labels for document relevance across regions, offering additional insight into cultural representation variance.
F QUALITY CONTROL
To evaluate the quality of the GPT-4o-generated captions, randomly sampled 50 captions from the subset Chinese images, and employ two local checkers to check the quality in Table 8. Each annotator assessed the accuracy of the captions, and report their individual accuracies along with the IAA to measure consistency. Beyond objective correctness, we also asked annotators to provide subjective ratings (5- (5-point Likert scale) of caption quality along two dimensions: naturalness (fluency and readability) and completeness (coverage of visual content). The results further show the GPT-4o caption is suitable for the next step retrieve the Wikipedia documents.
Table 8: Generated caption quality check. IAA denotes inter-annotator agreement.
Acc. (Annotator 1)
Acc. (Annotator 2) IAA
Naturalness (1-5)
Completeness (1-5)
92%
94%
0.85
4.98
4.66
To ensure the quality and consistency of annotations, implement several quality control methods. First of all, prior to the annotation process, all annotators are required to carefully review detailed instruction file outlining the relevance criteria and annotation guidelines. To ensure proper understanding of the guidelines, annotators are required to complete a mock annotation test and correctly answer all questions before proceeding with the actual annotation tasks (see Figure 13). We also perform an additional quality check on subset of the dataset. Specifically, for each selected countries, we employ an additional local quality checker who is tasked with manually reviewing the annotations to verify their accuracy and adherence to the guidelines. The quality checker reviews a random sample of annotated items, focusing on both the relevance labels and the justification behind any edge cases, such as borderline relevance or use of the "Cannot be determined" label. If inconsistencies or deviations from the annotation guidelines are identified, the affected samples are flagged for re-annotation. The overall acceptance rate from the meta quality checkers is 98.2%. The inter-annotator agreement (IAA) Cohen's Kappa (k) between the meta checker and annotator on the sampled annotations is 0.83.
19

---

Published as a conference paper at ICLR 2026
G CORRELATION BETWEEN AUTOMATIC METRICS AND HUMAN JUDGMENTS
To assess the correlation between human preferences and automatic evaluation metrics, we com- pute Kendall's т rank correlation. Human annotations segmented according to the output chunks produced by corresponding 14 VLMs (no Qwen3VL family) with each retriever variant CLIP-based, and non-RAG). Within each segment, we calculate the se- lection winning ratio for each retrieval method, yielding a human preference vector formed by concatenating these ratios across all evaluation instances, For the automatic evaluation, we extract BERTScore, CIDEr, ROUGE-L, and CLIPScore for each corresponding retrieval variant and VLM. These scores are similarly concatenated into metric-based vector. Finally, we compute Kendall's between the human and metric vectors to quantify the consistency between automatic rankings and human judgments.
H DETAILS OF HUMAN EVALUATION IN THE CIC TASK
We randomly sample 10 images and generate captions using 14 VLMs under three configurations: Ravenea -CIPP CLIP, and no-retrieval baseline, yielding 420 captions in total." Four expert an- notators participated in the evaluation. For each image, they were presented with caption triplets-one retrieval configuration and asked to assess approximately 35 such triplets each. Annotators are instructed select the caption that most accurately and uppropriately reflects the cultural context depicted. To evaluate annotation consistency, we randomly sample 30 triplets and assign them to fifth expert annotator. Inter-annotator agreement (IAA) between this annotator and the original four, measured using Cohen's Ki is 0.595.
I IMPLEMENTATION DETAILS
1.1 DATASET.
Validation
400
We integrate responses from three annotation questions per data point into continuous scale ranging from -3 to 3, where higher values in- dicate stronger cultural relevance. To ensure fair evaluation across regions, we adopt the train-validation-test split strategy from class-
Number Samples
350
300
250
200
150
imbalance loss (Cui et al., 2019), yielding an approximate 85-5-10 data split. We also per- formed strict de-duplication on the training and validation sets, guaranteeing that identical documents appear across the training, valida- tion, testing splits. This setup guarantees that validation and test sets contain a balanced set of images per country in per task (Figure 9), thereby mitigating evaluation bias from skewed geographic distributions. During training, we employ random cropping for images as a sample-level augmentation to normalize the distribution of training instances across countries, further mitigating region- aggion-specffic sampling biases.
50
India
Mexico Indonesia Nigeria
China
Korea
Spain
Russa
Country
Figure 9: Data distributions across eight countries. Restrict the dataset to the images paired with at least one document annotated by human raters as culturally relevant.
1.2 MULTIMODAL RETRIEVERS.
We finetune HIIPLLL14@22-1 (Radford et al., 2021), SigLIP2-SO/14e 384px (Tschannen et al., 2025). We also finetune a VisualBERTThased (Li et al., 2019; 2020) reranker following standard BERT-style setups (Nogueira & Cho, 2019), and adapt two multimodal generative models, such as VL-T5 (Cho et al., 2021) and LLaVA-One Vision-n- (Li et al., 2025), for end-to-end document retrieval (Chen et al., 2022: Yuan al., 2024; Sun al., 2023). We integrate responses from three annotation questions per data point into continuous scale ranging from -3 to 3, where higher values indicate stronger cultural relevance.
The variants of the Qwen3-VT. models used were released the time of this evaluation.
20

---

Published as a conference paper at ICLR 2026
Table 9: Hyperparameters for finetuning on five models.
Hyperparameters
VERT
VL-T5
LLaVA-OV-7B
CLIP-L/14@ 224px
SigLIP2:50/1140384xx
batch size
32
64
Ir
2e-4
le-4
le-4
le-5
le-5
Ir warmup ratio
0.1
0.03
0.03
Max Epoch
100
20
10
2
Patience
10
10
early stopping
Yes
Yes
Yes
No
No
optimizer
Adam
Using LORA
No
No
No
Hyperparameters. In this work, we adopt different sets of hyperparameters VisualBERT, VL- T5, LLaVA-OV-7B, CLIP-L/14@224px, SigLIP2-SO/14G 504144 For VisualBERT, VL-T5, and
LLaVA-OV-7B we follow the setting in (Chen et al., 2022; Yuan et al., 2024; Sun et al., 2023). We show the training hyperparameters rerankering experiments for all models in Table All experiments are conducted using maximum of 2 Nvidia H100 GPUs.
1.3 VISION-LANGUAGE MODELS
Table 10: Model details: Hugging Face model names.
Model
Hugging Face Model Name
LLaVV--OneVision-- (Li al., 2025)
1lava LLava-bf/laate wwwwla/alloleeeleeeallaeeeeaaeee mi mttrooetf//hhii--rutmmmttaaaa-i- mf disttallcoom ttsttral-coommmmmtyy/ttttttt -ta-
www.ooneevnnnnnneeeo owen2 ab-ov-hit
1hii4-Multimoda (Abouelenin al., 2025)
Pixtral (Agrawal et al., 2024)
Qwen2.5VL family (Bai et al., 2025b)
Dxen//Ween22.-5V2-BB---atttttte
Deen/Qwen2.55-21-78--11tt
Qwen3VL family (Bai et al., 2025a)
Dxen//wen33-VL-322-
DeepSeek- VL2 family (Wu al., 2024)
deepseeirall/eepssekk=222
In3erVL33 family (Zhu et al., 2025)
Openoviaab/Inteerr/333333
OpencVLaab/Inter::V133
Openntt.abbIInteeen133----11
Gemma3 family (Team et al., 2025)
2oogle/gemma-3-444---
google/ google/gemma-32200 gemmmmmmmmmm-m--m
We benchmark open and -eight widely used VLMS on RAVENEA, leveraging various re- trievers against non-RAG baselines, assessing retrieval effectiveness across models of different sizes. The open-sourced models include LLaVA- -isVA--Oneiiioon-77 (Li et al., 2025), Pixtral-12B (Agrawal al., 2024), Phi-4 Multimodll-Insrrutt (Abouelenin al., 2025), Gemma3-44-Instte Instruct (Team et al., 2025), Qwen2.5-VLLInnstruuc (3B, 7B, 72B) (Bai et al., 2025b), Qwen3-VL- Instruct(3B, 7B, 32B), Intern (2B, 8B, 78B) (Zhu al., 2025), and Deepseek- VL2 variants (Tiny and Base) (Wu et al., 2024). For the closed models, we adopt GPT-4. (Achiam et al., 2023) (accessed on 2025/04/14) For proprietary model, 1. we directly call the corresponding API. For open-sourced models, we use vilm (Kwon et al., 2023). During the evaluation, to ensure the stability of the results, we set the temperature parameter to 0.0 and the maximum output length to 2048. All open-weight models are listed in Table 10. To avoid the impact of the length the retrieved content, we use the first 256 words in the top-1 Wikipedia document.
and 27B-
J MORE RESULTS
As illustrated in Figures 1a and 11b, the models exhibit diverse cultural preferences. Notably, most models achieve relatively stronger performance on Indian cultural contexts in both cIC and
knowledge cutoff: June 2024; https:ttt//........ //p /p1aap form. fow..openaaaa....mmmooooooaaaeee com/ docs /odee1 a /g.tt4.1
21

---

Published as a conference paper at ICLR 2026
Regionscore
Accuracy
China
Russia
Spain
China
Indla
Indla
80 100
Mexico
Mexico
Nigeria
Nigeria
Indonesia
CLIP Ravenea-CLL Without RAG
Figure 10: Average improvement across 16 open-source VLMs for different countries with 2 retrievers on both cIC RegionScore and eVQA accuracy.
cVQA tasks. As shown in Figures 10, VLMs demonstrate varying degrees of performance shifts across cultural contexts in both the cIC and cVQA tasks. cIC, Ravenea-CIIT yields noticeable improvements over their original counterparts in Korea, India and Nigeria, indicating the effectiveness of retrieval adaptation. improvement compared to non-RAG baselines, certain countries, such as Indonesia, experience exhibit diminishing returns.
cVQA, while Ravenea MonnneaCCII CCLIP generally exhibit large performance
Table 11: Alation at cultural context. Models in gray are frozen retrievers, noisy characters, and
unrelated documents. VLMMs augmented with finetuned retriever generally perform better.
DeepSeek- Ting
Qwen2.5-VL.-72 -2B
Phi4-Multinodal
Qwen2.5- Sven2.5-VL-3BB
InternVL3-2B
VL3-88
IntemVLL3788B
Gemma3-4B
Gemmu3-2715
Pixtral-12B
LLaVA-OV-7B
Retriever
Average
W/O RAG
66.0
49.8
62.7
52.6
83.3
64.1
71.3
84.2
66.5
79.4
42.6
73.2
62.7
Unrelated documents
60.6
46.4
53.6
76.1
60.8
67.0
82.3
59.8
74.6
41.1
63.2
43.5
Random Characters
62.3
44.0
57.9
52.6
79.4
63.2
69.9
83.7
54.1
75.1
41.1
68.9
57.4
CLIP-I/14@224px
69.4
50.2
63.6
66.0
82.8
73.7
78.5
84.2
69.4
76.6
67.0
76.6
44.5
Ravenea-Cia (ours)
71.2
50.7
64.1
69.9
82.3
76.1
81.3
85.2
69.9
79.9
69.4
75.1
50.2
SJCrestonSsooect
W/O RAG
54.4
26.5
53.1
69.4
71.4
38.8
49.0
65.3
73.5
75.5
28.6
53.1
49.0
Unrelated documents
48.8
18.4
46.9
57.1
61.2
28.6
51.0
63.3
59.2
71.4
24.5
51.0
53.1
Random Characters
51.2
24.5
44.9
61.2
67.3
36.7
44.9
67.3
61.2
73.5
28.6
59.2
44.9
CLIP-L/14@22pps
38.8
71.4
65.3
65.3
714
73.5
73.5
61.2
69.4
51.0
Ravenea- CLLIP (ours)
67.7
46.9
61.2
77.6
75.5
69.4
67.3
73.5
75.5
75.5
63.3
71.4
55.1
22

---

Published as a conference paper at ICLR 2026
Nigeria
Indonesia
Mexico
Spain
Russia
China
India
100
80
Accuracy
60
20
DS-VL2
D5-VL2-T
Gemma3-48
Genima3-278
IVL3-28
IVL3-88
IVL3-786
LlaVA-Ov-78
100
80
Accuracy
60
20
Phi4-MM
Pixtral-128
Q2.5-38
Q2.5-7B
Q2.5-728
Q3-88
Q3-326
(a) Performance of the 16 VLMs equipped with Ravenea-CLIP in cVQA task.
China
India
Korea
Mexico
Nigeria
RegionScore
80
60
40
20
DS-VL2
DS-VL2-T
Gemma3-48 Gemma3-2779 -78
IVL3-28
IVL3-88
IVL3-788
LlavA-DV-78
80
RegionScore
60
40
Ph14-MM
Pixtral-128
Q2.5-38
Q2.5-76
Q2.5-728
Q3-28
Q3-88
Q3-326
(b) Performance of the 16 VLMs equipped with Ravenea -CLIP in cIC task.
Figure 11: Performance of the 16 VLMs equipped with Ravenea CCIIP across tasks and countries.
23

---

Published as conference paper at ICLR 2026
Table 12: Four metrics comparison of RAG and Non-RAG methods for CCUB task.
DeepsSeek Tlay
DeepSeak- VL2
Qwen2.5-V1-3B
Qwen2.5-VL-7B
Owen2.5-V1-722
InterVLL3-2B
InterVVL3-8B
InternVL.33888
Gemma3-4B
Genma3-277
Phit-Maltimodal
Pistral-128
MaVA-OV-78
Method
Average
CBUBe
W/O RAG
18.0
19.4
18.1
24.3
18.3
17.1
17.4
17.0
15.3
14.4
15.2
15.1
23.6
Frozen Models
SigL.1P-S00/446 384px
14.9
11.6
16.9
18.7
16.1
14.8
14.0
15.3
16.2
13.0
12.6
12.4
15.0
17.3
CLIP-L/14@ 224px
15.3
13.6
18.1
19.8
17.0
14.8
13.1
15.1
15.5
12.9
13.8
12.9
15.0
17.3
Finetuned Models
VisualBERT
11.1
16.1
17.4
15.7
14.7
12.2
15.6
12.9
13.7
11.8
18.0
VL-TS
14.4
10.6
15.9
18.0
16.8
14.3
13.0
15.1
15.6
12.6
13.1
10.1
14.8
17.2
LLaVA-Ov78
10.0
17.9
16.1
1244
15.8
12.7
Savenea-SigLIP
14.8
11.7
16.4
17.3
15.5
15.5
13.3
16.1
15.6
13.2
13.0
13.0
15.1
16.9
Ravenea-CLiP
15.3
12.1
16.7
19.5
18.1
15.6
12.8
15.6
16.1
12.5
13.3
13.6
14.7
18.3
CCUBEIDER
RAG
22.2
37.4
26.1
60.7
19.1
11.0
5.7
20.2
10.8
7.5
6.3
18.0
59.2
Frozen Models
SigLIP2-S0114@884p 384x
12.5
9.8
2.0
0.6
20.5
CLIP- 224px
9.0
21.5
17.0
18.9
11.5
4.3
0.6
3.2
3.3
3.4
2.7
1.6
10.6
18.0
Finetuned Models
VisuaBBRR
9.5
19.3
16.0
19.9
9.2
4.1
0.5
6.0
1.8
3.1
6.1
1.3
18.1
18.6
VL-TS
8.9
0.2
3.5
0.9
17.5
LLaVA-OV-7B
7.1
17.9
10.5
21.6
6.1
0.7
1.2
4.2
0.1
2.4
0.4
1.2
14.4
11.3
Ravenea-SigL73
8.0
16.8
12.9
12.5
10.0
5.4
0.4
5.3
3.6
1.8
1.8
0.7
13.3
19.0
Ravenea-CLiP
16.8
10.7
1.3
1.1
23.3
CCU DBERTSErr
W/O RAG
53.6
53.3
55.3
56.2
54,5
53.2
51.7
54.2
53.6
50.1
51.4
51.9
53.5
57.5
Frozen Models
109.122-S0/143884p
50.9
46.7
52.3
54.2
53.1
50.4
48.9
51.8
51.9
48.1
50.3
49.3
51.2
53.7
CLIP-L/14@ 224px
51.9
49.7
53.1
54.5
53.7
51.6
49.4
52.8
52.5
49.2
50.9
49.8
52.7
55.0
Finetuned Models
VisualBERT
50.6
46.6
52.4
53.9
$2.8
50.2
48.4
51.9
51.3
47.8
50.1
48.1
51.4
53.5
VL-TS
50.2
46.2
51.9
54.4
52.8
50.0
47.7
51.0
51.3
47.7
49.7
46.8
50.7
52.8
LLaVA-Ov-7B
50.2
46.1
51.4
54.5
52,4
49.5
47.7
51.2
51.5
47.3
49.6
47.4
50.4
53.3
Ravenea-SigL.r
51.2
53.0
52.8
48.9
50.5
Ravenea--lin
52.5
49.2
54.2
54.9
54.2
52.9
49.3
53.2
53.0
49.0
51.4
51.3
53.4
56.4
BeL.PSEe
W/O RAG
33.4
33.3
34.1
32.5
33.8
33.4
34.1
34.3
34.5
33.9
33.8
32.4
32.6
32.2
Frozen Models
Sigl (tt.I///00140884pp
32.7
29.6
32.9
33.4
32.7
33.0
33.8
33.0
33.7
32.1
33.7
31.8
31.9
33.3
CI.P--1114@24pp 224px
33.4
33.2
34.1
32.7
33.6
Finetuned Models
VisuaBBERT
32.7
33.2
VL-T5
32.8
29.7
33.0
33.4
32.8
33.0
33.7
33.3
33.7
32.0
33.9
31.7
31.8
33.2
LLaVA-OV-7B
32.7
29.6
32.8
33.6
32.7
33.1
33.6
32.0
33.8
31.7
33.0
Seveenaa-iil.ii
32.7
29.8
33.0
32.1
33.1
33.5
33.8
33.1
33.8
32.7
33.7
31.7
32.0
33.1
Ravenea-CLIP
33.5
31.6
33.4
34.0
33.6
33.7
34.7
34.2
34.2
33.0
33.9
32.8
32.8
33.3
24

---

Published as a conference paper at ICLR 2026
K
PROMPTS
The prompts that we used to collect the documents and two downstream tasks were designed to ensure consistency across different models. One prompt example for cultural caption generation is shown in Table 13. The prompts for both with and without retrieval augmentation for VLMs are shown in Table and 14.
Table 13: Prompt example for generating captions used to retrieve the Wiki documents.
Prompi for data cullection
GPT-4o
SYSTEM: Default
USER:
related caption given image around sentences. Please inglude the name of the thing shown in the picture within the caption the three most eulturally zelevant entities in the image and attach the name et the end. Follow the fornat of: captien. entity namel, antity name2; entity name3.
Generate z
it's stronçly zelated to the culture. Name
Table 14: Prompt examples without RAG. Multimodal prompt samples with interleaved image are shown for both CVQA and CCUB tasks.
CVQA task
CCUB task
SYSTEM:
SYSTEM:
You are helpful assistent.
You are helpful assistant.
USER:
USER:
Answer the follawing mu tiple choice question for the imaqe. The last line must be of the following format: "Answer: SLETTER' twithout quotea] where LETTER must be one of A, B, e, D.
Wrice a congl se, one-sentence caption the given image. The geterated caption must conta'n n the ví sual content, culturally-reevant elements, and the country'a a name. Avoid phrases like "This image" oz The image'.
Question: which city is famous for such artwork?
A1 Zhe Zhejiang Yivu
B) Jingdezhen, Jiangxi
Batong, Shanx!
D) Zibo, Shandong
25

---

Published as a conference paper at ICLR 2026
Table 15: Prompt examples with RAG. Multimodal prompt samples with interleaved image are shown for both CVQA and CCUB tasks.
CVQA task
CCUB task
SYSTEM:
SYSTEM:
You are helpful assistent.
You are hhe pful staistant.
USER:
USER:
Wrica a canci se, onp-sentence caption
given image. The geterated caption must contain the vi oiaull content, wolturally-relevan and the country's a name. Avoid phzaaes like "This image" or "The image'. please consider the fol following DOCUMANT as supplementary material for the image. Mention country's name of the culture.
elements,
Answer the following mu miptepe choice question for the imagu. last line
Document:
following format: 'Answer: SLETTER' wwithout quates) whare e,
Architecture of Nigeria was historically influenced by envi ronmental conditions well social and cultural factors. The coming missicnaries and political changes brought about by co ordalll fan prosipitased a change in architectaral style and utility of buildings.
LETTER
Use the shared culture betwean image and fellowing document to answer the question.
Document:
Rui dinnggg and st ructures
Gothig revival style was adopted for early shurches built in the colony of lagos. A storey timber house building made with pre fabricated material componerts and designed with the influence of classic antiquity styles served as miseion house for the missionaries...
Buildings about 80C Berobudur completed.
temple in
Haeinsa of Korea, is constructed. Palare of Charlenegns in Aachen, Carolingian Empire completed (bequn about 7901. The Falatine Chapel still stands. At Ovindo in the Kingdom Asturias Cámara Santa conter. First reconstruction of Oviedo Cathedral begun by Tioda, Second Temple of Somnath built in the Pratihara Empire, India. 816 Cathedral begun. 310s
Chape" of San ženo in Santa Peassede, Raze decorated. 818
old Cologne Cathedral built....
question: Khat is the name the square where the cathedral and the statue of the image are located?
A Ricgo Square
B) Trascozrales Square
The aquare of Alfonso the
Chaste
D) The Fontan square
26

---

Published as a conference paper at ICLR 2026
L HUMAN ANNOTATION & EVALUATION INTERFACE
The annotation interfaces are shown in Figure 13 and 14. The interface for human evaluation cIC task shown in Figure 12.
Image Description Evaluation
China 22)
Image
Decrlprion
Figure Human evaluation interface for cIC task.
Anadtabbe wwseecteeeeeeeteee..
Sooting Ortiete
Cample::
Figure 13: Human annotation instructions.
27

---

Published as a conference paper at ICLR 2026
(a) Annotation interface (1).
Misiing Wil Document Foire
(b) Annotation interface (2).
Figure 14: Human annotation interfaces.
28


Reviews:
[{"review_id": "0gGIA1qBAD", "reviewer": "Reviewer_cUbB", "summary": "The paper introduces RAVENEA, a retrieval-augmented, multimodal benchmark for cultural understanding across eight countries and eleven categories, built by pairing images from CVQA and CCUB with human-ranked Wikipedia passages and evaluating two tasks: culture-focused VQA (cVQA) and culture-informed captioning (cIC). The authors also propose RegionScore, a simple region-mention metric for cIC that correlates with human judgments better than standard captioning metrics, and Culture-Aware Contrastive (CAC) fine-tuning to adapt multimodal retrievers (producing CaCLIP/CaSigLIP) that notably improve retrieval and downstream performance, especially for smaller VLMs. This analysis highlights persistent cross-cultural gaps though there are diminishing returns for large models with RAG.", "questions": "Minor stuff:\n\n- The paper mentions that the benchmark was derived from an initial pool of 30 countries but does not explain the rationale for narrowing down to eight. Clarifying this selection process would improve transparency and help readers assess dataset representativeness.\nLine 126: “capabilities of of VLMs” → “of VLMs.”\n\n- Several “under­standing / culturE uNdErstAnding” line breaks; clean hyphenation in camera-ready.\n\n- Line 469: “incoming level (Nwatu et al., 2025)” likely should be “income level.”\n\n- The introduction and Table 6 alternately say “over 10,000 documents” vs. “11,580 documents / 18,680 pairs.” Clarify whether “10k” is rounded or represents unique documents.\n\n- Appendix A (LLM usage) and Section 2.2: consider reiterating that GPT-4o captions were not included in the released gold text; they were only used for retrieval augmentation. This is stated, but repeating it in the main text would prevent confusion.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Well-founded benchmark design with human relevance supervision. The pipeline uses GPT-4o captions → BM25 over ~6M English Wikipedia pages → human re-ranking with a clear, 3-question taxonomy (country association, topic alignment, explicit visual representation). The methodology of creation is also well outlined and easy to understand, with good documentation.\n\n- Transparent quality control and strong agreement. There is multiple evidence that supervision is reliable and that the QA process was conducted at an acceptable standard, ensuring the reliability of the proposed benchmark and methodology. Caption checks show 92%/94% accuracies and agreement = 0.85. Moreover, the meta-check on annotations reports 98.2% acceptance and agreement = 0.83 and this prolongs the trend of faithful agreement between the data collection methodology line in line with how humans should have done.\n\n- Substantive retriever improvements. The proposed CAC loss function works well in improving the performance of the new retriever when trained with the RAVENEA benchmark, signaling that the proposed auxiliary loss matters in fine-tuning the learning process for this specific use case. This method can potentially be extended into other tasks since the three losses are generic and widely used heuristics (contrastive, ranking, and intra-modal diversity regularization).\n\n- Insightful findings on scale vs. retrieval and culture-specific gaps. RAG helps small VLMs substantially but shows diminishing or negative returns for the largest family members. The cross-country analysis shows persistent disparities, with weaker performance on specific regional subsets.", "weaknesses": "- Generalizability of the method. The paper emphasizes CAC and CaCLIP, which are trained and evaluated on RAVENEA. Without testing on external cultural datasets, it is difficult to claim general culture-aware retrieval superiority beyond “RAVENEA-specific specialization.” Evaluation on other datasets capturing similar concepts (e.g., CVQA, CCUB, CultureVLM, WorldCuisines, ALMBench) would strengthen the claim.\n\n- The proposed RegionScore still measures lexical proxy rather than cultural semantic significance. By design, RegionScore is 1 if the caption contains the country or demonym, and 0 otherwise. This means it can reward trivial country mentions and miss high-quality culturally specific text without a country token. It limits what conclusions the cIC task can carry, and can even be harmful if a caption hallucinates an irrelevant country reference while describing a different cultural element.\n--  The W/O RAG caption is: “Three individuals wearing vibrant traditional attire perform a dance outdoors.” This is generic. RegionScore = 0.\n-- The With CaCLIP caption is: “A group of people wearing colorful traditional Igbo attire perform the Egedege dance at an event.” This is an excellent, culturally specific caption that captures deep nuance. However, it does not contain the word “Nigeria” or “Nigerian.” RegionScore = 0.\n-- A hypothetical poor caption such as “This is a photo from Nigeria” would receive a higher RegionScore due to the occurrence of Nigeria. The metric is rewarding geotagging, not the cultural description it claims to measure. This is a conceptual flaw that undermines the cIC task, which is defined as assessing “captions that are sensitive to and incorporate cultural nuances.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IX2MG1byNY", "reviewer": "Reviewer_nXd8", "summary": "This paper introduces RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a benchmark designed to evaluate how RAG enhances visual culture understanding in vision-language models (VLMs). The benchmark comprises 1.8K instances spanning eight countries (China, Nigeria, Russia, Spain, Mexico, India, Indonesia, and Korea) across eleven cultural categories, incorporating over 10K human-ranked Wikipedia documents. The authors evaluate seven multimodal retrievers and fourteen VLMs on two primary tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). Key findings include: (1) cultural grounding annotations enhance multimodal retrieval performance, (2) lightweight VLMs benefit substantially from culture-aware retrieval (3.2% improvement on cVQA, 6.2% on cIC), and (3) performance varies significantly across countries, with some models showing cultural biases.​", "questions": "- What criteria were used to select 8 countries from the original 30 in CVQA? \n\n- You observe systematic underperformance on Nigerian and Indian contexts—what analysis was conducted to determine whether this stems from (a) annotation bias, (b) Wikipedia coverage gaps, or (c) model training data biases?\n\n- RegionScore measures surface-level geographic mentions rather than deep cultural understanding. How do you validate that explicit country/demonym references correlate with actual cultural comprehension rather than superficial pattern matching?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Timely Contribution**:\nThe paper addresses the relatively underexplored intersection of multimodal retrieval and cultural understanding. While RAG has been effective in text-based cultural reasoning, its application to vision-language understanding remains sparse. RAVENEA directly addresses this omission and establishes a systematic evaluation framework.\n\n- **Comprehensive Experimental Design**: \nThe evaluation encompasses seven retrievers and fourteen VLMs across multiple model families, providing thorough empirical coverage. The inclusion of both open-source and proprietary models enhances the generalizability of findings.\n\n- **Well-motivated methodology**:\nThe integration of CAC learning effectively extends the CLIP-style alignment paradigm to explicitly encode cultural supervision. The accompanying diversity loss formulation (Eq. 5) is well-motivated and technically sound. The use of RegionScore in response to deficiencies in automatic caption metrics is empirically validated through human correlation studies.", "weaknesses": "- **Limited Novelty Beyond Dataset Construction**  \nDespite the benchmark’s quality, the methodological innovation (CAC loss and RegionScore) remains incremental. The CAC objective essentially adapts contrastive alignment to a culturally labeled setup a relatively modest technical contribution. RegionScore, while intuitive, captures surface-level lexical cues (country/demonym mentions) rather than deeper cultural semantics.  \n\n- **Inadequate Relevance Annotation Scheme**:   \nThe paper employs a fundamentally flawed binary annotation approach that fails to capture the graded nature of cultural relevance. By reducing cultural understanding to three binary questions and artificially combining them into a continuous scale, the authors lose crucial ranking information essential for cultural understanding tasks. Established information retrieval research demonstrates that graded relevance judgments provide superior training signals compared to binary labels, particularly for nuanced tasks like cultural understanding. The absence of proper graded annotations or pairwise ranking comparisons undermines the effectiveness of the contrastive learning framework and limits the model's ability to distinguish between documents with varying degrees of cultural relevance.\n\n- **Absence of ablations and bias audits for the retrieval pipeline.**  \nBM25 is a purely lexical retriever and cannot capture the semantic or conceptual associations central to cultural understanding, I do not feel its the optimal choice for getting to Wikipedia documents for further annotation. No experiments contrast retrieval quality using (a) GPT captions vs. human-written captions, or (b) BM25 vs. dense/hybrid retrieval. Without such comparisons, it remains unknown whether the current pipeline systematically overlooks semantically relevant documents or skews toward certain linguistic patterns. Furthermore, the paper does not analyze how often annotators added “missing” documents or how BM25’s coverage varied by language/country. These missing statistics weaken claims of balanced cross-cultural representation.   \n\n- **Potential data leakage due to image-only train/test split.**    \nThe authors report an 85 / 5 / 10 split by images but provide no evidence of a document-level split. If the same Wikipedia document appears as a positive for both training and test images (e.g., “Sushi” page reused across splits), retrieval models can trivially memorize text and inflate performance. The paper does not mention any de-duplication or partitioning of documents, nor does it measure document overlap across splits.  \n\n- **Critical Gaps in Evaluation setup:**  \nThe authors fail to report essential statistics about their evaluation setup, specifically the positive vs. negative document ratios within their 10-document sets per instance. Without knowing whether 8/10 documents are relevant or 2/10 are relevant, it is impossible to assess task difficulty or whether reported improvements represent meaningful advances or artifacts of trivial evaluation scenarios. The paper shows they tested against random characters and unrelated documents as controls, but:​ No analysis of hard negative examples (culturally similar but incorrect documents).  \n\n- **Insufficient Analysis of Cultural Bias and Representation:**   \nWhile identifying performance disparities across countries (e.g., poor performance on Nigerian contexts vs. better performance on Korean contexts), the paper provides no deeper investigation into the underlying causes of these systematic biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pg2SWLB9ns", "reviewer": "Reviewer_N9q4", "summary": "This paper introduces RAVENEA, a new benchmark dataset and evaluation framework designed to assess systematic generalization, reasoning, and visual understanding in AI models. The benchmark is inspired by the RAVEN or RPM (Raven’s Progressive Matrices)-style tasks but extends them by introducing multi-dimensional relational reasoning challenges that involve complex scene configurations, compositional attributes, and logical rule dependencies.\n\nThe authors propose:\n\n- A novel dataset generation pipeline that controls for compositional structure and visual diversity, ensuring that models cannot overfit to superficial cues.\n- Several generalization splits (e.g., attribute, relation, and logic-based splits) to test extrapolative reasoning.\n- A comprehensive evaluation of existing architectures (CNNs, ViTs, relational networks, multimodal transformers, and neuro-symbolic models) on this benchmark, revealing clear performance gaps in out-of-distribution (OOD) reasoning.\n- Discussion and visualization of error patterns, showing that models often rely on shortcut heuristics rather than true reasoning.\n\nThe benchmark is positioned as a rigorous and diagnostic tool for studying systematic generalization and compositional reasoning in visual intelligence.", "questions": "When large models exhibit performance degradation under the RAG setting, have the authors analyzed whether this results from limited retrieval relevance (e.g., using only Top-1 document and 256 tokens) or from suboptimal fusion/integration strategies?\n\nAdditionally, have you experimented with multi-document or paragraph-level retrieval, or with tool-augmented reasoning mechanisms to test if the issue lies in retrieval depth or model utilization?\n\nMoreover, the selection of large models in the current evaluation does not appear sufficiently frontier.\nModels such as Qwen3, Gemini 2.5 Pro, and GPT-5 are among the most representative current vision-language systems, yet their results are not included. Including these stronger baselines would help demonstrate the true upper bound of RAG-based cultural reasoning performance and provide a more complete picture of model scaling trends.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Problem Definition and Task Design\n\nThe paper clearly defines the problem and presents tasks that align well with practical applications. It focuses on the scenario of “visual content + external cultural knowledge”, using two downstream tasks—cVQA (multiple-choice visual question answering) and cIC (image captioning)—to evaluate the full pipeline of retrieving → consuming knowledge → answering/generating.\n\n- Data Scale and Annotation Quality Control\n\nThe dataset covers 8 countries and 11 categories, totaling 1,868 instances.\nEach image is paired with 10 Wikipedia documents, which are manually re-ranked, resulting in 18,680 image–text pairs.\nThe annotation process includes three dimensions of cultural relevance (national association, thematic alignment, and visual presence in the image), inter-annotator agreement checks, and spot audits.\n\n- Methodological Contribution: Culturally Aware Contrastive Learning (CAC)\n\nWithin the CLIP/SigLIP framework, the authors introduce a combination of binary classification loss, ranking-margin loss, and text-side diversity regularization, specifically designed to enhance cultural relevance ranking (CaCLIP / CaSigLIP).", "weaknesses": "- Limited Domain Coverage and Source Bias\n\nAs acknowledged by the authors in the “Limitations” section, the dataset covers only 8 countries and 11 categories, and relies primarily on English Wikipedia articles. This setup is prone to cultural and regional exposure bias, which may affect the generalizability of the claimed cross-cultural conclusions.\n\nFuture work should incorporate GLAM (Galleries, Libraries, Archives, Museums) institutional collections and non-English knowledge sources to achieve a more balanced and representative dataset — a direction the authors themselves briefly mention in the ethics and outlook discussion.\n\n- Small and Inconsistent Human Evaluation for cIC\n\nThe human evaluation for the cIC (cultural image captioning) task is based on only 10 images, 4 annotators, and 420 sentences in total, with a moderate inter-rater agreement of κ ≈ 0.595.\n\nGiven the evaluation spans 14 models and multiple retrieval variants, the sample size is too limited to support robust conclusions.\nThe authors are encouraged to expand the human evaluation sample and include statistical significance testing to strengthen reliability.\n\n- Conservative RAG Configuration Potentially Underestimates Strong Models\n\nDuring evaluation, the retrieval-augmented generation (RAG) setup uses only the Top-1 document truncated to the first 256 tokens.\nThis constrained configuration may explain the phenomenon where large models fail to benefit under RAG.\n\nIt would be beneficial to explore multi-document fusion, longer context windows, re-ranking and rerouting strategies, or tool-augmented reasoning chains, in order to more fairly assess the upper bound of large-model + RAG performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFj7daqKrK", "reviewer": "Reviewer_F5ZE", "summary": "This paper introduces RAVENEA, a manually curated benchmark for visual culture understanding in multimodal systems. The benchmark focuses on two key tasks: culture-focused Visual Question Answering (cVQA) and culture-informed Image Captioning (cIC). It contains 1,868 multimodal instances and 18,680 human-ranked Wikipedia passages (10 per instance).\nEach instance is drawn from existing datasets (CVQA and CCUB) and is augmented with culturally relevant Wikipedia passages. Annotators re-rank the top-10 BM25-retrieved documents based on GPT-4o captioning and can also add any missing articles they consider relevant.\nThe authors evaluate seven multimodal retrievers and examine their impact when integrated into 14 widely used Vision-Language Models (VLMs). They also introduce Culture-Aware Contrastive Learning (CaCL), a fine-tuning approach applied to CLIP and SigLIP, to produce culturally sensitive retrievers (CaCLIP and CaSigLIP2), which consistently outperform their frozen counterparts.\nEmpirical results show that retrieval-augmented generation (RAG) provides the most substantial gains for lightweight VLMs.", "questions": "1) Retrieval quality control:\nWas any evaluation conducted to compare the quality of BM25-retrieved documents against those retrieved by modern text-only or multimodal retrievers? It would be helpful to know whether better initial retrieval could enhance the benchmark’s representativeness.\n\n2) Annotation statistics:\nCould the authors provide summary statistics on the yes/no response distributions for the three cultural relevance annotation questions (country association, topic alignment, visual depiction)? These would clarify dataset balance and inter-question correlations.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Comprehensive exploratory analysis:\nThe paper conducts an analysis of RAVENEA across tasks, model scales, and cultural dimensions.\n\n2) Human annotation:\nThe dataset construction involves human ranking of retrieved documents, supplemented by quality control and reporting of strong inter-annotator agreement scores.\n\n3) Novel training approach (CaCL):\nThe proposed Culture-Aware Contrastive Learning objective produces retrievers that generalize well across architectures. The released CaCLIP model can become a valuable resource for future multimodal RAG research.\n\n4) Responsible dataset curation:\nThe authors demonstrate attention to ethical concerns by implementing automated face detection and blurring of identifiable individuals in all images.", "weaknesses": "1) Dependence on GPT-4o and BM25 in data construction:\nSince image captions are generated by GPT-4o and used as text queries for BM25 retrieval, the initial dataset quality depends heavily on the performance and biases of these models. This reliance may explain why larger VLMs show limited improvements with RAG, as the retrieved contexts may be redundant relative to their internal knowledge.\nFuture iterations could improve recall and diversity by incorporating more diverse sources of models for generation and retrieval, including multimodal retrievers rather than relying solely on text-based BM25 or including visually similar images and multilingual sources.\nRAVENEA’s document set is capped at the top-10 BM25 hits (plus any extras an annotator happens to recall). If a genuinely relevant article never appears in that shortlist, it cannot be selected, so the benchmark measures only how well models re-rank a limited candidate pool rather than how well they retrieve unseen but correct evidence. I would like to see more quantitative evidence on how often this recall limitation occurs to ensure it does not significantly impact the dataset’s completeness or the conclusions drawn from it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)