RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding

## Abstract

Abstract As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiveness of retrieval-augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application in multimodal scenarios remains underexplored. To bridge this gap, we introduce Ravenea ( R etrieval- A ugmented V isual cultur E u N d E rst A nding), a new benchmark designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). Ravenea extends existing datasets by integrating over 10,000 Wikipedia documents curated and ranked by human annotators. With Ravenea , we train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented with culture-aware retrieval, outperform their non-augmented counterparts (by at least 3.2 % percent 3.2 3.2\% 3.2 % absolute on cVQA and 6.2 % percent 6.2 6.2\% 6.2 % absolute on cIC). This highlights the value of retrieval-augmented methods and culturally inclusive benchmarks for multimodal understanding.

## 1 Introduction

Vision-language models (VLMs) are increasingly deployed in real-world applications, from education to assistive technologies di2024educational; de2024vqask; bates-etal-2025-template; yao2025countllm, where understanding not only visual content but also the surrounding cultural context is crucial. Despite achieving impressive performance on general tasks yue2024mmmu; lin2014microsoft; an2025multimodality; an2025generalized, VLMs often struggle to capture cultural nuances, such as traditions, symbols, and region-specific practices that require external, culturally grounded knowledge (nayak-etal-2024-benchmarking,; liu2025culturevlm,; khanuja-etal-2024-image,; cao2024exploring,). For example, as shown in Figure [1](https://arxiv.org/html/2505.14462v1#S1.F1), a VLM may incorrectly identify the season of a festival scene as ‘Autumn’, overlooking that the image depicts Kyoto’s Gion Festival, which occurs in July and corresponds to ‘Summer’. A promising approach to address this limitation is the integration of external knowledge through retrieval-augmented generation (RAG) lewis2020retrieval, which has shown success in improving cultural awareness in language models (seo2025valuesrag,; lertvittayakumjorn2025towards,). However, prior work in the culture domain has predominantly been confined to text-only settings. Meanwhile, existing culture-related multimodal datasets primarily focus on evaluating VLM outputs on culturally oriented tasks, with limited emphasis on the integration of external cultural knowledge. As a result, the potential of RAG to improve multimodal cultural understanding remains underexplored.

Figure: Figure 1: Effectiveness of culture-aware RAG. Given a culturally grounded visual question, VLMs enhanced with culture-aware RAG—retrieving relevant Wikipedia documents—generate more accurate answers than their non-RAG counterparts. (see Section [5.2](https://arxiv.org/html/2505.14462v1#S5.SS2)).
Refer to caption: x4.png

To bridge the gap, we introduce Ravenea (Retrieval-Augmented Visual culturE uNdErstAnding), a manually curated dataset designed to evaluate cultural understanding in VLMs with retrieval support. We construct Ravenea based on two existing datasets: CVQA (romero2025cvqa,), which includes culturally relevant visual questions and corresponding answers, and CCUB (liu2023towards,), offering culturally contextualized captions to foster inclusivity in text-to-image generation (^1^11We reuse the cultural captions from CCUB as ground-truth references for the inverse task, image-to-text generation, specifically for culture-aware image captioning.). For each instance drawn from the source datasets, we append a set of Wikipedia documents that have been human-ranked based on their cultural relevance to the associated image. This curation effort, designed to ensure broad cultural representation, contains data related to eight countries and spans eleven diverse categories, comprising more than 1,800 images and 10,000 human-ranked documents. Ravenea thus provides a retrieval-augmented benchmark for evaluating cultural sensitivity in multimodal retrievers, and further allows for assessing how well VLMs integrate and make use of retrieved cultural context. Specifically, we focus on two culturally grounded tasks: (i) culture-focused visual question answering (cVQA) and (ii) culture-informed image captioning (cIC). We select these two tasks for their centrality in assessing cultural understanding in VLMs—question answering tests context-aware reasoning, while captioning evaluates generation sensitivity to cultural nuances.

With Ravenea, we first train and evaluate seven multimodal retrievers that use both visual and textual inputs to retrieve Wikipedia documents for a given query image based on their cultural relevance. Then, we evaluate a diverse set of state-of-the-art (SOTA) VLMs, including GPT-4.1 achiam2023gpt, LLaVA-OneVision-7B li2025llavaonevision, Pixtral agrawal2024pixtral, Phi-4 Multimodal abouelenin2025phi, the Gemma3 family team2025gemma, the Qwen2.5-VL family bai2025qwen2, the InternVL3 family zhu2025internvl3, and the Deepseek-VL2 family wu2024deepseek, each with and without multimodal retrieval, to assess the impact of retrieval augmentation on cultural understanding. Our dataset provides a testbed for assessing the cultural relevance capabilities of multimodal retrievers and the effectiveness of VLMs in consuming and using such retrieved cultural context.

Our contributions and key findings include:

- •
Ravenea benchmark: We introduce Ravenea, the first benchmark aimed at evaluating VLMs and multimodal retrieval in leveraging external knowledge for visual culture understanding. It comprises a diverse, large-scale collection of human-curated, culturally related documents linked to images from eight countries across eleven categories, enabling evaluation on two tasks: culture-focused visual question answering and culture-informed image captioning. (Section [3](https://arxiv.org/html/2505.14462v1#S3))
- •
Cultural grounding annotations enhance multi-modal retrieval: We evaluate seven retrievers that integrate visual and textual cues to retrieve culturally relevant documents. We find fine-tuning retrievers on culture-targeted annotations leads to marked gains in retrieval accuracy, highlighting the value of explicit cultural supervision. (Section [6.3](https://arxiv.org/html/2505.14462v1#S6.SS3))
- •
Benefits of culture-aware retrieval: Culture-aware retrieval boosts task performance across VLMs, with lightweight models showing the greatest improvement. This suggests that such retrieval can seamlessly integrate into downstream VLM tasks, enhancing their performance. (Section [5.2](https://arxiv.org/html/2505.14462v1#S5.SS2))
- •
Cross-cultural variation: Evaluation across eight countries reveals that VLMs exhibit distinct cultural preferences, with each model favoring different regional contexts—suggesting model-specific cultural biases. (Section [6.2](https://arxiv.org/html/2505.14462v1#S6.SS2))

Figure: Figure 2: Ravenea: A Multimodal Retrieval-Augmented Visual culturE uNdErstAnding dataset. Left: Examples of cVQA and cIC tasks. Middle: Geographic and categorical distribution of cultural references. Right: Performance comparison of 14 VLMs, evaluated with and without integration of our culture-aware retriever. Here, CaCLIP="culture-aware CLIP-L/14@224px ".
Refer to caption: x5.png

## 2 Related Work

Retrieval augmentation for cultural understanding.
Retrieval augmentation has demonstrated significant efficacy for culture-related NLP tasks
(conia-etal-2024-towards,; hu-etal-2024-bridging,; fung-etal-2023-normsage,).
Prior work uses sources like the World Values Survey for cultural question answering seo2025valuesrag; yadav-etal-2025-beyond, or retrieves web and knowledge base content to enhance cultural contextualization lertvittayakumjorn2025towards. In multimodal settings, however, cultural retrieval remains underexplored. While some approaches retrieve culture-relevant images to fine-tune VLMs liu2025culturevlm, they require additional training. In contrast, our method introduces a plug-and-play retrieval system that enriches cultural grounding at inference time without modifying the base model.

Vision-language culture datasets.
Several recent studies examine cultural understanding in VLMs through multicultural VQA liu-etal-2021-visually; nayak-etal-2024-benchmarking, cuisine recognition winata-etal-2025-worldcuisines; li-etal-2024-foodieqa, and concept-based image retrieval bhatia-etal-2024-local. Others benchmark cultural entity recognition using Wikipedia-based prompts nikandrou-etal-2025-crope or curate culture-specific image sets for value-based tasks yadav-etal-2025-beyond. While these efforts highlight current VLM limitations, they often rely on static, manually curated datasets. A natural extension to address these deficiencies is through culture-relevant multimodal retrieval—an area lacking dedicated datasets. Our work specifically aims to fill this critical research gap.

## 3 Ravenea Dataset

We construct our dataset by building culturally relevant document lists for each image in two existing culture-grounded datasets: CVQA romero2025cvqa, a widely used dataset for culture-focused VQA, and CCUB liu2023towards, a dataset designed to mitigate cultural bias in text-to-image generation with culturally contextualized captions. To ensure broad geographic and cultural coverage, we curate a diverse subset comprising images from seven countries in CVQA: China, Nigeria, Russia, Spain, Mexico, India, and Indonesia, and all five countries in CCUB: China, Korea, India, Mexico, and Nigeria (see Figure [2](https://arxiv.org/html/2505.14462v1#S1.F2)). After that, we separate the dataset construction process into three critical stages shown in Figure [3](https://arxiv.org/html/2505.14462v1#S3.F3).

### 3.1 Dataset Construction

Data collection.
The data collection process consists of two main steps:
(i) culture-related captioning and (ii) document retrieval. For culture-related captioning, we generate culturally grounded captions for each image to facilitate more effective attachment of relevant documents. Since the CVQA lacks captions and the CCUB provides only brief descriptions, we employ GPT-4o to generate richer, culturally informative captions (see the prompt example in Table [10](https://arxiv.org/html/2505.14462v1#A13.T10)).
For document retrieval, we first conduct a coarse filtering using the generated cultural captions as queries for a BM25 Robertson2009ThePR retriever to extract the semantically relevant documents from a large-scale corpus comprising over six million English Wikipedia documents (^2^22[https://huggingface.co/datasets/wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)). To mitigate the impact of inaccurate captions and ensure precise document relevance, we then perform human annotation on the retrieved documents.

Figure: Figure 3: Ravenea construction pipeline. Left: A two-stage retrieval process to match each image with relevant documents. Middle: Decomposition of cultural relevance into three interpretable dimensions to improve human annotation. Right: Postprocessing methods for quality control.
Refer to caption: x6.png

Relevance annotation.
Based on the initial BM25 retrieval results, we refine the cultural relevance label of retrieved documents via human annotation. For each image-caption pair, annotators are presented with the top 10 Wikipedia documents retrieved by BM25. They are asked to assess whether each document provides meaningful background or contextual information that is relevant to the culture described in the caption or the image (see Appendix [K](https://arxiv.org/html/2505.14462v1#A11)).
Specifically, we decomposed cultural relevance into three interpretable and independently verifiable dimensions: Country association: Is the topic of the Wikipedia article associated with the same country as the image and its caption? Topic alignment: Does the topic of the Wikipedia article align with the semantic category of the image and its caption? Explicit visual representation: Is the topic of the Wikipedia article explicitly mentioned or visually represented in the image and its caption? Each dimension is framed as a binary (True / False) question to reduce ambiguity and improve annotation consistency. However, for the country association dimension (the first listed), we introduce an additional label, "Cannot be determined", to handle cases where this association is unclear from the annotator’s perspective. Additionally, annotators are also instructed to include the title and URL of any relevant Wikipedia article they believe is missing from the top-10 retrieved results. These manually suggested articles are treated as the most cultural references closely related to the given image (see details in Appendix [G](https://arxiv.org/html/2505.14462v1#A7)).

Quality control.
To ensure the quality and consistency of our annotations, we implement several quality control methods. First of all, prior to the annotation process, all annotators are required to carefully review a detailed instruction file outlining the relevance criteria and annotation guidelines.

**Table 1: Statistics of the Ravenea dataset. The dataset is constructed by curating existing sources and augmenting them with over 10,000 wiki-derived documents to broaden cultural knowledge coverage and enhance content diversity.**
| Dataset | Images | Documents | Pairs | Questions | Captions |
| --- | --- | --- | --- | --- | --- |
| CVQA | 1,213 | 8,319 | 12,130 | 2,331 | - |
| CCUB | 655 | 4,441 | 6,550 | - | 655 |
| Ravenea | 1,868 | 11,580 | 18,680 | 2,331 | 655 |

To ensure proper understanding of the guidelines, annotators are required to complete a mock annotation test and correctly answer all questions before proceeding with the actual annotation tasks (see Figure [14](https://arxiv.org/html/2505.14462v1#A11.F14)). We also perform an additional quality check on a subset of the dataset. Specifically, for each selected countries, we employ an additional local quality checker who is tasked with manually reviewing the annotations to verify their accuracy and adherence to the guidelines. The quality checker reviews a random sample of annotated items, focusing on both the relevance labels and the justification behind any edge cases, such as borderline relevance or use of the “Cannot be determined” label. If inconsistencies or deviations from the annotation guidelines are identified, the affected samples are flagged for re-annotation. The overall acceptance rate from the meta quality checkers is 98.2%. The inter-annotator agreement (IAA) Cohen’s Kappa ($\kappa$) artstein2017inter between the meta checker and annotator on the sampled annotations is 0.83.

Figure: Figure 4: Examples demonstrating the impact of CaCLIP Wikipedia retrieval integration on cVQA and cIC tasks using DeepseekVL2-Tiny. When augmented with culture-aware retrieval, the model exhibits enhanced sensitivity to cultural context.
Refer to caption: x7.png

### 3.2 Dataset Statistics

We present the statistics of our dataset in Table [1](https://arxiv.org/html/2505.14462v1#S3.T1).
The dataset comprises a total of 1,868 culturally diverse images, with approximately 65% originating from CVQA and the remaining 35% from CCUB. Each image is paired with a GPT-4o-generated cultural caption, as well as the top-10 ranked Wikipedia documents retrieved via a cultural relevance scoring pipeline (Figure [3](https://arxiv.org/html/2505.14462v1#S3.F3)), yielding a total of 18,680 image-document pairs (see illustrated examples in Figure [4](https://arxiv.org/html/2505.14462v1#S3.F4)). The collection spans eight countries and encompasses a broad spectrum of cultural domains, such as traditional attire, festivals, architecture, cuisine, and social practices (see more statistics in Appendix [E](https://arxiv.org/html/2505.14462v1#A5)).

## 4 Culture-aware Multimodal Retriever

Leveraging the Ravenea dataset, we train and evaluate seven multimodal retrievers to retrieve culturally relevant Wikipedia documents using both visual and textual inputs. We fine-tune five representative models—spanning both generative and discriminative paradigms—to optimize multimodal document retrieval. Performance is evaluated using standard retrieval metrics, including Mean Reciprocal Rank (MRR) shi2012climf, Precision@k (P@k) jarvelin2017ir, and Normalized Discounted Cumulative Gain (nDCG@k) wang2013theoretical, where $k\in\{1,3,5\}$. We integrate responses from three annotation questions per data point into a continuous scale ranging from $-3$ to $3$, where higher values indicate stronger cultural relevance. We fine-tune a VisualBERT-based li2019visualbert; li-etal-2020-bert-vision reranker following standard BERT-style setups nogueira2019passage, and adapt two multimodal generators—VL-T5 cho2021unifying and LLaVA-OneVision-7B li2025llavaonevision—for end-to-end document retrieval chen2022corpusbrain; yuan2024asking; sun2023learning. To enhance cultural awareness in the contrastive retrieval, we introduce Culture-Aware Contrastive (CAC) learning, a supervised learning framework compatible with both CLIP and SigLIP architectures. We denote the culture-aware fine-tuned versions of CLIP-L/14@224px and SigLIP2-SO/14@384px using CAC as CaCLIP and CaSigLIP2, respectively.

### 4.1 Culture-aware Contrastive Learning

Given an image $I_{i}$ associated with $T$ textual descriptions $\{D_{i1},D_{i2},\dots,D_{iT}\}$, each document $D_{it}$ is annotated with a binary label $y_{it}\in\{0,1\}$, where $y_{it}=1$ indicates cultural relevance and $y_{it}=0$ indicates irrelevance. For each image–text pair $(I_{i},D_{it})$, we employ a shared vision-language encoder—such as CLIP—to obtain modality-specific representations: $\mathbf{E}_{I_{i}}=\mathcal{E}_{V}(I_{i})$ for the visual input and $\mathbf{E}_{D_{it}}=\mathcal{E}_{L}(D_{it})$ for the textual input. We then compute the cosine similarity score $s_{it}$ between $\mathbf{E}_{I_{i}}$ and each corresponding $\mathbf{E}_{D_{it}}$, resulting in a similarity vector $\mathbf{S}_{i}=[s_{i1},s_{i2},\dots,s_{iT}]$.

Culture-awareness classification now amounts to:

$$ $\mathcal{L}_{\text{Culture Classify}}=-\frac{1}{B\cdot T}\sum_{i=1}^{B}\sum_{j =1}^{T}\left[y_{ij}\log\sigma(s_{ij})+(1-y_{ij})\log(1-\sigma(s_{ij}))\right],$ (1) $$

where B is the number of images; $\sigma(\cdot)$ denotes the sigmoid function.

To prioritize culturally relevant descriptions in the ranking, we apply a margin ranking loss between all pairs of descriptions with differing cultural relevance. For each image $I_{i}$, we compare all pairs $(D_{ij},D_{ik})$ such that $y_{ij}=1$ and $y_{ik}=0$, and encourage the model to assign a higher similarity score to the relevant description. The ranking loss is defined as:

$$ $\mathcal{L}_{\text{Rank}}=\frac{1}{B}\sum_{i=1}^{B}\sum_{\begin{subarray}{c}j, k=1\\ y_{ij}=1,y_{ik}=0\end{subarray}}^{T}\max\left(0,\delta-(s_{ij}-s_{ik})\right),$ (2) $$

To mitigate the risk of overly similar positive text embeddings for the same image, we introduce a penalty that encourages intra-modal diversity among textual representations. We apply a diversity-promoting loss that forces the similarity between different text embeddings to be reduced while keeping each embedding highly similar to itself. Specifically, the penalty is formulated using an exponential function to emphasize the dissimilarity between embeddings:

$$ $\mathcal{L}_{\text{Diversity}}\left(\mathbf{S}_{i}\right)=-\sum_{t=1}^{T}\log \left(\frac{\exp(s_{it})}{\sum_{j=1}^{T}\exp(s_{ij})}\right)$ (3) $$

Then we can get the culture-aware contrastive loss:

$$ $\mathcal{L}_{\text{CAC}}=\frac{1}{3}\left(\mathcal{L}_{\text{Culture Classify} }+\mathcal{L}_{\text{Rank}}+\mathcal{L}_{\text{Diversity}}\right).$ (4) $$

Table: Table 2: Performance with different retriever models. Fine-tuned contrastive models consistently outperform their frozen counterparts across tasks. Here, "CaSigLIP"=Culture-aware SigLIP2-SO/14@384px, and "CaCLIP"=Culture-aware CLIP-L/14@224px. Models in gray are frozen.

### 4.2 Multimodal Retrieval Results

We perform a comprehensive evaluation of both frozen and fine-tuned retrievers, and present the results in Table [2](https://arxiv.org/html/2505.14462v1#S4.T2). We find that fine-tuned models, particularly those based on contrastive learning, consistently outperform their frozen counterparts. For instance, CaCLIP achieves a substantial improvement in P@1, rising from 54.58% to 65.42%, and sets a new SOTA across all evaluation metrics. Although SigLIP2-SO/14@384px also benefits from fine-tuning, the performance gains are comparatively modest. In contrast, models such as LLaVA-OneVision-7B, VL-T5, and VisualBERT lag behind after fine-tuning, even underperforming relative to frozen baselines. This underperformance likely stems from the fact that models such as LLaVA-OneVision-7B and VisualBERT were originally pretrained for generative tasks with different objectives, whereas CLIP-L/14@224px and SigLIP2-SO/14@384px were explicitly trained for similarity-based alignment, providing them with a structural advantage in retrieval settings.

## 5 Multimodal Retrieval-augmented Visual Culture Understanding

We then evaluate the effectiveness of these retrievers with 14 SOTA VLMs, spanning a diverse set of architectures. We conduct experiments on two downstream tasks: cVQA and cIC, respectively.

### 5.1 Experimental Setup

#### Models.

We benchmark open and closed-weight SOTA VLMs on Ravenea, leveraging various retrievers against non-RAG baselines, assessing retrieval effectiveness across models of different sizes.
The open-weight models include LLaVA-OneVision-7B li2025llavaonevision, Pixtral-12B agrawal2024pixtral, Phi-4 Multimodal-Instruct abouelenin2025phi, Gemma3-4B-Instruct and 27B-Instruct team2025gemma, Qwen2.5-VL-Instruct (3B, 7B, 72B (^3^33Due to computational constraints, we use the quantized version Qwen2.5-VL-Instruct-72B-AWQ.)) bai2025qwen2, InternVL3 (2B, 8B, 38B) zhu2025internvl3, and Deepseek-VL2 variants (Tiny and Base) wu2024deepseek. For the closed models, we adopt GPT-4.1 achiam2023gpt (accessed on 2025/04/14) (^4^44Knowledge cutoff: June 1, 2024; [https://platform.openai.com/docs/models/gpt-4.1](https://platform.openai.com/docs/models/gpt-4.1)).

#### Evaluation metrics.

**Table 3: Kendall’s $\tau$ rank correlation kendall1938new between automatic metrics and human judgments for the CCUB task. Statistically significant correlations ($p<0.05$) are marked with ✓. Our proposed metrics correlate stronger with human evaluation than the others.**
| Rouge-L lin-2004-rouge | CIDER vedantam2015cider | BERTScore Zhang*2020BERTScore: | CLIPScore hessel-etal-2021-clipscore | RegionScore (ours) |
| --- | --- | --- | --- | --- |
| -0.172 ✗ | -0.316 ✗ | -0.011 ✗ | 0.139 ✗ | 0.442 ✓ |

For the cVQA task, we use accuracy as the primary evaluation metric, which measures the proportion of correctly predicted answers.
For the cIC task, we employ several evaluation metrics including ROUGE-L lin-2004-rouge, CIDEr vedantam2015cider, BERTScore Zhang*2020BERTScore:, and CLIPScore hessel-etal-2021-clipscore, to assess the alignment between generated and reference captions across lexical, syntactic, and embedding-based levels.
To further evaluate the cultural relevance and human-perceived quality, we conduct a human evaluation study. We employed four researchers to select the most accurate caption from 14 VLMs (see details in Appendix [I](https://arxiv.org/html/2505.14462v1#A9) and [L](https://arxiv.org/html/2505.14462v1#A12)), and find a significant mismatch between automatic metric scores and human judgments of cultural appropriateness (see Table [3](https://arxiv.org/html/2505.14462v1#S5.T3)). To bridge this gap, we further introduce RegionScore, a novel evaluation metric designed to quantify cultural grounding (see details in Appendix [H](https://arxiv.org/html/2505.14462v1#A8) and Table [9](https://arxiv.org/html/2505.14462v1#A10.T9)). It measures how well captions identify the correct country names tied to cultural elements, adding geographic and cultural specificity in image captioning.

### 5.2 Overall Performance

We present the main results in Table [4](https://arxiv.org/html/2505.14462v1#S5.T4). The results demonstrate the efficacy of incorporating culture-aware retrieval augmentation. Employing fine-tuned retrievers yields substantial performance gains over both non-RAG and frozen retrievers baselines. Specifically, the CaCLIP achieves the highest average performance across both tasks, improves the accuracy for cVQA from 67.7% to 71.5%, and substantially improving the RegionScore for cIC from 40.2% to 58.1%. While CLIP-L/14@224px also offers improvements, fine-tuning consistently unlocks further potential. Furthermore, in the cVQA task, among all evaluated models, GPT-4.1 achieves the highest accuracy (86.8%) without RAG. Within the category of open-weight models, Qwen2.5-VL-72B leads with an accuracy of 81.0%. For lightweight models ($\leq$ 8B parameters), Qwen2.5-VL-7B achieves the best performance without RAG, reaching 67.7%. However, incorporating a CaCLIP significantly boosts performance, which enables InternVL3-8B to achieve 74.2% and outperform Qwen2.5-VL-7B by 0.6% with identical reranking. Notably, across multiple model families, augmenting the smallest variant with CaCLIP consistently elevates its performance to match or even exceed that of the next larger model tier. In the cIC task, with culture-aware contrastive learning, CaCLIP demonstrates substantial gains in identifying the culture in country-level of visual content, especially when built on top of VLMs with strong vision-language priors. It achieves the highest average RegionScore (58.1%) among the six reranking methods evaluated, with peak performance reaching 76.3% on the Gemma3-4B backbone. CaCLIP achieves leading the scores on 9 of 14 diverse suits of VLMs. This result underscores CaCLIP’s robustness and adaptability, particularly in culture-aware image captioning and retrieval tasks that demand fine-grained multimodal alignment.

Table: Table 4: cVQA and cIC Performance w/ and w/o RAG. Models in gray are frozen retrievers. Results are colored as Best. VLMs augmented with finetuned retriever generally perform better.

## 6 Analysis and Further Discussion

Culture-aware retrieval augmentation substantially benefits VLMs across both cVQA and cIC tasks, compared to their no-RAG counterparts. In this section, we explore the margin of the improvement, cultural preference and the effectiveness of cultural annotation.

### 6.1 Scaling Models Yields Diminishing or Negative Returns across Retrievers

Figure: Figure 5: Performance improvements for smallest and largest models per family with multimodal retrievers. Scaling models yields marginal gains with various retrievers, even negative effects in both cVQA and cIC tasks. "ACC." denotes accuracy; "R.S." refers to the RegionScore; "$\Delta$" represents the change incorporated with RAG compared to the non-RAG baseline.
Refer to caption: x8.png

In the cVQA task, within the same VLM family, performance differences between RAG and non-RAG approaches exhibit non-monotonic trends as model size scales. For all four model families, larger models show marginal or even negative returns from RAG integration. What’s more, sensitivity to RAG varies across model families. Notably, DeepSeek-VL2 demonstrates the most pronounced performance gap: the smallest model benefits from RAG with an average improvement of approximately +5%, whereas the largest model in the same family suffers a degradation of around -6% on average. In the cIC task, the effectiveness of RAG exhibits a consistent trend with respect to model scale within a given model family. Across all four model families evaluated, larger models tend to benefit less—or at most comparably—from the integration of RAG, suggesting diminishing returns at higher capacity. Among them, Gemma3 models show the smallest relative improvement, achieving approximately a +7% performance gain on average, whereas InternVL3 models yield the highest benefit, with performance gains reaching up to +30%.

### 6.2 Differences across Countries

We evaluate all models using CaCLIP across a range of countries for both tasks, as shown in Figure [6](https://arxiv.org/html/2505.14462v1#S6.F6). In the cVQA setting, most VLMs exhibit substantially diminished performance on culture-specific questions regarding Nigeria and Indonesia, in contrast to their performance on questions under other national contexts.
Interestingly, questions related to Spanish culture reveal high inter-model variance,

Figure: Figure 6: Performance of 14 VLMs with CaCLIP across different countries. Despite the integration of CaCLIP, disparities in model performance persist across countries.
Refer to caption: x9.png

with accuracy differentials reaching up to 35%, underscoring significant discrepancies in cultural representation across models. In the cIC task, VLMs consistently underperform on images and documents associated with Indian cultural contexts, while achieving the highest RegionScores on Korean culture-related inputs. Model performance on Indian culture is particularly volatile, indicating inconsistent cultural grounding across architectures. By comparison, Korean and Chinese cultural inputs yield more stable performance across models, suggesting entrenched model-specific preferences in cultural alignment (see more results in Appendix [J](https://arxiv.org/html/2505.14462v1#A10)).

### 6.3 Ablation Study on Annotation Questions

Figure: Figure 7: Ablation for different annotation questions. Combining all three culture-relevant annotation questions yields the best performance.
Refer to caption: x10.png

We further perform ablation studies across diverse combinations of annotation questions to assess their impact on downstream performance. Specifically, we evaluate 13 open-weight VLMs equipped with either CaSigLIP or CaCLIP, each trained on datasets constructed using varying subsets of culture-relevant annotations.
From Figure [7](https://arxiv.org/html/2505.14462v1#S6.F7), we can observe that leveraging all three questions (Q1 regarding country association; Q2 for topic alignment; Q3 for visual representation fidelity) yields the strongest performance on both cVQA and cIC tasks.
For the cVQA task, we find Q1 provides the most significant benefit to CaSigLIP, whereas CaCLIP gains more from Q2. Among all pairwise combinations, the joint supervision from Q1 (country associations) and Q2 (topic alignment) proves slightly more effective than other pairs. In the cIC task, both CaSigLIP and CaCLIP achieve better performance improvements when trained with data derived from Q1, compared to other single-question sets. For pairwise combinations, CaCLIP benefits most from the Q1+Q3 combination, while CaSigLIP shows a clear preference for the Q2+Q3 setup.

## 7 Conclusion

We introduce Ravenea, a novel benchmark dataset designed to comprehensively evaluate the cultural sensitivity of diverse multimodal retrievers across 14 SOTA VLMs. The benchmark consists of culturally contextualized queries and image captions from eight countries, curated and paired with Wikipedia passages ranked by human annotators based on their cultural relevance. Our findings highlight the potential of RAG for visual cultural understanding, particularly when lightweight VLMs are enhanced with culturally-aware multimodal retrievers such as CaCLIP, which consistently outperform their non-augmented counterparts – with the caveat that the largest models in each VLM family often exhibit diminishing returns when integrated with RAG. Notably, the inclusion of culture-sensitive questions during data annotation significantly improves effectiveness of multimodal retrievers and enhances performance in downstream tasks.

## 8 Acknowledgment

We would like to thank all the annotators for their work. Special thanks to Nico Lang, Peter Ebert Christensen, Zhaochong An, Stella Frank, Srishti Yadav, for providing helpful research advice. Serge Belongie and Jiaang Li are supported by the Pioneer Centre for AI, DNRF grant number P1. Ivan Vulić is supported by a personal Royal Society University Research Fellowship ‘Inclusive and Sustainable Language Technology for a Truly Multilingual World’ (no 221137). Wenyan Li is supported by the Lundbeck Foundation (BrainDrugs grant: R279-2018-1145).

## References

- (1)
Elisa Di Nuovo, Manuela Sanguinetti, PIER Balestrucci, Luca Anselma, Cristian Bernareggi, Alessandro Mazzei, et al.
Educational dialogue systems for visually impaired students: Introducing a task-oriented user-agent corpus.
In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5507–5519. ELRA and ICCL, 2024.
- (2)
Maria De Marsico, Chiara Giacanelli, Clizia Giorgia Manganaro, Alessio Palma, and Davide Santoro.
Vqask: a multimodal android gpt-based application to help blind users visualize pictures.
In Proceedings of the 2024 International Conference on Advanced Visual Interfaces, pages 1–5, 2024.
- (3)
Luke Bates, Peter Ebert Christensen, Preslav Nakov, and Iryna Gurevych.
A template is all you meme.
In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10443–10475, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
- (4)
Ziyu Yao, Xuxin Cheng, Zhiqi Huang, and Lei Li.
Countllm: Towards generalizable repetitive action counting via large language model.
arXiv preprint arXiv:2503.17690, 2025.
- (5)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024.
- (6)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755. Springer, 2014.
- (7)
Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Min Wu, Ming-Ming Cheng, Ender Konukoglu, and Serge Belongie.
Multimodality helps few-shot 3d point cloud semantic segmentation.
In The Thirteenth International Conference on Learning Representations, 2025.
- (8)
Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Junlin Han, Ender Konukoglu, and Serge Belongie.
Generalized few-shot 3d point cloud segmentation with vision-language model.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
- (9)
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal.
Benchmarking vision language models for cultural understanding.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5769–5790, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
- (10)
Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang.
Culturevlm: Characterizing and improving cultural understanding of vision-language models for over 100 countries.
arXiv preprint arXiv:2501.01282, 2025.
- (11)
Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig.
An image speaks a thousand words, but can everyone listen? on image transcreation for cultural relevance.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10258–10279, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
- (12)
Yong Cao, Wenyan Li, Jiaang Li, Yifei Yuan, Antonia Karamolegkou, and Daniel Hershcovich.
Exploring visual culture awareness in gpt-4v: A comprehensive probing.
arXiv preprint arXiv:2402.06015, 2024.
- (13)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in neural information processing systems, 33:9459–9474, 2020.
- (14)
Wonduk Seo, Zonghao Yuan, and Yi Bu.
Valuesrag: Enhancing cultural alignment through retrieval-augmented contextual learning.
arXiv preprint arXiv:2501.01031, 2025.
- (15)
Piyawat Lertvittayakumjorn, David Kinney, Vinodkumar Prabhakaran, Donald Martin, and Sunipa Dev.
Towards geo-culturally grounded llm generations.
arXiv preprint arXiv:2502.13497, 2025.
- (16)
David Romero, Chenyang Lyu, Haryo Wibowo, Santiago Góngora, Aishik Mandal, Sukannya Purkayastha, Jesus-German Ortiz-Barajas, Emilio Cueva, Jinheon Baek, Soyeong Jeong, et al.
Cvqa: Culturally-diverse multilingual visual question answering benchmark.
Advances in Neural Information Processing Systems, 37:11479–11505, 2025.
- (17)
Zhixuan Liu, Youeun Shin, Beverley-Claire Okogwu, Youngsik Yun, Lia Coleman, Peter Schaldenbrand, Jihie Kim, and Jean Oh.
Towards equitable representation in text-to-image synthesis models with the cross-cultural understanding benchmark (ccub) dataset.
arXiv preprint arXiv:2301.12073, 2023.
- (18)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- (19)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
LLaVA-onevision: Easy visual task transfer.
Transactions on Machine Learning Research, 2025.
- (20)
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al.
Pixtral 12b.
arXiv preprint arXiv:2410.07073, 2024.
- (21)
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al.
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.
arXiv preprint arXiv:2503.01743, 2025.
- (22)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
arXiv preprint arXiv:2503.19786, 2025.
- (23)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
arXiv preprint arXiv:2502.13923, 2025.
- (24)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
arXiv preprint arXiv:2504.10479, 2025.
- (25)
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al.
Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.
arXiv preprint arXiv:2412.10302, 2024.
- (26)
Simone Conia, Daniel Lee, Min Li, Umar Farooq Minhas, Saloni Potdar, and Yunyao Li.
Towards cross-cultural machine translation with retrieval-augmented generation from multilingual knowledge graphs.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024.
- (27)
Tianyi Hu, Maria Maistro, and Daniel Hershcovich.
Bridging cultures in the kitchen: A framework and benchmark for cross-cultural recipe retrieval.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, November 2024.
- (28)
Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji.
NORMSAGE: Multi-lingual multi-cultural norm discovery from conversations on-the-fly.
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. Association for Computational Linguistics.
- (29)
Srishti Yadav, Zhi Zhang, Daniel Hershcovich, and Ekaterina Shutova.
Beyond words: Exploring cultural value sensitivity in multimodal models.
In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 7592–7608, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
- (30)
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott.
Visually grounded reasoning across languages and cultures.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
- (31)
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, and Chong-Wah Ngo.
WorldCuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines.
In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3242–3264, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
- (32)
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott.
FoodieQA: A multimodal dataset for fine-grained understanding of Chinese food culture.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
- (33)
Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, EunJeong Hwang, and Vered Shwartz.
From local concepts to universals: Evaluating the multicultural understanding of vision-language models.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024.
- (34)
Malvina Nikandrou, Georgios Pantazopoulos, Nikolas Vitsakis, Ioannis Konstas, and Alessandro Suglia.
CROPE: Evaluating in-context adaptation of vision and language models to culture-specific concepts.
In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7917–7936, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
- (35)
Stephen E. Robertson and Hugo Zaragoza.
The probabilistic relevance framework: Bm25 and beyond.
Found. Trends Inf. Retr., 3:333–389, 2009.
- (36)
Ron Artstein.
Inter-annotator agreement.
Handbook of linguistic annotation, pages 297–313, 2017.
- (37)
Yue Shi, Alexandros Karatzoglou, Linas Baltrunas, Martha Larson, Nuria Oliver, and Alan Hanjalic.
Climf: learning to maximize reciprocal rank with collaborative less-is-more filtering.
In Proceedings of the sixth ACM conference on Recommender systems, pages 139–146, 2012.
- (38)
Kalervo Järvelin and Jaana Kekäläinen.
Ir evaluation methods for retrieving highly relevant documents.
In ACM SIGIR Forum, volume 51, pages 243–250. ACM New York, NY, USA, 2017.
- (39)
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu.
A theoretical analysis of ndcg type ranking measures.
In Conference on learning theory, pages 25–54. PMLR, 2013.
- (40)
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
Visualbert: A simple and performant baseline for vision and language.
arXiv preprint arXiv:1908.03557, 2019.
- (41)
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
What does BERT with vision look at?
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265–5275, Online, July 2020. Association for Computational Linguistics.
- (42)
Rodrigo Nogueira and Kyunghyun Cho.
Passage re-ranking with bert.
arXiv preprint arXiv:1901.04085, 2019.
- (43)
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.
Unifying vision-and-language tasks via text generation.
In International Conference on Machine Learning, pages 1931–1942. PMLR, 2021.
- (44)
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng.
Corpusbrain: Pre-train a generative retrieval model for knowledge-intensive language tasks.
In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 191–200, 2022.
- (45)
Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, and Wai Lam.
Asking multimodal clarifying questions in mixed-initiative conversational search.
In Proceedings of the ACM Web Conference 2024, pages 1474–1485, 2024.
- (46)
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren.
Learning to tokenize for generative retrieval.
Advances in Neural Information Processing Systems, 36:46345–46361, 2023.
- (47)
Maurice G Kendall.
A new measure of rank correlation.
Biometrika, 30(1-2):81–93, 1938.
- (48)
Chin-Yew Lin.
ROUGE: A package for automatic evaluation of summaries.
In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
- (49)
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.
Cider: Consensus-based image description evaluation.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
- (50)
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
In International Conference on Learning Representations, 2020.
- (51)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
CLIPScore: A reference-free evaluation metric for image captioning.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
- (52)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.
- (53)
AI Anthropic.
The claude 3 model family: Opus, sonnet, haiku.
Claude-3 Model Card, 1:1, 2024.
- (54)
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.
Class-balanced loss based on effective number of samples.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
- (55)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023.
- (56)
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco captions: Data collection and evaluation server.
arXiv preprint arXiv:1504.00325, 2015.

## Appendix A Limitations

While Ravenea establishes a solid foundation for advancing the study of visual culture understanding with retrieval augmentation, it has three limitations that warrant future attention. First, due to budgetary constraints, the dataset’s cultural scope is currently limited to eight countries and eleven categories. Although this selection introduces meaningful diversity, it does not comprehensively represent the global spectrum of cultural perspectives, particularly those of underrepresented or marginalized communities. Second, our use of Wikipedia as the primary external knowledge source introduces inherent biases and may lack the depth, plurality, and contextual richness necessary for nuanced cultural interpretation. Finally, due to resource limitations, we were unable to include certain proprietary VLMs that require paid APIs, such as Gemini 2.5 Pro [52] and Claude Opus 3.7 [53]. We hypothesize that their performance would be comparable to GPT-4.1, which was included in our evaluation, but this remains an open empirical question.

## Appendix B Future Directions

Our work opens several avenues for advancing visual culture understanding in multimodal models. First, expanding Ravenea to include more countries, cultural categories, and diverse knowledge sources, beyond Wikipedia, would improve coverage and reduce institutional bias. Second, future benchmarks could include richer tasks beyond cVQA and cIC, such as culture-grounded object recognition, historical retrieval, and symbolic interpretation, to better capture cultural semantics. Third, our results suggest a need for culturally-aware evaluation metrics, particularly for text generation. The limited effectiveness of retrieval augmentation in larger models also warrants further study, especially regarding how cultural knowledge is integrated and utilized. Together, these directions aim to support the development of more culturally-sensitive and globally robust vision-language models.

## Appendix C Ethics Statement

This work focuses on improving the cultural awareness of VLMs through retrieval-augmented methods. All data used in the construction of the Ravenea benchmark were sourced from publicly available datasets and Wikipedia, a community-curated open-access knowledge base. To protect individual privacy, we apply automated face detection and blur all identifiable faces in images prior to release. To mitigate cultural bias and ensure broad representation, the benchmark includes images and documents spanning eight countries and eleven cultural domains, curated and annotated by a diverse group of annotators. We provide detailed documentation of the annotation process and guidelines to support transparency and reproducibility. While enhancing cultural understanding is a central goal of this work, we acknowledge that culture is inherently complex, dynamic, and context-dependent. Consequently, the benchmark cannot capture the full richness of any cultural context. Finally, this work does not involve any personally identifiable information (PII), biometric data, or sensitive attributes. All human annotators were compensated fairly, and data annotation adhered to ethical guidelines for responsible research.

## Appendix D Experimental Details for Multi-modal Retrievers

#### Dataset.

We integrate responses from three annotation questions per data point into a continuous scale ranging from $-3$ to $3$, where higher values indicate stronger cultural relevance. To ensure fair evaluation across regions, we adopt the train-validation-test split strategy from class-imbalance loss [54], yielding an approximate 85-5-10 data split. This setup guarantees that validation and test sets contain a balanced set of images per country (Figure [8](https://arxiv.org/html/2505.14462v1#A4.F8)), thereby mitigating evaluation bias from skewed geographic distributions. During training, we employ random cropping for images as a sample-level augmentation to normalize the distribution of training instances across countries, further mitigating region-specific sampling biases.

Figure: Figure 8: Data distributions across eight countries. Restrict the dataset to the images paired with at least one document annotated by human raters as culturally relevant.
Refer to caption: x11.png

**Table 5: Hyperparameters for finetuning on five models.**
| Hyperparameters | VisualBERT | VL-T5 | LLaVA-OneVision-7B | CLIP-L/14@224px | SigLIP2-SO/14@384px |
| --- | --- | --- | --- | --- | --- |
| batch size | 32 | 128 | 4 | 64 | 64 |
| lr | 2e-4 | 1e-4 | 1e-4 | 1e-5 | 1e-5 |
| lr warmup ratio | - | 0.1 | - | - | - |
| weight decay | - | - | - | - | - |
| Max Epoch | 100 | 20 | 10 | 50 | 50 |
| Patience | 10 | 10 | 3 | 5 | 5 |
| early stopping | Yes |  |  |  |  |
| optimizer | AdamW |  |  |  |  |
| Using LORA | - | - | Yes | - | - |

#### Hyperparameters.

In this work, we adopt different sets of hyperparameters as VisualBERT, VL-T5, LLaVA-OneVision-7B, CLIP-L/14@224px, SigLIP2-SO/14@384px. For VisualBERT, VL-T5, and LLaVA-OneVision-7B we follow the setting in [44, 45, 46]. We show the training hyperparameters in rerankering experiments for all models in Table [5](https://arxiv.org/html/2505.14462v1#A4.T5). All experiments are conducted using a maximum of 2 Nvidia H100 GPUs.

## Appendix E Data Statistics

**Table 6: Statistics of images in each country.**
| India | Mexico | Indonesia | China | Nigeria | Korea | Spain | Russia |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 408 | 349 | 309 | 303 | 223 | 151 | 142 | 77 |

**Table 7: Statistics of images in each category.**
| Architecture | Cuisine | History | Art | Daily Life | Companies | Sports & Recreation | Transportation | Religion | Nature | Tools |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 403 | 402 | 278 | 275 | 185 | 80 | 73 | 68 | 52 | 31 | 21 |

We then apply cryptographic hashing (SHA-256) to identify and remove duplicate images, resulting in a cleaner and more distinct cultural image set. Consequently, Ravenea comprises images collected from eight countries across four continents, spanning eleven distinct categories. The distribution of images by country and by category is detailed in Tables [6](https://arxiv.org/html/2505.14462v1#A5.T6) and [7](https://arxiv.org/html/2505.14462v1#A5.T7), respectively.

## Appendix F Evaluation of VLMs

For closed-weight model, GPT-4.1, we directly call the corresponding API. For open-source models, we use vllm [55]. During the evaluation, to ensure the stability of the results, we set the temperature parameter to 0.0 and the maximum output length to 256. All open-weight models are listed in Table [8](https://arxiv.org/html/2505.14462v1#A6.T8). To avoid the impact of the length the retrieved content, we use the first 256 words in the top-1 Wikipedia document.

**Table 8: Model details: Hugging Face model names.**
| Model | Hugging Face Model Name |
| --- | --- |
| LLaVA-OneVision-7B [19] | llava-hf/llava-onevision-qwen2-7b-ov-hf |
| Phi-4-Multimodal [21] | microsoft/Phi-4-multimodal-instruct |
| Pixtral [20] | mistral-community/pixtral-12b |
| Qwen2.5VL family [23] | Qwen/Qwen2.5-VL-3B-Instruct<br>Qwen/Qwen2.5-VL-7B-Instruct<br>Qwen/Qwen2.5-VL-72B-Instruct-AWQ |
| DeepSeek-VL2 family [25] | deepseek-ai/deepseek-vl2<br>deepseek-ai/deepseek-vl2-tiny |
| InternVL3 family [24] | OpenGVLab/InternVL3-2B<br>OpenGVLab/InternVL3-8B<br>OpenGVLab/InternVL3-38B |
| Gemma3 family [22] | google/gemma-3-4b-it<br>google/gemma-3-27b-it |

## Appendix G Annotation Details

Based on the initial BM25 retrieval results, we refine the cultural relevance label of retrieved documents via human annotation. We found that directly asking annotators to rate overall cultural relevance on a continuous scale (e.g., 0–10) led to unreliable and inconsistent labels. This difficulty arises from several factors: (1) the semantic meaning of intermediate scores is ambiguous, (2) annotators tended to overemphasize a few salient visual elements [56], and (3) small numerical differences (e.g., between 5 and 6) often fail to reflect meaningful distinctions, especially given the cognitive load of processing lengthy Wikipedia documents, resulting in intra-annotator variance even on repeated examples. Instead, given an image–caption–document triplet, we decomposed cultural relevance into three interpretable and independently verifiable dimensions: country association, topic alignment, and explicit visual representation.

Prior to the annotation process, all annotators are required to carefully review a detailed instruction file outlining the relevance criteria and annotation guidelines. To ensure proper understanding of the guidelines, annotators are required to complete a mock annotation test and correctly answer all questions before proceeding with the actual annotation tasks. For each image-caption pair, annotators are presented with the top 10 Wikipedia documents retrieved by BM25. They are asked to assess whether each article provides meaningful background or contextual information that is directly relevant to the cultural elements described in the caption or depicted in the image.

## Appendix H A New Metric: RegionScore

To evaluate the extent to which captions reference specific geopolitical regions, we introduce a RegionScore. This metric measures whether a caption contains explicit references to a country or its common demonyms. $N$ be the total number of samples. $\text{PRED}_{i}$ denote predicted captions for the $i$-th sample. $C_{i}$ be the country name associated with the $i$-th sample. $\text{Adj}(C_{i})$ denote the set of adjectives or demonyms associated with $C_{i}$. $\mathcal{T}_{i}=\{C_{i}\}\cup\text{Adj}(C_{i})$ denote the set of region-related terms for the $i$-th sample.
We define binary indicators for each sample:

$$ $\delta^{\text{pred}}_{i}=\begin{cases}1&\text{if any }t\in\mathcal{T}_{i}\text { appears in }\text{PRED}_{i}\\ 0&\text{otherwise}\end{cases}$ (5) $$

The RegionScore for predicted captions are then computed as:

$$ $\text{RegionScore}_{\text{PRED}}=\frac{1}{N}\sum_{i=1}^{N}\delta^{\text{pred}} _{i}$ (6) $$

These scores reflect the proportion of captions that include explicit regional identifiers. A higher score indicates stronger region-awareness in the captioning. For the ground truth in CCUB dataset, the $\text{RegionScore}_{\text{GT}}=99\%$.

## Appendix I Details of Human Evaluation in the cIC Task

We randomly sample 10 images and generate captions using 14 vision-language models (VLMs) under three configurations: CaCLIP, CLIP, and a no-retrieval baseline, yielding 420 captions in total. Four expert annotators participated in the evaluation. For each image, they were presented with caption triplets—one per retrieval configuration—and asked to assess approximately 35 such triplets each. Annotators are instructed to select the caption that most accurately and appropriately reflects the cultural context depicted. To evaluate annotation consistency, we randomly sample 30 triplets and assign them to a fifth expert annotator. Inter-annotator agreement (IAA) between this annotator and the original four, measured using Cohen’s $\kappa$, is 0.595.

## Appendix J Additional Results

Figure: Figure 9: Performance of the 14 VLMs equipped with CaCLIP in cVQA task.
Refer to caption: x12.png

As illustrated in Figures [9](https://arxiv.org/html/2505.14462v1#A10.F9) and [10](https://arxiv.org/html/2505.14462v1#A10.F10), the models exhibit diverse cultural preferences. Notably, most models achieve relatively stronger performance on Chinese and Indian cultural contexts in the cVQA task, and on Chinese and Korean contexts in the cIC task.

Figure: Figure 10: Performance of the 14 VLMs equipped with CaCLIP in cIC task.
Refer to caption: x13.png

We report the CIDEr, ROUGE-L, BERTScore, and CLIPScore metrics for 14 VLMs on the cIC task in Table [9](https://arxiv.org/html/2505.14462v1#A10.T9).

Table: Table 9: Four metrics comparison of RAG and Non-RAG methods for CCUB task.

Figure: Figure 11: Average improvement across 14 VLMs in different countries with 4 retrievers for cVQA task.
Refer to caption: x14.png

As shown in Figures [12](https://arxiv.org/html/2505.14462v1#A10.F12) and [11](https://arxiv.org/html/2505.14462v1#A10.F11), VLMs demonstrate varying degrees of performance shifts across cultural contexts in both the cIC and cVQA tasks. In the cIC setting, most fine-tuned retrievers yield noticeable improvements over their original counterparts, indicating the effectiveness of retrieval adaptation. Performance on the cVQA task reveals more nuanced outcomes. While fine-tuned retrievers generally exhibit large performance improvement compared to non-RAG baselines, certain countries, such as Spain and Indonesia, experience exhibit diminishing returns. These discrepancies may stem from the limited presence of culturally representative visual content in the training data. Although targeted image augmentation strategies were employed to alleviate this imbalance, the results suggest that data distribution remains a significant bottleneck. Understanding and addressing such cross-cultural performance disparities in multimodal tasks like cVQA will be an important direction for future work.

Figure: Figure 12: Average improvement across 14 VLMs in different countries with 4 retrievers in cIC task.
Refer to caption: x15.png

## Appendix K Human Annotation & Evaluation Interface

The annotation interface are shown in Figure [14](https://arxiv.org/html/2505.14462v1#A11.F14) and [15](https://arxiv.org/html/2505.14462v1#A11.F15). The interface for human evaluation in cIC task is shown in Figure [13](https://arxiv.org/html/2505.14462v1#A11.F13).

Figure: Figure 13: Human evaluation interface for cIC task.
Refer to caption: x16.png

Figure: Figure 14: Human annotation instructions.
Refer to caption: x17.png

Figure: (a) Annotation interface (1).
Refer to caption: x18.png

## Appendix L Correlation between Automatic Metrics and Human Judgments

To assess the correlation between human preferences and automatic evaluation metrics, we compute Kendall’s Tau rank correlation. Human annotations are segmented according to the output chunks produced by corresponding 14 VLMs with each retriever variant (CaCLIP-based, CLIP-based, and non-RAG). Within each segment, we calculate the selection winning ratio for each retrieval method, yielding a human preference vector formed by concatenating these ratios across all evaluation instances. For the automatic evaluation, we extract BERTScore, CIDEr, ROUGE-L, and CLIPScore for each corresponding retrieval variant and VLM. These scores are similarly concatenated into a metric-based vector. Finally, we compute Kendall’s Tau between the human and metric vectors to quantify the consistency between automatic rankings and human judgments.

## Appendix M Prompts

The prompts that we used to collect the documents and two downstream tasks were designed to ensure consistency across different models. One prompt example for cultural caption generation is shown in Table [10](https://arxiv.org/html/2505.14462v1#A13.T10). The prompts for both with and without retrieval augmentation for VLMs are shown in Table [12](https://arxiv.org/html/2505.14462v1#A13.T12) and [11](https://arxiv.org/html/2505.14462v1#A13.T11).

Table: Table 10: Prompt example for generating captions used to retrieve the Wiki documents.

**Table 11: Prompt examples without RAG. Multimodal prompt samples with interleaved image are shown for both CVQA and CCUB tasks.**
| CVQA task | CCUB task |
| --- | --- |
| SYSTEM:<br>You are a helpful assistant.<br>USER:<br>Answer the following multiple choice question. The last line of your response must be of the following format: Ánswer: $LETTER(́without quotes) where LETTER is one of ABCD.<br>Question: Which city is famous for such artwork?<br>A) Zhejiang Yiwu<br>B) Jingdezhen, Jiangxi<br>C) Datong, Shanxi<br>D) Zibo, Shandong | SYSTEM:<br>You are a helpful assistant.<br>USER:<br>Write a concise, one-sentence caption for the given image. The generated caption must contain the visual content and culturally relevant elements of the image. Avoid explicit references to the image itself (e.g., "This image shows...", "Pictured here is...", "In this photograph..."). Do not generate multiple options. |

**Table 12: Prompt examples with RAG. Multimodal prompt samples with interleaved image are shown for both CVQA and CCUB tasks.**
| CVQA task | CCUB task |
| --- | --- |
| SYSTEM:<br>You are a helpful assistant.<br>USER:<br>Answer the following multiple choice question. The last line of your response must be of the following format: Ánswer: $LETTER(́without quotes) where LETTER is one of ABCD.<br>The scope of the question is strictly limited to the given image. However, please analyze and incorporate information from both the image and the following document to answer the question.<br>Document:<br>Buildings and structures Buildings about 800 – Borobudur temple in Java completed. 802 Haeinsa of Korea, is constructed. Palace of Charlemagne in Aachen, Carolingian Empire completed (begun about 790). The Palatine Chapel still stands. At Oviedo in the Kingdom of Asturias Cámara Santa constructed. First reconstruction of Oviedo Cathedral begun by Tioda. 815 – Second Temple of Somnath built in the Pratihara Empire, India. 816 – Reims Cathedral begun. 810s – Chapel of San Zeno in Santa Prassede, Rome decorated. 818 – Old Cologne Cathedral built....<br>Question: What is the name of the square where the cathedral and the statue of the image are located?<br>A) Riego Square<br>B) Trascorrales Square<br>C) The square of Alfonso II the Chaste<br>D) The Fontan square | SYSTEM:<br>You are a helpful assistant.<br>USER:<br>Write a concise, one-sentence caption for the given image. The generated caption must contain the visual content and culturally relevant elements of the image. Avoid explicit references to the image itself (e.g., "This image shows...", "Pictured here is...", "In this photograph..."). Do not generate multiple options. Please consider the following context:<br>Architecture of Nigeria was historically influenced by environmental conditions as well as social and cultural factors. The coming of missionaries and political changes brought about by colonialism precipitated a change in architectural style and utility of buildings. A Gothic revival style was adopted for early churches built in the colony of Lagos. A one or two storey timber house building made with pre-fabricated material components and designed with the influence of classic antiquity styles served as mission house for the missionaries... |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Minor stuff:

- The paper mentions that the benchmark was derived from an initial pool of 30 countries but does not explain the rationale for narrowing down to eight. Clarifying this selection process would improve transparency and help readers assess dataset representativeness.
Line 126: “capabilities of of VLMs” → “of VLMs.”

- Several “under­standing / culturE uNdErstAnding” line breaks; clean hyphenation in camera-ready.

- Line 469: “incoming level (Nwatu et al., 2025)” likely should be “income level.”

- The introduction and Table 6 alternately say “over 10,000 documents” vs. “11,580 documents / 18,680 pairs.” Clarify whether “10k” is rounded or represents unique documents.

- Appendix A (LLM usage) and Section 2.2: consider reiterating that GPT-4o captions were not included in the released gold text; they were only used for retrieval augmentation. This is stated, but repeating it in the main text would prevent confusion.
soundness: 3
strengths: - Well-founded benchmark design with human relevance supervision. The pipeline uses GPT-4o captions → BM25 over ~6M English Wikipedia pages → human re-ranking with a clear, 3-question taxonomy (country association, topic alignment, explicit visual representation). The methodology of creation is also well outlined and easy to understand, with good documentation.

- Transparent quality control and strong agreement. There is multiple evidence that supervision is reliable and that the QA process was conducted at an acceptable standard, ensuring the reliability of the proposed benchmark and methodology. Caption checks show 92%/94% accuracies and agreement = 0.85. Moreover, the meta-check on annotations reports 98.2% acceptance and agreement = 0.83 and this prolongs the trend of faithful agreement between the data collection methodology line in line with how humans should have done.

- Substantive retriever improvements. The proposed CAC loss function works well in improving the performance of the new retriever when trained with the RAVENEA benchmark, signaling that the proposed auxiliary loss matters in fine-tuning the learning process for this specific use case. This method can potentially be extended into other tasks since the three losses are generic and widely used heuristics (contrastive, ranking, and intra-modal diversity regularization).

- Insightful findings on scale vs. retrieval and culture-specific gaps. RAG helps small VLMs substantially but shows diminishing or negative returns for the largest family members. The cross-country analysis shows persistent disparities, with weaker performance on specific regional subsets.
summary: The paper introduces RAVENEA, a retrieval-augmented, multimodal benchmark for cultural understanding across eight countries and eleven categories, built by pairing images from CVQA and CCUB with human-ranked Wikipedia passages and evaluating two tasks: culture-focused VQA (cVQA) and culture-informed captioning (cIC). The authors also propose RegionScore, a simple region-mention metric for cIC that correlates with human judgments better than standard captioning metrics, and Culture-Aware Contrastive (CAC) fine-tuning to adapt multimodal retrievers (producing CaCLIP/CaSigLIP) that notably improve retrieval and downstream performance, especially for smaller VLMs. This analysis highlights persistent cross-cultural gaps though there are diminishing returns for large models with RAG.
weaknesses: - Generalizability of the method. The paper emphasizes CAC and CaCLIP, which are trained and evaluated on RAVENEA. Without testing on external cultural datasets, it is difficult to claim general culture-aware retrieval superiority beyond “RAVENEA-specific specialization.” Evaluation on other datasets capturing similar concepts (e.g., CVQA, CCUB, CultureVLM, WorldCuisines, ALMBench) would strengthen the claim.

- The proposed RegionScore still measures lexical proxy rather than cultural semantic significance. By design, RegionScore is 1 if the caption contains the country or demonym, and 0 otherwise. This means it can reward trivial country mentions and miss high-quality culturally specific text without a country token. It limits what conclusions the cIC task can carry, and can even be harmful if a caption hallucinates an irrelevant country reference while describing a different cultural element.
--  The W/O RAG caption is: “Three individuals wearing vibrant traditional attire perform a dance outdoors.” This is generic. RegionScore = 0.
-- The With CaCLIP caption is: “A group of people wearing colorful traditional Igbo attire perform the Egedege dance at an event.” This is an excellent, culturally specific caption that captures deep nuance. However, it does not contain the word “Nigeria” or “Nigerian.” RegionScore = 0.
-- A hypothetical poor caption such as “This is a photo from Nigeria” would receive a higher RegionScore due to the occurrence of Nigeria. The metric is rewarding geotagging, not the cultural description it claims to measure. This is a conceptual flaw that undermines the cIC task, which is defined as assessing “captions that are sensitive to and incorporate cultural nuances.”

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: - What criteria were used to select 8 countries from the original 30 in CVQA? 

- You observe systematic underperformance on Nigerian and Indian contexts—what analysis was conducted to determine whether this stems from (a) annotation bias, (b) Wikipedia coverage gaps, or (c) model training data biases?

- RegionScore measures surface-level geographic mentions rather than deep cultural understanding. How do you validate that explicit country/demonym references correlate with actual cultural comprehension rather than superficial pattern matching?
soundness: 2
strengths: - **Timely Contribution**:
The paper addresses the relatively underexplored intersection of multimodal retrieval and cultural understanding. While RAG has been effective in text-based cultural reasoning, its application to vision-language understanding remains sparse. RAVENEA directly addresses this omission and establishes a systematic evaluation framework.

- **Comprehensive Experimental Design**: 
The evaluation encompasses seven retrievers and fourteen VLMs across multiple model families, providing thorough empirical coverage. The inclusion of both open-source and proprietary models enhances the generalizability of findings.

- **Well-motivated methodology**:
The integration of CAC learning effectively extends the CLIP-style alignment paradigm to explicitly encode cultural supervision. The accompanying diversity loss formulation (Eq. 5) is well-motivated and technically sound. The use of RegionScore in response to deficiencies in automatic caption metrics is empirically validated through human correlation studies.
summary: This paper introduces RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a benchmark designed to evaluate how RAG enhances visual culture understanding in vision-language models (VLMs). The benchmark comprises 1.8K instances spanning eight countries (China, Nigeria, Russia, Spain, Mexico, India, Indonesia, and Korea) across eleven cultural categories, incorporating over 10K human-ranked Wikipedia documents. The authors evaluate seven multimodal retrievers and fourteen VLMs on two primary tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). Key findings include: (1) cultural grounding annotations enhance multimodal retrieval performance, (2) lightweight VLMs benefit substantially from culture-aware retrieval (3.2% improvement on cVQA, 6.2% on cIC), and (3) performance varies significantly across countries, with some models showing cultural biases.​
weaknesses: - **Limited Novelty Beyond Dataset Construction**  
Despite the benchmark’s quality, the methodological innovation (CAC loss and RegionScore) remains incremental. The CAC objective essentially adapts contrastive alignment to a culturally labeled setup a relatively modest technical contribution. RegionScore, while intuitive, captures surface-level lexical cues (country/demonym mentions) rather than deeper cultural semantics.  

- **Inadequate Relevance Annotation Scheme**:   
The paper employs a fundamentally flawed binary annotation approach that fails to capture the graded nature of cultural relevance. By reducing cultural understanding to three binary questions and artificially combining them into a continuous scale, the authors lose crucial ranking information essential for cultural understanding tasks. Established information retrieval research demonstrates that graded relevance judgments provide superior training signals compared to binary labels, particularly for nuanced tasks like cultural understanding. The absence of proper graded annotations or pairwise ranking comparisons undermines the effectiveness of the contrastive learning framework and limits the model's ability to distinguish between documents with varying degrees of cultural relevance.

- **Absence of ablations and bias audits for the retrieval pipeline.**  
BM25 is a purely lexical retriever and cannot capture the semantic or conceptual associations central to cultural understanding, I do not feel its the optimal choice for getting to Wikipedia documents for further annotation. No experiments contrast retrieval quality using (a) GPT captions vs. human-written captions, or (b) BM25 vs. dense/hybrid retrieval. Without such comparisons, it remains unknown whether the current pipeline systematically overlooks semantically relevant documents or skews toward certain linguistic patterns. Furthermore, the paper does not analyze how often annotators added “missing” documents or how BM25’s coverage varied by language/country. These missing statistics weaken claims of balanced cross-cultural representation.   

- **Potential data leakage due to image-only train/test split.**    
The authors report an 85 / 5 / 10 split by images but provide no evidence of a document-level split. If the same Wikipedia document appears as a positive for both training and test images (e.g., “Sushi” page reused across splits), retrieval models can trivially memorize text and inflate performance. The paper does not mention any de-duplication or partitioning of documents, nor does it measure document overlap across splits.  

- **Critical Gaps in Evaluation setup:**  
The authors fail to report essential statistics about their evaluation setup, specifically the positive vs. negative document ratios within their 10-document sets per instance. Without knowing whether 8/10 documents are relevant or 2/10 are relevant, it is impossible to assess task difficulty or whether reported improvements represent meaningful advances or artifacts of trivial evaluation scenarios. The paper shows they tested against random characters and unrelated documents as controls, but:​ No analysis of hard negative examples (culturally similar but incorrect documents).  

- **Insufficient Analysis of Cultural Bias and Representation:**   
While identifying performance disparities across countries (e.g., poor performance on Nigerian contexts vs. better performance on Korean contexts), the paper provides no deeper investigation into the underlying causes of these systematic biases.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: When large models exhibit performance degradation under the RAG setting, have the authors analyzed whether this results from limited retrieval relevance (e.g., using only Top-1 document and 256 tokens) or from suboptimal fusion/integration strategies?

Additionally, have you experimented with multi-document or paragraph-level retrieval, or with tool-augmented reasoning mechanisms to test if the issue lies in retrieval depth or model utilization?

Moreover, the selection of large models in the current evaluation does not appear sufficiently frontier.
Models such as Qwen3, Gemini 2.5 Pro, and GPT-5 are among the most representative current vision-language systems, yet their results are not included. Including these stronger baselines would help demonstrate the true upper bound of RAG-based cultural reasoning performance and provide a more complete picture of model scaling trends.
soundness: 3
strengths: - Problem Definition and Task Design

The paper clearly defines the problem and presents tasks that align well with practical applications. It focuses on the scenario of “visual content + external cultural knowledge”, using two downstream tasks—cVQA (multiple-choice visual question answering) and cIC (image captioning)—to evaluate the full pipeline of retrieving → consuming knowledge → answering/generating.

- Data Scale and Annotation Quality Control

The dataset covers 8 countries and 11 categories, totaling 1,868 instances.
Each image is paired with 10 Wikipedia documents, which are manually re-ranked, resulting in 18,680 image–text pairs.
The annotation process includes three dimensions of cultural relevance (national association, thematic alignment, and visual presence in the image), inter-annotator agreement checks, and spot audits.

- Methodological Contribution: Culturally Aware Contrastive Learning (CAC)

Within the CLIP/SigLIP framework, the authors introduce a combination of binary classification loss, ranking-margin loss, and text-side diversity regularization, specifically designed to enhance cultural relevance ranking (CaCLIP / CaSigLIP).
summary: This paper introduces RAVENEA, a new benchmark dataset and evaluation framework designed to assess systematic generalization, reasoning, and visual understanding in AI models. The benchmark is inspired by the RAVEN or RPM (Raven’s Progressive Matrices)-style tasks but extends them by introducing multi-dimensional relational reasoning challenges that involve complex scene configurations, compositional attributes, and logical rule dependencies.

The authors propose:

- A novel dataset generation pipeline that controls for compositional structure and visual diversity, ensuring that models cannot overfit to superficial cues.
- Several generalization splits (e.g., attribute, relation, and logic-based splits) to test extrapolative reasoning.
- A comprehensive evaluation of existing architectures (CNNs, ViTs, relational networks, multimodal transformers, and neuro-symbolic models) on this benchmark, revealing clear performance gaps in out-of-distribution (OOD) reasoning.
- Discussion and visualization of error patterns, showing that models often rely on shortcut heuristics rather than true reasoning.

The benchmark is positioned as a rigorous and diagnostic tool for studying systematic generalization and compositional reasoning in visual intelligence.
weaknesses: - Limited Domain Coverage and Source Bias

As acknowledged by the authors in the “Limitations” section, the dataset covers only 8 countries and 11 categories, and relies primarily on English Wikipedia articles. This setup is prone to cultural and regional exposure bias, which may affect the generalizability of the claimed cross-cultural conclusions.

Future work should incorporate GLAM (Galleries, Libraries, Archives, Museums) institutional collections and non-English knowledge sources to achieve a more balanced and representative dataset — a direction the authors themselves briefly mention in the ethics and outlook discussion.

- Small and Inconsistent Human Evaluation for cIC

The human evaluation for the cIC (cultural image captioning) task is based on only 10 images, 4 annotators, and 420 sentences in total, with a moderate inter-rater agreement of κ ≈ 0.595.

Given the evaluation spans 14 models and multiple retrieval variants, the sample size is too limited to support robust conclusions.
The authors are encouraged to expand the human evaluation sample and include statistical significance testing to strengthen reliability.

- Conservative RAG Configuration Potentially Underestimates Strong Models

During evaluation, the retrieval-augmented generation (RAG) setup uses only the Top-1 document truncated to the first 256 tokens.
This constrained configuration may explain the phenomenon where large models fail to benefit under RAG.

It would be beneficial to explore multi-document fusion, longer context windows, re-ranking and rerouting strategies, or tool-augmented reasoning chains, in order to more fairly assess the upper bound of large-model + RAG performance.

---

confidence: 2
contribution: 3
initial_rating: 4
presentation: 3
questions: 1) Retrieval quality control:
Was any evaluation conducted to compare the quality of BM25-retrieved documents against those retrieved by modern text-only or multimodal retrievers? It would be helpful to know whether better initial retrieval could enhance the benchmark’s representativeness.

2) Annotation statistics:
Could the authors provide summary statistics on the yes/no response distributions for the three cultural relevance annotation questions (country association, topic alignment, visual depiction)? These would clarify dataset balance and inter-question correlations.
soundness: 3
strengths: 1) Comprehensive exploratory analysis:
The paper conducts an analysis of RAVENEA across tasks, model scales, and cultural dimensions.

2) Human annotation:
The dataset construction involves human ranking of retrieved documents, supplemented by quality control and reporting of strong inter-annotator agreement scores.

3) Novel training approach (CaCL):
The proposed Culture-Aware Contrastive Learning objective produces retrievers that generalize well across architectures. The released CaCLIP model can become a valuable resource for future multimodal RAG research.

4) Responsible dataset curation:
The authors demonstrate attention to ethical concerns by implementing automated face detection and blurring of identifiable individuals in all images.
summary: This paper introduces RAVENEA, a manually curated benchmark for visual culture understanding in multimodal systems. The benchmark focuses on two key tasks: culture-focused Visual Question Answering (cVQA) and culture-informed Image Captioning (cIC). It contains 1,868 multimodal instances and 18,680 human-ranked Wikipedia passages (10 per instance).
Each instance is drawn from existing datasets (CVQA and CCUB) and is augmented with culturally relevant Wikipedia passages. Annotators re-rank the top-10 BM25-retrieved documents based on GPT-4o captioning and can also add any missing articles they consider relevant.
The authors evaluate seven multimodal retrievers and examine their impact when integrated into 14 widely used Vision-Language Models (VLMs). They also introduce Culture-Aware Contrastive Learning (CaCL), a fine-tuning approach applied to CLIP and SigLIP, to produce culturally sensitive retrievers (CaCLIP and CaSigLIP2), which consistently outperform their frozen counterparts.
Empirical results show that retrieval-augmented generation (RAG) provides the most substantial gains for lightweight VLMs.
weaknesses: 1) Dependence on GPT-4o and BM25 in data construction:
Since image captions are generated by GPT-4o and used as text queries for BM25 retrieval, the initial dataset quality depends heavily on the performance and biases of these models. This reliance may explain why larger VLMs show limited improvements with RAG, as the retrieved contexts may be redundant relative to their internal knowledge.
Future iterations could improve recall and diversity by incorporating more diverse sources of models for generation and retrieval, including multimodal retrievers rather than relying solely on text-based BM25 or including visually similar images and multilingual sources.
RAVENEA’s document set is capped at the top-10 BM25 hits (plus any extras an annotator happens to recall). If a genuinely relevant article never appears in that shortlist, it cannot be selected, so the benchmark measures only how well models re-rank a limited candidate pool rather than how well they retrieve unseen but correct evidence. I would like to see more quantitative evidence on how often this recall limitation occurs to ensure it does not significantly impact the dataset’s completeness or the conclusions drawn from it.


Decision: Accept (Poster)