Venue: NAACL 2025

# The Power of Many: Multi-Agent Multimodal Models

for Cultural Image Captioning

Longju Bai\({}^{1*}\) **Angana Borah\({}^{1*}\)****Oana Ignat\({}^{2*}\)****Rada Mihalcea\({}^{1}\)**

\({}^{1}\)University of Michigan - Ann Arbor, USA

\({}^{2}\)Santa Clara University - Santa Clara, USA

{_longju, anganab, mihalcea_}@umich.edu oignat@scu.edu_

###### Abstract

Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MoSAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adapitable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research. Our dataset and models can be accessed at [https://github.com/MichiganNLP/MoSAIC](https://github.com/MichiganNLP/MoSAIC).

+
Footnote †: _(Volume 1: Long Papers)_, pages 2970–2993

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Large Multimodal Models (LMMs) demonstrate remarkable performance across various multimodal tasks. Despite these achievements, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models Hershcovich et al. (2022); Bhatia et al. (2024). Conversely, multi-agent models have proven to be highly capable, often excelling in solving complex tasks Guo et al. (2024). In this paper, we propose to evaluate and analyze the collective performance of LMMs as multi-agent models in the novel multimodal task of culturally enriched image captioning.

Culture is a complex and elusive concept. As Adilazauda et al. (2024) show, _Culture is a multifaceted concept meaning different things to different people at different times_. This complexity is apparent in various cultural expressions, such as proverbs, social norms, and other context-dependent elements. In our work, we adopt the definition provided by Nguyen et al. (2023) and focus on visual cultural elements such as food, drinks, clothing, traditions, rituals, and behaviors.

Culture is strongly tied to our group-oriented human nature, which allows us to learn from one another over generations. Furthermore, as sociologists and anthropologists have demonstrated, our progress as a species is primarily due to our cooperative nature, rather than individual knowledge Henrich (2015). Inspired by the success of human collective intelligence, we conceptualize the culturally enriched image captioning task as

Figure 1: In a multi-agent setting, three LMM agents, each embodying a _curious_ persona and drawing upon knowledge from distinct countriesIndia, China, and Romania), participate in a question-and-answer dialogue centered around an image. A fourth agent then summarizes their discussion, creating a culturally enriched image caption.



a "social task". Specifically, we frame it as a dialogue between three agents from different cultures who seek to learn about each other's cultures through an image. They engage in asking questions and sharing insights, akin to human collaborative problem-solving. A moderator provides examples of initial questions and highlights key visual cultural aspects to focus on. The conversation is then summarized into a comprehensive cultural image description (see Figure 1). Our findings indicate that this multi-agent approach yields better results than single-agent methods.

We summarize our contributions as follows. First, inspired by collective intelligence, we propose MosAIC, **a novel multi-agent framework to improve cross-cultural image captioning performance**. Second, we share **a dataset of 2,832 culturally enriched image captions** in English, for images from three different countries: China, India, and Romania, across three datasets: GeoDE, GDVCR, and CVQA. Third, we introduce a **culture-adaptable metric for evaluating cultural information within image captions.** Finally, we show that **multi-agent interaction surpasses single-agent (and culturally fine-tuned) models across different metrics**, and provide actionable steps for future work.

## 2 Related Work

Large Multi-Agent Multimodal Models.The inspiring progress of Large Language Models (LLMs) has led to the proposal of LLM-based multi-agents that leverage the collective intelligence and specialized skills of multiple agents Guo et al. (2024). In this context, multiple independent agents discuss and make decisions, mirroring the cooperative human nature. This approach has facilitated progress on various tasks such as software development Hong et al. (2023), society simulation Park et al. (2022), game simulation Xu et al. (2023), debate simulation Chan et al. (2023), and polarization Ohagi (2024).

At the same time, Large Multimodal Models (LMMs) have extended the capabilities of traditional language models by integrating several data modalities such as text, videos, and images. LMMs such as LLaVA Liu et al. (2023), GPT-4 OpenAI, 2023) or LENS Berrios et al. (2023) have shown promising results in complex vision-language tasks due to their pretraining on terabytes of image and language data with billion-parameters Bai et al. (2023); Zhang et al. (2023) To the best of our knowledge, our study is the first to employ LMMs in a multi-agent setting for a cross-cultural multimodal understanding task.

Cross-Cultural Multimodal Understanding.Even though LLMs and LMMs are already instrumental in various real-life applications, such as recommender systems Li et al. (2023b) and customer service Pandya and Holia (2023), these models often mirror Western-centric perspectives, leading to reinforcing stereotypes and algorithmic monoculture Kleinberg and Raghavan (2021); Hershcovich et al. (2022); AlKhamissi et al. (2024).

Several efforts have been made in the AI community to enhance the diversity of data and models, both linguistically and visually. Specifically, recent language studies have developed cross-cultural benchmarks such as CultureBank Shi et al. (2024) and NORMAD Rao et al. (2024) to enhance LLMs' cultural awareness. The vision-language community also has started to focus on creating multilingual, geographically, income, and culturally diverse multimodal datasets such as Dollar Street Rojas et al. (2022), GeoDE Ramaswamy et al. (2023), GD-VCR Yin et al. (2021), CVQA Romero et al. (2024), MaRVL Liu et al. (2021), and WIT Srinivasan et al. (2021).

Despite the increased availability of cultural benchmarks, the current evaluation metrics and methods are not suited to capture cultural information Awal et al. (2023). Evaluation metrics such as Accuracy or F1 score do not focus on the cultural nuances in LMMs' generations and, therefore, cannot reflect their cultural awareness in practice. Generation-focused metrics such as ClipScore Hessel et al. (2021), LongCLIP Zhang et al. (2023), and Completeness score Zhang et al. (2023) also do not account for cross-cultural variations. However, more culture-focused metrics are emerging, such as Culture Noise Rate (CNR) Yun and Kim (2024), which measures the ratio of cultural words among all words generated in a caption. The cultural words are extracted from a cultural commonsense knowledge base (CCSK), which contains several cultural facets like food, drinks, clothing, traditions, rituals, and behaviors Nguyen et al. (2023). Our work aligns with Yun and Kim (2024), as both studies address the task of culturally enriched image captioning. However, our approach diverges by focusing on multi-agent settings and evaluating the models based on three culturally diverse benchmarks 



## 3 MosAIC: A Framework for Cultural Image Captioning

We introduce Mosaic, a framework for Multi-Agent Interactions, as shown in Figure 2, to tackle cultural image captioning, a complex task that involves not only describing the visual content of the image but also capturing the cultural elements it represents. The framework consists of a multi-agent model, a cultural benchmark, and evaluation metrics, as described below.

### Muti-Agent Interaction Model

We introduce a multi-agent setup (Figure 3) to emulate collaboration in a culturally diverse group. Our multi-agent model consists of five agents, each with specific _roles_: three Social agents, a Moderator agent, and a Summarizer agent.

**Moderator.** The Moderator agent has two primary tasks. First, it generates questions based on the image to which the Social agents respond. Second, it guides the Social agents to focus on aspects relevant to their cultures, promoting more comprehensive and culturally diverse image descriptions.

**Social.** Each of the Social agents assumes a persona from three cultures: China (\(C\)), India (\(I\)), and Romania (\(R\)). Furthermore, the agents are encouraged to embody a _curious_ persona to facilitate more interaction in their conversation. In the first conversation round, each agent shares their initial description (\(d\)) of the given image and a question (\(q\)) about the image from the ones provided by the Moderator. In the next conversation rounds, the agents learn from one another, enriching the image description with more comprehensive and detailed content. Specifically, each agent answers the questions raised by the other agents in the current and previous rounds and asks a new question. For example, in Figure 3_Round 2_, agent \(R\) answers all the questions from the other agents posed in _Round 1_ and _Round 2_. Note that agent \(R\) answers more questions than the others as it responds last. To balance the number of questions each agent answers, we randomize the order of agents for each round and image. In the final round of conversation, Figure 3_Round 3_, each Social agent summarizes (\(s\)) everything learned from the previous rounds, including all the initial image descriptions (\(d\)), the questions (\(q\)) and the corresponding answers (\(a\)) from all agents. The summaries distill the most important information gained from the interaction, helping to condense and focus the key insights.

**Summarizer.** The Summarizer agent collects all the summaries from the Social agents and generates a summary representing the final image description.

Agent Memory.Each agent has its own memory. The Moderator agent generates questions stored in a shared question memory that is accessible to the Social agents. Initially, the Social agents independently analyze the image without memory of/ knowing their peers' responses, minimizing potential bias. In the conversation rounds, each Social agent can access the responses from all agents in previous rounds and those preceding them in the current round. Finally, each agent's memory is erased after the Summarizer agent completes the image caption. We also tested a longer-term memory across multiple images but found no performance improvement, likely due to the significant differences between the images.

Figure 3: Multi-Agent Interaction Model. The Moderator presents questions to the Social agents, who engage in three conversation rounds. The Summarizer creates the final image caption by compiling the conversation summaries from the Social agents.

Figure 2: Overview of MoSAIC, our proposed framework for Multi-Agent Image Captioning. The framework consists of a multi-agent interaction model, cultural benchmarks and evaluation metrics. The input is an image and the output is a cultural image caption.



Setup.We use LLaVA-1.5 13b1Liu et al. (2023) to simulate the agents in our interaction model. Each agent is initialized as a separate LMM, so parameters are not shared among the agents. Each agent has an individual memory, where generated outputs by all agents are stored.

Footnote 1: [https://huggingface.co/liuhaotian/LLaVA-v1.5-13b](https://huggingface.co/liuhaotian/LLaVA-v1.5-13b)

### Cultural Benchmark

We introduce a new dataset of cross-cultural captions for 2,832 images from three cultures: China, India, and Romania generated by MosAIC and other models. To achieve this, we use images from three geographically diverse datasets: GeoDE, GD-VCR, and CVQA.2 We provide image captions generated by MosAIC, our top-performing model, alongside LLaVA-13b captions to facilitate comparisons between single-agent and multi-agent approaches. Furthermore, for a subset of the images (25 images per dataset and culture), we provide human-generated captions as described in section 4.1.

GeoDE.GeoDERamaswamy et al. (2023) is a geo-diverse dataset for object recognition with crowd-sourced 61,940 images from 40 classes and 6 world regions, namely West Asia, Africa, East Asia, South-East Asia, the Americas, and Europe.

Gd-Vcr.GD-VCRYin et al. (2021) is a geodiverse visual commonsense reasoning dataset with 328 cultural and geo-location-specific images from Western, East Asian, South Asian, and African countries.

Cvqa.CVQA Romero et al. (2024) is a culturally diverse multilingual visual question-answering dataset with 5,239 images from 30 countries across Asia, Africa, South America, and Europe.

### Evaluation Metrics

We employ both automated metrics (alignment, completeness, cultural information) and human evaluation (Turing test and caption correctness) to comprehensively assess the image captions.

Alignment.We measure text-to-image alignment using LongCLIPZhang et al. (2024). This metric builds on CLIPScoreHessel et al. (2021), a popular reference-free evaluation metric for image captioning that outperforms existing reference-based metrics Vedantam et al. (2015). LongCLIP uses a knowledge-preserved stretching of positional embedding to increase the maximum input length of CLIPScore from 77 to 248 tokens.

Completeness.We evaluate the completeness of the image captions by calculating the ratio of words mentioned in both the image and the caption to the total number of words (tags) in the image. To generate a comprehensive list of image tags, we use the Recognize Anything Model (RAM) Zhang et al. (2023) and expand it with their corresponding synonyms from WordNet Miller (1994) .3

Footnote 3: RAM is a state-of-the-art image tagging model with exceptional accuracy and scope, recognizing 6,400 common tags from OpenImages V6 Kuznetsova et al. (2020) with impressive zero-shot performance.

Cultural Information.We propose a new metric to quantify the presence of cultural information in image captions. This approach is inspired by the Culture Noise Rate (CNR) Yun and Kim (2024), which measures the proportion of cultural words in image captions. However, given that the captions generated by our model tend to be longer than those from other models,4 a ratio-based metric like CNR may disproportionately affect performance. To address this, we instead compute the count of unique cultural words in a caption, a length-invariant metric, to better capture cultural specificity. Further, to improve the metric coverage, we generate and include 700 additional cultural words from 14 categories, such as Traditions and Festivals (50 words per category).5 Human validation (one native annotator per country) confirmed that all GPT-generated words aligned with the provided cultural categories. Our final cultural information metric integrates the filtered cultural terms from CNR with the additional GPT-generated words. This metric is straightforward to compute and adaptable for assessing cultural specificity across various countries.

Figure 4: Human Annotation Guidelines for Cultural Image Captioning.

 

Turing Test Accuracy.We evaluate the similarity of the LMM-generated captions to human-generated captions. For 30 images per culture, evenly distributed across datasets, three annotators are tasked with distinguishing between a human-generated caption, as described in Section 4.1, and an LMM-generated caption. Lower accuracy indicates that the LMM-generated captions are more similar to those generated by humans.

Caption Correctness.We assess the image caption correctness by considering both the correctness of image content descriptions and the cultural information. Specifically, for 30 images per culture, evenly distributed across datasets, three annotators evaluate the percentage of correct captions generated by LIMs, identifying issues such as hallucinations, mislabeling of instances, and inaccuracies in cultural representation.

## 4 Evaluation and Results

We assess the influence of multi-agent interaction on image captioning by comparing our multi-agent interaction model, MosAIC, with single-agent models (BLIP-2, LLaVA-13b) and a human baseline.

### Baseline Models

Blip-2.BLIP-26[19] leverages frozen pre-trained image encoders ViT-L/14 from CLIP [7] and a FlanT5 LLM [8] by training a lightweight, 12-layer Transformer encoder in between them. It achieves an impressive state-of-the-art zero-shot performance on image captioning.

Footnote 6: [https://huggingface.co/Salesforce/blip2-opt-2.7b](https://huggingface.co/Salesforce/blip2-opt-2.7b) 



### Ablation Studies

**MosAIC Setting.** Our model, MosAIC, employs CoT prompting and operates through three rounds of conversation (see Figure 3). It functions in a zero-shot learning setting without the need for fine-tuning. We also perform ablation studies to assess MoSAIC's performance across various settings: the number of conversation rounds, prompting techniques, fine-tuning, and different cultures and datasets, as shown in Figure 6.

a) Number of Conversation Rounds.Figure 6 a) shows that increasing the number of agent conversations from three to four rounds improves Cultural Information (26.1 vs. 31.1) while keeping Alignment and Completeness stable. The slight decrease in Cultural Information from rounds two to three is attributed to the Summarizer's failure to synthesize key cultural aspects, instead concatenating the conversations.

b) Prompt Techniques.Given the challenges in achieving cross-cultural alignment between the agents Ananthram et al. (2024), we experiment with different prompt techniques:8

Footnote 8: All prompts are in Appendix A.4 and in our repository.

SimpleThis strategy offers straightforward instructions, such as asking a social agent to describe an image and its cultural significance.

MultilingualWe prompt agents from specific cultures by translating the Simple prompt into the dominant languages of their countries, such as _Mandarin Chinese_, _Hindi_, and _Romanian_.9 The generated responses are in English for consistency.

AnthropologicalThis prompting technique considers emic and etic perspectives, cultural context, socioeconomic background, individual values, personal experience, cultural relativism, spatial and temporal dimensions in a nuanced manner as introduced by AlKhamissi et al. (2024).

Chain of Thought (CoT).CoT prompting involves generating intermediate reasoning steps, mimicking human problem-solving to arrive at a final answer. Wei et al. (2023). Inspired by multi-modal CoT Zhang et al. (2017), we guide agents in making detailed image observations.

InspistsAs shown in Figure 6 b), CoT prompting outperforms other strategies, while Anthropological prompting--designed to enhance cultural alignment in LLMs--performs similarly to or worse than Simple prompting. This suggests LMMs need

Figure 6: Ablations on (a) conversation rounds, (b) prompt types, (c) fine-tuning, (d) cultures, and (e) datasets. The zero-shot Multi-agent MoSAIC outperforms the Human Baseline and the fine-tuned single-agent LLLAVA–13b–ft-all, highlighting the value of multi-agent interactions in cultural image captioning.



further refinement for effective cross-cultural alignment. Additionally, Multilingual prompting ranks lowest in Cultural Information, likely due to confusion from inputs in three languages, highlighting the need for better multilingual alignment in LMMs.
**c) Fine-tuning Impact.** Current LLMs and LMMs struggle to align with diverse global cultures, often reflecting predominantly WEIRD [10] cultural norms [23, 47]. To improve our model's cultural alignment, we apply fine-tuning, which has previously shown promise [56]. For fine-tuning data, we utilize the Wikipedia-based Image-Text (WIT) dataset from Srinivasan et al. [8]. We implement two fine-tuning setups:

1. We fine-tune a LLaVA-13b model on 9000 WIT images and captions across three cultures, creating two models: the non-interaction LLaVA-13b-ft-all, which only summarizes, and the interaction MosAIC-ft-all, where the fine-tuned agents collaborate.
2. We fine-tune three LLaVA-13b models, one for each culture, using 3000 WIT images and their corresponding captions. The interactions among these agents yield the multi-agent model MoSAIC-ft-specific.

**Insights.** Fine-tuning generally enhances Cultural Information (Figure 6 c), with a modest 4-point improvement for the multi-agent model (MosAIC vs. MoSAIC-ft-specific) and a more substantial 9-point gain for the single-agent model (LLaVA-13b vs. LLaVA-13b-ft-all), considering the compute-intensive nature of the process. Furthermore, MossAIC outperforms the non-interaction LLaVA-13b-ft-all model (Cultural Info: 26.01 vs. 23.18), underscoring the benefits of multi-agent interaction over fine-tuning. The fine-tuned models show lower performance in Alignment and Completeness, as the fine-tuning primarily focuses on enhancing cultural alignment. Among fine-tuned models, ft-specific setting performs the best as each agent in interaction has specific cultural knowledge about the country they represent.

**d) Performance across Cultures.** Figure 6 d) reveals similar trends across the three cultures: China, India, and Romania.11 Notably, Mosaic achieves the highest Cultural Information performance across all cultures, underscoring the significance of incorporating diverse cultural perspectives in generating image captions.

Footnote 11: Details regarding the dataset and fine-tuning hyperparameters are provided in Appendix A.2.

**e) Performance across Datasets.** Figure 6 e) shows that Cultural Information is highest for GD-VCR, followed by CVQA, and lowest for GeoDE, which aligns with expectations since GD-VCR and CVQA contain more cultural information than GeoDE. Although MoSAIC scores lower than Humans on Alignment, it achieves higher scores for Completeness and Cultural Information across all datasets.12

Footnote 12: footnotemark: Detailed results across all models in Appendix A.3.1.

### Human Evaluation and Error Analysis

We assess the human-likeness of generated captions using Turing Test accuracy (Section 3.3). Mosaic scores lower than LLaVA-13b (83.1 vs. 87.9), suggesting Mosaic's captions are more human-like. However, the high overall scores indicate LMMs still struggle to match human captioning, mainly due to stylistic differences, as humans tend to use a more casual, direct style, as shown in the qualitative results (Section 4.5).

We evaluate caption correctness (Section 3.3), finding 94.5% of human captions correct, compared to 60.2% for Mosaic and 64.56% for LLaVA-13b. At the dataset level, we observe that Mosaic performs equally or better than LLaVA-13b on GD-VCR (Human - Machine correctness difference (lower is better): 28.5 vs. 28.5) and CVQA (34.2 vs. 37.1). We hypothesize that Mosaic underperforms on GeoE (40.0 vs. 25.0) because this dataset contains less culturally rich information. Mosaic's lower correctness, compared to LLaVA-13b, may also result from compound hallucinations caused by the interaction of multiple LMMs. Future work can address this issue by making each agent less susceptible to hallucinations, as detailed in the Limitations section. Common errors include incorrect country, object recognition, people counting, and overly general descriptions (examples in Appendix A.5).

### Qualitative Results

In Figure 7, we compare image captions generated by Mosaic, Humans, and LLaVA-13b for images from China, India, and Romania.

Compared to LLaVA-13b, MoSaic shows closer alignment with Human captions, capturing more cultural elements. For example, in the Chinese image, Mosaic identifies the giant panda as a national treasure, similar to the Human caption. In the 

Indian image, both MosAIC and Human captions recognize the religious significance of bells, highlighting MoSAIC's greater cultural sensitivity, while LLaVA-13b provides Western-centric descriptions.

MosAIC excels at identifying country-specific information, whereas LLaVA-13b fails to recognize the country in any of the images, resulting in vaguer descriptions. For example, MosAIC connects the panda to China and accurately describes its cultural symbolism, while LLaVA-13b remains overly general. Furthermore, MosAIC consistently employs relevant cultural terminology, such as "Hinduism" and "pilgrimage" in the Indian image. In contrast, LLaVA-13b uses vaguer language, like "festive atmosphere", indicating less cultural specificity.

However, instances of hallucinated content were still observed, with MosAIC incorrectly mentioning a person not present in the Romanian image, while LLaVA-13b associated the bells in the Indian image with Christmas, showing cultural inaccuracy.

In summary, MosAIC generated more accurate and culturally aligned captions, although it occasionally hallucinated. In contrast, LLaVA-13b struggled with cultural specificity and country identification.

## 5 Lessons Learned and Actionable Steps

Our findings reveal the performance of multi-agent LMMs in cultural image captioning, highlighting lessons learned and suggesting steps to enhance cultural richness in future models.

Prioritize multi-agent modelsWhile LMMs excel in tasks like generation and retrieval, they fall short in cross-cultural performance, even with culture-centric prompting strategies [2]. Our findings show that even Simple and CoT prompts in multi-agent LLMs are helpful and outperform Anthropological and Multilingual prompts (Figure 6 b). Additionally, increasing the number of conversation rounds between agents enhances cultural information (Figure 6 a). To further improve cross-cultural understanding, future work should focus on developing equitable frameworks using multi-agent LMMs and cross-cultural benchmarks.

Develop efficient cross-cultural techniquesCurrent approaches to improving cross-cultural understanding in LMMs often rely on fine-tuning. However, we show that interaction-based models outperform fine-tuned, non-interaction models (Figure 6 c), highlighting both the effectiveness and efficiency of multi-agent LMMs. For instance, LLaVa-13-ft-all requires 54 hours on a single NVIDIA A100 GPU to generate captions (12h for fine-tuning on 9000 WIT images and 42h for inference). In contrast, MosAIC completes the task in 47 hours with only inference. Given these findings, future work should focus on multi-agent models to improve sustainability and accessibility.

LMMs focus on culture; humans focus on correctnessLMMs tend to be more culturally specific in their responses, while humans provide more accurate answers (Section 4.4). The main sources of errors in LMMs stem from poor object recognition and hallucinations--instances where the model generates incorrect or fabricated information. Future work can integrate a state-of-the-art object recognition system to enhance caption accuracy.

## 6 Conclusion

In this paper, we leverage LMM agents interaction to enhance cross-cultural image captioning, introducing MosAIC. We presented a comprehensive framework for implementing the proposed LMM agent-agnostic approach to alleviate the problem of generating multiple-agent LMMs. This is an important step towards exploring the impact of different 

sive analysis of three cultures across three datasets, using various prompting techniques. Additionally, we demonstrate the advantages of multi-agent LMM interactions, comparing their performance with compute-intensive methods like fine-tuning for improving cultural alignment. We also open-source our culturally enriched captions dataset generated by our proposed framework, MosAIC, alongside baseline models. Finally, we create a comprehensive and culture-adaptable metric for evaluating cultural information within image captions. Based on our findings, we share ideas for future work. Our dataset and models can be accessed at [https://github.com/MichiganNLP/MOSAIC](https://github.com/MichiganNLP/MOSAIC).

### Limitations and Ethical Considerations

The Complexity of Defining and Evaluating Cultural Information.Our approach utilizes multi-agent LLM interactions, where each LLM represents distinct cultural personas based on specific countries. While we explore various prompting strategies and fine-tuning techniques to align the models with different cultural contexts, it is important to acknowledge that culture is an inherently complex and multifaceted concept. Relying solely on countries as proxies for cultural identity oversimplifies the rich variation in cultural experiences and individual perspectives [1]. Using country and language information represents only the tip of the iceberg when it comes to capturing cultural diversity. While these factors provide a basic framework for understanding cultural distinctions, they do not fully account for the deeper, more nuanced aspects that define human cultures. We encourage future work to delve into these deeper dimensions, extending beyond simple national or linguistic markers. Important areas to explore include values, attitudes, and biases, which shape individual and collective worldviews.

Multi-Agent Setup affects Correctness.Multi-agent models are more prone to hallucinations compared to single-agent models due to the compound effect, where errors or hallucinations from one agent can influence and amplify those in other agents. This cumulative effect results in a lower Caption Correctness score. Future research could explore ways to mitigate this issue by making each agent less susceptible to hallucinations. This might involve improving the architecture of individual agents for better accuracy, using grounding techniques or external knowledge to verify information, and creating stronger communication protocols between agents to prevent errors from spreading. These improvements could enhance the overall correctness and reliability of the model's outputs.

Further Assessing the Impact of Conversation Rounds and Fine-Tuning on Cultural Metrics.While our ablation studies demonstrate that both conversation rounds and fine-tuning lead to enhanced performance on cultural metrics, additional analysis is necessary to evaluate the impact of various configurations (e.g., interaction vs. non-interaction settings). This deeper investigation will allow us to better evaluate how effective each setup is and reveal the key factors behind the observed improvements. Understanding these factors will be essential for refining our approach and enhancing the model's ability to adapt across different cultural contexts.

Limited Cultural Alignment in LLMs.Cultural alignment in LLMs and LMMs is a well-researched area. While it is not the central focus of our research, we recognize that the prompt engineering techniques and fine-tuning methods we employ may not achieve perfect cultural alignment. This could lead to inconsistencies in how each multi-cultural agent produces responses across different cultural contexts. Additionally, our study is limited to just three countries, each with relatively high representation in the training data, due to the lack of a broader, more diverse set of human evaluations. This limitation highlights the need for more comprehensive validation across a wider range of cultures to ensure better alignment and more reliable cross-cultural performance.

## Acknowledgments

We thank the anonymous reviewers for their constructive feedback. We also thank the members of the Language and Information Technologies lab at the University of Michigan for the insightful discussions during the early stage of the project. This project was partially funded by a Microsoft Foundational Model grant. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Microsoft.

## References

* Adilazuarda et al. (2017) Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh 

Dwivedi, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling" culture" in lms: A survey. _arXiv preprint arXiv:2403.15412_.
* Alkhamissi et al. (2024) Badr Alkhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. _arXiv preprint arXiv:2402.13231_.
* Ananthram et al. (2024) Amith Ananthram, Elias Stengel-Eskin, Carl Vondrick, Mohit Bansal, and Kathleen McKeown. 2024. See it from my perspective: Diagnosing the western cultural bias of large vision-language models in image understanding. _arXiv preprint arXiv:2406.11665_.
* Atari et al. (2023) Mohammad Atari, Mona J Xue, Peter S Park, Damian Blasi, and Joseph Henrich. 2023. Which humans? Rabibul Awal, Le Zhang, and Asishway Agrawal. 2023. Investigating prompting techniques for zero-and few-shot visual question answering. _ArXiv_, abs/2306.09996.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. _Preprint_, arXiv:2308.12966.
* Berrios et al. (2023) William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. 2023. Towards language models that can see: Computer vision through the lens of natural language. _ArXiv_, abs/2306.16410.
* Bhatia et al. (2024) Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, and Vered Shwartz. 2024. From local concepts to universals: Evaluating the multi-cultural understanding of vision-language models. _arXiv preprint arXiv:2407.00263_.
* Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llvm-based evaluators through multi-agent debate. _arXiv preprint arXiv:2308.07201_.
* Chung et al. (2024) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. _Journal of Machine Learning Research_, 25(70):1-53.
* Guo et al. (2024) Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. _arXiv preprint arXiv:2402.01680_.
* Henrich (2015) Joseph Henrich. 2015. _The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter_. Princeton University Press, Princeton, NJ.
* Henrich et al. (2010) Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? _Behavioral and Brain Sciences_.
* Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhonueux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Iias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Sogaard. 2022. Challenges and strategies in cross-cultural NLP. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6997-7013, Dublin, Ireland. Association for Computational Linguistics.
* Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7514-7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. _arXiv preprint arXiv:2308.00352_.
* Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Ke et al. (2024) Luoma Ke, Song Tong, Peng Chen, and Kaiping Peng. 2024. Exploring the frontiers of llms in psychological applications: A comprehensive review. _arXiv preprint arXiv:2401.01519_.
* Kleinberg and Raghavan (2021) Jon Kleinberg and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. _Proceedings of the National Academy of Sciences_, 118(22):e2018340118.
* Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. _International journal of computer vision_, 128(7):1956-1981.
* Li et al. (2024) Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024. Culturedlm: Incorporating cultural differences into large language models. _arXiv preprint arXiv:2402.10946_.
* Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. _Preprint_, arXiv:2301.12597.
* Li et al. (2023b) Lei Li, Yongfeng Zhang, and Li Chen. 2023b. Prompt distillation for efficient llm-based recommendation. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_, pages 1348-1357 

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Eliott. 2021. Visually grounded reasoning across languages and cultures. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In _NeurIPS_.
* Miller (1994) George A. Miller. 1994. WordNet: A lexical database for English. In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
* Mukherjee et al. (2024) Anjishnu Mukherjee, Ziwei Zhu, and Antonios Anastasopoulos. 2024. Crossroads of continents: Automated artifact extraction for cultural adaptation with large multimodal models. _arXiv preprint arXiv:2407.02067_.
* Nguyen et al. (2023) Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting cultural commonsense knowledge at scale. In _Proceedings of the ACM Web Conference 2023_, pages 1907-1917.
* Ohagi (2024) Masaya Ohagi. 2024. Polarization of autonomous generative ai agents under echo chambers. _arXiv preprint arXiv:2402.12212_.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* OpenAI (2024) OpenAI. 2024. Chatgpt. [https://chat.openai.com/](https://chat.openai.com/). Accessed: 2024-09-18.
* Pandya and Holia (2023) Keivalya Pandya and Mehfuza Holia. 2023. Automating customer service using langchain: Building custom open-source gpt chatbot for organizations. _arXiv preprint arXiv:2310.05421_.
* Park et al. (2022) Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhnii Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Ramaswamy et al. (2023a) Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 2023a. Geode: a geographically diverse evaluation dataset for object recognition. _Preprint_, arXiv:2301.02560.
* Ramaswamy et al. (2023b) Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 2023b. Geode: a geographically diverse evaluation dataset for object recognition. In _NeurIPS Datasets and Benchmarks_.
* Rao et al. (2024) Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models.
* Rojas et al. (2022) William A Gaviria Rojas, Sudhya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Romero et al. (2021a) David Romero, Chenyang Lyu, Haryo Akbarianto Wibowe, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Ashikh Mandal, Alina Dragometti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farrooqui, Frederico Belcavelio, Ganorig Batnasan, Gisela Vallejo, Grainine Caulfield, Guido Ivetta, Haiyve Song, Henok Biagglian dementwe, Herman Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyun Dunstan, Laura Alonso Alemany, Kumarandage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Virdiano, MarcoS Estecha-Gariegogitia, Maria Camla Bulitrago Cabrera, Mario Rodriguez-Canellar, Melanie Joutiteau, Mihail Mihalov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adiliazuarda, Munkhjargal Gochoo, Munkh-Erdone Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Monica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryantido Dinadaru, Samuel Calwayujaya, Santiago Gongara, Soyeon Jeong, Sukannya Purkasatha, Tatuski Kuribayashi, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kenentchedjibeva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nuwat, Rada Mahlaca, Thaharm Solario, and Alham Fiki Ariji. 2024. Cycqa: Culturally-diverse multilingual visual question answering benchmark. _Preprint_, arXiv:2406.05967.
* Shi et al. (2024) Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua yu, Raya Horesh, Rog"ero Abreu de Paula, and Diyi Yang. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies.
* Srinivasan et al. (2021a) Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021a. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. SIGIR '21, page 

2443-2449, New York, NY, USA. Association for Computing Machinery.
* Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4566-4575.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Xu et al. (2023) Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. _arXiv preprint arXiv:2310.18940_.
* Yin et al. (2021) Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. 2021. Broaden the vision: Geo-diverse visual commonsense reasoning. _Preprint_, arXiv:2109.06860.
* Yun and Kim (2024) Youngsik Yun and Jihie Kim. 2024. Cic: A framework for culturally-aware image captioning. _Preprint_, arXiv:2402.05374.
* Zhang et al. (2024) Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. _arXiv preprint arXiv:2403.15378_.
* Zhang et al. (2023a) Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuanguriu Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2023a. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. _Preprint_, arXiv:2309.15112.
* Zhang et al. (2023b) Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchum Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. 2023b. Recognize anything: A strong image tagging model. _arXiv preprint arXiv:2306.03514_.
* Zhang et al. (2021) Zhuosheng Zhang, Aston Zhang, Mu Li, George Karypis, Alex Smola, et al. Multimodal chain-of-thought reasoning in language models. _Transactions on Machine Learning Research_.
* Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36.

## Appendix A Appendix

### Cultural Information Metric

CNR's cultural words are derived from the CANDLE commonsense knowledge base Nguyen et al. (2023), which covers various cultural facets like _Food, Clothing,_ and _Traditions_. However, we identified generic terms, such as occupations (e.g., "attorney"), that lack cultural specificity. Additionally, CNR includes countries outside our focus--Romania, India, and China--and does not include Romania. To refine this, we filtered out occupation-related terms from CNR and utilized ChatGPT OpenAI, 2024) to generate additional country-specific cultural words (50 words per category). We use the following prompt: _Give a comprehensive list of 50 cultural words related to CATEGORY in COUNTRY. Make sure to include words that are related to both traditions and festivals_

The 14 categories provided are: Traditions and Festivals, Cuisine, Langauge, Religion and Spirituality, Art and Literature, Science and Education, History, Social Norms and Values, Architecture and Design, Clothing and Fashion, Music and Dance, Sports and Recreation, Festivals and Holidays, Icons and Symbols.

### Fine-tuning Details

For fine-tuning, we use the WIT dataset Srinivasan et al. (2021). WIT comprises a curated set of 37.5m entity-rich image-text examples with 11.5m unique images across 108 Wikipedia languages. For fine-tuning, we choose Romanian, Hindi, and Chinese languages for Romania, Hindi and Chinese respectively.

For fine-tuning LLaVA-13b, we utilize the Transformer Reinforcement Library13 with LoRA Hu et al. (2021) enabled allowing for parameter-efficient fine-tuning. We use a 4-bit quantization, which reduces memory consumption, and a 'bf16' precision for training. This reduces memory footprint. We train the ft-specific models for 3 epochs and ft-all model for 5 epochs, with a batch size of 16 and a learning rate of \(1.4e^{-5}\). Total compute time on NVIDIA A100 is 2.5 GPU hours for each ft-specific model and 6.5 GPU hours for ft-all model.

Footnote 13: [https://huggingface.co/docs/rtl/en/index](https://huggingface.co/docs/rtl/en/index)

### Results

**On the caption length impact on Alignment score.** LLaVA-based image captions can extend 

up to three sentences, frequently surpassing LongCLIP's input limit of 248 tokens, negatively impacting Alignment performance. Moreover, this metric penalizes aspects not directly visible in the image, such as cultural context (e.g., traditions, social norms, and values). To mitigate this, we instruct the LLaVA models to focus on describing the image content in the initial sentence and to address cultural elements in subsequent sentences. Conversely, BLIP-2-generated captions are constrained to a single sentence and lack cultural information, leading to higher performance in Alignment. Consequently, Alignment scores should be evaluated alongside the other metrics to provide a comprehensive assessment.

#### a.3.1 Quantitative Results

See results on data annotated by humans (Figure 8). The trends are consistent with performance on all the data (Figure 5).

See results across each dataset:

1. Table 1 for Alignment metric
2. Table 2 for Completeness metric
3. Table 3 for Cultural Information metric

#### a.3.2 Qualitative Analysis

Across Different Datasets.For relatively simple datasets (Figure 9) with minimal cultural content or complex scenes, both MoSAIC and LLaVA-13b exhibit performance comparable to that of human annotators, displaying fewer hallucinations and higher levels of agreement. However, when applied to more complex datasets like GD-VCR10, which consist of movies from diverse cultural backgrounds, MoSAIC continues to effectively identify country-specific information and cultural elements, maintaining greater alignment with human annotators. In contrast, LLaVA-13b tends to deviate from human-like behavior, generating more hallucinations (e.g., referencing individuals not present in the image).

Across Different Conversation Rounds.With only two conversation rounds, MoSAIC tends to merely compile the perspectives of three agents from different countries without offering substantial insights, often struggling to identify the correct country information accurately. As the number of conversation rounds increases to four, MoSAIC provides more comprehensive cultural information, though this comes at the cost of increased hallucinations. Notably, three conversation rounds achieve the optimal balance between the richness of cultural descriptions and the accuracy of the information provided (Figure 11).

Across Different Prompt Strategies.Across various prompt strategies12, the CoT approach yields the best performance, delivering accurate cultural information with minimal hallucinations through step-by-step guidance. The anthropological prompt achieves a comparable level of cultural richness, though it is accompanied by more hallucinations. When given a simple prompt, MoSAIC tends to merely compile conversations without providing substantial extensions on image-related cultural insights. The multilingual prompt results in the poorest performance, offering less cultural information and producing more hallucinations, highlighting LLaVA's limitations in handling multimodal multilingual tasks effectively.

Across Different Fine-Thuning Strategies.Under different fine-tuning strategies13, MoSAIC demonstrates improved performance, generating more culturally relevant information while maintaining comparable accuracy in describing image contents.

### Prompts

See:

1. Figure 16 for LLaVA-13b prompts.
2. Figure 17 for Simple prompts.

Figure 8: Comparison across data annotated by humans. The trends are consistent with performance on all the data. Our interaction-based model, MoSAIC, surpasses non-interaction models and Humans on Completeness and Cultural Info while performing on par with the other models in Alignment. _For clarity, the Alignment and Completeness scores are normalized to a [0,1] scale, whereas the Cultural Info score ranges from 0 to the total number of words in a caption._





\begin{table}
\begin{tabular}{l l c r r r r r r r r r r r} \hline \hline  & & \multicolumn{4}{c}{CVQA} & \multicolumn{3}{c}{GDVCR} & \multicolumn{3}{c}{GeoDE} \\ \cline{2-15}  & & CN & IN & RO & All & East-Asia & South-Asia & West & All & CN & IN & RO & All \\ \hline \multirow{6}{*}{**Courage**} & human & 0.31 & 0.30 & 0.30 & 0.30 & 0.29 & 0.29 & 0.29 & 0.29 & 0.31 & - & 0.31 & 0.31 \\  & blip-2 - no inter. & 0.19 & 0.19 & 0.20 & 0.19 & 0.30 & 0.30 & 0.30 & 0.30 & 0.31 & - & 0.31 & 0.31 \\  & LLaVA-7b - no inter. & 0.22 & 0.22 & 0.23 & 0.22 & 0.31 & 0.30 & 0.30 & 0.30 & 0.28 & - & 0.30 & 0.29 \\  & LLaVA-13b - no inter. & 0.22 & 0.23 & 0.23 & 0.23 & 0.31 & 0.30 & 0.31 & 0.31 & 0.31 & - & 0.30 & 0.31 \\  & Simple & 0.23 & 0.24 & 0.24 & 0.24 & 0.27 & 0.28 & 0.27 & 0.27 & 0.28 & - & 0.29 & 0.29 \\  & Anthro. & 0.23 & 0.24 & 0.24 & 0.23 & 0.27 & 0.27 & 0.27 & 0.27 & 0.28 & - & 0.27 & 0.27 \\  & Multi. & 0.26 & 0.25 & 0.24 & 0.25 & 0.27 & 0.28 & 0.26 & 0.27 & 0.28 & - & 0.28 & 0.28 \\  & MoSAIC & 0.30 & 0.28 & 0.29 & 0.29 & 0.27 & 0.26 & 0.27 & 0.27 & 0.30 & - & 0.30 & 0.30 \\  & ft-all - no inter. & 0.28 & 0.29 & 0.29 & 0.29 & 0.27 & 0.28 & 0.27 & 0.27 & 0.30 & - & 0.30 & 0.30 \\  & MoSAIC-ft-all & 0.23 & 0.24 & 0.24 & 0.24 & 0.28 & 0.28 & 0.28 & 0.28 & 0.23 & - & 0.23 & 0.23 \\  & MoSAIC-ft-specific & 0.29 & 0.29 & 0.29 & 0.29 & 0.29 & 0.29 & 0.29 & 0.29 & 0.28 & - & 0.27 & 0.27 \\  & 2r & 0.30 & 0.30 & 0.29 & 0.29 & 0.26 & 0.26 & 0.27 & 0.27 & 0.31 & - & 0.31 & 0.31 \\  & 3r & 0.30 & 0.28 & 0.29 & 0.29 & 0.27 & 0.26 & 0.27 & 0.27 & 0.30 & - & 0.30 & 0.30 \\  & 4r & 0.30 & 0.28 & 0.28 & 0.29 & 0.26 & 0.26 & 0.26 & 0.26 & 0.30 & - & 0.30 & 0.30 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of **Alignment** across datasets and models (No interaction frameworks include ‘no inter.’, otherwise they include interaction)

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r} \hline \hline  & & \multicolumn{4}{c}{CVQA} & \multicolumn{4}{c}{GDVCR} & \multicolumn{4}{c}{GeoDE} \\ \cline{2-15}  & & CN & IN & RO & All & East-Asia & South-Asia & West & All & CN & IN & RO & All \\ \hline \multirow{6}{*}{**Courage**} & human & 0.18 & 0.25 & 0.21 & 0.23 & 0.24 & 0.24 & 0.23 & 0.24 & 0.24 & - & 0.26 & 0.25 \\  & blip-2 - no inter. & 0.03 & 0.04 & 0.04 & 0.04 & 0.19 & 0.20 & 0.19 & 0.19 & 0.19 & - & 0.20 & 0.20 \\  & LLaVA-7b no inter. & 0.09 & 0.11 & 0.13 & 0.11 & 0.39 & 0.46 & 0.44 & 0.44 & 0.27 & - & 0.26 & 0.26 \\  & LAA-13b - no inter. & 0.14 & 0.10 & 0.15 & 0.13 & 0.36 & 0.40 & 0.38 & 0.38 & 0.34 & - & 0.34 & 0.34 \\  & Simple & 0.11 & 0.11 & 0.16 & 0.12 & 0.40 & 0.44 & 0.40 & 0.41 & 0.35 & - & 0.35 & 0.35 \\  & Anthrop. & 0.11 & 0.08 & 0.11 & 0.10 & 0.36 & 0.38 & 0.39 & 0.38 & 0.32 & - & 0.32 & 0.32 \\  & Multi. & 0.41 & 0.32 & 0.29 & 0.35 & 0.31 & 0.35 & 0.32 & 0.33 & 0.30 & - & 0.28 & 0.29 \\  & MosAIC & 0.40 & 0.38 & 0.35 & 0.38 & 0.49 & 0.48 & 0.48 & 0.48 & 0.37 & - & 0.39 & 0.38 \\  & – & 0.40 & 0.38 & 0.35 & 0.38 & 0.49 & 0.48 & 0.48 & 0.48 & 0.37 & - & 0.39 & 0.38 \\  & 4r & 0.42 & 0.35 & 0.37 & 0.40 & 0.54 & 0.46 & 0.47 & 0.49 & 0.36 & - & 0.39 & 0.37 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of **Completeness** across datasets and models (No interaction frameworks include ‘no inter.’, otherwise they include interaction)

Figure 10: Comparison of image descriptions from MosAIC, Human Baseline, and LLaVA-13b across three cultures in the **GD-VCR dataset**. The cultural words are **bolded**, blue shows agreement with human captions, orange shows the identified country, and red shows incorrect and hallucinated content.

Figure 9: Comparison of image descriptions from MosAIC, Human Baseline, and LLaVA-13b across three cultures in the **GeoDE dataset**. The cultural words are **bolded**, blue shows agreement with human captions, orange shows the identified country, and red shows incorrect and hallucinated content.

 

Figure 12: Comparison of image descriptions from different **prompt strategies across three cultures in the CVQA dataset. The cultural words are bolded**, blue shows agreement with human annotators, orange shows the identified country, and red shows incorrect and hallucinated content.

Figure 13: Comparison of image descriptions from different **fine-tuning strategies** across three cultures in the CVQA dataset. The cultural words are bolded, blue shows agreement with human annotators, orange shows the identified country, and red shows incorrect and hallucinated content.

 

3. Figures 18, 19, 20 for CoT prompts across different rounds of conversation.
4. Figure 21 for Multilingual prompts.
5. Figure 22 for Anthropological prompts.

### Error Examples

See:

1. Figure 23 for incorrect country identification.
2. Figure 24 for incorrect object recognition.
3. Figure 25 for incorrect people counting.
4. Figure 26 for vague description error.

 

A multimodal CoT example when asking the agent to observe the image and give description and questions:

1. Remember you are a person from _{agent.role}_. **First**, observe the image and think what you first see in the image;
2. **Then**, think of how the object you saw is related to your culture in _{agent.role}_;
3. **Next**, think of cultural related questions about this image
4. **Finally**, generate human-like conversational language to describe the object you first saw in the image and how is this related to your culture in _{agent.role}_ and also the question you would like to ask.
5. Remember to be conversational and in human-like dialogue style. Limit answer to two sentences.

Figure 14: Chain-of-Thought Prompt from Wei et al. (2023); Zhang et al.

Figure 15: Anthropological Prompt from AlKhamissi et al. (2024)



Figure 16: LLaVa-13b Prompts

Figure 17: Simple Prompts



* [26] M. C. Bader, J. A. Krizhevsky, and D. R. Lowe. Towards real-time object detection using deep neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1378-1390, 2014.
* [27] S. Ghemawatwathan, I. Goodfellow, Y. Bengio, and H. Courville. Generative adversarial nets for image recognition. In _Advances in Neural Information Processing Systems (IJCNN)_, pages 539-546, 2014.
* [28] L. Bottou, T. Darrell, E. Farhadi, and V. Ramanan. Deep residual learning with convolutional neural networks. In _NIPS_, volume 1, pages 1378-1389, 2014.
* [29] P. Dollar, R. Girshick, and Z. Donahue. Imagenet: a large-scale hierarchical approach to image segmentation. In _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1381-1389, 2014.
* [30] W. Dong, X. Li, and Q. Weinberger. Unsupervised feature extraction via deep neural network. In _ICLR_, pages 139-140, 2014.
* [31] J. Deng, F. Ren, and J. Sun. Faster rgbg: a simple way to prevent fake 

* [26] M. C. Bader, R. D. Kiefer, and A. J. S. Russell, "A survey of the image-based object detection system for large scale visual recognition," in _IEEE Transactions on Image Processing Systems_, vol. 103(4):579-580, 2002.
* [27] G. F. Hinton, P. H. Friedman, and E. Hinton, "Deep learning with deep convolutional neural networks," _Advances in Neural Information Processing Systems_, vol. 103(4):513-523, 2002.
* [28] L. Bottou, Y. Bengio, and V. Vanhoucke, "Batch normalization by reducing internal covariate information," _arXiv preprint arXiv:1412.6098_, 2014.
* [29] T. Darrell, N. Ramanan, Z. Chen, W. Dong, X. Li, and Q. Weinberger, "Image denoising via a simple network," _CoRR_, abs/1412.05580, 2014.
* [30] I. Goodfellow, I. Guyon, J. Shlens, and R. Fergus, "The MITPS'04 conference on computer vision, speech, and machine translation," _IEEE transactions on pattern analysis and machine intelligence_, vol. 103(4):1155-1160, 2004.
* [31] I. Goodfellow, I. Sutskever, and I. Goodfellow, "Learning to recognize objects from tiny images," _Journal of Machine Learning Research_, vol. 103(4):1155-1166, 2004.
* [32] I. Goodfellow, I. Sutskever, and I. Goodfellow, "Generative adversarial nets for image classification," _Neural Networks_, vol. 103(4):1-10, 2002.

[MISSING_PAGE_POST]

 

* [28] M. C. Bader, A. K. Grafner, and R. J. S. Schultz, _A new approach to the design of a single-layer perceptron_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-490. Springer, Cham, 2007.
* [29] D. P. Kingma, I. Ba, L. Vanhoucke, and T. Darrell, "Batch normalization for stochastic optimization: A simple way to prevent neural networks from overfitting," in _Advances in Neural Information Processing Systems (NIPS'06)_, vol. 10, no. 1, pp. 1-10, Jan. 2006.
* [30] H. Li, Y. Zhang, X. Wang, and Z. Sun, "Deep residual learning for image recognition," _IEEE Transactions on Image Processing_, vol. 10, no. 1, pp. 1-1, Jan. 2006.
* [31] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [32] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [33] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [34] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [35] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [36] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [37] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [38] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [39] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [40] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [41] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [42] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [43] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [44] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [45] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [46] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [47] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [48] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [49] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [50] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [51] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [52] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [53] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [54] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [55] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [56] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [57] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [58] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [59] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [60] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [61] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [62] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [63] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [64] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [65] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 1036 of _Lecture Notes in Computer Science_, pages 457-468. Springer, Cham, 2007.
* [66] W. Liu, Y. Chen, and Q. Yu, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE Conference 

* [26] M. C. Bader, R. D. Kiefer, and A. J. S. Russell, "A new approach to the 

Figure 23: Error examples for incorrect country identification

Figure 24: Error examples for incorrect object recognition

Figure 25: Error examples for incorrect people counting