Vision-Language Models under Cultural and Inclusive Considerations (2024)

Abstract: Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available. coastalcph/vizwiz-culture

text: Proceedings of the 1st Human-Centered Large Language Modeling Workshop, pages 53–66 August 15, 2024 ©2024 Association for Computational Linguistics
coastalcph/vizwiz-culture

heading: 1 Introduction
text: With the increasing integration of AI applications into our lives, it is important to consider humancentered use cases when evaluating such systems. Large multimodal language models are now used as visual assistants for blind and visually impaired individuals. Given that people across different cultures use such applications, it is essential to ensure not only their accuracy and faithfulness (Brady et al., 2013; Gonzalez et al., 2024) but also their cultural representation and inclusion (Hershcovich et al., 2022; Shi et al., 2024).
Existing evaluation benchmarks for VLMs focus primarily on English with few, implicit mutlicultural references. Although multicultural evaluation datasets like MaRVL (Liu et al., 2021) and XM3600 (Thapliyal et al., 2022) include culturespecific images (e.g., traditional wedding costumes), they also contain images with minimal cultural significance (e.g., a bag of carrots). Consequently, these datasets may not accurately measure
the cultural knowledge of VLMs, despite being useful for assessing their multilingual capabilities. Additionally, evaluating these systems as visual assistants presents further challenges due to varying photo quality, user goals, and photo content (Chiu et al., 2020; Jung et al., 2022). Recently, Gonzalez et al. (2024) conducted a diary study with blind and low-vision individuals using an AI-powered scene description application, revealing that significant improvements are still needed for satisfying and trustworthy user experiences.
To address both cultural and visual challenges, we first surveyed visually impaired individuals to gather their caption preferences and determine if cultural details are necessary. Then, we filtered an existing dataset with images taken from people who are blind, identifying implicit cultural concepts. This is used as a challenging benchmark to evaluate image captioning performance on cultural images of state-of-the-art models across different prompt settings. With these experiments, we investigate how AI applications, such as image captioning, can foster a more inclusive and culture-aware experience for all.
53
Background Current models are trained without consideration for the subjective perspectives and cultural influences of those who provided the image descriptions (Ye et al., 2023). This raises the need for carefully curated sources of data and annotation paradigms that are more culturally aware and inclusive (Arora et al., 2023; Cao et al., 2023). Lately, there has been a growing body of work releasing multicultural multimodal datasets for visiolinguistic reasoning (Liu et al., 2021), text to image generation (Liu et al., 2023b; Ventura et al., 2023), and image captioning (Thapliyal et al., 2022). Beyond the focus on the multilingualism of the captions, concurrent work also addresses the cultural concepts depicted in the images (Cao et al., 2024; Burda-Lassen et al., 2024; et al., 2024a; Mukherjee et al., 2024; Bhatia et al., 2024). However, they still do not take into account specific use cases, such as visual assistance. Gurari et al. (2020) released the first image-captioning dataset with photos from people who are blind, and a series of challenges for multimodal systems across different tasks (Gurari et al., 2018). After this initiative, there have been many works trying to improve current models for a specific use-case, to assist people with visual disabilities (Dognin et al., 2022; Ahsan et al., 2021; Delloul and Larabi, 2023). There has also been research in human-computer interaction (HCI) and accessibility on designing image descriptions for visually impaired individuals, primarily focusing on screen readers and functional descriptions of online, publicly available images (Morris et al., 2018; Bennett et al., 2021; Schaadhardt et al., 2021). Despite these efforts, there still seems to be a lack of focus on image captioning for the visually impaired (Ghandi et al., 2023), especially in multi-cultural settings.

heading: 2 Methodology
text: We first created a survey seeking to understand the preferences of visually impaired individuals for image captions, focusing on the inclusion of cultural information and the desired level of detail (see Appendix A). We aggregate the participants’ assessments of the helpfulness and importance of cultural information in Figure 1.
We then focused on two lines of contribution: (1) We filtered the VizWiz dataset for implicit cultural concepts. VizWiz is a widely used visual question answering and image captioning dataset representing a real-world use case, where examples consist
of images and questions submitted by people who are blind, together with crowdsourced answers and image captions (Gurari et al., 2020). The selection of this dataset serves two main purposes. Firstly, it is a challenging dataset specifically tailored to realworld challenges faced by people seeking to access visual information. Secondly, VizWiz might contain implicit cultural references that are currently not captured due to the lack of culture-specific captions. (2) We evaluated the image captioning performance of state-of-the-art close-sourced and opensourced models in a culturally diverse setting using our filtered VizWiz dataset. We performed both an automatic scoring of model-generated captions against two sets of annotations using the COCO evaluation package1 and a human evaluation.

heading: 2.1 Data Filtering
text: To filter the data we hired a total of 165 annotators through the Prolific platform.2 We first asked participants to specify their country of origin, location, and their cultural background. Then, we asked them to retrieve images from the VizWiz dataset visualizer3 related to their cultural background, provide the image name, the reason they think the image is culture-related, and their preferred caption from the dataset (VizWiz provides five different image captions per image). We also gave them the option to suggest a better caption that includes cultural aspects. After collecting all the culture-specific candidate images, we proceeded to a second step of verification. In this step, we retained only those images that had received consensus agreement from at least two individuals. We collected a total of 324 images and 648 captions spanning 60 different identified cultures. It should also be noted that more than 96% of the annotators suggested a cultural revision of the original captions. We refer to Appendix B for further information about the annotation guidelines and data filtering approach and results.

heading: 2.2 Models and evaluation
text: We conducted experiments on the image captioning task in the zero-shot setting, in which a pretrained model is queried to produce a textual description for an image without finetuning on the same dataset. We relied on four commonly used open-access
1https://github.com/tylin/coco-caption 2https://www.prolific.com/ 3https://vizwiz.cs.colorado.edu/VizWiz_
visualization/view_dataset.php
Model BLEU-4 METEOR CIDEr SPICE
Prompt Default Cultural Default Cultural Default Cultural Default Cultural Annotation Original Cultural Original Cultural Original Cultural Original Cultural Original Cultural Original Cultural Original Cultural Original Cultural
BLIP-2 8.0± 0.4 4.8 7.0± 0.4 4.6 12.6± 0.2 10.2 12.3± 0.3 10.3 51.3± 3.2 39.9 44.0± 3.0 36.7 13.8± 0.4 12.5 12.8± 0.5 11.5 InstructBLIP 14.0± 0.5 8.7 14.1± 0.4 9.0 17.3± 0.3 13.2 17.7± 0.3 13.3 77.1± 3.4 60.0 78.8± 3.2 60.2 18.5± 0.4 15.6 18.2± 0.5 14.9 Idefics2 12.0± 0.5 10.1 9.8± 0.5 10.7 18.1± 0.3 15.1 18.9± 0.3 17.1 80.2± 1.9 78.4 74.1± 2.2 78.2 18.0± 0.5 16.7 18.8± 0.2 17.8 LLaVA-1.6 10.0± 0.5 11.4 6.7± 0.3 7.7 18.9± 0.4 17.3 18.4± 0.3 17.0 60.2± 2.3 75.2 40.3± 1.7 56.3 16.3± 0.6 16.5 15.8± 0.5 15.4 Gemini-1.5-Pro 10.8± 0.3 14.1 5.8± 0.1 8.7 20.8± 0.4 21.3 18.2± 0.1 21.0 71.5± 2.1 88.8 14.8± 0.5 34.1 19.6± 0.4 21.6 14.9± 0.3 17.7 GPT-4o 11.9± 0.6 16.4 8.1± 0.3 12.2 22.4± 0.4 23.4 19.9± 0.3 22.6 66.8± 2.8 99.8 40.4± 1.0 72.8 19.1± 0.4 21.8 16.6± 0.3 20.1
Table 1: Performance of various VLMs on our filtered VizWiz dataset across captioning prompts (default & culture-specific) and annotations (original & culture-specific). We use 2 reference annotations per image. Since the original VizWiz has 5 annotations per image, we report the mean and standard deviation over all 10 combinations with two references. We underline the best result for each model and display the top result for each metric in bold.
models:4 BLIP-2 6.7B (Li et al., 2023a) with OPT as LLM backbone (Zhang et al., 2022), InstructBLIP 7B (Dai et al., 2023) with Vicuna backbone (Chiang et al., 2023), Idefics2 8B (Laurençon et al., 2024), and LLaVa-1.6 7B (Liu et al., 2023a) with Mistral backbone (Jiang et al., 2023). We also used two state-of-the-art closed-access models: GPT-4o (OpenAI, 2024) and Gemini Pro 1.5 (et al., 2024b). For all of these models, we experimented with two different prompt types including a culture-specific prompt following Shi et al. (2024) and a default captioning prompt taken from Dai et al. (2023). The exact prompts can be found in App D. We evaluated the model-generated captions in two ways: (1) via the COCO evaluation suite and (2) through human evaluation. The COCO evaluation suite was first introduced by (Chen et al., 2015) as a framework to assess image captions using numerous automatic metrics, including BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), METEOR (Denkowski and Lavie, 2014), and SPICE (Anderson et al., 2016). For consistency with our culture-specific re-annotations (two captions per image), we also used two reference captions per image to score models on the original annotations. Since each image has five original captions, we report aggregate results over all ten two-caption combinations. Our human evaluation had two stages. In the first stage, we asked 60 participants to determine if a caption is accurate (on a binary scale) given the corresponding image. In the second stage, we asked the same participants to rank all captions (human-generated, and model-generated) according to their preference. We did not make the annotators aware that one caption was model-generated to minimize bias. We provide further details on the human evaluation in Appendix F.
4We used implementations and model weights from HuggingFace (Wolf et al., 2020).

heading: 3 Results
text: Automatic evaluation We present the results of our automatic evaluation of model-generated captions in Table 1. Note that due to using two reference captions per image, results for the original annotations are slightly different than when using all five at once; we report the latter in Appendix E for completeness.
As expected, the closed-access models (Gemini and GPT-4o) score best overall. Slightly lower performance is achieved by the instruction-tuned openaccess VLMs (LLaVa, Idefics2, and InstructBLIP). BLIP-2, which has not been instruction-tuned, is lagging behind across all metrics. Since VizWiz is naturally noisy due to the high ratio of low-quality, blurry images, the increased scale and overall multimodal reasoning capabilities of the closed-source models appear to give a significant advantage.
Strikingly, Gemini and GPT-4o achieve much better performance on our newly annotated captions that include cultural information than on the original captions (e.g., 11.9 vs. 16.4 BLEU-4 and 66.8 vs. 99.8 CIDEr for GPT-4o with the default prompt), while we observe the opposite for the open-access models (e.g, 14.0 vs. 8.7 BLEU-4 and 77.1 vs. 60.0 CIDEr for InstructBLIP with the default prompt). One possible explanation is that the closed-source models have been tuned to generate more descriptive captions that are aligned better with human preferences and our cultural caption annotations, whereas the open-access models have been tuned to generate slightly more concise captions that align well with benchmark datasets like COCO Captions. Our new cultural annotations are also guaranteed to not have leaked into the VLMs’ training data, thus favoring more objectively capable models such as GPT-4o.
Next, while individual models (Idefics2 and InstructBLIP in particular) seem amenable to cul-
Figure 2: Examples of various images from the filtered VizWiz dataset with the original ( ) and culture-specific ( ) annotations, and generated captions from Gemini-1.5-Pro, GPT-4o, InstructBLIP, and LLaVA-1.6 with default ( ) and culture-specific ( ) prompting.
tural prompting, leading to improved performance even on the original image captions, the cultural prompting strategy is overall largely ineffective at improving performance on the cultural captions. This result may be due to the models’ tendency for sycophantic behavior and them being primed to point out cultural information over other relevant content in the image (Sharma et al., 2023). Alternatively, cultural prompting might elicit more verbose captions that are disfavored by the automatic evaluation metrics, in which case the automatic evaluation results paint an incomplete and potentially misleading picture.
Human evaluation The results of the human evaluation are shown in Figure 3. In line with the automatic metrics, our human annotators tend to prefer the captions produced by closed-access models, GPT-4o and Gemini-Pro, with the BLIP-family models having the lowest ranking. The former are rated as accurate in more than 90% of the cases, while the latter are deemed inaccurate in more than half of the cases. Despite the strong performance of the closed-access models, our preference comparison also shows that the culture-specific humanannotated captions are still preferred over all of the models, suggesting there is ample room for improvement.
In spite of the often stark differences in automatic evaluation scores between cultural and default prompting (with a preference for the latter), human participants prefer the model generations obtained via cultural prompting in 4/6 cases (for both the ranking and the accuracy assessment), supporting our hypothesis that cultural prompting simply elicits an answer format that is disfavored by automatic metrics.
Overall, our results are promising in regard to the reliability of VLMs at zero-shot generating captions that are accurate and useful to users who are blind in culturally diverse scenarios.

heading: 4 Further Analysis
text: To further analyze our results and assess the modelgenerated captions in a more fine-grained manner, we manually inspected all generated captions for our 324 images filtered VizWiz dataset and provided some examples in Figure 2.
We find that InstructBLIP and BLIP-2 captions tend to be very short, lack a lot of information, and are often irrelevant hallucinations. This is, to an extent, expected as we perform zero-shot captioning, so the models are not necessarily accustomed to the desired captioning style. In this case, few-shot prompting or finetuning the models would likely improve model performance (Brown et al., 2020; Mañas et al., 2023; Ramos et al., 2023). The closedaccess models, in contrast, largely provide further or more useful and culture-specific details about the image than given by the human captioners. They also seem to provide more accurate captions compared to the open-sourced models. These points may explain why GPT-4o and Gemini-1.5-Pro and were overall preferred in our human evaluation.
Overall, we observed that the closed-access models can transcribe various language scripts from books, food or beverage packages, giving them an advantage over the smaller models. In most cases, in both culture-specific and default prompts, the models can identify culture-specific beverages like Japanese matcha tea, Chinese jelly grass or lychee juice, and food such as the Indian lijjat papad, Japanese mochi, Tom Kha Gai Thai soup, Korean kimchi, etc. There are also cases where they identify religious or folk items like the Wayang Golek puppets, a jar with traditional Norwegian costumes, or a delft plaque with traditional Dutch costumes.
There is, however, a tendency to generate longer text in the culture-specific prompts by adding generic phrases such as ‘hinting at the drink’s cultural origin’, ‘suggesting a celebration of their shared heritage’, ‘highlighting its appeal across age groups in Indian culture’, etc. The most challenging cases for the closed-source models seem to be foreign currencies (especially the Arabic ones), historic figures, and paintings. For example, models seem to confuse Bahraini, Jordan, and Egyptian banknotes, and they do not recognize the Chinese historical figure of Sun Yat-sen, or paintings of Joan Miró or Frederick Morgan. We provide further examples in Appendix G.

heading: 5 Discussion
text: Given the current integration of VLMs as virtual assistants for people who seek sighted support, their performance on culture-specific image captioning seems promising. Examples from our error analysis and case studies highlight some remaining challenges. Measured by automatic evaluation metrics, the performance of the models is overall relatively low compared to results in existing studies evaluating (finetuned) VLMs for image captioning on the full VizWiz and other datasets (Gurari et al., 2020; Chen et al., 2023; Wang et al., 2022). On the other hand, our human evaluation and error analysis show that the generated captions by Gemini-1.5Pro and GPT-4o are accurate and preferred in many cases. There also seems to be an extended hallucination problem, which remains an existing major challenge not only for VLMs (Li et al., 2023b) but across various language model applications (Bang et al., 2023; Ji et al., 2023).

heading: 6 Conclusion
text: We evaluated the cultural performance of various models on image captioning using a multicultural dataset tailored to a real-world use case. Although the performance of state-of-the-art closed-source models is promising, there is plenty room for improvement. Examples from our error analysis provide insights into the models’ performance, helping us identify some of their weak spots. In our use case, we find that automatic evaluation metrics might not be fully representative of model performance, and therefore encourage researchers to reconsider a more comprehensive assessment framework. For future work, we aim to extend our small filtered cultural dataset by including questionanswering tasks with POV cultural questions.
Limitations
Our work focuses primarily on data curation and empirical analysis of large multimodal language models. Our survey, while aimed at determining caption preferences, may not capture the full range of needs and preferences of all people with visual impairment. Further, through our analysis, we gained insights into some weak spots with respect to what cultures and cultural concepts are well recognized by the models. However, since we use a finite amount of data, there might be a data bias in identifying particular cultures or cultural
concepts as problematic. Lastly, cultural complexities and variations make it difficult to develop a standardized approach to cultural inclusion in AI. We do, however, hope that our culture-centric approach in the data filtering and annotation process can serve as an initial step towards evaluating and understanding the cultural awareness and abilities of vision-language models for real-world uses.
Ethics Statement
The motivation behind this study is that large vision-language models have rapidly become mainstream and are used even by those who seek sighted support and cannot easily assess model hallucinations or inaccuracies. The primary purpose of our experiments is to assess the performance of visionlanguage models in the task of image captioning using a multicultural dataset of images taken from people who are blind. However, it is crucial to recognize that results from our current filtered dataset may not be representative of model performance across cultures. Furthermore, our refined dataset might retain biases present in the original source dataset.
We find it improbable that our experiments and the filtered dataset will meaningfully benefit those intending to create deceptive models for malicious purposes. Additionally, the VizWiz dataset may lack coverage of highly specific subjects, offering only a general overview of factual topics. People who intend to use our resources, however, should state their purpose of usage and be accountable for their own work.

heading: A Survey on Caption Preferences
text: We created a survey aiming to understand the preferences of individuals who seek sighted support regarding image captioning. Our interest was particularly focused on whether they prefer image captions to include cultural information, and how detailed they prefer the descriptions to be. We published our survey through the Prolific platform, by choosing 60 participants with an equal gender sample of and representative across countries compensated with 18$ per hour. We also added a screener and selected participants without corrected/normal vision. Overall, the participants were positive regarding the helpfulness and importance of cultural information in the captions with average ratings of 4.1 and 3.9 respectfully.5 Participants also tended to prefer short captions compared to longer ones. After the anonymity period, we are going to release our survey link and full results.
B VizWiz Data Filtering—Human Annotation
As mentioned in the experimental set-up section, to filter the data we created a survey through the Prolific annotation platform. All annotators were compensated with 18$ per hour. We ran this survey 4 times asking for 40 participants each time.
We asked people to identify images from the VizWiz dataset based on their cultural background, provide an original and a corrected caption, and specify the reason they selected the image as culture-specific. We grouped the reasons that the annotators provided for selecting culture-specific images in Figure 4.
5The scale is from 1. Not important/helpful at all to 5. Very important/helpful)
The cultural concepts identified by our annotators can be found in Figure 5.
The full annotation guidelines were the following:
Creating datasets that reflect a variety of cultures is a challenging task. This is why we will try to filter an existing dataset. Your task is to find culture-related images from a dataset called VizWiz. You need to: - Visit the dataset website[link]. - Browse the dataset or use the search bars on the left side of the page and search key-terms related to your culture ’Within visual question’, ’Within visual answer’ or ’Within captions’. - Try to find an image that is related to your culture/cultural background (i.e. food brand, currency, books, culturespecific locations etc.) - Provide your answers to the 5 following questions.
1. Copy and paste the image name (VizWiz_train_**number**.jpg).
2. Based on your cultural background, specify what culture you think is the image related to.
3. Select a caption for the image from the suggested Image Captions.
4. Do you have a better suggestion for the image caption? To guide your caption generation, imagine that you are describing the image to a visually impaired friend. The caption should explain the whole image, including all the main objects, activities, and their relationships, and reflect the culture information of the image.
5. Provide a reason as to why the image is culture-specific.
After this, we collected information about the annotators’ cultural backgrounds. We asked for both home-country of origin and current country location information since sometimes both can affect our cultural beliefs and practices. The distribution of the annotators counties of origin and location are presented in Figure 6b.
The last step is to answer some final questions about your cultural background, and age. We do not collect any other personal information. Your answers will only be used for statistical research purposes.
• What is your country of origin that you consider your ’home’, influencing your cultural beliefs and other aspects of your identity?
• Is there a country in which you are currently located for a long period of time?
• How old are you? Fill in years in numbers.
After collecting all the responses, we kept only the images where at least two annotators agreed to select the image as culture-specific. After this extra validation, we resulted in a total of 324 images spanning 60 different identified cultures. We compared the similarity between the suggested captions by the annotators and the original VizWiz captions and the results can be found in Table 2 indicating a high similarity between the new culture-specific suggested captions.
Annotators Country Location

heading: C Model Overview
text: We list models with their API identifiers in Table 3 below.

heading: D Model Prompting
text: We provide the templates we used to prompt our models. The default templates have been sourced from Dai et al. (2023) and Shi et al. (2024).
E Vizwiz Results – 5 Original References
We report model performance on our filtered VizWiz dataset when using all five original captions per image (rather than combinations of two references at a time) in Figure 5.

heading: F Human Evaluation
text: To conduct the human evaluation of the model generated responses we created a survey and hired 54 annotators through the Prolific platform compensated with 18$ per hour. We added a screening in the platform for a representative sample of countries and an even distribution of male and female
participants. Each annotator evaluated 12 images and their captions and for each image, we assigned two annotators and averaged their scores. We provided the following instructions to the annotators for evaluating the captions:
This study involves evaluating captions. To guide your ratings, imagine that you are describing the image to a visually impaired friend. Then consider: How well does the caption describe the image to this friend? Does it take into account cultural considerations? You will be given two sets of captions describing an image.
1. Specify which caption you prefer for the given image (1, 2 or both).
2. Determine if each caption is accurate and relevant to the given image.
As a general guidance you should consider a caption as bad when it has one or more of the following issues: a) Caption misses the main topic of the image. b) Caption has major grammatical errors (such as being incomplete, words in the wrong order, etc). Please ignore the capitalization of words and punctuation. c) Caption includes hallucinations and mentions objects, activities, or relationships that are definitely not in the image. d) Caption is not as informative. e) Caption does not reflect the cultural information depicted in the image.

heading: G Error Analysis II
text: We provide further examples from currency-related images in Figure 7. We can see that for countries such as US, or Australia, the original VizWiz captions provide culture-specific information, but this is not the case for Japanese or Arabic currencies. Moreover, the models seem robust in western and Asian currencies, but not with all the Arabic ones. The example provided in Figure 7 shows how the models confuse a Jordan currency with Egyptian or Saudi Arabian currencies and how the smaller open-source models are more prone to hallucinations.
Default prompting
<Image> A short image description:
<Image> Write a caption that describes the photo.
Format your response in JSON as follows: {
"caption": "Caption for the image" }
<Image> A photo of
<Image> Can you briefly describe the content of the image?
<Image> Write a caption that describes the photo.
Culture-specific prompting
<Image> A short, culture-aware image description:
<Image> Cultural information encompasses content that showcases the distinctive characteristics, artifacts, or manifestations of a specific group, community, or region.↪→ This includes, but is not limited to, practices, behaviors, norms, values, beliefs, habits, customs, architectural styles, environmental engagements, and any other elements that are emblematic of a particular cultural setting.↪→ It does not include generic information or widespread practices that are not distinctly tied to a specific cultural identity.
For this task, consider information as "cultural" if:

heading: 1. It is associated with or characteristic of a specific identified group (e.g., Americans, Italians, midwestern Americans, etc.).

heading: 2. It reveals a unique aspect of that group’s way of life, including social conventions, physical creations, or interactions with their surroundings that are not typically seen in other cultures.↪→

heading: 3. It provides insight into the cultural uniqueness, whether through social practices, material culture, or other culturally significant elements.↪→
text: Please exclude generic or ubiquitous statements or observations that do not clearly relate to the unique cultural context of a specific group.↪→
Given this image, do two things:

heading: 1. Determine whether the provided example contains cultural information.

heading: 2. Write a caption that describes the photo and includes the cultural information extracted.
text: Format your response in JSON as follows: {
"caption": "Caption for the image", "is_cultural": true/false, "justification": "Why or why not the image contains cultural information"
}
<Image> Write a caption that describes the photo and includes any cultural information present.
Table 4: Image captioning templates used to prompt our models.

heading: H Case Study
text: We illustrate the value of cultural and inclusive VL models via a case study on evaluating GPT-4V as a visual assistant integrated into the ‘Be My Eyes’ platform. In this case study, we took a random sample of 20 images from the MaRVL dataset (Liu et al., 2021). Here we provide a selection of images we tried in our case study. Each figure includes the
target culture behind each image and the GPT-4 Vision output after loading the image in the Be My Eyes application.
Figure 7: Examples from images related to currency comparing original ( ) with culture-specific ( ) annotations and generated captions from Gemini-Pro and GPT-4o with default ( ) and culture-specific ( ) prompting.
Figure 8: A picture extracted from MaRVL depicting a statue of Agamas and the GPT-4V image description provided in BeMyEyes.
Figure 9: A picture extracted from MaRVL depicting Buddhist statues and the GPT-4V image description provided in BeMyEyes.
Figure 10: A picture extracted from MaRVL depicting sambar, a traditional Tamil dish, and the GPT-4V image description provided in BeMyEyes.
Figure 11: A picture extracted from MaRVL depicting a döner, a traditional Turkish dish, and the GPT-4V image description provided in BeMyEyes.

heading: Future Work
text: For future work, we aim to extend our small filtered cultural dataset by including questionanswering tasks with POV cultural questions. Limitations
Our work focuses primarily on data curation and empirical analysis of large multimodal language models. Our survey, while aimed at determining caption preferences, may not capture the full range of needs and preferences of all people with visual impairment. Further, through our analysis, we gained insights into some weak spots with respect to what cultures and cultural concepts are well recognized by the models. However, since we use a finite amount of data, there might be a data bias in identifying particular cultures or cultural
concepts as problematic. Lastly, cultural complexities and variations make it difficult to develop a standardized approach to cultural inclusion in AI. We do, however, hope that our culture-centric approach in the data filtering and annotation process can serve as an initial step towards evaluating and understanding the cultural awareness and abilities of vision-language models for real-world uses