Venue: NAACL 2025

WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines

Genta Indra Winata\({}^{,}\)1\({}^{,}\)2\({}^{,}\)3\({}^{,}\)4\({}^{,}\)5\({}^{,}\)6\({}^{,}\)7\({}^{,}\)8\({}^{,}\)9\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)10\({}^{,}\)12\({}^{,}\)13\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)15\({}^{,}\)17\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)10\({}^{,}\)12\({}^{,}\)13\({}^{,}\)15\({}^{,}\)17\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)16\({}^{,}\)18\({}^{,}\)19\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)19\({}^{,}\)12\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)12\({}^{,}\)13\({}^{,}\)19\({}^{,}\)13\({}^{,}\)15\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)12\({}^{,}\)13\({}^{,}\)19\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)15\({}^{,}\)17\({}^{,}\)19\({}^{,}\)10\({}^{,}\)11\({}^{,}\)12\({}^{,}\)13\({}^{,}\)15\({}^{,}\)16\({}^{,}\)18\({}^{,}\)19\({}^{,}\)12\({}^{,}\)13\({}^{,}\)19\({}^{,}\)14\({}^{,}\)15\({}^{,}\)16\({}^{,}\)17\({}^{,}\)18\({}^{,}\)19\({}^{,}\)12\({}^{,}\)13\({}^{,}\)19\({}^{,}\)15\({}^{, 

guage, which serves as a proxy for cultural knowledge Freedman (2021). Food choices often reflect intricate community histories, societal transformations, and both individual and collective identities, thereby creating a rich tapestry of cultural expression Almerico (2014). The relationship between culture and food is dynamic; both evolve in tandem over time, resulting in the emergence of new dishes that are influenced by historical culinary traditions Anderson (2014).

As a result, similar food concepts can be found across different countries, reflecting a shared human culinary heritage. Researchers use food as a proxy to model and analyze cultural dynamics, helping to quantify cultural differences across regions Adilazuarda et al. (2024). Many cultures have developed their own versions of "stuffed pasta" or "dumplings", each with unique ingredients and preparation methods, often known by different names Gallani (2015), as illustrated in Figure 1. Small details like how the dumping is shaped can signal the cultural background. Conversely, some dishes share the same name but have different meanings; for example, "jelly" in the U.S. refers to a fruit spread, while in the U.K. and parts of Asia, it refers to a gelatinous dessert Poppe (1992); Abe (2013). This culinary diversity presents a challenge for Vision Language Models (VLMs), which must accurately recognize and differentiate food items based on cultural context for applications like food recognition. These models navigate the complexities of names, ingredients, and preparation styles that vary widely across regions. VLMs have shown effectiveness in text captioning Liu et al. (2024),c) and have been adapted to support multiple languages Geigle et al. (2023); Shin et al. (2024).

However, there is limited research on evaluating the multicultural capabilities of VLMs, particularly in terms of multilinguality. The study by Romero et al. Romero et al. (2024) introduce visual question answering (VQA) from a multicultural perspective, but it mainly focuses on knowledge and situational context at a specific moment, which does not fully assess the ability of VLMs to reason and differentiate between cultures within a single question. Moreover, another study on food VQA is limited to Chinese culture and does not explore the broader spectrum of global cultures Li et al. (2024). An earlier investigation into cultural bias in language models also found that cultural knowledge is lacking Naous et al. (2023). Therefore, further research is necessary to address these limitations and enhance our understanding of VLMs' multicultural and multilingual capabilities.

To facilitate a comprehensive analysis of multilingual and multicultural research, we develop resources for evaluating VLMs. Table 1 summarizes how our work compares to previous studies. Our benchmark stands out for its cultural diversity, offering more VQA datasets and broader language and dialect coverage. Our major contributions can be summarized in three-fold:

* We present WorldCuisines, the first massive scale benchmark consisting of 1 million high-quality multilingual and multicultural text-image pairs annotated by native speakers in their local languages. We publicly release our resources, i.e., datasets, 1 code,2 and leaderboard3 to advance future research in this rapidly evolving field. Footnote 1: We release WC-VQA at [https://huggingface.co/datasets/worldcuisines/vqa](https://huggingface.co/datasets/worldcuisines/vqa) and WC-KB consisting food, location, cuxiuse, and prompt templates at [https://huggingface.co/worklociusines](https://huggingface.co/worklociusines).

Footnote 2: We release our code at [https://github.com/wrolduciusines/worldcuisines](https://github.com/wrolduciusines/worldcuisines).

Footnote 3: We release our leaderboard at [https://huggingface.co/spaces/worldcuisines/worldcuisines](https://huggingface.co/spaces/worldcuisines/worldcuisines).

* We evaluate open-source and commercial

\begin{table}
\begin{tabular}{l r c c c c c l}  & **\# VQA** & **\# Lang.Dilec!** & **\# Countries** & **\# Food Entries** & **\# Images** & **Parallel Data** & **License** \\ \hline FooderoA Li et al. (2024) & 659 & 2 & 1 & 60 & 389 & × & CC BY-XC-ND J 4.0 \\ World Wide Dasgupta et al. (2024) & 765 & 131 & 63 & 765 & 301 & × & CC BY 4.0 \\ AGOA Pfeiffer et al. (2022) & 12578 & 8 & 8 & NRA & 398 & ✓ & CCBY 4.0 \\ MaXLi Changpinyo et al. (2023) & 2,142 & 7 & 7 & N/A & 335 & × & Custom \\ EVYQNA Nguyen et al. (2023) & 33,790 & 3 & 1 & N/A & 4,909 & × & N/A \\ CulturalVAO Vasak et al. (2024) & 2,378 & 1 & 11 & N/A & 2,328 & × & N/A \\ SEA-VOA Urualterpasteri et al. (2024) & 1,999 & 1 & 8 & N/A & 515 & × & Custom \\ CYQA Romeo et al. (2024) & 9,000 & 26 & 28 & 1,834 & 4,560 & ✓ & Various \\ IndifWordQA Agrawal et al. (2024) & 16,716 & 1 & 1 & 255 & 414 & × & N/A \\ \hline WC-VQA & **1,152,000** & 30 & 189 & 2,414 & 6,045 & ✓ & CC BY-SA A.0 \\ \end{tabular}
\end{table}
Table 1: Data statistics for WC-VQA compared to existing VQA datasets. The data samples are sourced from their respective publications. \({}^{\dagger}\)The reported numbers are based on their human-annotated test set. \({}^{\dagger}\)This entry includes the language variations we collected for all languages.

 

VLMs for cultural awareness through two VQA tasks: predicting dish names from images and context, and identifying their geographical origin. We also assess the impact of context, including adversarial scenarios.
* We create multilingual templates for queries and context (such as the questions in QA pairs) while preserving language varieties, including dialects and registers. This is achieved by creating translations that incorporate different inflections, articles, and contractions. Our goal is to ensure naturalness in each translation and to use appropriate inflections for place names.

## 2 WorldCuisines

We propose WorldCusines, an open-source benchmark designed to evaluate the cultural relevance and understanding of VLMs. Figure 2 displays VQA examples in English, alongside selected parallel translations in Japanese and French.

### Overview

We develop both a VQA dataset (WC-VQA) and a curated KB for world cuisines (WC-KB). The WC-VQA dataset is constructed using WC-KB, which serves as the primary data source. We design two tasks as follows:

* **Task 1:** Dish Name Prediction. This task involves predicting the name of a dish based on its image, a question, and contextual information. It comprises three subtasks, each with distinct query types: _(a) no-context_ question, _(b) contextualized_ question, and _(c) adversarial contextualized_ question.
* **Task 2:** Location prediction. The task is to predict location where the food is commonly consumed and originated given the dish image, question, and a context.

Wc-Kb.A KB encompassing 2,414 dishes worldwide includes 6,045 images and metadata,

Figure 2: WC-VQA in WorldCuisines comprises two primary tasks: (1) predicting dish names and (2) predicting regional cuisines. Task 1 is further divided into three subtasks: (a) no-context, (b) contextualized, and (c) adversarial. We also include two answer types: multiple-choice question (MCQ) and open-ended question (OEQ).

 

covering both coarse-grained (e.g., stew) and fine-grained categories (e.g., beef stew), locations, and regional cuisines. It also features multilingual translations of 90 crowd-sourced prompt templates and 401 parallel data entries (i.e., multilingual information) for location and regional cuisine information.

Wc-Vqa.A multilingual parallel VQA dataset with 1 million samples encompassing over 30 languages and dialects, including various varieties and registers, such as formal and casual styles, with high-quality human annotations. The VQA is designed to evaluate models' ability to understand cultural food names and their origins.

### WC-KB Construction

Our data sources are gathered from Wikipedia4 and Wikimedia Commons5 to ensure they can be easily redistributed under an accepted open-source license. The data construction process involves four key steps: (1) dish selection, (2) metadata annotation, (3) quality assurance, and (4) data compilation. Figure 3 provides statistics on the regions covered in our dataset, with detailed information available in Table 9 in the Appendix. Figure 4 shows the distribution of dish frequencies, highlighting the top 50 countries with the most dishes.

Footnote 4: Wikipedia web pages can be accessed at [https://wikipedia.org](https://wikipedia.org).

#### 2.2.1 Dish Selection

We compile a comprehensive list of dish names sourced from Wikipedia. We manually review pages that feature lists of dishes to determine whether each dish is a specialty unique to a specific culture, as we aim to focus on dishes that have distinct cultural significance. We exclude generic categories, such as ice cream, which lacks a specific cultural association. We ensure that each dish on our list has its own dedicated Wikipedia page. If a dish does not have a Wikipedia page, it is also excluded from our compilation. This meticulous approach ensures that our dataset is both culturally relevant and well-documented.

Figure 4: Countries by number of assigned dishes, showing the top 50 countries.

Figure 3: WorldCuisines distribution of food entries by country in the World Map. The food entries are distributed across 189 countries, with the highest concentration found in Asia, Europe, and North America. There are also some entries from the continents of Africa, Oceania, and Central and South America.

 



#### 2.2.2 Metadata Annotation

Given a dish name and its corresponding Wikipedia page link, we then ask annotators to manually compile metadata based on the provided information. This metadata includes:

* **Visual Representation**: Images sources from Wikimedia Commons are included, along with their license information.
* **Categorization**: Dishes are classified into both coarse-grained (e.g., rice, bread) and fine-grained (e.g., fried rice, flatbread) categories.
* **Description**: Annotators provide a description of each dish based on the content from its Wikipedia page, avoiding the use of the dish's name, origin, or any distinctive keywords that uniquely identify the dish.
* **Cuisine**: The dish's origin cuisine and any cuisines with which it is strongly associated.
* **Geographic Distribution**: This includes the dish's associated countries, area (city or region), and broader continental region.

The metadata description, along with the example, is further elaborated in the Appendix Table 4.

#### 2.2.3 Quality Assurance

Before starting the quality assurance process, we first identify common issues that arise during the annotation and develop automated rules to detect easily identifiable annotation errors, such as incorrect string formatting. Annotators are then asked to correct these errors. To further ensure data quality and validity, we conduct several rounds of quality assurance. Initially, we focus on image quality by removing instances where images are blurry, dark, or contain distracting elements such as people or other dishes. We also verify image licenses by cross-referencing them with information on Wikimedia Commons. Next, we refine the dish categorization and descriptions, ensuring consistency in category assignments and maintaining descriptions free from "information breaches" (e.g., excluding regional details from the description). We standardize cuisine names and eliminate any redundancies. Finally, we meticulously review all country and area information to ensure its accuracy. This comprehensive approach guarantees the integrity and reliability of our dataset.

#### 2.2.4 Data Compilation

In this phase, we verify the overall quality check done by annotators, and identify any potential inconsistencies that are missed during the quality assurance. Then, we compile the dataset by collecting the metadata into a single file.

### VQA Generation

In this phase, we generate VQA data by sampling from WC-KB. An entry of VQA data comprises visual image, question text, and answer text. This process involves four stages: (1) conducting a similarity search for dish names, (2) constructing questions and contexts, (3) translating these elements into multiple languages, and (4) generating the VQA triplets.

#### 2.3.1 Dish Names Similarity Search

To identify similar dishes in our dataset, we follow the approach from Winata et al. (2024) to employ a multilingual model E5LARGE Instruct Wang et al. (2024) for computing text embedding. Formally, given a dish \(x\) with name \(x_{\text{name}}\) and text description \(x_{\text{desc}}\), we use a multilingual model \(\theta\) to compute the embedding vector \(v_{x}=\theta(\{x_{\text{name}};x_{\text{desc}}\})\), then apply cosine similarity to compute a score \(s=\text{similarity}(v_{i},v_{j})\) between dish \(i\) and dish \(j\). For each dish, we consider the top-\(k\) most similar dishes to generate distractors in the multiple choice question.

#### 2.3.2 Question and Context Construction

Dish name prediction (Task 1) is divided into three question variations depending on the context: (1a)

\begin{table}
\begin{tabular}{l r r r r r|r|r|r} \hline \multicolumn{1}{c}{\multirow{2}{*}{**Data**}} & \multicolumn{6}{c|}{**Task 1 (Dish Name)**} & \multicolumn{3}{c|}{**Task 2**} & \multicolumn{1}{c}{**Total**} \\  & \multicolumn{2}{c}{**(a) no-context**} & \multicolumn{2}{c}{**(b) contextualized**} & \multicolumn{2}{c}{**(c) adversarial**} & \multicolumn{2}{c}{**(Location)**} & \multicolumn{1}{c}{**\# VQA**} \\  & \multicolumn{1}{c}{\# VQA} & \multicolumn{1}{c}{\# Images} & \multicolumn{1}{c}{\# YQA} & \multicolumn{1}{c}{\# Images} & \multicolumn{1}{c}{\# VQA} & \multicolumn{1}{c}{\# Images} & \multicolumn{1}{c}{\# IVQA} & \multicolumn{1}{c}{\# Images} & \\ \hline Train (1M) & 270,300 & 3,383 & 267,930 & 3,555 & 271,770 & 3,589 & 270,000 & 3,361 & 1,080,000 \\ Test Small (12k) & 3,000 & 100 & 3,000 & 100 & 3,000 & 100 & 3,000 & 100 & 12,000 \\ Test Large (60k) & 15,000 & 500 & 15,000 & 500 & 15,000 & 499 & 15,000 & 499 & 60,000 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics for WC-VQA tasks for train, test small, and test large data splits. Total #VQA represents the total number of VQA from Task 1 and Task 2.



no-context question_, where we simply ask for the name of the dish without any provided context; (1b) _contextualized question_ where we provide additional information related to cuisine or location; and (1c) _adversarial contextualized question_ which are similar to the contextualized questions but may include misleading location information to assess the model's robustness to irrelevant details.

For example, consider _coxinha_ from **Brazil**, shown in Figure 2 (1b). A query with additional context here would be: "What is the common name for this dish in Brazil?" Here, the origin of _coxinha_, **Brazil**, serves as the context. In contrast, adversarial context involves providing misleading or irrelevant information in terms of location or type of cuisine to assess the model's robustness to such distractions. For instance, in the case of eggs benedict shown in Figure 2 (1c), an adversarial context would be: "Yesterday I had a nice lunch at a Korean restaurant. I am about to have this dish now. What is this dish called?" In this scenario, the model should ignore the irrelevant detail ("nice lunch at a Korean restaurant") and focus solely on the image and the question.

Only basic question without any provided context is available for regional cuisine prediction (Task 2). The data statistics for each task are presented in Table 2.

#### 2.3.3 Multiple Language Translation

Question and ContextAll questions and contexts are initially collected in English, which are then carefully translated by native human speakers into 30 language varieties: 23 different languages with 7 languages having two different varieties each. We instructed the translators to prioritize the naturalness, and then followed by the diversity of translations when the duplication occurs.

Food Name AliasUsing Wikipedia pages as our primary source, we can verify if the English page has translations available in other languages. This enables us to extract dish names in multiple languages and compile them as translations for each dish. We utilize both the Wikipedia page titles in various languages and the alias text found on the English page. These translations are especially valuable for multilingual prompt translation, as they allow us to use the dish's native name instead of its English equivalent, enhancing cultural relevance and accuracy. We use the English name as default when the translation is unavailable.

Locations and CuisinesAs there are more than 400 unique locations, including countries, cities,

\begin{table}
\begin{tabular}{l c c c c c|c c c|c c} \hline \hline  & \multicolumn{6}{c|}{**Task 1 (Dish Name)**} & \multicolumn{3}{c|}{**Task 2**} & \multirow{3}{*}{**Average**} \\ \cline{3-9}
**Model** & & \multicolumn{2}{c}{**(a) no-context**} & \multicolumn{2}{c}{**(b) contextualized**} & \multicolumn{2}{c}{**(c) adversarial**} & \multicolumn{2}{c|}{**(Location)**} & \\  & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ \\ \hline
**Open-Source** & & & & & & & & & & \\ Llaval.6 Vicuna 7B & 34.57 & 1.59 & 43.48 & 4.03 & 34.84 & 1.41 & 32.24 & 9.29 & 36.28 & 4.08 \\ Llaval.6 Vicuna 13B & 40.17 & 2.79 & 48.17 & 5.85 & 39.05 & 2.57 & 37.79 & 10.16 & 41.30 & 5.34 \\ Qwen2 VL Instruct 2B & 41.65 & 7.98 & 42.29 & 8.13 & 39.69 & 6.74 & 47.85 & 14.55 & 42.87 & 9.35 \\ Qwen2 VL Instruct 7B & 61.48 & 6.76 & 67.85 & 10.36 & 53.52 & 6.12 & 55.90 & 21.03 & 59.69 & 11.07 \\ Qwen2 VL Instruct 72B & 74.19 & 12.67 & 80.79 & 21.31 & 62.43 & 8.37 & 61.90 & 27.27 & 69.83 & 17.40 \\ Lima 3.2 Instruct 1H1 & 59.93 & 18.75 & 64.12 & 22.96 & 53.17 & 13.39 & 57.93 & 31.58 & 58.79 & 21.67 \\ Llama 3.2 Instruct 90B & 77.69 & 16.93 & 82.92 & 23.60 & 63.96 & 10.87 & 67.87 & 31.31 & 73.11 & 20.68 \\ Molmo-E 1B & 18.81 & 0.01 & 24.22 & 0.23 & 19.55 & 0.01 & 18.97 & 1.54 & 20.39 & 0.45 \\ Molmo-D 7B & 46.01 & 2.89 & 55.95 & 3.66 & 41.61 & 2.31 & 33.35 & 11.45 & 44.23 & 5.08 \\ Molmo-O 7B & 39.96 & 5.15 & 44.93 & 6.03 & 38.41 & 3.51 & 29.81 & 10.07 & 38.28 & 6.19 \\ Pangae 7B\({}^{\ddagger}\) & 52.35 & 1.52 & 63.07 & 2.73 & 49.17 & 1.57 & 48.71 & 20.15 & 53.33 & 6.49 \\ Ariza 2SB & 58.61 & 4.99 & 69.29 & 9.17 & 52.82 & 3.39 & 42.82 & 16.20 & 55.89 & 8.44 \\ Phi-3.5 Vision 4B & 43.37 & 2.91 & 48.71 & 4.23 & 40.87 & 2.07 & 35.01 & 9.22 & 41.99 & 4.61 \\ Pixtral 12B & 56.65 & 1.22 & 70.69 & 2.94 & 52.12 & 1.09 & 46.67 & 14.43 & 56.53 & 4.92 \\ NVLM-D 72B & 69.82 & 4.71 & 78.93 & 10.29 & 52.12 & 2.89 & 51.97 & 16.68 & 63.21 & 8.64 \\ \hline
**Proprietary** & & & & & & & & & & \\ GPT-4o & **88.45** & **21.88** & **91.57** & **37.51** & **82.29** & **14.79** & 66.52 & **37.13** & **82.21** & **27.83** \\ GPT-4o Mini & 72.80 & 10.28 & 81.65 & 20.87 & 57.76 & 5.72 & 52.37 & 25.79 & 66.14 & 15.66 \\ Gemini 1.5 Flash & 77.05 & 12.81 & 80.97 & 15.16 & 69.13 & 6.46 & **71.53** & 30.03 & 74.67 & 16.12 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy (%) results of WC-VQA for Test Large (60k). MCQ and OEQ indicate multiple-choice question and open-ended question, respectively. Best and second-best are **bolded** and underlined, respectively. \({}^{\ddagger}\)We employ an optimized prompt provided by the authors (see Subsection E.1 in the Appendix for further details).

 

and areas, we first translate the English locations into other languages by using GPT-4o, followed by proofreading each translation by the native speakers. The string values for the regional cuisines, i.e., the adjective form of the location in English, are translated in the same manner as location.

Morphological Inflections.Indo-European languages, such as Czech or Spanish, are rich in inflectional morphology which involves word modification to express different grammatical categories, such as number, gender, or case. For example, the equivalents of the phrases "in Japan" and "from Japan" in Czech are "v Japonsk\(u\)" and "z Japonska\(a\)", respectively. We provide a framework for the human translators to use the inflections in the prompt template to prioritize the naturalness while keeping the inflections as few as possible.

#### 2.3.4 Generating VQA Triplets

To ensure no overlap in train and test subsets, we split the dishes and the multilingual-questions into two subsets each, to ensure no dish or multilingual questions leakage between train and test. For every subset, we apply random sampling to get a pair of dish and its multilingual-questions. We use the dish entry in our WorldCuisines KB dataset to pick the image and the location to be injected to the context, if any. The answer candidates for multiple-choice were picked by utilizing similarity search (Section 2.3.1). We repeat this process until we reach the desired number of training or test samples, or until all possible dish and question combinations are used, discarding any duplicates.

## 3 Experiments

### Experimental Setup

Metrics.We use accuracy as the primary metric to evaluate predictions. For Task 2 (open-ended), we employ BERTScore Zhang et al. (2019) with XLM-R Large Conneau and Lample (2019) as a secondary metric to determine if the model-generated content includes food names similar to those in the gold labels. For open-ended questions, we compute the accuracy of each test sample against multiple references, including translations of the dish in different languages. This approach allows us to accommodate predictions that may not be in the expected language.

Models.We evaluate our benchmark on various available VLMs, including 15 open-source models and 3 proprietary models. During the inference

Figure 5: Accuracy (%) categorized by language **(left)**, language vitality **(center)**, and language family **(right)**. We classify the language vitality by following the classification proposed by Joshi et al. (2020).

 

of the open-source model, we use 16-bit floating point and employ greedy decoding. We access the proprietary models through API. The complete list of the models is available in Table 3.

## 4 Results and Discussion

### Overall Results

The results for WC-VQA are summarized in Table 3. The multiple-choice question (MCQ) results without any context exhibit significant variability, ranging from 30% to 80%, indicating considerable differences in model performance. This variability indicates that predicting MCQs remains a challenging task for many models. Notably, proprietary models, particularly GPT-4o, demonstrate exceptional performance, outperforming all other models. In the open-ended question (OEQ) setting, the task proves even more difficult than the MCQ, with models achieving a maximum accuracy of less than 20% for dish name predictions and slightly higher for location predictions when no context is provided. However, incorporating context enhances performance across all settings, highlighting that context effectively guides the models in making better predictions. Interestingly, when the adversarial context is introduced, it misleads the models, leading to incorrect predictions and adding further complexity to the task. Among the models evaluated, Llama 3.2 Instruct significantly outperforms other open-source model families, while Qwen2 performs relatively better than Llava 1.6 and Molmo, despite having smaller model sizes.

### The Role of Context

For dish name prediction (Task 1), incorporating more relevant context significantly enhances performance across all language families. However, when adversarial context is introduced, performance drops significantly. The adversarial context included in the prompt significantly affects the prediction. Instead of relying solely on the image input, the model often sways and makes predictions based on incorrect location or cuisine information, even when the context is unrelated to the query. This observation is particularly intriguing, as it signifies that such prompts can shift the model's attention and influence its generation process.

### Results by Language

In Task 1 with OEQ setting (Figure 5b), some languages with non-Latin scripts, such as Arabic, Korean, Japanese, and Marathi, tend to perform poorly, with the exception of Chinese. For Task 2 with OEQ setting, most models struggle with Sino-Tibetan languages (i.e., Chinese, Cantonese, and Hokkien) and Niger-Congo languages (i.e., Yoruba). In contrast, the models demonstrate relatively strong performance with Japonic, Koreanic, Kra-Dai (i.e., Thai), and Turkic (i.e., Azerbaijan) languages. We also observe that answering OEQs in underrepresented languages remains particularly challenging for the models, as shown by the relatively lower results for the "left behind", "scraping by", and "hopeful" languages. Interestingly, lower performance in the OEQ does not necessarily translate to the lower performance in the MCQ setting (Figure 5a) where the performance gap between language categories is less pronounced. The gap between OEQ and MCQ, especially for underrepresented languages, suggests that the bottleneck might lie in the factors beyond cultural understanding, such as text generation capabilities.

### Scaling Law

It is evident that large models perform better than smaller ones, showing the scaling law still exists in this experiment, as shown in Figure 6. It is very interesting to see the same trend across different model families (e.g., Llava, Owen, and even GPT-4o series). However, it is pretty clear for open-source models, Llama 3.2 Instruct has the lead for overall performance, which may be due the coverage of multilingual data used in its training, although it is still unclear since there is no evidence or supporting information that can back up the finding. Regardless, NVLM-D model does not perform as good as their base model Qwen2 VL Instruct in our benchmark. One reason could be the NVLM model is highly tuned for English, but not in languages other than English.

## 5 Related Work

**Cultural VQA.** Several prior studies have focused on developing culturally relevant VQA benchmarks, including FM-IQA Gao et al. (2015), MCVQA Gupta et al. (2020), xGQA Pfeiffer et al. (2022), MaXM Changpinyo et al. (2023), MTVQA Tang et al. (2024), MABL Kabra et al. (2023), MAPS Liu et al. (2024), and MaRVL Liu et al. (2021). Additionally, CVQA Romero et al. (2024) and CulturalVQA Nayak et al. (2024) provide VQA datasets that cover various regions and 

diverse topics, including food, with CVQA also offering questions in multiple languages alongside English translations. SEA-VQA (Urailertprasert et al., 2024) specifically benchmarks the South East Asian region. In contrast, FoodieQA Li et al. (2024) and World Wide Dishes Magomere et al. (2024) are benchmark focusing on food. Our work is similarly motivated by using food as a cultural proxy, but it distinguishes itself with a significantly larger dataset and broader coverage of languages and cultures.

Multi-modal LLMs.Recent advancements in VLMs have led to the emergence of multi-modal LLMs that can process both images and text. LLaVA Liu et al. (2024) exemplifies this approach by utilizing Vicuna Zheng et al. (2023) as an image encoder, thereby enhancing visual understanding. This architecture has set a precedent for other VLMs, including Qwen2-VL Bai et al. (2023), Llama 3.2 Dubey et al. (2024), Pixxtral Agrawal et al. (2024), Phi-3.5 Vision Abdin et al. (2024), Molmo Deitke et al. (2024), Aria Li et al. (2024), Pangea Yue et al. (2024), and NVLM Dai et al. (2024), each leveraging their respective large language models for multi-modal tasks. In a specialized application, FoodLMM Yin et al. (2023) focuses specifically on the food domain, training on publicly available food datasets and conversational data generated by GPT-4 Achiam et al. (2023). Our work evaluates the capabilities of these models within the food domain, offering insights into their performance and potential applications in culinary-related tasks across multicultural settings.

## 6 Conclusion

We introduce WorldCuisines, an open-source, large-scale benchmark designed for multilingual and multicultural, visually grounded language understanding. It comprises over 1 million data points across 30 languages and dialects. Our findings reveal that this benchmark remains challenging for VLMs, particularly with dishes from specific regions and in low-resource languages. This provides insight into how well models understand regional cuisines. To enhance usability, we offer a dedicated evaluation split with two datasets of varying sizes. Our evaluation shows that while VLMs perform better with the correct context, they struggle with adversarial contexts intended to mislead them. Additionally, we are releasing a comprehensive knowledge base, VQA dataset, code, and leaderboard as open-source resources to support future research.

## Acknowledgements

We extend our gratitude to everyone who has supported our project, especially the numerous annotators who provided meticulous and comprehensive annotations and conducted thorough quality checks. Special thanks to Francesca Porcu for her assistance with the Sardinian language and to Shintaro Ozaki for his help with Japanese. We are also deeply appreciative of Nayeon Lee and Wenliang Dai for their insightful discussions and for integrating NVLM into our benchmark. Additionally, we thank Xiang Yue and Yueqi Song for their help in integrating Pageea into our benchmark.

## Limitations

In this paper, we limit our investigation to avoid exhaustively evaluating all possible models due to resource constraints. Our primary focus is on developing a benchmark that facilitates exploration for future research. We also provide a training data split for reference, allowing other researchers to utilize it to enhance their VLMs and evaluate their

Figure 6: Scaling matters for MCQ (6a) and OEQ (6b).

 

models against our test sets. Currently, we include 30 different languages and dialects, establishing one of the largest and most diverse benchmarks for comprehensive multilingual VQA. We aim to extend this benchmark to encompass additional languages in the future, making it more inclusive and representative of a broader range of linguistic diversity.

It is important to note that our food entries are currently sourced from English Wikipedia. Although we aim to include as many diverse dishes as possible, we acknowledge that this approach limits the coverage of some regions. This is due to language affects commonsense and its specific knowledge Sakai et al. (2024), which in turns suggesting insufficiency of sourcing only English Wikipedia. Nevertheless, our dataset serves as a valuable starting point. In future work, we plan to incorporate entries from non-English Wikipedia pages to improve regional representation and cultural diversity. For evaluation purposes, we include accuracy metrics for overall model performance and BERTScore for more detailed analysis. However, we recognize that evaluating VQA model performance on multicultural data remains an open challenge. Appropriate evaluation metrics are needed to effectively model the diversity of cultural contexts and linguistic variations. Addressing this issue will be a key focus of our future research efforts.

## Ethical Considerations

Our research focuses on evaluating VLMs within the context of multilingual and multicultural VQA, a field that holds significant implications for diverse multilingual communities. We are committed to conducting our data collection and evaluations with the highest standards of transparency and fairness. To achieve this, we have adopted a crowd-sourcing approach for the annotation process, inviting volunteers to contribute and become co-authors if they provide significant contributions. We follow the guidelines from ACL for authorship eligibility as shown in [https://www.aclweb.org/adminwiki/index.php/Authorship_Changes_Policy_for_ACL_Conference_Papers](https://www.aclweb.org/adminwiki/index.php/Authorship_Changes_Policy_for_ACL_Conference_Papers). In line with our commitment to openness and collaboration, we will release our dataset under an open-source license, CC-BY-SA 4.0.

## References

* Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadalah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. _arXiv preprint arXiv:2404.14219_.
* Abe (2013) Marie Abe. 2013. Tokyo, japan. In _The Ethnomusicologists' Cookbook_, pages 40-45. Routledge.
* Achim et al. (2023) Josh Achim, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Andkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Adilazuarda et al. (2024) Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyunma Lavania, Siddharn Singh, Ashutosh Dwivedi, Alham Fikiri Ajj, Jacki O'Neill, Ashtudsh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling" culture" in llms: A survey. _arXiv preprint arXiv:2403.15412_.
* Agarwal et al. (2024) Pulkit Agarwal, Settaluri Savanthi, and Pushpak Bhattacharyya. 2024. Indifoodvqa: Advancing visual question answering and reasoning with a knowledge-infused synthetic data generation pipeline. In _Findings of the Association for Computational Linguistics: EACL 2024_, pages 1158-1176.
* Agrawal et al. (2024) Pravesh Agrawal, Szymon Antoniak, Emna Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amelie Heliou, Paul Jacob, et al. 2024. Pixtral 12b. _arXiv preprint arXiv:2410.07073_.
* Almerico (2014) Gina M Almerico. 2014. Food and identity: Food studies, cultural, and personal identity. _Journal of International Business and Cultural Studies_, 8:1.
* Anderson (2014) Eugene Newton Anderson. 2014. _Everyone eats: Understanding food and culture_. NYU Press.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Owen-vl: A frontier large vision-language model with versatile abilities. _arXiv preprint arXiv:2308.12966_.
* Campbell and Grondona (2008) Lyle Campbell and Veronica Grondona. 2008. Ethnologue: Languages of the world. _Language_, 84(3):636-641.
* Changpinyo et al. (2023) Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023. Maxm: Towards multilingual visual question answering. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2667-2682.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. _Advances in neural information processing systems_, 32 

* Dai et al. (2024) Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolong Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. lvlm: Open frontier-class multimodal lms. _arXiv preprint arXiv:2409.11402_.
* Deitke et al. (2024) Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadzeza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldani, et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. _arXiv preprint arXiv:2409.17146_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Freedman (2021) Paul Freedman. 2021. _Why Food Matters_. Yale University Press.
* Gallani (2015) Barbara Gallani. 2015. _Dumplings: A global history_. Reaktion Books.
* Gao et al. (2015) Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question. _Advances in neural information processing systems_, 28.
* Geigle et al. (2023) Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavas. 2023. mblip: Efficient bootstrapping of multilingual vision-llms. _arXiv preprint arXiv:2307.06930_.
* Gupta et al. (2020) Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya. 2020. A unified framework for multilingual and code-mixed visual question answering. In _Proceedings of the 1st conference of the Asia-Pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing_, pages 900-913.
* Holtzman (2006) Jon D Holtzman. 2006. Food and memory. _Annu. Rev. Anthropol._, 35(1):361-378.
* Joshi et al. (2020) Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293.
* Kabra et al. (2023) Anubha Kabra, Emmy Liu, Simran Khanuja, Albham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anoulwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and multi-cultural figuring language understanding. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8269-8284.
* Li et al. (2024a) Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. 2024a. Aria: An open multimodal native mixture-of-experts model. _arXiv preprint arXiv:2410.05993_.
* Li et al. (2024b) Wenyan Li, Xinyu Zhang, Jia Liang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Sogaard, et al. 2024b. Food-iota: A multimodal dataset for fine-grained understanding of chinese food culture. _arXiv preprint arXiv:2406.11030_.
* Liu et al. (2024a) Chen Liu, Fajir Koto, Timothy Baldwin, and Iryna Gurevych. 2024a. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2016-2039.
* Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10467-10485.
* Liu et al. (2024b) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024b. Improved baselines with visual instruction tuning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 26296-26306.
* Liu et al. (2024c) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024c. Visual instruction tuning. _Advances in neural information processing systems_, 36.
* Magomere et al. (2024) Jabez Magomere, Shu Ishida, Tejumade Afonja, Aya Salama, Daniel Kochin, Foutse Yuehgoh, Imane Hamzaioui, Raesetie Sefala, Aisha Alaagib, Elizaveta Semenova, et al. 2024. You are what you eat? feeding foundation models a regionally diverse food dataset of world wide dishes. _arXiv preprint arXiv:2406.09496_.
* Naous et al. (2023) Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. _arXiv preprint arXiv:2305.14456_.
* Nayak et al. (2024) Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishway Agrawal. 2024. Benchmarking vision language models for cultural understanding. _arXiv preprint arXiv:2407.10920_.
* Nguyen et al. (2023) Ngan Luu-Thuy Nguyen, Nghia Hieu Nguyen, Duong TD Vo, Khanh Quoc Tran, and Kiet Van Nguyen. 2023. Vlysp2022-eviyqa challenge: Multilingual visual question answering. _arXiv preprint arXiv:2302.11752_.
* Pfeiffer et al. (2022) Jonas Pfeiffer, Gregor Geigle, Aishshway Ramath, Jan Martin Steitz, Stefan Roth, Ivan Vulic, and Iryna Gurevych. 2022. xgga: Cross-lingual visual question answering. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2497-2511.
* Poppe (1992) J Poppe. 1992. Gelatin. In _Thickening and gelling agents for food_, pages 98-123. Springer 

David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliiev, Anatrik Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. _arXiv preprint arXiv:2406.05967_.
* Sakai et al. (2024) Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2024. mcsqa: Multilingual commonsense reasoning dataset with unified creation strategy by language models and humans. In _Findings of the Association for Computational Linguistics: ACL 2024_, pages 14182-14214. Association for Computational Linguistics.
* Shin et al. (2024) DongJae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, SeungWoo Song, HanGyeol Yoo, SangMin Kim, and KyungTae Lim. 2024. X-lava: Optimizing bilingual large vision-language alignment. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 2463-2473.
* Tang et al. (2024) Jinggun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. 2024. Mtvaq: Benchmarking multilingual text-centric visual question answering. _arXiv preprint arXiv:2405.11985_.
* Urailertprrasert et al. (2024) Norawit Urailertprrasert, Peerat Limkonchotiwat, Supasorn Suwajanakorn, and Sarana Nutanong. 2024. Sea-vqa: Southeast asian cultural context dataset for visual question answering. In _Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR)_, pages 173-185.
* Wahlqvist (2007) Mark L Wahlqvist. 2007. Regional food culture and development. _Asia Pacific journal of clinical nutrition_, 16:2.
* Wang et al. (2024) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. _arXiv preprint arXiv:2402.05672_.
* Winata et al. (2024) Genta Indra Winata, Ruochen Zhang, and David Ifeolouwa Adelani. 2024. Miners: Multilingual language models as semantic retrievers. _arXiv preprint arXiv:2406.07424_.
* Yin et al. (2023) Yuehao Yin, Huiyan Qi, Bin Zhu, Jingjing Chen, Yu- Gang Jiang, and Chong-Wah Ngo. 2023. Foodmm: A versatile food assistant using large multi-modal model. _arXiv preprint arXiv:2312.14991_.
* Yue et al. (2024) Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutwikia, Sathyaranayanan Ramamoorthy, and Graham Neubig. 2024. Pangea: A fully open multilingual multimodal llm for 39 languages. _arXiv preprint arXiv:2410.16153_.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zheng et al. (2023)Linamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhangao Wu, Yonghao Zhuang, Zi Lin, Zhouhan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-juage with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595-46623.

## Appendix A Data Statement

### Executive Summary

WorldCuisines is a vision-language benchmark comprised of two resources: (1) WC-VQA, a multilingual parallel question answering dataset covering 30 languages and dialects where each dish image is accompanied by questions and context constructed through human translation; and (2) WC-KB, a knowledge base containing images and metadata associated with the dishes.

### Curation Rationale

The goal of WorldCuisines is to evaluate the cultural understanding of vision-language models (VLMs) within the food domain. To achieve this, we develop WC-VQA and WC-KB. Dish names and their information are collected from English Wikipedia, and the images are selected from Wikimedia Commons to ensure a permissive license, with an emphasis on representing a wide range of food categories and geographic origins (or where the dish is popular). This selection strategy aims to provide insights into the VLMs' ability to generalize across diverse culinary and cultural contexts.

### Language Variety

WorldCuisines covers 30 languages and dialects spoken across diverse countries and regions. The complete list of languages and dialects is shown in Table 5. An example of the multilingual prompt is shown in Table 6.

### Annotator Demographic

Over 30 annotators are involved in building WorldCuisines, specifically in translating the query and context for the WC-VQA dataset. Most annotators are native speakers of the target languages or dialects included in our data; some are L2 speakers with more than 10 years of study in their respective languages. The detailed demographics for each language are elaborated below.

#### a.4.1 Austronesian

IndonesianTwo native Indonesian speakers are involved as translators. One is in the 26-35 age 

range, and the other is in the 16-25 age range.

TagalogOne native Tagalog speaker in the 16-25 age range is involved as a translator.

SundaneseTwo L2 Sundanese speakers contribute to the translation. One, in the 16-25 age range with 15 years of experience with the Sun

\begin{table}
\begin{tabular}{l l c l} \hline \hline
**Attribute** & **Value** & **Description** & **Example** \\ \hline Name & String & Name of the dish. & “Dozyak”: \\ Alias & ListClebTcp. & Name alas., i.e. the name in the original language. & [“\(\geq\)0\% b\# ”; Japanese”] \\ Coarse-grained categories & ListScrimp–Complex-level categories. & “Pancake”: "Desester" \\ Fine-grained categories & ListScrimp-Type-Free-led categories. & [“Wagushi Pancake”] \\ Cvinines & String & Name of cuinsense. & “Japanese” \\ Associated Cuinines & String & Associated cuinses to the dish. & “Japanese” \\ Area & String & Specific region where the dish is originated & “Ueno” \\ Countries & String & Specific region where the dish is originated & “Japan” \\ Region[...] & String & Specific content where the dish is originated & “Eastern Asia” \\  & & Short description of the dish, including the ingredients & “The dish consists of two small pancake-like entities made from castals wrapped around a filling of sweet penate. \\ Image[...] URL & String & Image link to Wikimedia Commons. & “.../commens/93/pc/Droryaki_001_(3).jpg” \\ Image[...] License & String & Licence of the image & “CC BY-SA 3.0” \\ \hline \hline \end{tabular}
\end{table}
Table 4: WC-KB attributes in WorldCuisines.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Language Name** & **Language Vitality\({}^{\dagger}\)** & **Resource Classification\({}^{\ddagger}\)** & **Linguistic Register** & **Additional Notes** \\ \hline
* Rising Star & Casual & \\ Tagalog & Institutional & 3
- Rising Star & & \\ Sundanese & Stable & 1
- Scraping by & Loma & Common speech form \\ Javanese & Institutional & 1
- Scrapping by & Krama & Central-Java dialect, polite form \\ \hline
* Winners & Formal & Polite form or teinei-go \\  & & & Casual & Daily conversation \\ \hline
* Winners & & \\ Cantonese & Institutional & 1
- Scraping by & & \\ Hokkien & Institutional & 0
- Left Behind & Written & Medan dialect \\ \hline
* Underdog & Formal & \\  & & & Casual & \\ \hline
* Rising Star & & \\ \hline
* Winners & & Latin-American dialect \\ Spanish & Institutional & 5
- Winners & & \\ French & Institutional & 5
- Winners & & \\ Russian & Institutional & 4
- Undergd & Formal & \\  & & & Casual & \\ Czech & Institutional & 4
- Undergd & & \\ Italian & Institutional & 4
- Undergd & & \\ Hindi & Institutional & 4
- Undergd & & \\ Bengali & Institutional & 3
- Rising Star & & \\ Marthi & Institutional & 2
- Hopeful & & \\ Sardinian & _Endangered_ & 1
- Scraping by & & Logudores (src) \\ Sinhala & Institutional & 0
- Lef Behind & Formal & Spoken form \\ \hline
* Winners & & \\ \hline
* Hopeful & & \\ \hline
* Scraping by & North Variety (azj) \\ \hline \hline \end{tabular}
\end{table}
Table 5: The details of languages used in the prompt generation for our VQA dataset. \({}^{\dagger}\)Taken from Ethnologue (Campbell and Grondona, 2008). \({}^{\ddagger}\)Based on Joshi et al. (2020).
 



\begin{table}
\begin{tabular}{l l|c c|c} \hline


danese language, assists with translation. The other, in the 26-35 age range with 25 years of experience with the language, primarily serves as the proofreader.

JavanesOne native Javanese speaker with Central Java dialect in the 16-25 age range translates for both registers of the language Krama and Ngoko).

a.4.2 Japonic JapaneseThree L2 Japanese speakers with over 10 years of language study contribute to the Japanese translation. Two are in the 26-35 age range, and one is in the 36-45 age range. A native Japanese speaker then proofreads the translated sentences. Additionally, one native Japanese speaker from Western Japan in the 16-25 age range gives input for the casual form.

#### a.4.3 Sino-Tibetan Chinese

ChineseOne native Chinese speaker in the 16-25 age range is involved as a translator.

CantoneseTwo native Cantonese speakers are involved as translators. One is in 36-45 age range, and the other is in the 16-25 age range.

HokkienTwo native Hokkien speakers in the Medan dialect translate for both written and spoken versions of the language. Both are in the 26-35 age range.

#### a.4.4 Koreanic

KoreanOne native Korean speaker in the 16-25 age range translates the formal and casual versions of the language.

#### a.4.5 Kra-Dai

ThaiOne native Thai speaker in the 26-35 age range is involved as a translator.

a.4.6 Indo-European EnglishQuery and context in English are constructed. All are L2 English speakers with over 20 years of study and have lived in the English speaking countries. Four of the annotators are in the 26-35 age range, and one is in 36-45 age range. Two native English speakers skimed through the prompt templates.

SpanishOne native Spanish speaker in the 26-35 age range translates the Latin-American versions or dialects of the language.

FrenchOne native French speaker and one L2 speaker are involved as translators. The native speaker is in the 26-35 age range, and the L2 speaker is in the 36-45 age range.

RussianOne native Russian speaker in the 26-35 age range is involved as translators. One L2 speaker in 36-45 proofreads the template for inflection.

Figure 8: Model performance evaluated with different references on open-ended question.

Figure 7: BERTScore (%) categorized by language **(left)**, language vitality **(center)**, and language family **(right)**. We classify the language vitality by following the classification from Joshi et al. (2020).

 

CzechOne native Czech speaker in the 36-45 age range is involved as a translator.

ItalianTwo native Italian speakers, both in the 36-45 age range, are involved as translators.

HindiOne native Hindi speaker in the 26-35 age range is involved as a translator.

BengaliOne native Bengali speaker in the 26-35 age range is involved as a translator.

MarathiOne native Marathi speaker in the 26-35 age range is involved as a translator.

SardinianOne native Logudorese Sardinian speaker in the 36-45 age range is involved as a translator.

SinhalaOne native Sinhala speaker in the 26-35 age range is involved as a translator.

#### A.4.7 Afro-Asiatic Arabic (MSA)

One native Arabic speaker in the 26-35 age range is involved in the Modern Standard Arabic (MSA) translation.

#### A.4.8 Niger-Congo

YorubaOne native Yoruba speaker in the 16-25 age range is involved as a translator.

#### A.4.9 Turkic

AzerbaijaniOne native Azerbaijan speaker in the 16-25 age range is involved as a translator.

## Appendix B Open-Source Collaborative Effort

The WorldCuisines data collection and benchmark construction is a fully open-source project. We invite contributions from researchers, practitioners, and grassroots communities, such as local NLP communities, who are interested in participating. Contributions can include data collection, annotation, quality checks, and evaluation. To ensure high-quality data, we engage native speakers of local languages in the annotation process with strict quality control (QC). The contributors who provide substantial contribution are invited to have co-authorship on this paper. We follow the guidelines from ACL for authorship eligibility.6 Our goal is to develop a resource and benchmark that will have a meaningful impact on future research. To achieve this, we are dedicated to expanding language coverage and ensuring that contributions are as _inclusive and diverse_ as possible.

Footnote 6: The ACL guidelines can be found at [https://www.aclweb.org/adminwiki/index.php/Authorship_Changes_Policy_for_ACL_Conference_Papers](https://www.aclweb.org/adminwiki/index.php/Authorship_Changes_Policy_for_ACL_Conference_Papers).

## Appendix C Detailed Dataset Construction

### Dataset Compilation

Our dataset, comprising 2,414 dishes and 6,084 images, was meticulously compiled and verified manually. Key metadata includes dish name, alias, coarse- and fine-grained categories, cuisines, regions, descriptions, images, and their licenses. The compilation process followed these steps:

* We listed dish names for annotation.
* Annotators filled metadata fields and selected up to 8 licensed images per dish, guided by instruction documentation with examples for consistent accuracy.
* Post-annotation, annotators formed subgroups to verify specific metadata categories, ensuring detailed and consistent data across fields such as categories, cuisines, regions, descriptions, and images.

### Negative Sampling

Recall that from our annotations, we have detailed metadata for all 2,414 dishes, including the dish name, coarse-grained categories, fine-grained categories, countries, and text descriptions. The negative answers were sampled using the following procedure:

1. We used a multilingual model, specifically E5-LARGE Instruct, to compute the text embeddings. Each embedding was generated by concatenating the dish name with its corresponding text description.
2. To identify negative samples, we computed the cosine similarity between the embeddings of the target dish and those of all other dishes in the dataset. The top-K most similar dishes were selected under three different conditions: * Same Fine-grained Category: Select top-K dishes from the same fine-grained category as the target dish. * Same Coarse-grained Category: Select top-K dishes from the same coarse-grained category but potentially different fine-grained categories 

* No Category Restriction: Select top-K dishes from the entire dataset without any restriction on categories.

Here, we used K=15, resulting in 45 candidate dishes in total.
3. Each MCQ consists of five options: one correct answer and four negative answers. The negative answers were chosen as follows: * Two Difficult Options: The first two negative answers were selected from dishes in the same fine-grained category. These are intended to be more challenging for the model to distinguish. * One Medium Option: The third negative answer was selected from dishes in the same coarse-grained category. * One Easy Option: The fourth negative answer was selected from dishes without any category restriction, making it likely to be easier to identify as incorrect. This approach ensures a balanced difficulty among the negative options, with two difficult, one medium, and one easy negative answer.
4. _(Optional)_ Specifically for task 2, where the question involves identifying the correct location (country) of a dish, we followed a slightly modified approach: * From the previously retrieved 4 negative options, we identified the countries associated with each dish. * We then excluded the countries that are valid locations for the correct dish. The

Figure 9: Regression Analysis for BERTScore OE vs. Accuracy OE.

 

remaining countries were used to create the negative options for the location-based question.

## Appendix D More Results

### Primary Metric: Accuracy (%)

Table 7 presents the comprehensive results of WC-VQA for both Test Small and Test Large. Additionally, we examine the performance gap between different references used in the evaluation, with the results displayed in Figure 8.

### Secondary Metric: BERTScore

As a secondary metric, we employ BERTScore using XLM-R Large as the base model. Table 8 presents the comprehensive results of WC-VQA for both Test Small and Test Large. Figure 7 illustrates the model's performance categorized by language, language vitality, and language family.

Robustness and Error Analysis.Figure 9 illustrates the correlation between BERTScore and accuracy in the open-ended setting through regression analysis. The R-squared value is 0.41, indicating a low correlation between BERTScore and accuracy. Despite this, BERTScore remains a useful metric for assessing whether the model's predictions have semantic similarity to the gold labels, even if they are not exact matches.

## Appendix E Evaluation

### Prompt Sensitivity

We use the same prompts for all models, with the exception of the Pangea 7B model Yue et al. (2024). This model is particularly sensitive and lacks robustness in handling diverse prompt instructions, often struggling to follow instructions accurately, especially in multiple-choice questions (MCQs), unless a specific template is applied. In contrast, models like Llama 3.2 Instruct and Qwen2 VL Instruct are more adaptable to varied instructions. After consulting with the authors, we adopted the prompt "Answer with the option letter from the given choices directly." for MCQ queries when using the Pangea 7B model 



\begin{table}
\begin{tabular}{l c r r r r r|r r r r|r r} \hline \hline  & \multicolumn{6}{c|}{**Task 1 (Dish Name)**} & \multicolumn{3}{c|}{**Task 2**} & \multicolumn{3}{c}{**Average**} \\
**Model (Accuracy \%)** & \multicolumn{3}{c|}{**(a) no-context**} & \multicolumn{3}{c|}{**(b) contextualized**} & \multicolumn{3}{c|}{**(c) adversarial**} & \multicolumn{3}{c|}{**(Location)**} & \multicolumn{3}{c}{**Average**} \\  & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ & MCQ & OEQ \\ \hline
**Test Small (12k)** & & & & & & & & & & & \\ \hline
**Open-Source** & & & & & & & & & & & \\ Llavan.16 Vicuna 7B & 33.63 & 0.87 & 43.13 & 2.83 & 28.67 & 0.60 & 27.77 & 7.93 & 33.30 & 3.06 \\ Llavai.6 Vicuna 13B & 40.87 & 1.00 & 50.30 & 4.17 & 38.37 & 1.60 & 31.07 & 8.63 & 40.15 & 3.85 \\ Qwen2 VL Instruct 2B & 40.97 & 3.33 & 44.40 & 4.60 & 47.07 & 3.43 & 48.37 & 12.50 & 45.20 & 5.96 \\ Qwen2 VL Instruct 7B & 63.83 & 4.07 & 67.20 & 8.57 & 57.00 & 3.90 & 56.80 & 21.23 & 61.21 & 9.44 \\ Qwen2 VL Instruct 72B & 76.13 & 10.40 & 81.63 & 17.43 & 67.23 & 6.27 & 56.73 & 26.07 & 70.43 & 15.04 \\ LIama 3.2 Instruct 11B & 57.93 & 14.37 & 65.57 & 19.20 & 56.27 & 9.50 & 46.60 & 27.23 & 56.59 & 17.58 \\ Llama 3.2 Instruct 90B & 77.33 & 14.27 & 83.43 & 22.30 & 71.23 & 9.00 & 64.70 & 29.73 & 74.17 & 18.82 \\ Molmo-E IB & 21.87 & 0.00 & 24.53 & 0.13 & 20.23 & 0.00 & 19.60 & 1.27 & 21.56 & 0.35 \\ Molmo-D TB & 50.67 & 1.00 & 57.00 & 2.23 & 48.67 & 1.73 & 36.73 & 11.70 & 48.27 & 4.16 \\ Molmo-O 7B & 46.03 & 2.13 & 43.27 & 4.37 & 41.60 & 2.10 & 26.83 & 9.03 & 39.43 & 4.41 \\ Pangea 7B & 45.33 & 0.43 & 59.40 & 1.33 & 22.17 & 0.63 & 34.10 & 17.90 & 40.25 & 5.07 \\ Pangea 7B\({}^{\ddagger}\) & 54.87 & 0.43 & 65.77 & 1.33 & 55.00 & 0.63 & 48.47 & 17.90 & 56.03 & 5.07 \\ Aria 25B & 65.77 & 2.67 & 71.43 & 6.47 & 57.13 & 1.80 & 39.60 & 15.70 & 58.48 & 6.66 \\ Phi-3.5 Vision 4B & 49.27 & 1.90 & 53.03 & 3.03 & 42.90 & 1.33 & 31.23 & 8.43 & 44.11 & 3.67 \\ Pixtral 12B & 57.57 & 0.60 & 72.33 & 1.83 & 55.40 & 0.57 & 44.73 & 12.83 & 57.51 & 3.96 \\ NVLM-D 72B & 75.50 & 3.13 & 78.20 & 7.37 & 54.67 & 1.37 & 54.13 & 17.40 & 65.62 & 7.32 \\ \hline
**Proprietary** & & & & & & & & & & \\ GPT-to & **88.40** & **16.60** & **90.43** & **35.47** & **82.23** & **12.60** & 63.60 & **35.53** & **81.17** & **25.05** \\ GPT-4o Mini & 75.33 & 7.30 & 83.00 & 17.67 & 64.83 & 3.53 & 52.87 & 26.90 & 69.01 & 13.85 \\ Gemini 1.5 Flash & 78.17 & 16.30 & 82.07 & 23.53 & 71.33 & 7.33 & **66.00** & 32.30 & 74.39 & 19.86 \\ \hline
**Test Large (60k)** & & & & & & & & & & \\ \hline
**Open-Source** & & & & & & & & & & \\ Llavan.16 Vicuna 7B & 34.57 & 1.59 & 43.48 & 4.03 & 34.84 & 1.41 & 32.24 & 9.29 & 36.28 & 4.08 \\ Llava1.6 Vicuna 13B & 40.17 & 2.79 & 48.17 & 5.85 & 39.05 & 2.57 & 37.79 & 10.16 & 41.30 & 5.34 \\ Owen2 VL Instruct 2B & 41.65 & 7.98 & 42.29 & 8.13 & 39.69 & 6.74 & 47.85 & 14.55 & 42.87 & 9.35 \\ Qwen2 VL Instruct 7B & 61.48 & 6.76 & 67.85 & 10.36 & 53.52 & 6.12 & 55.90 & 21.03 & 59.69 & 11.07 \\ Qwen2 VL Instruct 72B & 74.19 & 12.67 & 80.79 & 21.31 & 62.43 & 8.37 & 61.90 & 27.27 & 69.83 & 17.40 \\ Llama 3.2 Instruct 11B & 59.93 & 18.75 & 64.12 & 22.96 & 53.17 & 13.39 & 57.93 & 31.58 & 58.79 & 21.67 \\ LLama 3.2 Instruct 90B & _77.69_ & 16.93 & 82.92 & 23.60 & 63.96 & 10.87 & 67.87 & 31.31 & 73.11 & 20.68 \\ Molmo-E IB & 18.81 & 0.01 & 24.22 & 0.23 & 19.55 & 0.01 & 18.97 & 1.54 & 20.39 & 0.45 \\ Molmo-D T/B & 46.01 & 2.89 & 55.95 & 3.66 & 41.61 & 2.31 & 33.35 & 11.45 & 44.23 & 5.08 \\ Molmo-O YB & 39.96 & 5.15 & 44.93 & 6.03 & 38.41 & 3.51 & 29.81 & 10.07 & 38.28 & 6.19 \\ Pangea 7B & 41.38 & 1.52 & 57.95 & 2.73 & 21.77 & 1.57 & 37.15 & 20.15 & 39.56 & 6.49 \\ Pangea 7B\({}^{\ddagger}\) & 52.35 & 1.52 & 63.07 & 2.73 & 49.17 & 1.57 & 48.71 & 20.15 & 53.33 & 6.49 \\ Aria 25B & 58.61 & 4.99 & 69.29 & 9.17 & 52.82 & 3.39 & 42.82 & 16.20 & 55.89 & 8.44 \\ Phi-3.5 Vision 4B & 43.37 & 2.91 & 48.71 & 4.23 & 40.87 & 2.07 & 35.01 & 9.22 & 41.99 & 4.61 \\ Pickrat 12B & 56.65 & 1.22 & 70.69 & 2.94 & 52.12 & 1.09 & 46.67 & 14.43 & 56.53 & 4.92 \\ NVLM-D 72B & 69.82 & 4.71 & 78.93 & 10.29 & 52.12 & 2.89 & 51.9

Figure 10: Model performance with different references on open-ended question.

 



\begin{table}
\begin{tabular}{l c c|c|c|c} \hline \hline
**Model (BERTScore)** & \multicolumn{3}{c|}{**Task 1 (Dish Name)**} & \multicolumn{2}{c|}{**Task 2**} \\  & **(a) no-context** & **(b) contextualized** & **(c) adversarial** & **(Location)** & **Average** \\ \hline
**Test Small (12k)** & \multicolumn{4}{c}{} \\ \hline
**Open-Source** & \multirow{6}{*}{81.49} & \multirow{6}{*}{82.13} & \multirow{6}{*}{81.56} & \multirow{6}{*}{85.45} & \multirow{6}{*}{82.66} \\ Lava1.6 Vicuna 7B & & & & & \\ Lava1.6 Vicuna 13B & & & & & \\ Qwen2 VL Instruct 2B & & & & & \\ Qwen2 VL Instruct 7B & & & & & \\ Qwen2 VL Instruct 72B & & & & & \\ Qwen2 VL Instruct 72B & & & & & \\ Llama 3.2 Instruct 11B & & & & & \\ Llama 3.2 Instruct 90B & & & & & \\ Molmo-E 1B & & & & & \\ Molmo-D 7B & & & & & \\ Molmo-O 7B & & & & & \\ Molino-O 7B & & & & & \\ Pangea 7B & & & & & \\ Aria 25B & & & & & \\ Phi-3.5 Vision 4B & & & & & \\ Pixtral 12B & & & & & \\ NVLM-D 72B & & & & & \\ \hline
**Proprietary** & \multirow{6}{*}{81.29} & \multirow{6}{*}{81.78} & \multirow{6}{*}{80.19} & \multirow{6}{*}{86.31} \\ GPT-4o & & & & & \\ GPT-4o Mini & & & & & \\ Gemini 1.5 Flash & & & & & \\ \cline{2-2} \cline{



\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Continents/Regions** & **\# Countries** & **\# Food Entries** & **\% in Our Data** \\ \hline
**Global*** & N/A & **96** & **3.98\%** \\
**Africa** & **52** & **190** & **7.87\%** \\ Eastern Africa & 18 & 40 & 1.7\% \\ Middle Africa & 6 & 17 & 0.7\% \\ Northern Africa & 7 & 67 & 2.8\% \\ Southern Africa & 5 & 33 & 1.4\% \\ Western Africa & 16 & 60 & 2.5\% \\
**America** & **37** & **472** & **19.55\%** \\ Caribbean & 15 & 60 & 2.5\% \\ Central America & 8 & 134 & 5.6\% \\ Northern America & 2 & 230 & 9.5\% \\ South America & 12 & 109 & 4.5\% \\
**Europe** & **47** & **808** & **33.47\%** \\ Eastern Europe & 10 & 164 & 6.8\% \\ Northern Europe & 15 & 237 & 9.8\% \\ Southern Europe & 13 & 300 & 12.4\% \\ Western Europe & 9 & 233 & 9.7\% \\
**Asia** & **53** & **1,052** & **43.58\%** \\ Central Asia & 5 & 10 & 0.4\% \\ Eastern Asia & 9 & 420 & 17.4\% \\ South Eastern Asia & 12 & 362 & 15.0\% \\outhern Asia & 9 & 200 & 8.3\% \\ Western Asia & 18 & 155 & 6.4\% \\

- \\ Polynesia &
- \\ \hline \hline \end{tabular}
\end{table}
Table 9: Geographical distribution of WC-KB, corresponds to Figure 3. Note that there are food entries linked to multiple regions, with some linked to multiple continents. *_Global_ denotes entries with more than five regions.




\begin{table}
\begin{tabular}{l r l c l c l c l} \hline