Cultural Concept Adaptation on Multimodal Reasoning
Authors: Zhi Li
Yin Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Cross-cultural
Adaptation
Low-resource
Multi-modal
Data augmentation

Abstract: Developing cultural adaptation methods is important, which can improve the model performance on the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past methods primarily focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, we navigate this uncharted territory by leveraging high-resource cultures to facilitate comprehension of low-resource ones. We first introduce an annotation-free method for cultural-concept adaptation and construct a concept mapping set. To facilitate the model's comprehension of cultural-concept mappings, we propose a new multimodal data augmentation called CultureMixup. This approach employs a three-tier code-switching strategy on textual sentences. Additionally, it uses a cultural concept-based mixup method for the images. This combination effectively generates new data instances across culture, phrase, word, and image levels. For visually grounded reasoning across languages and cultures, experimental results on five languages show that our method consistently improves performance for four existing multilingual and multimodal models on both zero-shot and few-shot settings.

Cultural Concept Adaptation on Multimodal Reasoning
Zhi Li and Yin Zhang*
College of Computer Science and Technology, Zhejiang University, China (zhili, zhangyin98) juz edu. cn
Abstract
Molulingual
MiBERT (Dvin et 2019)
XLM M.MoaeCC at 2029)
Vilton (Laas al. 20.30000 2019)
Developing cultural adaptation methods is im- portant. which can improve the model perfor- mance the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past meth- ods primarily focused on multilingual and mul- timodal capabilities, and the improvement of multicultural competence is still unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, navigate this uncharted territory by leveraging high-resource cultures to facili- comprehension of low-resource ones. We first introduce an annotation-fin method for cultural-concept adaptation and construct con- cept mapping To facilitate the model's comprehension of cultural-concept concept mappings, we propose new multimodal data augmenta- tion called Mixuu This approach em- ploys three-tier code-switching strategy on textual sentences. Additionally, uses cul- tural concept-based mixup method for the im- ages. This combination effectively generates culture, phrase, word, and image levels. For visually grounded reason- ing across languages and cultures, experimental results five languages show that method consistently improves performance for four ex- isting multilingual and multimodal models on both zero-shot and few-shot settings.
Unicader (L 2029)
MP at., 202la)
UCC (ichar ol 2021)
CCLM (Zeng 2022)
Minhal
Table 1: The different settings between our method and previous work.
However, despite these advancements, the multicul- tural element is often neglected. The development of cultural adaptation methods is critical as they enhance model performance for low-resource lan- guages and democratize the benefits of advanced technology. Hershcovich al. (2022) underscores two major hurdles in cross-cultural NLP: cultural concepts and common sense. Our focus primar- ily on the former: models trained on high-resource languages and images struggle to comprehend low- resource cultural concepts.
A number of previous works (Vilares and Gómez-Rodríguee 2018; Acharya al., 2020; Yin al., 2021; Liu et al., 2021a; Cao et al., 2023) have delved into cultural topics, largely focusing on cul- tural differences or evaluating the cross-cultural competency of computational models instead enhancing them. The primary reason is the com- plexity of improving cross-cultura abilitics, as low- resource languages and their cultural concepts are inherently searce, exacerbating the data scarcity issue. Moreover, annotating cross-cultural data and establishing links between concepts across cultures an expensive process given the limited number of annotators well-versed in various countries' cul- tures.
data instances
1 Introduction
The computer vision (CV) and natural language processing (NLP) communities have witnessed sig- nificant strides in multilingual and multimodal re- search in recent years. For instance, XLMs (Con- neau and Lample, 2019) have substantially sur- passed the previous benchmarks in cross-lingnn cross-lingual tasks. The UC² model (Zhou et al., 2021) learns to represent input tokens context-aware man- ner by leveraging both linguistic and visual content.
To overcome this challenge, we initially propose an annotation-free method for cultural-coneept adaptation, which constructs concept mapping set. An instance of cultural-concept adaptation involves the Chinese concept Erhu ', which has no corresponding English translation. Explaining
*Corresponding author: Yin Zhang.

---

it to English-speaking individuals unfamiliar with Chinese culture would likely involve likening the Erhu to a Chinese violin. This an example of cultural adaptation. Leveraging the relationships of hypernyms, hyponyms, and synonyms from pub- licly accessible semantic dictionaries, our method maps source cultural concepts to their correspond- ing target concepts, thereby eliminating the need for costly manual annotation.
2 Related Work
2.1 Cultural Research
Human language and visual content are intimately entwined with their respective cultures, evolving together, mirroring, and reciprocally influencing them. Culture is typically tied to specific geo- graphic region or locality, with distinct cultures characterizing different countries. The extant lit- crature on culture tends to concentrate on three key aspects: examining cultural differences or similarities (Vilares and RodriegzZ 2018; Acharya et al., 2020; Kozlowski et al., 2018; Sun et al., 2020), developing cross-cultural benchmarks (Peskov et al., 2021; Yin al., 2021; Liu et al., A021a), and evaluating the cross-cultural compe- tence of computational models (Nguyen al., 2022: Arora et al., 2022; Cao et al., 2023). For instance, Liu et al. (2021a) outline the multifaceted challenges involved in reasoning visually across languages and cultures, encompassing cross-modal, cross-lingual, and cross-cultural aspects. In con- trast to the majority of prior research focusing predominantly analysis and evaluation, work confronts the issue directly by enhancing the model's adaptability to low-resource cultural con- cepts from both a visual and textual standpoint.
To support the model's understanding of cultural- concept mappings, we subsequently introduce a novel cultural concept-based multimodal data aug- mentation technique. This technique features a three-tier code-switching strategy on textual sen- tences and cultural concept-based mixup method for images 2. By training the model on both orig- inal and augmented data, manage to signifi- cantly boost the model's 's performance on visually grounded reasoning tasks across languages and cul- tures. This improvement is reflected by a mini- mum increase of 2 points over existing multilingual multimodal models. Furthermore, our method can be adapted to improve specific languages or cul- tural topics by modifying the sampling distribution, thereby mitigating model bias.
Our contributions are encapsulated as follows:
Leveraging web resources, we propose an annotation-free cultural adaptation method. By utilizing relationships hypernyms, hy- ponyms, and synonyms from openly accessi- ble semantic dictionaries, we construct a cul- tural adaptation graph that facilitates mapping between source and target cultural concepts.
2.2 Code-switching and Mixup Methods
Code-switching is a common occurrence in mul- tilingual communities, wherein the lexicon and morphemes of two or more languages are inter- changeably utilized in oral written communica- tion. Training models using code-switched data encourages the alignment of source and target lan- guage representations by blending their contextual information. This approach has been used to chal- lenge multilingual models (Tan and Joty, 2021), 1), enhance Neural Machine Translation (NMT) tasks (Yang et al., 2020a; Liu et al., 202 b;; Yang et al., 2020b), and further cross-lingual tasks (Qin et al.. 2020; Zhang al., 2021; Lee et al., 2021). In this study, broaden the conventional perception of code-switching, transitioning from a solely linguistic phenomenon to a cultural one.
To combat data scarcity and foster the model's understanding of cultural adaptation map- pings, introduce novel cultural concept- based multimodal data augmentation tech- nique, generating new data instances at the concept, phrase, word, and image levels.
Key results for the task of visually grounded reasoning across languages and cultures reveal that our methods consistently and significantly outperform baseline measures. Additionally, our technique can be tailored to enhance spe- cific languages or cultural topics by adjust- ing the sampling distribution, thus reducing model bias.
While code-switching operates on sentences. mixup methods are utilized in variety of con-
"Code-switching is widespread phenomenon multi- lingual communities. characterized by switching words and morphemes from two or more languages in speech or writing. The switched elements usually hear semantic similarity to the originals.
!=#l (Erhu) is two-stringed howed musical instrument, which may also he called Southern Fiddle and is sometimes known in the Western world as the Chinese violin.

---

hyperym
hyponyme
synonym
Chinese
Benal strin: Intrnnnn
English
Collo
Saxophons
-
Wind Instriment
Culuma Adeptation
Obce
Musical Inserneen
Drum
Vait, Drmm Piano
Plana
Figure 1: Consider the cultural adaptation of the term (Erruu, . translates "Erhu" in its phonetic English form, but lacks direct English equivalent. How might this be explained to someone living in an English-speaking country, unfamiliar with Chinese culture? You might refer to Erhu as the "Chinese violin," " as form of cultural adaptation. We advocate comprehensive method for cultural adaptation, utilizing relationships of hypernyms, hyponyms, and synonyms from open and freely-availabe semantic dictionaries to construct a cultural adaptation graph. Each leaf node within this graph could represent potential cultural adaptation of [Br(urr The shorter the path distance in the graph, the more precise the adaptation. For instance, "violin" and "cello" are better adaptations than "saxophone" and "drum".
texts, such as mixup (Zhang et al., 2017), cutmix (Yun et al., 2019), attentive cutmix (Walawalkar al., 2020), and alignmixup (Venkataramanan al., 2022). Hao al. (2023) introduced joint data augmentation method, which generates new image-text pairs while maintaining semantic co- herence through image interpolation and text con- catenation. In contrast to these approaches, we substitute the target portion of the image with one that corresponds to low-resource cultural concept.
cultural-concptt adaptation, which constructs a concept mapping set. To support the model's un- derstanding of cultural-coneey mappings, we sub- sequently introduce novel cultural concept-based multimodal data augmentation technique. By train- ing the model on both original and augmented data, we significantly boost the model's 's performance.
3.1 Cultural Adaptation
To illuminate the research methodology, we now formally define the central components:
2.3 Multilingual Multimodal Models
Definition 1 (Cultural Adaptation Set)
Most multimodal models based on transformer ar- chitecture (Ni et al., 2021b; Zhou et al., 2021; Jain et al., 2021; Liu et al., 2021a; Khan et al., 2021: Song et al., 2021; Zeng et al., 2022) pre-trained using self-supervised objectives. While these methodologies advance the multilingual and multimodal capabilities of models, they often over- look cross-cultraa aptitude. We propose a cultural concept adaptation upproach to improve model per- formance across different cultures. By extending the code-switching mechanism to cultural concepts during fine-tuning, our method can help mitigate biases towards language and culture that may arise from imbalanced pre-training resource distribution, an issue that is challenging address using self- supervised pre- pre-training objectives alone.
Consider a low-resource cultural concept x and high-resource cultural concepts set Y {yi,.... Ynt. The cultural adaptation set of Y2 {.I....., 9ma is subset of Y such that each E Yz shares similar category attributes' 2.
Problem Statement Given x C X, where X is set of cultural concepts, how can we identify the cultural adaptation set Yg?
We denote Ho. S, and He functions query hyponyms, synonyms, and hypernyms respectively from publicly accessible semantic networks such Conceptnet (Speer al., 2017) and Wordnet (Miller, 1995). We construct a cultural adaptation graph G2 by applying a com- posite function F(x), which is defined as follows:
Proposed Solution
3 Method
F(x) = Ho(. (S. (H+()))))) (1)
To overcome data scarcity and costly labeling, we initially propose an annotation-fie method for
For example, the 'violin" and "erhu" are hoth classified under the category of orchestral instruments.

---

Cultural Concepts
Retrieval
Viulin
The image on the left shows vialin player and a flute player sitting side by side.
Dabavet
Calture
Rersbin:
Yigi glvps
The image on the left shows
08
player and flute player sitting side by side.
Qumisasi
Sumpling
level
Samplagg
The image kushoto shows
player and flute player sitting side by side.
Cahtza Adagaiion
Violin,
Word level
A
Drum,
Piano
resim leushoto shows
player flute peserta sitting side by side.
Figure 2: A schematic of three-tier code-switching strategy the textual sentences. We commence by collecting data from web resources and forming set of cultural concepts, each represented by oval of a distinct color. Following this, a cultural concept is sampled for adaptation, based on a probability distribution. This selected adaptation is used retrieve a corresponding data instance from the training dataset. Subsequently, we apply three levels of code-switching to the retrieved sentences. In these code-switched sentences, words highlighted in yellow, green, blue, and purple denote languages Chinese, Swahili, Turkish, and Indonesian, respectively.
Definition 2 (Cultural Adaptation Graph)
cultural adaptation sets can serve as a kind of multi- cultural resource.
A cultural adaptation graph G2 = (V, F)). where V denotes the set of vertices and E the set of edges, is directed graph created by the iterative application of the Hypon, Syn, and Hyper functions starting from the cultural concept x. Each vertex 0; V represents cultural concept linked to through a path of hyponym, synonym, or hypernym relationships, while each directed edge Eij E E depicts one of these relationships {Ho.S, He) be- (ween the cultural concepts and Vig.
3.2 Culture Mixup
To mitigate the issue of data scarcity and prompt the model to comprehend the mapping of cultural adap- tation, we propose cultural concept-based mul- timodal data augmentation named CultureMixup. It includes three-tier code-switching strategy on the textual sentences and cultural concept-based mixup method on the image and thereby generates new data instances from the culture, phrase, word, and image levels. We initially gather data from online resources to construct a set X of cultural concepts, each paired with corresponding images. Refer to the appendix A for further details. Sub- sequently, we select cultural concept E X for cultural adaptation according to a predefined prob- ability distribution. The sampling of an adaptation 3/k Y, from the high-resource cultural adapta- tion set is inversely proportional to its path distance in the cultural adaptation graph G₂ (as defined Equation 1), i.e., P( (r) x T The adapted concept is then used to retrieve data instances from the training dataset. Each data instance com- prises both textual sentences and images.
Given a cultural adaptation graph G2 = (V, each leaf node v in G, potentially represents cultural adaptation of the cultural concept æ. An example of this process is illustrated in Figure 1, showing the cultural adaptation (Erhu) The 'Erhu' has a hypernym "3 l gz " synony- mous in English with "Bowed string instrument". This term has various hyponyms, including the 'vi- olin' and 'cello', making them potential cultural adaptations of the 'Eru'. Similarly, "#"" the hy- and by extension second- order hypernym of 'Erhu'. translates to "Musical Instrument" English. Thus, other instruments like the 'saxophone' and 'oboe' may also serve as potential cultural adaptations of the 'Erhu'. In fact, every leaf node in the cultural adaptation graph could potentially represent a cultural adaptation of 'Erhu', where shorter path distances indicate higher accuracy. For instance, 'violin' and 'cello" would provide more accurate cultural adaptations than 'saxophone' and 'drum'. A simple iterative traversal algorithm can yield all leaf nodes and their respective path distances from 'Erhu". These
pernym of " = 32 oh
3.2.1 Three-tier Code-switching on Text
We apply a three-tier code-switching strategy to the textual sentences, as displayed in Figure 2.
3 PZ कT. is bowed string instruments in English, which are subcategory of string instruments that are played by bow rubbing the strings. The bow rubbing the string causes vibration which the instrument cmits as sound.

---

Cultural-level We extend the conventional derstanding of code-switching from a purely lin- guistic phenomenon
applied seamlessly integrate the new ob- ject into the original image and make more compatible with the background.
cultural one. The cultur- ally adapted concept yk in the sentence is replaced by the original low-resource cultural concept x. For instance, in Figure 2, the word "violin" is code- switched with its Chinese cultural adaptation "_ "t!".
This introduces cultural diversity at the visual level, thereby enriching the data instances for train- ing.
3.3 Reduce Model Bias
Phrase-level For the phrase-level code- switching, we collect and identify higu-fryy phrases pertinent to reasoning and translate them into different languages. If a sentence contains these selected phrases, they are code-switched based on given probability. For example, in Figure 2, the phrase "on the left" is code-switched by "kushoto" in Swahili.
Pretrained multilingual multimodal models often demonstrate disparate performance across various languages and cultural contexts in test datasets, a discrepancy likely attributable to uneven resource distribution during pretraining. These imbalances pose challenges that are not readily addressed by sulf-super pretraining objectives. Our method offers a mechanism to ameliorate these language and cultural biases by manipulating the sampling distribution. Essentially, can enhance model performance on specific language cultural top- a controllable manner. For instance, if the model anticipated to be applied a Turkish con- text, the sampling probability for Turkish can be increased during data augmentation. Likewise, if the model will be deployed in scenarios involving traditional Chinese musical instruments, like the Erhu, we can elevate the sampling probability of these specific Chinese musical concepts. In sum- mary, our approach provides statistically signifi- cant, fine-grained performance boost for the model over predefined language or cultural categories.
Word-level At the word level, each word sentence undergoes code-switching in a different language based on specified probability. For ex- ample, in Figure 2, the words "image" and "player" are code-switched by "resim" in Turkish and "pe- serta" in Indonesian, respectively.
3.2.2 Concept-Based Mixup for Images
For the image-level cultural adaptation, first ap- ply an object detection algorithm the images. We identify visual elements that correspond high- resource cultural concepts yk, and replace them with visual elements associated with the original low-resource cultural concept æ. The specific pro- cess is as follows:
4
Experiment
1. Visual Element Detection: We use the high- resource concept as the vocabulary term and deploy the object detection model to locate the corresponding visual elements in the image.
4.1
Dataset
We collect and evaluate cultural concepts from di- verse cultures (Indonesian, Swahili, Tamil, Turkish, Chinese) using crowd-sourced workers. The pro- involves gathering wide range of cultural concepts manually and validating them through a majority vote. For detailed methodology, please refer to the appendix. This approach, while cost- effective, emphasizes accuracy by requiring a sig- nificant consensus among evaluators and prioritiz- ing manual collection to capture cultural nuances. This dataset is publicly available 5. In evaluat- ing the resulting cultural adaptation graphs, about 84% aligned with human judgment, confirming the method's effectiveness. For the less accurate graphs, issues were primarily due to translation lim- itations for certain cultural concepts' hypernyss.
2. Bounding Box Extraction: Based on the ob- ject detection results, we extract the object that needs to be replaced from the original high-resource image using bounding boxes.
3. Resizing: We adjust the size of the low- resource image element, which is collected together with concepts themselves, to match the size of the bounding box.
4. Pasting: The resized new low-resone lowrresource image is then pasted into the corresponding location of the original image.
5. Smoothing and Harmonization: Finally, smoothing and harmonization techniques are

---

Proportion
NLVR2
MaRVL
Model
::yy
EN
ZH
TA
SW
ID
TR
avg.
1:1
69.23
54.85
53.44
53.01
53.43
57.2
54.79
2:1
69.74
56.99
55.81
53.26
54.45
56.67
53.44
mh MUNITER
3:1
70.45
57.33
56.02
53.97
55.14
58.67
56.23
4:1
70.82
58.04
55.57
53.61
54.92
57.99
56.03
5:1
71.24
56.88
55.41
52.99
53.24
56.98
55.10
3:1
70.45
57.33
56.02
53.97
55.14
58.67
56.23
MUNITER
9:3
71.22
58.47
56.11
54.19
55.24
58.91
56.58
15:5
71.56
59.71
58.45
54.35
55.24
58.81
57.31
Table 2: Different proportion model performance. y means that the total dataset is y times larger than the origin English training data. And ズサツ of the total dataset is the origin English training data (just copy origin dataset x times) and _ of the total dataset is augmented data. The results cover five languages: Indonesian (ID), Swahili (SW), Tamil (TA), Turkish (TR) and Mandarin Chinese (7.H).
Zero Shot
NIVRE EN
MoRVI.
Translate: Test
Model
Method
1A
10
MaRVLELu dt 20216)
73.2
50.3
32.2
31.5
35.0
54.7
34.0
63.3
MUNITER IGLUE (Bigharelle
71,91
55.34
52,66
51,17
$4.79
54.66
53.72
63.82
Our Reprodace
70.93
$4.05
53.14
51,72
55.41
53.20
53.51
Medhod
71,56
60.44 (59.71)
58.92(58.45)
54.87(54335)
55.72(55,24)
59.53188881
37.99)57.31)
dann 2011a)
72.8
55.1
57.1
>UNTTRE
KHII (Hoghamilinarll
71,55
$3.06
$3.06
55.51
55.14
$6.19
$4.59
64.04
Our Bapeoduas
Oar Mesteal
70.34
$7.91 (56.78)
$8.0557.888)
60.01(59.06)
58,
58.78158.21)
59.67159.44)
58.88/58.27)
UCV
(Boglarllo e
60:47
56.74
Out Repoodacc
69.05
58.1
60.06
$2.97
57.18
56.102
$7.07
(59.73)
(1.61660.88)
58.55577977
(5(875)
60.72(60034)
59.98)59.333
ICLUEE (Reghanettee 2007)
68.22
$5.04
$6.04
55.69
56.47
56.78
56.00
62.52
Reprodace
Dur Mtateal
68.10
59.21(58733)
(0.89159.96
61.23(60.6
59.18(58.99)
61.31160.65)
60.36(59.79)
Table 3: Zero-shot statistics. The best scores are in bold. For our method, we not only report the best performance but also report statistical mean. The digits in "O" represent the average value different random seeds. The results cover five languages: Indonesian (ID), Swahili (SW). Tamil (TA), Turkish (TR) and Mandarin Chinese (ZH).
Strategies like restricting the use of higher-order hypernyms and implementing an exponential decay of sampling probability for concept inclusion were employed to enhance accuracy and authenticity, en- suring the graphs' overall quality and reliability. See the appendix B for details.
2019). not only spans five typologically diverse languages (Chinese, Tamil, Swahili, Indonesian, and Turkish) but also adopts different basic-level concepts across different cultures. Thus, the chal- lenges are multi-faceted, including cross-modal, cross-lingual and cross-cultura aspects. In this task, given two images (Teeft and Fright) and de- scription D. the model needs to assess the validity of the description given the images, which can be cast as a classification problem. See the appendix C for sampled examples and detailed descriptions.
We use the Detic model (Zhou et al., 2022) avail- able at Facebook Research's GitHub repository for object detection. It employs the CLIP (Radford et al., 2021) classifier and offers open opea-vocyyulary capabilities. With this model, ure the vocabulary, allowing us detect specific cultural concepts within images. you want more precise result, recommended to use the segment anything model(Kirillor et al., 2023), but it may require some manual clicking.
freely config-
4.2 Baseline
We assess the efficacy of our approach using four existing multilingual multimodal pretrained models that have been relcased so far:
Following Liu et al. (2021a), we employ NLVR2 in English for training and MaRVI. for testing. MaRVL (Liu et al., 2021a) is a visually grounded reasoning dataset similar to NLVR2 (Suhr et al.,
mUNITER (Liu al., 2021a), which extends the UNITER architecture (Chen et al., 2020) to support multilingual functionality.

---

Shot
2
4
10
10x2
ZH 59.41/56.52
59.23/54.84
60.18/54.55
60.60/55.34
61.63/54.05
MUNITER
TR
59.66/52.54
59.24/55.93
59.73/55.59
60.48/54.66
60.97/54.75
ID
55.50/51.42
55.88/52.13
56.34/54/34
56.71/56.83
56.97/51.42
ZH
57.02/54.45
56.72/53.66
57.63/54.94
57.66/53.44 46
57.82/55.73
EUNITER
TR
59.19/55.93
59.58/57.46
59,48/57.54
59.86/57.8 80
60.60/58 05
ID
58.60/56.12 59.48/57.18
59.55/58.87
60.51/58.87
60.75/58.60
ZH 59.34/58.99
59.40/57.00
59.29/58.99
59.99/57.21
60.91/60.18
UC
TR
59.64/55.76
59.92/54.39
60.71/52.63
61.15/55.93
61.53/56.27
ID 59.33/56.12
59.49/56.29
59.52/57.53
60.11/57.62
60.89/58.51
ZH
58.92/52.47
59.32/53.85
59.71/52.47
60.03/54.45
60. 12/54.44
M
TR
59.90/57.88
59.58/58.98
59.81/57.97
60.59/58.39
61.19/57.54
ID
58.69/57.36
59.40/56.92
59.61/57.27
59.57/58.51
59.91/58.51
Table 4: Different shots statistics. Results of Our methods/baseline. The baseline results come from IGLUE
(Bugliarello al., 2022). The results three languages: Indonesian (ID), Turkish (TR), and Mandarin Chinese (ZH).
+UNITER (Liu et al., 2021a), which is identi- cal m UNITER, except for its initialization: while m UNITER initialized from mBERT, xUNITER originates from XLM-R.
4.3.1 Proportion
We conduct two groups of experiments on MUNITER in a zero-shot setting: one examines the impact of the proportion of augmented data, while the other investigates the effect of the total volume of augmented data model performance while keeping the proportion constant. In Table 2, y implies that the total dataset is + y times larger than the original English training data. Furthermore, z the total dataset the orig- inal English training data (essentially replicating the original dataset æ times), and _ is the code- switched generated data. To maintain the total num- ber of training steps, we reduce the training epochs and train the models for 20 æ+y epochs.
M (Ni et al., 2021b), which introduces pretraining tasks using multimodal code- switching.
UC (Zhou al., 2021), which uses masked region- region-to-toker to-keenn modeling and visual transla- tion language modeling as pretraining tasks.
To provide fair comparison with baselines (Liu al., 2021a; Bugliarello al., 2022), we adopt nearly identical experimental setups and hyperpa- rameters except that we finetune models on the origin and augmented NLVR2 (Suhr et al., 2019) dataset. Despite augmenting the dataset, we main- tain the same total number of training steps by reducing the training epochs. For more detailed information about settings and the implementation of the model, please refer the appendix D. Our code is based on VOLTA (Bugliarello al., 2020).
For the first group of experiments, establish five different ratios y=1 1, 2 1, 1, 4 1. Results in Table 2 suggest that the volume of original data increases, the model's performance on the English test set consistently improves. How- ever, performance other languages and cultures initially increases then decreases. This reveals a trade-off: while substantial proportion of English data aids the model's task comprehension, propriate amount of augmented data from other cultures facilitates the model's transfer ability. A ratio of roughly 3:1 yields optimal results. We further investigate this by holding the scale con- stant and incrementally increasing the English and augmented datasets.
ap-
4.3 Results
In this part, mainly discuss four parts with ex- perimental results. (1) What proportion is appropri- ate for the augmented data? (2) The zero-shot and few-shot performance of models with our proposed m3thods.(3) Ablation studies. (4) Controllability and reduce model bias.
In order to examine the influence of the total volume of augmented data on model performance

---

Ablation
NLVR2
MaRVL
Model
Method
EN
ZH
TA
SW
ID
TR
avg.
Baseline
70.93
54.05
53.14
51.72
55.41
53.22
53.51
Our Method
71.56
59.71
58.45
54.35
55.24
58.81
57.31
All w/o Concept
71.18
58.87
57.3
53.82
54.61
57.52
56.42
in MUNITER
All w/o Phrase
70.71
57.95
56.94
53.18
54.02
57.19
55.86
All w/o Word
70.65
58.69
57.22
53.55
54.38
57.55
56.28
All w/o Image
70.85
58.14
56.81
52.95
54.69
56.33
55.78
Table 5: Ablation studies across five languages.
while keeping the proportion constant, we establish three different ratios y=3 : 1, 3, and 15 5. As indicated in Table 2, the performance of the model improves with an increase in the volume of augmented data. Taking into account the results from these two sets of experiments, we decide to amplify the dataset to twenty times the original size and choose a ratio of x y 15 for our subsequent zero-shot and few-shot experiments. Although we do not contend that the ratio of x y = 15 : is optimal, we assert that this choice is adequate to demonstrate the effectiveness of our approach.
method enhances the M and mUNIER scores by about ~ points, while UC and zUNITER gain only about ~ 3 points. This disparity might stem from the fact that UC and zUNITER have acquired better-aligned representations during the pre-training phase.
Table 4 displays the results of few-shot perfor- mance on three languages, demonstrating that our method also achieves the state-of-the-art perfor- mance the few-shot setting. Nevertheless, simi- lar to Liu et al. (2021a); Bugliarello et al. (2022), our results corroborate the finding that unlike text- only multilingual tasks, where even a handful of examples in the target language substantially enhance model performance, this phenomenon is largely absent in multimodal multilingual settings (Bugliarello et al., 2022). As the number of shots increases, the model's performance remains largely unchanged or shows slight growth. We attribute this to two main factors. Firstly, the inherent com- plexity of the task, and secondly, within the same language, data samples embodying diverse cultural concepts may vary significantly. The model may overfit to data samples associated with specific cul- tural concepts, a phenomenon that warrants further investigation future studies.
4.3.2 Zero-shot and Few-shot
As previously mentioned, we employ configura- tion of x twenty times its original size. This involves repli- cating the original English NLVR2 dataset (Suhr al., 2019) 15 times and generating augmented dataset five times larger. Consequently, the final dataset is 20 times the size of the original English NLVR2 dataset. To maintain a consistent number of training steps with Liu et al. (2021a); Bugliarello et al. (2022), who trained models for 20 epochs, we train our models on this expanded dataset for single epoch. We present both the best results and statistical mean for our method. 'Translate- test' refers to the setup in which the multilingual MaRVL datasets are translated into English. The re- sults of Liu et al. (2021a); Bugliarello et al. (2022) are used directly as zero-shot benchmarks.
15 5 to augment the dataset to
4.3.3 Ablation Study
To examine the impact of cultural concept- based multimodal data augmentation approach, we conducted an ablation study on MUNTER, main- taining the 15:5 setting for consistency, The results, as presented in Table represent the statistical mean of various random seeds and underscore that each component of our method significantly con- tributes to enhancing the model's performance.
Table 3 displays the zero-shot performance of the four models and demonstrates that our method consistently and statistically surpasses the basc- lines. Our approach considerably diminishes the disparity between the performance in the transla- tion test and the transfer performance, validating the effectiveness of our code-switching method. We observe that, compared with the baselines, our
4.3.4 Reduce Model Bias
Our method can be utilized to mitigate model bias on specific target languages or cultural topics by

---

gressive generation, such as GPT-4-V is also highly worthwhile to explore.
61
60
59.83
59
58.46
Limitations
58-
Accuracy
57
Our approach principally amplifies the conceptual adaptability of models to low-resource cultures. Nevertheless, cultural differences are complex and multidimensional. encompassing not only concep- tual elements but also elements of common sense. The comprehensive acquisition of such common sense across diverse cultures is vital yet challeng- ing endeavor. Therefore, our community still has a considerable path to tread in order to fully en- hance the multicultural competencies of Al models. Simultaneously, we only conducted experiments on multi-modal models based on masked language modeling. Further investigation is needed to de- termine the effectiveness multi-modal models based autoregressive generation.
56
55.04
55
54
52.97
53
52.62
52
51
Baseline
Ours
+5k
Figure U on Swahili language. "Baseline" rep- resents the score reported by Bugliarello et al. (2022) and "Ours" represents our replication of basline. "+Ik" /"+I0K" represent adding 1k/10k Swahili-English code- switched data to training dataset. "Best" represent the performance when we add more code-switched data examples.
adjusting the sampling distribution. For instance, most models exhibit relatively lower scores in Swahili compared to other languages, with UC2 being particularly affected. To address this, we specifically focus improving U perfor- mance in Swahili. By sampling, obtain 1 and 10k examples of "wahili-Englist code-switched data, which we then merge with the original En- glish dataset to create new training dataset. As Figure illustrates, our method can significantly enhance the performance of U in Swahili. We also enhance the model's performance on the topic "Speech and language" in Chinese. For further details, please refer to appendix D.
Acknowledgments
This work was supported by Zhejiang Provin- cial Natural Science Foundation of China un- der Grant No. L723FF00009, the NSFC project (No. 62072399), MoE Engincering Research Cen- ter of Digital Library, China Research Centre on Data and Knowledge for Engineering Sciences and Technology, and the Fundamental Research Funds for the Central Universities. We would like to ex- press our sincere gratitude the anonymous viewers for their invaluable feedback and construc- tive comments, which significantly contributed to the improvement of this paper.
5 Conclusion
References
To attack the difficulties of data annotation and scarcity, propose annotation-free cultural adaptation method and design novel cultural concept-based multi-modal data augmentation to generate the new data example. By training the model on the augmented dataset, key results indi- cate that our methods consistently and statistically outperform the baselines. In the future, we plan to apply our method to more downstream tasks related to culture. Employing curriculum lcarning and de- signing more refined training strategies according the difficulty of different languages and cultural concepts also worth exploring. the same time, how to further extend our method to make it more applicable to multi-mon modal models based on autore-
Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2020. Towards atlas of cultural com- monsense for machine reasoning. arXiv preprint arXiv:2009006664,
Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augen- stein. 2022. Probing pre- pre-trained language models for cross-cultural differences in values. arXiv preprint arXiv:2203. 13722.
Emanuele Bugliarello. Ryan Cotterell, Naoaki Okazaki. and Desmond Elliott. 2020. Multimodal pretrain- ing unmasked: A meta-analysis and unified frame- work of vision-and-language berts. arXiv preprint arXiv:2011.51124.

---

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic. 2022. IGLUE: A benchmark for trans- fer learning across modalities, tasks, and languages. CoRR, abs/2201.11732.
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian enror linear units (gelus). arXiv preprint arXiv: 1rXiv:6606.0841.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and An- ders Søgaard. 2022. Challenges and strategies in cross-cultural nip. ArXiv, 0bs/2203.10020
Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and hu- man societies: An empirical study. urXiv preprint arXiv:2303. 17466.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: universal image-text representation learning. In Computer Vision ECCVV 2020 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020. Proceedings, Part XXX. volume 12375 of Lecture Notes in Computer Science, pages 104-120. Springer.
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. 2021. MURAL: multimodal, multitask retrieval across languages. CoRR, 0tbs/2199051122
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? preliminary study. arXiv preprint arXiv:2301.08755
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pages 8440 8451, Association for Computational Linguistics.
Humair Raj Khan, Deepak Gupta, and Asif Ekbal. 2021. Towards developing a multilingual and code-mixed visual question answering system by knowledge dis- tillation. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November. 2021, pages 1753-1767. Association for Computa- tional Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. 2023. Segment anything. arXiv preprint 20X.v22304.02643.
Austin C. Kozlowski, Matt Taddy, and James A. Evans. 2018. The geometry of culture: Analyzing meaning through word embeddings. CoRR, Dos//803309920
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Dohyeon Lee, Jaeseong Lee, Gyewon Lee, Byung- Gon Chun, and Seung-won Hwang. 2021. SCOPA: soft code-switching and pairwise alignment for zero- shot cross-lingual transfer. In CIKM '1:: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pages 3176-3180. ACM.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLI 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages 4171-4186. Association for Computational Linguistics.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on A Artificial Intelligence, volume 34, pages 1133-111444
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju. Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 379-389.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond EI- liott. 2021a. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November. 2021. pages 10467-10485. Association for Computa- tional Linguistics.
Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu Sharath Chandra Guntuku, and Lyle Ungar. 2023. Multilingual language models are not multiculural:: case study in emotion. arXiv preprint arXiv:2307.013:: 01.007

---

Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021b. Continual mixed-lanee -angugge pre-training for extremely low-resource neural machine translation. In Findings of the Association for Computational Lin- guistiex: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/UCNLP 2021 of Findings of ACI., pages 2706-2718. Association for Computa- tional Linguistics.
Virtual Event, China, October 20 24, 2021, pages 2443-2852.45 ACM.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-Fire AAAI Conference on Artificial Intelligence, February 4-9, 2017. San Francisco, California, USA, pages 4444 4451. AAAI Press.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining lask-agnostic visiolinguis- tic representations for vision-and-languee tasks. Ad- vances in neural information processing systems, 32.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- une 1: Long Papers, pages 6418-6428. Association for Computational Linguistics.
George A Miller. 1995. Wordnet: lexical database for english. Communications of the ACM, 38(1 1):39441.
Tuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, and Gerhard Weikum, 2022. Extracting cul- tural commonsense knowledge at scale. Proceedings of the ACM Web Conference 2023.
Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R Mortensen. 2020. Cross- cultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. arXiv preprint arXiv:2006.09336
Minheng Ni, Haoyang Huang, Lin Su. Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. 2021a. M3p: Learning universal rep- resentations via multitask multilingual multimodal pre-training. In Proceedings of the IEFF/CVT Con- ference on Computer Vision and Pattern Recognition, pages 3977-3986.
Samson Tan and Shafq R. Joty. 2021. Code-mixing on sesame street: Dawn of the adversarial polyglots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3596 3616. Association for Computational Linguis- tics.
Minheng Ni. Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang. Dongdong Zhang, and Nan Duan. 2021b. M3P: learning universal repre- sentations via multitask multilingual multimodal pre- training. IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3977-3986. Computer Vision Foundation JEEE.
Shashanka Venkataramanan, Ewa Kijak, Laurent Am- saleg, and Yannis Avrithis. 2022. Alignmixup: Im- proving representations by interpolating aligned fea- tures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 19174-19183.
Denis Peskov, Viktor Hangya, Jordan L. Boyd-Graber, and Alexander Fraser. 2021. Adapting entities across languages and cultures. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16- 20 November, 2021, pages 3725 37500. Association for Computational Linguistics.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance. arXiv preprins 1.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi- Multi-lingual
David Vilares and Carlos Rodríguz. 2018. Grounding the semantics of pfrt-of-day nouns world- wide using Twitter. In Proceedings of the Second Workshop on Computational Modeling of People's 's Opinions, Personality, and Emotions in Social Me- dia, pages 123-128, New Orleans, Louisiana, USA. Association for Computational Linguistics.
switsshiithhng data augmentation for zero-shot cross-ling -oossll NLP. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3853-3860. ijcai.org.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. 2020. Attentive cutmix: An hanced data augmentation approach for deep learn- ing based image classification. arXiv preprint arXiv:2003.1304
Yuqing Song, Shizhe Chen, Qin Jin, Wei Luo, Jun Xie, and Fei Huang. 2021. Product-oriented ma- chine translation with cross-modal cross-lingual pre- training. In MM 21: ACM Multimedia Conference,
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020a. CSP: code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural

---

Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages Computational Linguistics.
A Collecting the Cultural Concepts
Association for
First, we need to collect the cultural concepts of different countries. In detail, we choose a diverse set of cultures and languages following: Indone- stan, Swahili, Tamil, Turkish, Chinese. Each cul- ture contains about ten chapters: festival, music, religion and belief, animal and plant, food, cloth- ing, building, agriculture, tool, and sport. Each chapter contains several to dozens of cultural con- cepts. The collection of cultural concepts is primar- ily carried out mamually. Importantly, this approach doesn't 't necessitate cross-cultural specialists; rather, it employs crowd-sourced workers familiar with the respective culture, making it relatively cost- effective and simple process. The procedure is twofold:
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020b. CSP:code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2624 2636. Online. Association for Computational Linguistics.
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-We Chang. 2021. Broaden the vision: Geo- diverse visual commonsense reasoning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 21 15-21299, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference com- puter vision, pages 6032.
1. Diverse Candidate Collection: For each culture under consideration, involve five crowd-sourced workers and require them explore a minimum of three types of online resources. (1) Wikipedia. For example, the article "Culture of Turkey" on Wikipedia lists many cultural concepts including foods, fes- tivals, architecture, and so on. (2) Official websites. Most countries provide official web- sites to introduce their culture. (3) Search engines. Some websites retrieved by search engines such as travel guides will introduce the local culture. They collect as many cul- tural concepts as possible for each category. The collected data from each worker is then aggregated.
Yan Zeng. Wangchunshu Zhou, Ao Luo, and Xinsong Zhang. 2022. Cross-view language modeling: To- wards unified cross-lingual modal pre-training. arXiv preprint arXiv:2206.0066 00621.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17 1710.094.2.
Wenxuan Zhang, Ruidan He, Haiyun Peng, Lidong Bing, and Wai Lam. 2021. Cross-lingual aspect- based sentiment analysis with aspect term code- switching. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9220-9230 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
2. Voting for Filtering:
Next, an additional 10 crowd-sourced workers from the respective country or region assess whether each gath- ered concept genuinely belongs to the local culture. If seven more evaluators agree, the concept is added the final 'culture concept set.'
Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021. UC2: universal cross-Tingual cross-modal vision-and- language pre- training. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 202), pages 4155-4165. Com- puter Vision Foundation IEEE.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. 2022. Detecting twenty-thousand classes using image-level supervi- sion. In European Conference on Computer Vision, pages 350-368. Springer.
This method ensures quality in several ways:
Inclusion of Multiple Evaluators: By in- volving multiple people from the respective culture in both the collection and the valida- tion process, we minimize individual bias and enhance the dataset's reliability.
Threshold for Inclusion:
The use of a voting system provides a safety net against inaccura- cies and biases. If a concept is included, it's

---

because a significant majority (at least out of 10) of the evaluators from that culture have vouched for its relevance.
NEVRZ
MARVL
Manual Over Automatic: While automated methods may miss nuances or make errors, manual collection engages those who under- stand the cultural intricacies best-the the people from that culture.
Dove
Figure 4: Sampled examples from NLVR2 (Suhr et al., 2019) and MaRVI,(Liu al., 2021a). The NLVR2 is semantically-diii diverse dataset for reasoning about natural language descriptions of photos. The task is to determine if caption true with regard pair of images. MaRVL is similar to NLVR2, except that descriptions of NLVR2 are in English and MaRVL spans typologically diverse languages (Chinese, Tamil, Swahili, Indonesian, and Turkish).
Hence, our approach offers a robust, yet eco- nomical way of collecting high-quality cultural concepts.
B Human Evaluation of Cultural Adaptation Graphs
The evaluation of the generated cultural adaptation graphs yielded quite reassuring results:
These measures are designed to ensure that the cultural adaptation graph generated is of high qual- ity, both in terms of capturing authentic cultural elements and in conforming to human judgment. Therefore, while the graph is not perfect, con- structed with numerous safeguards to ensure its utility and accuracy.
Approximately 84% of the generated cultural adaptation graphs were in alignment with human judg- ment. This high rate of agreement underscores the validity of our approach in capturing cul- turally meaningful concepts.
High Human Agreement:
Addressing Inaccuracies: For the remain- ing graphs that were less accurate, the pri- mary issue often lay in the lack of direct En- glish translations for the low-resource cultural concepts' hypernyms. To address this, the method would need consider higher-order hypernyms, making the entire path too distant and potentially distorting cultural similarity according to human evaluators.
C NLVR2 and MaRVL Dataset
Liu al. (2021a) points that most of the synsets employed by NLVR2 (Suhr al., 2019) and Im- ageNet (Deng al., 2009) are only present in 30 or fewer languages and they contain overly spe- cific concepts that belong to Icaf nodes in WordNet. Given the biases in ImageNet-dett deriive or inspired datasets, they define a protocol to collect data that is driven by native speakers of language, consisting of concepts arising from their lived experiences. As consequence, the descriptions languages: Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish, and the concepts are selected be culturally relevant. Both multilingual and mono- lingual models perform comparably well in English (NLVR2). When these models are evaluated on the languages in MaRVL, however, the performance of zero-shot multilingual baselines dramatically drops lot, floating just above the chance level. Fur- ther analysis shows that there are sources of difficulty that make MaRVL challenging: 1) cross- cultural transfer cout-of-distribune concepts with respect to English datasets) and 2) cross- tross-lingual transfer.
To counteract these issucs, we implemented the following strategies:
Hypernym Limitation: As described in the paper, we restrict the method to considering at most three-order hypernyms during the con- struction of the cultural adaptation graph. If suitable translation or hypernym cannot be identified within this constraint, the concept is discarded.
written in five
Exponential Decay of Sampling Probabil- ity: The paper also outlines that the sampling probability for including concepts in the cul- tural adaptation set decays exponentially with path distance. This mechanism serves to mit- igate possible inaccuracies by giving greater weight to more closely related concepts.

---

D The Implementation in Details
D.I Experimental Setting
We provide detailed experimental setting. First, collate and merge data from three web resources and build initial dataset containing cultural con- cepts of different countries. Refer appendix for detail. For cultural adaptation. " "/r/IsA" in Conceptnet can be used to query hypernyms in target languages, and Srr/Symomm" is em- ployed to query synonyms in English. And we use the hyponeyms() function of Wordnet to query hyponyms of English concepts. We only consider three-order hypernym most. And the sampling probability decays exponentially with path distance. For phrase level, we count n-gram phrases. n range from to We select phrases related to reasoning and translate them to other languages via Google Translate. For word level, we employ the #/r/Symonymn in Conceptnet.
59.46
Accuracy
59
57.65
57
55
53.8
53.47
53
Baseline
Ours
+1k
+10k
Figure 5: mUNITER on the topic Speech and language of Chinese test dataset in MARVL. "MaRVL" represent score reported by et (2021a) and we also reproduce this result. "+Ik" /"+10k" represent adding 1k/10k code-switched data about traditional Chinese musical instruments to the training dataset. "Best" rep- resent the best performance when add more code- switched data examples.
D.2 Model Implementation
E Error Case Analysis
Most multilingual multimodal pre-trained models architecture consists of a stack of Transformer layers are similar to BERT. Their inputs are the concatenation of language and vision embeddings. The language inputs are tokenized and surrounded by two special tokens {[CLS), ti.., fn, {.EP]}. The language embeddings are then obtained by index token ids just like the original BERT. The vision input consists of a visual features pro- duced by well-trained object detector. Following (Liu et al., 2021a) and (Bugliarello et al., 2022), we also add a special feature [IMG] that encodes the entire image, G), vi,..., }. Each fea- ture is embedded using a BERT-like embedding layer by using its bounding box coordinates as the input position. Then these language and vision embeddings will be fed into BERT-like encoder and get M b Mccs) MiMaI W cCs) representa- tions at the last layer. We follow Liu et al. (2021a) and employ the cross-entropy loss function. We apply two-layer MLP with a GeLU activation function (Hendrycks and Gimpel, 2016) on top of the image- text representation. The probability that they are both correct is predicted by softmax over classes (representing true and false labels):
E.1 Performance Across Languages
As demonstrated in Table3 of the paper, per- formance diverges significantly across languages. This discrepancy is due largely to the imbalanced distribution of pre-training data, linguistic idiosyn- crasies, and cultural distinctions. Furthermore, the ease of language transfer from English to other languages varies considerably.
F.2 Within-Language Category Differences
An intra-language examination reveals that perfor- mance fluctuates across different categories. For instance, the model demonstrates heightened effi- cacy in the "animal and plant" and "building" cate- gories relative to others. Section 4.3.4 elaborates on this phenomenon, attributing primarily the distribution of pre-training data.
E.3 Instance-Level Analysis
We sampled a subset of the test data, and when we conducted a detailed error investigation at the individual instance level, we discovered several illuminating patterns:
Poor Spatial Awareness: The model predom- inantly falters in spatial reasoning tasks. No- tably, descriptions formatted as "one image.... another image..." are more challenging for the model than those specifying "left image...
([ Mipeay Merss ML.P
I))
Fg D)-
Minua Mfcus
(2)

---

right image...". with performance improving statistically across languages in the latter case (58.4% vs 53.8%).
Language Resource Limitations: Prior stud- ies (Jiao al., 2023; Vilar al., 2022; Cao et., 2023; Havaldar al., 2023) and our case studies on GPT-3.5's initial version indicate subpar performance in certain low-resource languages, such as Swahili (sw), despite profi- ciency in languages like Chinese (zh).
Limited Textual Reasoning: Performance enhances when the description is condensed into a single sentence ("in both images...") rather than dispersed across two ("one im- age..., another image...)), with performance leap to 60.4%.
Dictionary lookups are generally more efficient and economical than operating large language models.
Dost-Effeettveeess::
False Bias: The models exhibit pronounced propensity predict 'false' in the test set, con- stituting 57% of the model's predictions, con- trasted with 43% of 'true" predictions. This imbalance, especially conspicuous given the actual 50-50 distribution of 'true" and 'false' labels in the training and test sets, stems pre- dominantly from the model's unfamiliarity with certain cultural concepts and images, prompting 'false' prediction.
Reproducibility & Hallucinations: Inher- ent weaknesses in LLMs include instability based prompts and difficulty ensuring producibility. Verifying outputs is particularly challenging when hallucinations occur.
Dataset Compatibility: Our downstream dataset, NLVR2, revolves around a WordNet subset, facilitating queries. Relying on Chat- GPT would necessitate additional output pro- cessing.
These analyses collectively provide a compre- hensive insight into model's limitations, illumi- nating potential avenues for refinement.
Leveraging Existing Dictionary Resources: Our method can fully exploit existing dictio- nary resources for extremely low-resource lan- guages, like Navajo, through platforms such ConceptNet.
F Reduce Model Bias
According to Liu al. (2021a), each language test dataset can divided into multiple topics based on the categories of cultural concepts. For example, the topic "Speech and language" of the Chinese test dataset in MARVL contains many traditional Chinese musical instruments, such as Erhu. Similar adjusting the sampling probability of language, can generate Chinese-Engliss code-switched by adding the sampling probability of assigned cul- tural concepts and phrases of Chinese. Assuming models need to improve their reasoning ability on datasets about traditional Chinese musical instru- ments, all we need is to improve the sampling prob- ability of concepts about this topic in our collected cultural concepts set. Figure 5 shows that as add more related data, the model performance is becoming better on corresponding topics.
G.1 Evaluating Recent Models
Recently, we evaluated the latest GPT-3.5 and GPT- models for cultural adaptation capabilities. Using a chain of prompt approach to construct a cultural adaptation graph yielded promising results, partic- ularly with GPT-4. Nonetheless, these models also encounter challenges, such as potential hallucina- tions, and necessitate meticulous prompt design for diverse cultural adaptations. We believe that the integration of dictionary- cultural adap- tation methods with large language models repre- sents a promising hybrid approach. For instance, dictionary-based method could produce broad set of cultural adaptations, subsequently filtered for relevance by large language model. Incorporating LLMs to devise our cultural adaptation graph is in- deed promising and under consideration for future exploration.
LLM for Cultural Concept Adaptation
This section discusses the potential of using large language models (LLMs) like ChatGPT for cul- tural concept adaptation. Despite their proficiency in explaining various cultural concepts, several fac- tors discourage their exclusive use as bascline or expert system.


Reviews:
[{"review_id": "C4m9defxfq", "reviewer": "Reviewer_eiJt", "summary": "", "questions": "Question A: Although the author(s) provide a description of cultural concept collection in Appendix A, the description primarily focuses on the data resources and considered geo-cultures as well as cultural categories. I'm still confused about how the author(s) selected low-resource cultural concepts from the mentioned resources. E.g., manually or automatically?How to guarantee the collected concepts' quality? \n\nQuestion B:  Regarding cultural-level code-switching for data augmentation, the author(s) only considered the replacement of high-resource cultural concepts by low-resouce ones. In fact, this strategy can further extend to any two low-resource cultures (e.g., 'Erhu' in China versus 'Kongahyan' in Indonesian), possibly augmenting more data and benefiting the model with a deep understanding of low-resource culural concepts. I'm wondering if there are some potential reasons for author(s) emphasize high-resource versus low-resource cultural concept code-switching?\n\nQuestion C: Although the authors show the benefits of cultural-adapted data augmentation on baselines for multimodal reasoning, the quality of the generated cultural adaptation graph is uncertain. Did the author(s) conduct any evaluation (e.g., human evaluation) for this generated graph quality?\n\nQuestion D: Did the author(s) conduct any error analyis to explore potential patterns of model prediction errors?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eiJt,\n\nThank you for taking the time to review our paper and we appreciate the positive feedback and your detailed questions regarding our methodology and experiments. We would like to address your concerns as follows:\n\n**Question A**: Although the author(s) provide a description of cultural concept collection in Appendix A, the description primarily focuses on the data resources and considered geo-cultures as well as cultural categories. I'm still confused about how the author(s) selected low-resource cultural concepts from the mentioned resources. E.g., manually or automatically?How to guarantee the collected concepts' quality?\n\n**Reply**: Thank you for your question regarding the method of collecting cultural concepts outlined in Appendix A. We understand your concern about how the concepts are selected, especially in the context of low-resource cultural information, and the assurance of quality.\n\nTo clarify, the collection of cultural concepts is primarily carried out manually. Importantly, this approach doesn't necessitate cross-cultural specialists; rather, it employs crowd-sourced workers familiar with the respective culture, making it a relatively cost-effective and simple process. The procedure is twofold:\n\n1. **Diverse Candidate Collection**: For each culture under consideration, we involve five crowd-sourced workers who follow the guidelines specified in the appendix A. They collect as many cultural concepts as possible for each category. The collected data from each worker is then aggregated.\n2. **Voting for Filtering**: Next, an additional 10 crowd-sourced workers from the respective country or region assess whether each gathered concept genuinely belongs to the local culture. If seven or more evaluators agree, the concept is added to the final 'culture concept set.'\n\nThis method ensures quality in several ways:\n\n- **Inclusion of Multiple Evaluators**: By involving multiple people from the respective culture in both the collection and the validation process, we minimize individual bias and enhance the dataset's reliability.\n- **Threshold for Inclusion**: The use of a voting system provides a safety net against inaccuracies and biases. If a concept is included, it's because a significant majority (at least 7 out of 10) of the evaluators from that culture have vouched for its relevance.\n- **Manual Over Automatic**: While automated methods may miss nuances or make errors, manual collection engages those who understand the cultural intricacies best—the people from that culture.\n\nHence, our approach offers a robust, yet economical way of collecting high-quality cultural concepts. We hope this clears up any confusion you may have had.\n\n**Question B**: Regarding cultural-level code-switching for data augmentation, the author(s) only considered the replacement of high-resource cultural concepts by low-resouce ones. In fact, this strategy can further extend to any two low-resource cultures (e.g., 'Erhu' in China versus 'Kongahyan' in Indonesian), possibly augmenting more data and benefiting the model with a deep understanding of low-resource culural concepts. I'm wondering if there are some potential reasons for author(s) emphasize high-resource versus low-resource cultural concept code-switching?\n\n**Reply**: Your insight about extending the data augmentation strategy to low-resource versus low-resource cultural concept code-switching is both perceptive and innovative. However, there are several underlying reasons why we emphasized high-resource versus low-resource cultural concept code-switching.\n\n1. **High-Resource as a \"Bridge\"**: For most open-source multilingual and multimodal models, a significant proportion of the pre-training data is in English, which naturally incorporates a wealth of cultural knowledge. Using this high-resource English data as a \"bridge\" is logical for aiding the adaptation of low-resource cultural concepts.\n2. **Downstream Task Constraints**: Many downstream task datasets used for evaluation have limited representation of low-resource cultures. For instance, the NLVR2 dataset is solely in English, making quantitative evaluation between low-resource cultures difficult.\n\nTo address your constructive suggestion:\n\n1. **Theoretical Feasibility**: Theoretically, the idea holds water. The term \"low-resource\" is relative, and code-switching between different low-resource cultures could potentially benefit each other.\n2. **Proof of Concept**: In line with your suggestion, we translated the NLVR2 dataset into Chinese via machine translation and applied the Cultural Adaptation method (as detailed in section 3.1 of the paper). We then created cultural adaptation sets from Indonesian (ID), Swahili (SW), Tamil (TA), and Turkish (TR) to Chinese. Following this, we employed the Concept-Based Data Augmentation from section 3.2, using a 15:5 ratio for augmentation and the model is mUNITER. Compared to a baseline model trained solely on the translated Chinese training set, our method exhibited improved performance on the MARVL test across all the involved languages:\n    - Chinese (ZH): Baseline 62.61, Our method 61.88\n    - Indonesian (ID): Baseline 53.20, Our method 56.17\n    - Swahili (SW): Baseline 51.42, Our method 53.34\n    - Tamil (TA): Baseline 52.11, Our method 57.58\n    - Turkish (TR): Baseline 54.26, Our method 57.93\n\nThus, our proof-of-concept trial supports the merit in extending the data augmentation strategy beyond just high-resource to low-resource cultural concept substitution. We hope this clarifies why we initially emphasized high-resource vs. low-resource concepts while also acknowledging the validity and potential of your suggestion.\n\n**Question C**: Although the authors show the benefits of cultural-adapted data augmentation on baselines for multimodal reasoning, the quality of the generated cultural adaptation graph is uncertain. Did the author(s) conduct any evaluation (e.g., human evaluation) for this generated graph quality?\n\n**Reply**: You raise an important question regarding the quality of the generated cultural adaptation graph, a key component of our methodology. Indeed, we share your concern for ensuring high-quality output in this aspect of our work.\n\nBefore proceeding with data augmentation experiments, we did conduct a human evaluation of the generated cultural adaptation graphs. The results were quite reassuring:\n\n1. **High Human Agreement**: Approximately 84% of the generated cultural adaptation graphs were in alignment with human judgment. This high rate of agreement underscores the validity of our approach in capturing culturally meaningful concepts.\n2. **Addressing Inaccuracies**: For the remaining graphs that were less accurate, the primary issue often lay in the lack of direct English translations for the low-resource cultural concepts' hypernyms. To address this, the method would need to consider higher-order hypernyms, making the entire path too distant and potentially distorting cultural similarity according to human evaluators.\n\nTo counteract these issues:\n\n- **Hypernym Limitation**: As described in the paper, we restrict the method to considering at most three-order hypernyms during the construction of the cultural adaptation graph. If a suitable translation or hypernym cannot be identified within this constraint, the concept is discarded.\n- **Exponential Decay of Sampling Probability**: The paper also outlines that the sampling probability for including concepts in the cultural adaptation set decays exponentially with path distance. This mechanism serves to mitigate possible inaccuracies by giving greater weight to more closely related concepts.\n\nThese measures are designed to ensure that the cultural adaptation graph generated is of high quality, both in terms of capturing authentic cultural elements and in conforming to human judgment. Therefore, while the graph is not perfect, it is constructed with numerous safeguards to ensure its utility and accuracy. I hope this clarifies any concerns you might have had.\n\n**Question D**: Did the author(s) conduct any error analyis to explore potential patterns of model prediction errors?\n\n**Reply**: You make a valid point about the importance of error analysis to understand the model's limitations. While we did conduct an error case analysis, we had to omit it from the paper due to space constraints. However, we are eager to share some of the key findings here to address your concerns.\n\n1. **Performance Across Languages**: As Table 3 in the paper shows, the performance varies across languages. This is influenced by the imbalanced distribution of the pre-training data as well as linguistic and cultural differences. The ease of transfer from English to other languages is also non-uniform.\n2. **Within-Language Category Differences**: When looking within a single language, we found that the difficulty varies across different categories. For instance, the model performs better on \"animal and plant\" and \"building\" categories compared to others. As detailed in section 4.3.4 of the paper (\"Reduce Model Bias\"), this also relates to the distribution of pre-training data.\n3. **Instance-Level Analysis**: We conducted a more granular error analysis at the instance level and found insightful patterns:\n    - **Poor Spatial Awareness**: The model generally struggles with spatial reasoning. Specifically, if the description format is \"one image..., another image...\" instead of specifying \"left image..., right image...\", the models statistically perform better across languages (58.4%  vs 53.8% ).\n    - **Limited Textual Reasoning**: The model also fares better when the description is given in a single sentence (\"in both images...\") as opposed to two separate ones (\"one image..., another image...\"). Here, the model's performance increased to 60.4%.\n    - **False Bias**: The models have a higher tendency to predict 'false' in the test set. Specifically, 57% of the model predictions were 'false' and 43% were 'true', even though the actual distribution of 'true' and 'false' labels in both the training and test sets is balanced at 50%. The main reason for this skew is the model's unfamiliarity with specific cultural concepts and images, leading it to predict 'false'.\n\nThese analyses offer a multi-level perspective into the shortcomings of the model and help us understand where improvements can be made. We hope this clarifies any concerns you may have had regarding the model's error characteristics.\n\n**Missing References:**\n\nThank you for pointing out the missing reference. We are aware of Sun et al.'s work and, upon reflection, agree that it should be included in our discussions. We will make sure to incorporate this in our revised version.\n\n**Looking Forward to Your Reply and Further Discussion**\n\nWe sincerely hope that our clarifications adequately address your concerns. As we move to revise our manuscript, we will integrate your valuable suggestions to further strengthen the research presented. If our responses have been satisfactory in resolving your questions, we kindly ask that you consider re-evaluating our paper. Specifically, we would be extremely grateful if you could revisit your scoring for 'Soundness' and 'Excitement.'\n\nShould you have any further concerns or require additional clarifications, we are more than willing to engage in further discussions to ensure the quality of our research.\n\nWarm regards,\n\nAll Authors of the Paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of capturing concept pairs for cultural adaptation is interesting.\n\n2. The author(s) conducted a bunch of experiments to evaluate and analyze the proposed method.\n\n3. The paper is easy to follow.", "reasons_to_reject": "Some fundamental details in the method and experiment design are missing. Please see questions in the next section.", "questions_to_authors": "Question A: Although the author(s) provide a description of cultural concept collection in Appendix A, the description primarily focuses on the data resources and considered geo-cultures as well as cultural categories. I'm still confused about how the author(s) selected low-resource cultural concepts from the mentioned resources. E.g., manually or automatically?How to guarantee the collected concepts' quality? \n\nQuestion B:  Regarding cultural-level code-switching for data augmentation, the author(s) only considered the replacement of high-resource cultural concepts by low-resouce ones. In fact, this strategy can further extend to any two low-resource cultures (e.g., 'Erhu' in China versus 'Kongahyan' in Indonesian), possibly augmenting more data and benefiting the model with a deep understanding of low-resource culural concepts. I'm wondering if there are some potential reasons for author(s) emphasize high-resource versus low-resource cultural concept code-switching?\n\nQuestion C: Although the authors show the benefits of cultural-adapted data augmentation on baselines for multimodal reasoning, the quality of the generated cultural adaptation graph is uncertain. Did the author(s) conduct any evaluation (e.g., human evaluation) for this generated graph quality?\n\nQuestion D: Did the author(s) conduct any error analyis to explore potential patterns of model prediction errors?", "paper_topic_and_main_contributions": "This paper focuses on multimodal reasoning over images and languages, with an emphasis on low-resource cultural concepts. Specifically, the author(s) introduce a cultural-concept adaptation method to align low-resource cultural concepts with high-resource ones based on their shared semantic properties. With the resulting concept mapping set, the author(s) further propose a multimodal data augmentation method based on the code-switching strategy. Experiments on a popularly used multicultural multimodal reasoning benchmark (MaRVL) show that the proposed method improves the baselines in both zero- and few-shot settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Sun, J., Ahn, H., Park, C. Y., Tsvetkov, Y., & Mortensen, D. R. (2021, April). Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 2403-2414)."}}, {"review_id": "PDQxfExGYg", "reviewer": "Reviewer_sDGb", "summary": "", "questions": "* The possibility and portion of components in the text being replaced by Three-tier Code-switching can be stated in more detail. It will be even better when there are hyper-parameter experiments about the possibility or the portion of the replaced components gradually changing on a scale. \n* As LLMs are generally good when doing something like culture concepts explanation. E.g., they can answer what is an erhu. Why not adapt ChatGPT to do the cultural concept adaption? If ChatGPT is not a good expert, it can also be a baseline for the scenario.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sDGb,\n\nThank you for your thorough review and feedback on our manuscript. We truly appreciate the time and effort you dedicated to understanding our work, and your constructive comments. We'd like to address each of your concerns:\n\n**Question 1**: The possibility and portion of components in the text being replaced by Three-tier Code-switching can be stated in more detail. It will be even better when there are hyper-parameter experiments about the possibility or the portion of the replaced components gradually changing on a scale.\n\n**Reply** : Thank you for your question regarding the need for more detailed information on the possibility and portion of text components replaced by the Three-tier Code-switching technique. You've raised a valid point, and I appreciate the opportunity to provide further clarification.\n\n**Regarding Possibility:**\nWe have collected detailed statistics from multiple angles to address your query. In our dataset tailored for low-resource cultural adaptation:\n\n*Concept*: The probability that a cultural concept from low-resource cultural adaptation set appears at least once in the training set is 64.3%. The sentences that could potentially be retrieved to implement this concept constitute around 75% of the entire training dataset.\n\n*Phrase*: Among these retrievable sentences, at least one phrase that can be replaced exists with a probability of 78.5%.\n\n*Word*:  In terms of at least one  words that can be replaced, the probability stands at 97.8%. On average, about 70% of the words in a sentence can find possible low-resource replacements.\n\n**Regarding Portion:**\nIn our paper, we have formalized the methodology in such a way that it is adaptable to a broad range of scenarios, making use of probability distributions for a more nuanced representation of the methods involved. However, in the experiments, specific data requires specific analysis. For instance, in the context of our dataset:\n1. In terms of the dataset composition, we follow a 15:5 ratio. That means 75% of the data in the training set is in English, and the remaining should be as diverse as possible.\n2. For retrievable sentences with cultural concepts and phrases, usually only one to two phrases per sentence can be replaced. Also, only 70% of the words can be queried for possible replacements, thereby setting a natural upper limit.\n\nGiven these, we set the replacement probability for Three-tier Code-switching at 100%. This not only yielded the best results but also eliminated the need for complicated hyperparameter tuning.\n\n**Hyperparameter Experiments:**\n\nAs per your suggestion, we also experimented with different probabilities for the replacement and observed consistent performance improvements across all three models as the sampling probability increased:\n\n| Sampling Probability | mUNITER | UC2 | M3P |\n| --- | --- | --- | --- |\n| 0% | 54.0 | 57.28 | 56.0 |\n| 30% | 55.32 | 58.14 | 57.46 |\n| 50% | 56.27 | 58.85 | 58.79 |\n| 70% | 57.04 | 59.16 | 59.34 |\n| 100% | 57.31 | 59.33 | 59.79 |\n\nAlso, modifying the sampling ratios at individual levels—such as phrases, concepts, or words—does have an impact on the final performance, as clearly evidenced by our ablation studies.\n\nIn summary, we find that higher probabilities for replacement are beneficial for the model performance. We hope this addresses your question comprehensively. Thank you for the insightful inquiry, which has provided an opportunity for us to clarify our methodology.\n\n**Question 2**:  As LLMs are generally good when doing something like culture concepts explanation. E.g., they can answer what is an erhu. Why not adapt ChatGPT to do the cultural concept adaption? If ChatGPT is not a good expert, it can also be a baseline for the scenario.\n\n**Reply** : Thank you for the thoughtful question regarding the potential of using large language models (LLMs) like ChatGPT for cultural concept adaptation. While it's true that LLMs can perform well in explaining various cultural concepts, there are several considerations that guided our decision to not solely rely on them as a baseline or as an expert system. Allow me to elaborate:\n\n1. **Language Resource Limitations**: Previous studies [1][2][3] and our own  case studies on the GPT-3.5 of initial version have shown that the model performs sub-optimally on some low-resource languages. For example, it may perform well for languages like Chinese (zh), it shows limitations for languages like Swahili (sw).\n2. **Cost-Effectiveness**: Dictionary lookup operations are generally more efficient and cheaper compared to running large language models.\n3. **Reproducibility & Hallucinations Issues**: LLMs have inherent weaknesses. They are not very stable based on prompts, making it difficult to guarantee reproducibility. Additionally, verifying the model's outputs can be challenging, particularly when hallucinations occur.\n4. **Dataset Compatibility**: Our downstream dataset, NLVR2, is constructed around a subset of WordNet, making it easier to query. If we were to rely on ChatGPT, additional processing would be needed to match the output.\n5. **Leveraging Existing Dictionary Resources**: Our method have the potential to make full use of existing dictionary resources for extremely low-resource languages, such as Navajo, through platforms like ConceptNet.\n\nBeyond these concerns, recently, we have evaluated the latest GPT-3.5 and GPT-4 models for their capability in cultural adaptation. We employed a chain of prompt approach to construct a cultural adaptation graph and found promising results, especially with GPT-4. However, these models also face the challenge of potential hallucinations and require careful prompt design to generate diverse cultural adaptations.\n\nIn conclusion, we believe that a more reasonable approach would be to explore the combination of dictionary-based cultural adaptation methods with large language models. For example, the dictionary-based method could generate a large set of cultural adaptations, which could then be filtered for relevance by the large language model. Incorporating large language models to construct our proposed cultural adaptation graph does indeed hold promise and is a direction we are considering for future work.\n\n[1] Is ChatGPT A Good Translator? A Preliminary Study\n\n[2] Prompting PaLM for Translation: Assessing Strategies and Performance\n\n[3] Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study\n\n[4] Multilingual Language Models are not Multicultural: A Case Study in Emotion\n\n**Reproducibility Concerns:**\n    \nWe understand the importance of reproducibility in research. To this end, we'll work on providing a more detailed specification of our experimental setup, including a clearer delineation of parameter settings and the rationale behind them. And we will release both the code and the data to aid the scientific community in replicating our results.\n    \n\nOnce again, we are grateful for your constructive feedback and insights. They provide us with a clear direction on how to refine and enhance our paper. We are optimistic that the revisions which integrated your valuable suggestions will further elevate the impact of our work.\n\nWarm regards,\n\nAll Authors of the Paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Proposed to enhance the multicultural realization and inability of models to alleviate the undesired multicultural performance due to the absence of other non-English languages' sufficient data. \n* Proposed a multimodal culture-related augmentation method with two steps to solve the observed problem. \n* Promising performance on the target multimodal reasoning task with reasonable experiments. \n* Good writing.", "reasons_to_reject": "* More details and even more experiments are required to explore the boundary of the proposed method.", "questions_to_authors": "* The possibility and portion of components in the text being replaced by Three-tier Code-switching can be stated in more detail. It will be even better when there are hyper-parameter experiments about the possibility or the portion of the replaced components gradually changing on a scale. \n* As LLMs are generally good when doing something like culture concepts explanation. E.g., they can answer what is an erhu. Why not adapt ChatGPT to do the cultural concept adaption? If ChatGPT is not a good expert, it can also be a baseline for the scenario.", "paper_topic_and_main_contributions": "This paper aims to alleviate the problem of unsatisfactory multicultural competence of low-resource languages in multimodal reasoning. To this end, it proposes Cultural Concept Adaptation with two steps: firstly, adapting an annotation-free method for cultural-concept adaptation and constructing a concept mapping set; secondly, utilizing the three-tier code-switching strategy and cultural concept-based mix-up for the purpose of multimodal data augmentation. The proposed method achieved improved performance in zero-shot and few-shot settings on several low-resource languages (relative to English).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Yz2nH117ty", "reviewer": "Reviewer_bqm3", "summary": "", "questions": "1. Compared with the baseline method mUNITER, the performance degradation of all ablation models was not obvious. Which data augmentation strategy played a role in performance improvement? Might it be possible to ablate multiple data augmentation strategies simultaneously to observe changes in model performance? \n\n2. I did not find relevant details about the Concept-Based Mixup method in the paper. What object detection algorithm do you use to identify images? How do you replace identified high-resource image elements with low-resource cultural image elements?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bqm3,\n\nThank you for your thoughtful and thorough review of our manuscript. We highly appreciate the time and effort you put into evaluating our work, and your comments are instrumental in improving the paper. Below, we respond to each point of concern you raised.\n\n**Question1**: Compared with the baseline method mUNITER, the performance degradation of all ablation models was not obvious. Which data augmentation strategy played a role in performance improvement? Might it be possible to ablate multiple data augmentation strategies simultaneously to observe changes in model performance?\n\n**Reply**:The impression that the degradation in performance for our ablation models is not significant compared to the mUNITER baseline deserves clarification.\n\n1. **Degradation is More Significant than It Appears**:  While it may appear that the decline in performance is subtle (about 1-2 points), it's crucial to consider this within the context of the overall performance gain, which is less than 4 points. Given this, a 1-2 point drop is indeed substantial. As depicted in Table 5 of our paper, the impact of omitting phrase and image information is substantial. For example, when we removed concept and phrase information, the performance showed a more significant decrease compared to other ablations.\n2. **Multiple Ablations Conducted**: Taking your suggestion to heart, we conducted additional experiments to ablate multiple data augmentation strategies simultaneously. The table provided showcases these results.\n\n| Method | NLVR2 | MaRVL (ZH, TA, SW, ID, TR, avg.) |\n| --- | --- | --- |\n| Baseline | 70.93 | 54.05, 53.14, 51.72, 55.41, 53.22, 53.51 |\n| Our Method | 71.56 | 59.71, 58.45, 54.35, 55.24, 58.81, **57.31** |\n| All w/o Concept | 71.18 | 58.87, 57.30, 53.82, 54.61, 57.52, **56.42** |\n| All w/o Phrase | 70.71 | 57.95, 56.94, 53.18, 54.02, 57.19, 55.86 |\n| All w/o Word | 70.65 | 58.69, 57.22, 53.55, 54.38, 57.55, 56.28 |\n| All w/o Image | 70.85 | 58.14, 56.81, 52.95, 54.69, 56.33, 55.78 |\n| All w/o Concept+Phrase | 69.91 | 56.27, 54.92, 53.05, 53.78, 55.63, 54.73 |\n| All w/o Concept+Word | 71.08 | 57.91, 57.03, 53.44, 53.72, 54.41, 55.30 |\n| All w/o Concept+Image | 70.39 | 57.21, 55.99, 52.75, 54.36, 55.07, 55.07 |\n| All w/o Phrase+Word | 70.94 | 57.81, 55.22, 53.37, 53.89, 56.51, **55.36** |\n| All w/o Phrase+Image | 70.85 | 57.25, 55.77, 52.31, 53.11, 55.48, 54.78 |\n| All w/o Image+Word | 69.88 | 57.94, 55.35, 52.41, 53.49, 56.27, 55.09 |\n| All w/o Concept+Phrase+Word | 71.10 | 56.01, 54.38, 52.88, 53.56, 54.35, 54.24 |\n| All w/o Concept+Word+Image | 70.68 | 56.95, 55.01, 52.51, 53.36, 54.27, **54.42** |\n| All w/o Concept+Image+Phrase | 70.72 | 56.12, 54.36, 51.91, 53.07, 53.83, 53.86 |\n| All w/o Phrase+Word+Image | 70.45 | 55.97, 54.81, 52.05, 53.16, 54.93, 54.18 |\n\n\n\n**Key Insights from Multiple Ablations:**\n\n1. **Two-Feature Ablations**:\n    - Removing both 'Concept' and 'Phrase' has the most significant impact on performance.\n    - Conversely, the least effect is seen when 'Phrase' and 'Word' are removed, indicating that pairs of cultural concepts and their corresponding images are most beneficial for performance improvement.\n2. **Three-Feature Ablations**:\n    - Ablating 'Concept', 'Image', and 'Phrase' together has the largest impact, suggesting that the 'Word' feature contributes least to the performance gains.\n    - Removing 'Concept', 'Word', and 'Image' has the least impact, signifying that the 'Phrase' feature contributes most to performance gains.\n\nThese multiple ablations provide nuanced insights into the importance of each feature in our model. They allow us to isolate and understand how individual or combinations of features contribute to performance. We hope this detailed breakdown addresses your question effectively.\n\n**Question2**: I did not find relevant details about the Concept-Based Mixup method in the paper. What object detection algorithm do you use to identify images? How do you replace identified high-resource image elements with low-resource cultural image elements?\n\n**Reply**: Thank you for your question regarding the Concept-Based Mixup method, a topic that, unfortunately, was omitted in our original paper due to space constraints. Your inquiry has made us realize the importance of including this information, and we apologize for the oversight. Rest assured, this section will be incorporated into the revised version of the paper.\n\nTo address your question:\n\n1. **Object Detection Model**: We use the Detic[1] model available at [Facebook Research's GitHub repository](https://github.com/facebookresearch/Detic) for object detection. It employs the CLIP[2] classifier and offers open-vocabulary capabilities. [Here is their official demo using Colab for further reference.](https://colab.research.google.com/drive/1QtTW9-ukX2HKZGvt0QvVGqjuqEykoZKI)\n2. **Vocabulary Configuration**: With this model, we can freely configure the vocabulary, allowing us to detect specific cultural concepts within images.\n\nFor the actual mixup process, here's our approach:\n\n1. **Visual Element Detection**: We use the high-resource concept as the vocabulary term and deploy the object detection model to locate the corresponding visual elements in the image.\n2. **Bounding Box Extraction**: Based on the object detection results, we extract the object that needs to be replaced from the original high-resource image using bounding boxes.\n3. **Resizing**: We adjust the size of the low-resource image element, which is collected together with concepts themselves, to match the size of the bounding box.\n4. **Pasting**: The resized new low-resource image is then pasted into the corresponding location of the original image.\n5. **Edge Smoothing**: Finally, Poisson blending is applied to smoothly integrate the new object with the original image.\n\nWe hope this clarifies how the Concept-Based Mixup method is implemented in our experiments. Thank you for bringing attention to this crucial aspect of our work.\n\n[1]Detecting Twenty-thousand Classes using Image-level Supervision\n\n[2]Learning Transferable Visual Models From Natural Language Supervision\n\n **Reproducibility**\nWe have taken note of your valuable feedback concerning the reproducibility of our study. Should our paper be accepted, we will release both the code and the data to aid the scientific community in replicating our results. We believe this will address your concerns regarding the replicability of the study. Thank you for bringing this important issue to our attention.\n\nIn conclusion, we truly value your constructive feedback, which has guided substantial improvements in our manuscript. If our responses have been satisfactory in resolving your questions, we kindly ask that you consider re-evaluating our paper. Specifically, we would be extremely grateful if you could increase your scoring for 'Soundness' and 'Excitement.' We look forward to any further comments you may have.\n\nSincerely,\n\nAll Authors of the Paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to understand.\n\n2. This paper builds an external cross-language concept mapping graph, replacing the words, phrases and concepts in the dataset based on the graph path distance, so as to expand the dataset for training. Experiments show that the data augmentation method can significantly improve the performance of the pre-trained models on the multilingual multimodal reasoning task.\n\n3. Experiments and details are rich. ", "reasons_to_reject": "1. Compared with the baseline， the performance degradation of the ablation models is not obvious, and it is unknown which data augmentation strategy plays a role in performance improvement.\n\n2. This work proposed a concept-based mixup method to expand image data, which identifies visual elements that correspond to high-resource concepts and replaces them with visual elements associated with the original low-resource concept. However, I did not find the relevant details in the paper.", "questions_to_authors": "1. Compared with the baseline method mUNITER, the performance degradation of all ablation models was not obvious. Which data augmentation strategy played a role in performance improvement? Might it be possible to ablate multiple data augmentation strategies simultaneously to observe changes in model performance? \n\n2. I did not find relevant details about the Concept-Based Mixup method in the paper. What object detection algorithm do you use to identify images? How do you replace identified high-resource image elements with low-resource cultural image elements?", "paper_topic_and_main_contributions": "This work proposes a data augmentation approach for multilingual multimodal reasoning. Specifically, it constructs an external cross-language concept mapping graph to associate the concepts between different cultures (language), and calculates the correlation between concepts of different cultures through the graph path distance. Then, based on the correlation between cross-language concepts, the concepts of original dataset are replaced at word-level and phrase-level for training; image objects are also considered as replacement objects. Experimental results show that the proposed method achieves significantly better performance than the baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: (I've used bqm3's updated soundness score of 4, which was mentioned in the comments but not reflected in the numeric score).

The paper focuses on the problem of cultural adaptation in multimodal reasoning, in which some concepts are difficult to translate directly from one language to another. They suggest an annotation-free process for augmenting data using a "code switching" approach. The reviewers are unanimous that the experiments provided are well done and provide support for the authors' claims about the benefits of their approach. The detailed author responses clarify outstanding questions and suggest careful thinking and thorough work.

The reviewers raised several significant questions about the details of the work, including the source of the cultural concepts they use. The authors provide more details about the crowdsourced data collection effort, which is also described in Appendix A.

Several other questions that reviewers raised are thoroughly answered by the authors. Due to space constraints, not all of the answers may appear in the final text, but seem sufficient to address the reviewer questions.
Decision: Accept-Main