GD-COMET: A Geo-Diverse Commonsense Inference Model
Authors: Mehar Bhatia
Vered Shwartz
Venue: EMNLP 2023 Main (2023)
Keywords: commonsense reasoning
culture-aware NLP
geo-diverse applications
TL;DR: In this paper, we introduce GD-COMET, a geo-diverse commonsense inference model that generates culturally nuanced knowledge, enabling more inclusive and culturally aware AI systems.

Abstract: With the increasing integration of AI into everyday life, it's becoming crucial to design AI systems to serve users from diverse backgrounds by making them culturally aware. In this paper, we present GD-COMET, a geo-diverse version of the COMET commonsense inference model. GD-COMET goes beyond Western commonsense knowledge and is capable of generating inferences pertaining to a broad range of cultures. We demonstrate the effectiveness of GD-COMET through a comprehensive human evaluation across 5 diverse cultures, as well as extrinsic evaluation on a geo-diverse task. The evaluation shows that GD-COMET captures and generates culturally nuanced commonsense knowledge, demonstrating its potential to benefit NLP applications across the board and contribute to making NLP more inclusive.

GD-COMET: A Geo-Diverse Commonsense Inference Model
Mehar Bhatia and Vered Shwartz
University of British Columbia
Vector Institute for AI (meharb23, vshwartz)@cs ubc. ca
Abstract
PersonX eats a dutch baby
greedy, disgurting,
With the increasing integration Al into every- day life, it's becoming crucial design sys- tems that serve users from diverse backgrounds by making them culturally aware. In this pa- per, present GD-COMET, geo-diverse sion of the COMET commonsense inference model. GD-COMET goes beyond Western com- monsense knowledge and is capable of gener- ating inferences pertaining to a broad range of cultures. We demonstrate the effectiveness of GD-COMET through comprehensive human evaluation across 5 diverse cultures, as well extrinsic evaluation on a geo-diverse task. The evaluation shows that GD-COMET captures and generates culturally nuanced commonsense knowledge, demonstrating its potential to bene- fit NLP applications across the board and con- tribute to making NLP more inclusive.
Penorlaerh
throw up, water, wash the babl
Parssk
satisfied, full hungry. happy, guilty
Craac tel
sad, disgusting, happy, angry
Pescekia soen
hungry, ravenous, starving, gready
wenh
stomach a full belly, to satiffy hurger. be full, to have snack
satisted, full, happy
empty, craving, happy, keen
Figure Inferences from COMET and GD-COMET for the sentence "PersonX cats a dutch baby". demonstrat- ing lack of culture awareness COMET.
model (LM) and further trained on commonsense inferences collected from North American crowd- source workers (Sap et al., 2019). Consequently, the model exhibits a certain bias towards the North American cultural perspective. As evidenced by Fig. 1, COMET displays limited familiarity with the concept of German pancake, erroneously in- terpreting the term "dutch baby" in a literal sense. We identify a need for more inclusive common- sense reasoning models and propose GD-COMET: Geo-Diverse COMET. As demonstrated in Fig 1, GD-COME CO-MEMMTT gained the culturally relevant knowl- edge to interpret "dutch baby" as legitimate dish. English LM but is trained on knowledge base of cultural knowledge (Nguyen et al., 2023) prior to training on COMET's original training data. This simple approach is effective, as judged by both human evaluations as well as extrinsic evaluation on a geo- diverse task (Yin et al., 2021). GD-COMET can potentially benefit many downstream NLP applica- tions where the user population is diverse.'
1 Introduction
Culture plays significant role in shaping an indi- vidual's worldviews, beliefs, behaviours, and com- munication styles (Spradley, 1987). A considerable portion of what is commonly referred to as com- monsense knowledge is not universal but rather culture-specific, including social norms, values, tra- ditions, and more. An example of cultural differ- ences is greetings, which may involve a handshake in Western cultures, bowing in some Asian cultures, 'namaste' gesture in India, or 'wai' in Thailand. With AI systems becoming increasingly ubiqui- tous in society, is imperative to go beyond the Western cultural perspective (Hershcovich et al., 2022). Lack of cultural awareness may lead to models perpetuating stercotypes and reinforcing societal inequalities (Hutchinson et al., 2020; Ross et al., 2021; Søgaard, 2022), impeding their effec- tiveness for users from non- non-Western countries.
GD-COMET is similarly based
2 Background
2.1
Commonsense Inference Models
Many NLP tasks require reasoning beyond what is explicitly stated in the text. People fill in those gaps
In this paper, we focus on a popular model for commonsense reasoning, COMET (Bosselut et al., 2019), which is based on an English language
'oode available at giinhh.cmmneearbhatitCCCCCCT

---

with their commonsense knowledge. NLP models attempt to do the same by leveraging commonsense knowledge bases (KBs) such as ConceptNer (Speer et al., 2017) and ATOMIC (Sap et al., 2019). To achieve better coverage, knowledge models such as COMET (Bosselut et al., 2019) are based on pre-trained LMs and further fine-tuned on KBs, enabling contextually-relevan inferences along the KB's dimensions for new contexts.
2023). Finally, there are several recent cultural KBs such as StereoKG (Deshpande al., 2022). Quasimodo (Romero et al., 2019), and CANDLE (Nguyen et al., 2023). CANDLE, which we use in this work, is the most comprehensive among them, containing 1.1M assertions in English about 386 cultures (e.g. "A Dutch baby is a German pancake that is baked instead of cooked on the stove top"). CANDLE assertions were extracted from large web corpus and clustered into 5 facets of culture: food, drinks, clothing, rituals, and traditions.
COMET's hybrid approach proved useful for various tasks (e.g.. Chakrabarty et al., 2020; Am- manabrolu et al., 2021: Ravi et al., 2023a). Sub- sequent versions of COMET have been developed draw inferences from paragraphs (Gabriel et al., 2021), images (Park et al., 2020), and complex sen- tences (Ravi et al., 2023b). Further improvements include obtaining additional training data through crowdsourcing (Hwang et al., 2021) or generating synthetic data from LMs (West et al., 2022).
GD-COMET
We present GD-COMET, geo-diverse version of COMET. The goal of GD-COMET is to gen- erate high-quality commonsense inferences for concepts and events pertaining to both Western and non-Western cultures. Rather than collecting large-scale geo-diverse dataset in the style of ATOMIC, we split the training into two phases: (1) training the underlying LM on geo-diverse data; (2) continue training on the large-scale original COMET training data. This is motivated by Bosse- lut et al. (2019) that showed that implicit com- monsense knowledge from underlying LM's pre- training transfers to COMET. We similarly hypoth- esize that encoding geo-diverse data into the under- lying LM prior to training on COMET data will transfer this knowledge to GD-COMET.
COMET and its successors assume the universal- ity of commonsense knowledge, yet much of this knowledge may differ among cultures, in traditions (e.g., duration of a wedding ceremony; Acharya et al., 2021), foods (e.g., what counts as breakfast food; Speer et al., 2017), social norms, and more.
2.2 Culture-Aware NLP
While multilingual NLP a popular topic, culture- aware NLP is explreex.. It is crucial for lan- guage technologies to not only serve speakers of a wide variety of languages but also acknowledge that users come from diverse cultures (Hershcovich et al., 2022). Cultural norms and pragmatic aspects differ across speakers from different cultures (Zhou et al., 2023). reflect a Nevertheless, English LMs primarily lens due to training on web data with a US user bias (Cao et al., 2023).
Geo-Diverse Training (GD-BART). We pick 770,000 assertions from CANDLE with a com- bined score greater than 0.5. This threshold selects highly distinctive assertions specific and relevant to the specific region. We fine-tune BART-Large, the underlying LM of the latest COMET model (Hwang al., 2021), on this data, using BART's original pre-training objectives (token masking, to- ken deletion, text infilling and sentence permuta- tion). We save the model checkpoint with the low- est validation loss after training for 50 epochs on two NVIDIA A40 GPUs.
Current work in culture-aware NLP addresses various aspects. One line of work focuses on cul- tural stereotypes and biases, and ways mea- sure and mitigate them (e.g.. Hutchinson et al., 2020; Ross et al., 2021; Søgaard, 2022). Another line of work analyzes the differences in culture- specific commonsense knowledge, including rela- tional knowledge (Yin et al., 2022), grounding of time expressions (Shwartz, 2022), food-related cus- toms (Palta and Rudinger, 2023) and social values (Lin et al., 2021: Arora al., 2023). At the same time, there have been efforts to develop bench- marks (Yin et al., 2021; Liu et al., 2021), and adapt models to new cultures (Zhou et al., 2023; Yin et al.,
COMET Training. We proceed fine-tuning GD-BART on the large-scale ATOMIC-2020 dataset, using the same training method and hyper- parameters as Hwang et al. (2021). Appendix A lists the 34 COMET relations used in this paper.
4 Intrinsic Evaluation
To evaluate the quality of GD-COMENT -O-MOMET, we construct a set of input sentences pertaining to diverse cul-

---

5
Extrinsic Evaluation
Average
India
2.32
2.16
2.65
0.71
Traditional benchmarks often fall short in testing models' knowledge and comprehension of diverse cultural contexts. To show GD-COMET's 's utility for downstream tasks, evaluate
COMET
S Korea
1.93
1.86
2.32
0.67
Nigeria
1.97
1.98
2.27
0.61
Iran
2.09
2.31
2.42
0.63
multimodal task, GD-VCR (Sec 5.1). We develop model inspired by VIC--ERT (Ravi et al., 2023a) that generates inferences and incorporates them into a vision and language &&L) model (Sec 5.2). We show that GD-COMET improves the performance on GD-VCR upon array of baselines (Sec 5.3) and demonstrate the inferences contributing to the performance gains (Sec 5.4).
Indonesia
2.28
2.36
2.55
0.66
Average
GO-COMET
India
2.62
2.54
2.73
0.74
S Korea
2.13
1.92
2.35
0.65
Nigeria
2.25
1.92
2.35
0.59
2.27
2.38
0.76
Indonesia
2.43
2.46
2.58
0.77
Table I: Evaluation of COMET and GD-COMET infer- ences, judged by annotators from the respective cultures.
5.1 Dataset
Visual Commonsense Reasoning (VCR; Zellers et al., 2019) is a benchmark for testing V&L mod- els' ability to understand and reason beyond a vi- sual scene. Each example consists of an image extracted from movies or TV series and a multiple- choice question about the actions or people de- picted in the image. This dataset focuses solely on Western, primarily North American movies.
tures (Table 1). We sample 5 concepts for each facet and use facet-specific templates (Appendix B) create 20 sentences for each culture. For each of COMET and GD-COMET, we use beam search to generate inferences for each of the 34 dimensions and convert them natural language statements us- ing relation-specific templates based on prior work (Bosselut et al., 2019). The correctness of both in- ferences were judged by 10 graduate students, two students from each of the respective cultures. An- notators were asked to grade inferences along the following criteria scale of 0 (worst) to 3 (best):
Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR; Yin et al., 2021) follows the same setup of VCR but extends diverse regions. This evaluation-only dataset includes 328 images from movies and TV series in East Asian, South Asian, African and Western countries (See Ap- pendix C). We follow the original setup and train our model on VCR before testing on GD-VCR.
① Cultural Relevance: The inference is factu- ally accurate and reflects the values, customs, traditions, and societal norms associated with the given culture.
5.2 Model (VLC-BERT with GD-COMET)
2 Stereotype Avoidance:
The inference does not perpetuate stereotypes about the culture.
We take inspiration from VLC-BERT (Ravi et al., 2023a), that incorporated COMET inferences into VL-BERT (Su et al., 2019). Instead, we integrate GD-COMET as a source of contextualized cultural commonsense knowledge for GD-VCR. Figure 2 illustrates the model. We describe below VLC- BERT and where our model deviates from it.
3 Linguistic Accuracy:
The inference is gram-
matical, and the vocabulary and idiomatic pressions appropriate in that culture.
The annotations yielded a substantial inter- annotator agreement with 0.656 for COMET and 0.702 for GD-COMET, measured with average Cohen's Kappa (Cohen, 1960) across cultures.
VLC-BERT Tranaformer using CD-COMET
Results. Table reveals that GD-COMET consis- tently outperforms the standard COMET model. Specifically, GD-COMET excels in generating cul- turally aligned inferences across chosen diverse cultures, and is more likely than COMET to avoid biased assumptions. However, there is still room for improvement for South Korea and Nigeria.
Figure 2: A model using GD-COMET for GD-VCR.

---

VLC-BERT GD-COMET
Datasets
Human
VisualBERT*
ViLBERT
VL-BERT
GD-BART
COMET
GD-VCR
88.84
53.27
58.47
58.63
52.69
59.59
63.51
West
91.23
65.82
64.37
65.27
57.69
66.78
69.93
South Asia
92.98
52.04
62.9
64.92
54.35
64.25
68.17
Africa
87.93
51.85
62.04
58.17
51.87
57.71
64,81
East Asia
83.05
45.39
46.45
47.88
41.87
49.64
53.07
Table 2: Accuracy (%) of the different models the subset of each region in GD-VCR. We report the average
(see Appendix for the results of individual seeds). Results marked with have been reported in Yin (2021).
Knowledge Generation and Selection. VLC-
VLC-BERT w/COMET modestly improves upon VL-BERT across most regions, with an overall im- provement of 1.2 points in accuracy. This sug- gests that COMET provides some commonsense inferences that are universal. Conversely, GD- COMET shows a substantial improvement of nearly 5 points over VL-BERT and 4 points over VLC- BERT w/COMET. This highlights the effectiveness of incorporating GD-COMET for downstream tasks that require culture-specific knowledge across di- verse regions. Furthermore, GD-BART performs less effectively than other methods, underscoring the importance of training on structured knowledge to generate contextually relevant responses.
BERT uses the question and the object tags as input to COMET. Instead of object tags, we generate an image caption using BLIP (Li et al., 2023) and extract noun phrases from the caption using SpaCy (Honnibal et al., 2020). We found that the noun phrases provide more detailed description of the depicted activities within the image (e.g. "family. burn" in Fig. 2). We additionally append a country tag the input. During training on VCR, we use the tag "North America", the primary source of movies in the dataset. For the images in GD-VCR, we extracted country tags from Wikipedia.
We use beam search to generate five inferences for each of the 34 dimensions. To select the most relevant inferences,
5.4 Qualitative Analysis
convert the inferences to natural language sentences using relation-specifer templates and select the inferences that are the most similar to the question using SBERT embeddings (Reimers and Gurevych, 2019).
Figures 3 presents several GD-VCR instances along with the models' predictions, and the infer- ences generated by COMET and GD-COMET for them. In Figure 3a, GD-COMET accurately asso- ciates girl wearing henna in Somalia with mar- riage. In Figure 3b, it understands that folding palms during an Indian festival signifies a greet- ing or welcome. Finally, in Figure 3c, recog- nizes that bowing in South Korea gesture of apology, leading to VLC-BERT w/ GD-COMET to be the only model that provides correct answer. In contrast, COMET's inferences for this exam- ple are generic and irrelevant. These examples highlight GD-COMET's effectiveness in identifying the cultural context and dynamically generating culturally-relevant commonsense inferences across ATOMIC's relations.
Overall Architecture. The generic input VL- BERT for VCR is <question, answer tokens, image regions>. Following Ravi et al. (2023a), we embed each inference with SBERT and sum- marize them into a single token with a weighted average based on learned attention scores. Finally, we feed the output of the [CLS] token into a classi- fier to predict score for each answer choice. We train the model using binary cross-entropy loss for 20 epochs on 4 NVIDIA RTX6000 GPUs.
5.3 Results
Table 2 compares our model's 's performance on GD- VCR with baselines that: (i) do not make use of commonsense knowledge (VL-BERT); (ii) gen- crate inferences using GD-BART; and (iii) use COMET (VLC-BERT w/COMET). Note that the same signals (i.e., country tag and noun phrases) were used for the GD-BART and COMET base- lines. We also include prior results reported us- ing VisualBERT and ViLBERT for completeness.
Conclusion
This work challenges the current notion of uni- versally applicable commonsense knowledge by introducing GD-COMET, geo-diverse variant of COMET. GD-COMET can generate culturally- nuanced commonsense inferences for a broad range of cultures. Our comprehensive evaluation con-

---

perccoz
Question Why does presond want honna pointing
Question Why does peesond Said her palms?
Qsestion way gaasent
and persond bowing?
Seap co AERT
Maa COBEET
Thlny connct
Usira 23 BART
Usbg CAM
Uhine CEMET
ELETTT
Using E3
Vsey S1 COMET
Extracted gid, bezna
Country: Somelaan
feationl, women
Country: India
Extracted MPs: mea suits
Country: South Korea
Toys Inferenam CONST
Dogos Jabnnzess OCOAET
Doya Iafiemone COOKEZ
Maeee Perssx b0y
(0.32)
musital 20:00)
wasted get
pmeety B 4N
greating (B 32)
This Pesmica Lave good
0..31
Dedace Pe gideeitse O6
201
mantee.
(8.55]
(3.31)
Postuct ain getling reanted, (f.4
syalnggian PernooT
(0.30)
gerllng tallen.
(8.10)
Cootival (6.12)
Inaated 3. shadiom (8.10)
320
colea... (0.80)
(9.00)
Pasce oo the tarple, (8.14)
Eadaa hogen (5.50)
sdeded desse up. (9.10)
Perso
This
Iadia
powunti
(0.00)
Pertuo go kint Roass (6.1))
Poraatt
Figure 3: Attention analysis of commonsense inferences generated by COMET and GD-COMET for testing samples in GD-VCR.
firms the effectiveness of GD-COMET in incorpo- rating and leveraging cultural cues. We view our work as a step towards developing more inclusive and culturall-awaae AI systems.
to go beyond testing models on knowledge about concepts from diverse cultures to understanding cultural nuances.
Ethics Statement
Limitations
While GD-COMET represents a significant ad- vancement in incorporating cultural commonsense knowledge into AI models, few limitations need to be acknowledged.
Despite being designed to be more culturally inclu- sive, GD-COMET runs the risk of unintentionally perpetuating biases present in CANDLE data. In particular, CANDLE might misrepresent cultures with stereotypes or underrepresent cultures. Ad- dressing these concerns requires proactive mea- sures such as identifying biases using methods such as Mehrabi et al. (2021) and mitigating them through filtering and additional data collection.
First, the availability of comprehensive, high- quality data remains a challenge in training culturally-aware models. While resources like CANDLE provide a step forward in curating di- verse cuitural knowledge, it is essential to note that merely capturing the existence of concepts within a culture is insufficient. Future efforts should aim to collect data that reflects the presence of certain con- cepts and encompasses how people perceive and interpret those concepts within their specific cul- tural contexts. This would require extensive data collection efforts that go beyond surface-level un- derstanding, and delve into the nuances of cultural perspectives.
Additionally, the size evaluation benchmarks means they don't always account for cultural vari- ations within the same region. For example, GD- VCR images in the African region are concentrated in East Africa. Similarly, addressing this issue would require additional annotation efforts.
Acknowledgement
A second limitation is the availability of suitable benchmarks for testing models' knowledge and un- derstanding of cultural variations. In particular, two such tasks, GD-VCR and MarVL (Liu et al., 2021), focus on vision and language, while Nguyen et al. (2023) proposes a cultural knowledge quiz. We hope to see more language-only datasets developed
This work was funded, in part, by the Vector Insti- tute for AI, Canada CIFAR AI Chairs program, NSERC discovery grant, and research gift from A12. Finally, we sincerely thank Sahithya Ravi, Aditya Chinchure, Ward Pennink and Jan Zimny for valuable feedback and discussions.

---

References
Daniel Hersheovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross- cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guisties (Volume I: Long Pupers), pages 6997-7013, Dublin, Ireland. Association for Computational Lin- guistics.
Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2021. Towards an atlas of cultural com- monsense for machine reasoning. In Proceedings of the Workshop on Common Sense Knowledge Graphs (CSKGs) at AAAI
Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated sto- rytelling via causal, commonsense plot ordering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5859-5867.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, Adriane Boyd, et al. 2020. spacy: Industrial- strength natural language processing in python.
Arnay Arora, Lucie-aimée Kaffee, and Isabelle Augen- stein. 2023. Probing pre-trained language models for cross-cultura differences in values. In Proceedings of the First Workshop on Cross- Cuss-Culuurra Considera- tions in NIP (C3NLP), pages 14-130,, Dubrovnik, Croatia. Association for Computational Linguistics.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 4591-5501, Online. Association for Computational Linguistics.
Antoine Bosselut, Hammah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (COMET-JATOME) On symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2020:
Yong Cao, Li Zhou, Scolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-culurral alignment between ChatGPT and hu- man societies: An empirical study. In Proceedings of Cross-Cultural Cultural Considerations in NLP (C3NLP), pages 53-67, Dubrovnik, Croatia. Association for Computational Linguistics.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprini arXiv:2301.1 12597.
the First Workshop on Cross-
Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xi- ang Ren. 2021. Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 1274-1288 Online. Association for Computational Linguisties.
Tuhin Chakrabarty, Dehanjan Ghosh, Smaranda Mure- san, and Nanyun Peng. 2020. R33 Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7976-7986, Online. Association for Computa- tional Linguisties.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-46
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier. and Desmond EI- liott. 2021. Visually grounded reasoning across lan- guages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. 2022. StereoKG: Data-driven knowledge graph construction for cultural knowledge and stereotypes. In Proceedings of the Sixth Work- shop on Online Abuse and Harms (WOAH), pages 67-78, Seattle, Washington (Hybrid). Association for Computational Linguistics.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pu- jara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 2021 Conference on Empirical Methods in Nat- wral Language Processing, pages 5016-5033, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, and Yejin Choi. 2021. Paragraph-Ievel commonsense transformers with recurrent memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12857-12865.

---

Tuan-Phong Nguyen, Simon Razniewski, Aparn Varde, and Gerhard Weikum. 2023. Extracting cultural com- monsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, WWW '23, page 1907-1917, New York, NY, USA. Association for Computing Machinery.
pages 2842-2853, Dublin, Ireland, Association for Computational Linguistics.
Anders Søgaard. 2022. Should we ban English NLP for a year? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5254-5260, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Lingwisties: ACL 2023, pages 9952-9962, Toronto, Canada. Associa- tion for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mot- taghi, Ali Farhadi, and Yejin Choi. 2020. Visual- comet: Reasoning about the dynamic context of still image. In Proceedings of the European Con- ference on Computer Vision (ECCV).
James P Spradley. 1987. Culture and cognition: Rules, maps, and plans. Waveland PressInc.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre- training generic visual- visual--inguistic representations. arXiv preprint 7008::900.080
Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023a. Vlc-bert: Visual question answering with contextualized common- sense knowledge. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, pages 1155-1165.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4625 Seat- tle, United States. Association for Computational Linguistics.
Sahithya Ravi, Raymond Ng, and Vered Shwartz. 2023b,
Comet-m: Reasoning about multiple events in com- plex sentences. arXiv preprint arXiv:2305. 14617.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP", pages 3982-399 -992, Hong Kong, China. Association for Com- putational Linguistics.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai-We Kai-Wei Chang. 2022. GeoM- LAMA: Geo-diverse commonsense probing on multi- lingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Language Processing, pages 2039-2055 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Julien Romero, Simon Razniewski. Koninika Pal, Jeff Z. Pan, Archit Sakhadeo, and Gerhard Weikum. 2019. Commonsense properties from query logs and ques- tion answering forums. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1411 1-120..
Da Yin, Feng Gao, Govind Thattai, Michael John- ston, and Kai-Wei Chang. 2023. Givl: Improv- ing geographical inclusivity of vision-language mod- els with pre-training methods. arXiv preprint arXiv:2301.0188.
Candace Ross, Boris Katz, and Andrei Barbu. 2021. Measuring social biases in grounded vision and lan- guage embeddings. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 998-1008, Online. Asso- ciation for Computational Linguistics.
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. 2021. Broaden the vision: Geo- diverse visual commonsense reasoning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2115-2129, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi, 2019. Atomic: An atlas of machine commonsense for if- then reasoning. In Proceedings of the AAA/ con- ference on artificial intelligence, volume 33, pages 3027-3035.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Vi- sual commonsense reasoning, In Proceedings of the IEFF/CVF conference on computer vision and pat- tern recognition, pages 6720-6731.
Vered Shwartz. 2022. Good night at 4 pm?! time ex- pressions different cultures. In Findings of the As- sociation for Computational Linguistics: ACL 2022,
Li Zhou, Laura Cabello, Yong Cao, and Daniel Her- shcovich, 2023. Cross-cultural transfer learning for

---

Chinese offensive language detection. Proceed- ings of the First Workshop on Cross- Cultural Consid- erations in NLP (C3NLP), pages 8-15, Dubrovnik, Croatia, Association for Computational Linguisties.
Cross-Cultural
COMET Relations
Table 3 lists COMET relations used in this work.
Attlocation
Copableor
1sBefore istiileddy
Couses
Cousestesire
CreatedBy
Desires
offfect ofeact
HasPreraqulsite
RadFi rstSubevent
HasProperty
Instanceof
ISA
xAttr
LozatedNear
MadeDf
xE ffect
Madelpaff
Motivatadiytiaa
xintent
ObjectUse
10f
xNeed
Rece RecelvesAction
Symbol0r
xReact
UsedFor
isAfter
xReason
Table COMET relations used in this work.
B Facet Templates
Table presents the templates used for creating input sentences to GD-COMET for each concept associated with a cultural facet.
clothing
"PersonX wears [cancept] [cauntry]*
Food festival
"PersonX eats Lconcept] in [country]"
"PersonX [cancept] [country]*
"PersonX celebrates [soncept] in [country]"
Table 4: Templates used to create input sentences for GD-COMET for each CANDLE facet.
C VCR and GD-VCR Statisties
Table 5 displays the statistics of the VCR and GD- VCR datasets. The bottom half shows the number of images for cach region in GD-VCR.
Images
avg Q length
arg. length
VCR (dev)
9929
26534
6.77
7.67
GD-VCR
328
886
7.38
7.68
West
100
275
7.36
7.19
Asia
South Asia
87
221
6.85
8.00
Africa
40
108
7.98
8.54
Table 5: Statistics the VCR and GD-VCR benchmarks.
D Full Performance
Extending upon Table 2, we provide a complete summary of the results individual seeds on GD- VCR in Table 6.

---

Models
Overall
West
South Asia
East Asia
Africa
Human Performance
88.84
91.23
92.98
83.05
87.93
VisualBERT ViLBERT
53.27
62.91
52.04
45.39
51.85
58.47
65.82
62.9
46.45
62.04
VL-BERT (seed 1)
58.8
64,73
67.42
48.58
54.78
VL-BERT (seed 2)
58.92
65.82
63.8
47.16
62.04
VE-BERT (seed 3)
58.19
65.27
63.54
47.91
59.7
VL-BERT (average)
58.63
65.27
64.92
47.88
58.17
VEC-BERT with GD BART (seed 1)
53.63
58.54
54.08
41.42
$2.78
VLC-BERT with GD-BART (seed 2)
52.92
$7.27
54.08
42.09
$2.13
VEC-BERT with GD BART (seed 3)
51.55
57.27
54.89
42.09
$0.70
VLC-BERT with GD-BART (average)
52.69
57.69
54.35
41.87
51.87
VLC-BERT with COMET (seed
59.71
67.27
64.71
50.00
55.56
VLC-BERT with COMET (seed 2)
59.82
67.27
64.71
50.36
55.56
VLC-BERT with COMET (seed
59.25
65.82
63.35
48.58
62.04
BEC-BERT with COMET (average)
59,59
66.78
64,25
49.64
$7.71
VICC-ERT with GD-COMET (seedl)
62.87
67.64
70.14
51.77
64.81
VLC-BERT with GD-COMET (sced2)
65.01
72.36
67.42
$4.97
67.59
VLC-BERT with GD-COMET (seed3)
62.64
69.82
66.97
$2.48
62.03
VLC-BERT with GD-COMET (average)
63.51
69.93
68.17
53.07
64.81
Table Accuracy (%) of the various models on the subset of each region in GD-VCR, Results marked with have been reported in Yin et al. (2021).


Reviews:
[{"review_id": "NzBwORwY27", "reviewer": "Reviewer_La5x", "summary": "", "questions": "A. How does the original COMET perform when adding the country information as in Section 4?  Is there a human evaluation done on that as well to show the GD-COMET generates better assertions than the baseline? \n\nB. In section 5.4, what I've seen in CANDLE is \"In Korea, the bow is the standard greeting.\" and nothing about apology. Is the qualitative example just the effect of the randomness of the training starting point?\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s time and effort in providing valuable feedback on our paper. We have carefully considered each of the comments and suggestions and have made appropriate revisions to address the concerns raised. Below, we outline our response:\n\n**Figures are blurry. Visual effects like the pencil brush and shadow make the figures a little bit more messy.**\n\nWe enhanced the resolution of the figures and updated them in the paper.\n\n**At the beginning of section 5, add a short explanation of why a multi-modal setting is used as the downstream task. (Maybe move a few sentences from limitation to the main paper.)**\n\nWe added a brief explanation at the beginning of Section 5 to elucidate the rationale for employing a multi-modal setting in the downstream task.\n \n**Question A. How does the original COMET perform when adding the country information as in Section 4? Is there a human evaluation done on that as well to show the GD-COMET generates better assertions than the baseline?**\n\nThanks for pointing out the missing human evaluation of COMET. We repeated the same intrinsic evaluation for the outputs of the original COMET model. Across all five cultures, we see average scores of 2.118, 2.134, and 2.442 for our three metrics (compared to average scores of 2.34, 2.244, and 2.518 of GD-COMET assertions). This human evaluation reassesses that GD-COMET generates improved cultural-specific assertions compared to vanilla COMET. The results will be added in Section 4 in the camera-ready version. \n\n**Question B. In section 5.4, what I've seen in CANDLE is \"In Korea, the bow is the standard greeting.\" and nothing about apology. Is the qualitative example just the effect of the randomness of the training starting point?**\n\n**But for the GD-VCR downstream task, it is not very likely that the extracted noun phrases hit into the concepts in CANDLE, and there may need some extra explanations.**\n\nWe appreciate the attention to detail! We would like to clarify that while the specific example mentioned might not be explicitly present in the CANDLE dataset, it is not solely attributed to the randomness of the training starting point. Instead, it underscores one of the strengths of the GD-COMET model. Similar to COMET, GD-COMET can dynamically generate novel and relevant (and culturally aligned, unlike COMET) commonsense inferences across ATOMIC's relations. We will clarify this in the paper that, in essence, our approach encourages the model to grasp a foundational cultural understanding, allowing it to make inferences that might not be explicitly present in CANDLE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The cultural-awareness aspect of LM is an issue that requires more collected effort of the community.  This work provide a validated useful tool for followed-up researches (We've seen how COMET is widely-used for commonsense related tasks).\n\n2. The evaluation is comprehensive. An human evaluation is carried out, instead those meaningless automatic metrics. The model is also tested on a multimodal downstream task as the knowledge generation module, where there is a decent improvement of vanilla COMET. \n\n3. The content is well-presented with a few illustrations that makes the paper easy to read.", "reasons_to_reject": "1. For the intrinsic evaluation, the baseline of the original COMET is missing, which is very crucial. Otherwise, we don't have a quantitative proof (on the intrinsic side) that the model generates better cultural-aware commonsense knowledge than the original model. \n2. There is a lack of analysis or intuition presented on the effect of pretraining for different COMET inputs (especially containing cultural-specific concepts or not).  To be more specific, CANDLE is mainly about noun concepts around clothing, food & drink, and some traditions. But for the GD-VCR downstream task, it is not very likely that the extracted noun phrases hit into the concepts in CANDLE, and there may need some extra explanations. \n", "questions_to_authors": "A. How does the original COMET perform when adding the country information as in Section 4?  Is there a human evaluation done on that as well to show the GD-COMET generates better assertions than the baseline? \n\nB. In section 5.4, what I've seen in CANDLE is \"In Korea, the bow is the standard greeting.\" and nothing about apology. Is the qualitative example just the effect of the randomness of the training starting point?\n\n", "paper_topic_and_main_contributions": "This paper presents a way to improve commonsense knowledge models, such as COMET, into a Geo-diverse aware model. \n\nThe author pretrains the backbone LM on culture commonsense assertions and then trains on ATOMIC-2020 as the original COMET did. The culture commonsense originates from CANDLE, which is an existing dataset collected through web mining and consists of assertions explicitly with country names. \n\nAs for the evaluation, the author did a human evaluation to test the model's ability to generate cultural-relevant commonsense assertions. The author also tests their model with a multi-modal downstream task, which is Geo-diverse visual commonsense reasoning, using their model as the knowledge generation component. \n\nMain Contribution:\n1. The work provides a cultural-aware knowledge model that can be easily used. \n2. This paper validates the intuition that pretraining LM on cultural commonsense assertions can improve the model's cultural-awareness, which is evaluated both on the assertion generation itself and a multi-modal downstream task. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figures are blurry. Visual effects like the pencil brush and shadow make the figures a little bit more messy.\n2. At the beginning of section 5, add a short explanation of why a multi-modal setting is used as the downstream task. (maybe move a few sentences from limitation to the main paper)"}}, {"review_id": "rTa5aaxOD4", "reviewer": "Reviewer_J9XN", "summary": "", "questions": "Lines 134-137: Do you think it makes a difference whether you first train on 1 vs. 2?\nLines 145-156: Do you specify the region when fine-tuning?\nLines 165-168: will you publish the evaluation set?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thorough review and valuable feedback, and we are grateful for your recognition of the importance of training culturally aware NLP models. We have carefully considered each of the points raised and have made corresponding adjustments to enhance the quality and clarity of our paper. Below, we address the reviewer's comments:\n\n**Lines 134-137: Do you think it makes a difference whether you first train on 1 vs. 2?**\n\nThis is an interesting question.  We did not study the effect of reversing the order of training phases since we hypothesized that if we train on ATOMIC-style data first as the first step and later on CANDLE’s unstructured data, we might lose the ability to generate commonsense inferences along ATOMIC’s relations. In the future, we will explore more advanced scheduling and multi-task training techniques. \n\n**Lines 145-156: Do you specify the region when fine-tuning?** \n\nWe do not explicitly add a region tag again when fine-tuning BART with CANDLE data. The reason is that most sentences in CANDLE data already contain the region name (e.g. “Paneer is a type of cheese that is popular in Indian cuisine”). We will clarify this and include some examples from CANDLE in the camera-copy version. \n\n**Lines 165-168: Will you publish the evaluation set?** \n\nYes!\n\n**Missing References** \n\nThank you for the missing reference! We became aware of FORK at ACL’23 and will make sure to cite it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper discusses an important, and often neglected aspect of NLP: how to to train culturally pluralistic models. It offers an interesting approach towards solving this issue and convincingly evaluates the model on intrinsic and extrinsic tasks. The evaluation seems to be well thought out and they also make sure to check whether the inference does perpetuate stereotypes about a given culture. ", "reasons_to_reject": "Besides the clarification questions asked below I don’t see any obvious risks with having this paper accepted. For future work it would be interesting to see an analysis on why the model does better on certain cultures than others.", "questions_to_authors": "Lines 134-137: Do you think it makes a difference whether you first train on 1 vs. 2?\nLines 145-156: Do you specify the region when fine-tuning?\nLines 165-168: will you publish the evaluation set?\n", "paper_topic_and_main_contributions": "This paper trains a culturally-aware commonsense inference model. It does so by first training a LM on geo-diverse data, which it obtained from CANDLE (a cultural KB). The model then gets further trained on COMET training data. The paper evaluates their model intrinsically using human evaluation and extrinsically on a visual commonsense reasoning. In both cases they show that their model produces more culturally aligned outputs compared to other approaches.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models\nShramay Palta, and Rachel Rudinger\nFindings of the Association for Computational Linguistics: ACL 2023\n\n(might be concurrent work, but in any case would be good to cite it for the camera-ready)"}}, {"review_id": "8YJUDc5jCo", "reviewer": "Reviewer_3GPm", "summary": "", "questions": "1.  In Section 4, could you provide some examples of the inference statements used for annotation and explain in more detail how the \"culture relevance\" is scored? It is not clear in the paper whether the relevance score indicates correctness or just topical relevance.\n\n2. What is your intuition regarding the difference in scores among the selected cultures?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the detailed and thorough review of our paper and will use the constructive feedback to enhance the quality of our work. Below, we provide our responses to your comments: \n\n**In Section 4, could you provide some examples of the inference statements used for annotation and explain in more detail how the \"culture relevance\" is scored? It is not clear in the paper whether the relevance score indicates correctness or just topical relevance.**\n\nThe \"culture relevance\" score aims to assess the extent to which the inference generated by our model aligns with the values, customs, traditions, and societal norms associated with a specific culture. In essence, it is correct to consider both correctness (i.e., being factually accurate) and topical relevance (i.e., pertinent to the culture's themes, practices, or historical background). We will add the specific instructions and examples that were provided to the annotators to the appendix. \n\n**I think the proposed mitigating method (pretraining on CANDLE data) is simple, which is ok, but there is a lack of analysis regarding how and why pretraining on CANDLE contributed to the geo-diverse commonsense inference.**\n**(1) The intrinsic evaluation section lacks a comparison of how the baseline models performed, making it difficult to determine whether GD-COMET improved cultural-specific inference.** \n\nAs also requested by Reviewer 1, we repeated the intrinsic evaluation using the original COMET model. Across all five cultures, we see average scores of 2.118, 2.134, and 2.442  for our three metrics (compared to average scores of 2.34, 2.244, and 2.518 of GD-COMET assertions), supporting our claim that GD-COMET demonstrates improved cultural-specific assertions compared to vanilla COMET. We will update the paper with the results. We also believe that this evaluation provides additional empirical evidence on how CANDLE contributes to generating culturally-aligned inferences. \n\n**(2) The extrinsic evaluation looks more sound, but it is unclear whether the same signals were used for the VLC-BERT w GD-BART or COMET baselines.**\n\nYes, We used the same inputs, i.e. country tag and noun phrases, for training both other baselines.  We will clarify this in the paper.\n\n**(3) I think, in general, the paper could benefit from evaluating more datasets if possible, as the dataset used in the extrinsic evaluation is small.** \n\nAs we mention in the limitation section, there are very few appropriate benchmarks for assessing models' cultural knowledge and comprehension.  However,  given that this topic is now gaining popularity and new datasets are being released, we plan to test GD-COMET on new datasets such as Fork (Palta and Rudinger, 2023) in the future.\n\n**What is your intuition regarding the difference in scores among the selected cultures?**\n\nWe appreciate the insightful question about the observed score variations among the selected cultures in our evaluation. As can be seen by comparing the results of VL-BERT in Table 2, the improvement from GD-COMET is relatively uniform across cultures, suggesting that the gap originates in the underlying LM (i.e. BERT) itself. Understanding why models represent some cultures better than others is an important question that we plan to address in future work. We envisage that there are several factors, such as the availability of cultural data, cultural complexity and diversity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper focus on an interesting problem, that commonsense knowledge should be cultural-specific.They proposed a simple solution to mitigate the problem, which is to pre-train the common sense inference model on geo-diverse statements.\n2. The problem statement and description of their method are clear.\n", "reasons_to_reject": "1. I think the proposed mitigating method (pretraining on CANDLE data) is simple, which is ok, but there is a lack of analysis regarding how and why pretraining on CANDLE contributed to the geo-diverse commonsense inference. \n\n2. The evaluation conducted in the paper is not entirely convincing:\n(1) The intrinsic evaluation section lacks a comparison of how the baseline models performed, making it difficult to determine whether GD-COMET improved cultural-specific inference.\n(2)  The extrinsic evaluation looks more sound, but it is unclear whether the same signals were used for the VLC-BERT w GD-BART or COMET baselines. For example, the authors introduced additional signals such as country tag and extracted NPs when evaluating with GD-COMET, and it was not clear if the same signals have been used for the other two baselines. \n(3) I think in general the paper could benefit from evaluating on more datasets if possible, as the dataset used in the extrinsic evaluation is small. \n\n3. The paper does not provide an explanation for the difference in model performance across the selected cultures in both human and downstream task evaluations.\n", "questions_to_authors": "1.  In Section 4, could you provide some examples of the inference statements used for annotation and explain in more detail how the \"culture relevance\" is scored? It is not clear in the paper whether the relevance score indicates correctness or just topical relevance.\n\n2. What is your intuition regarding the difference in scores among the selected cultures?\n", "paper_topic_and_main_contributions": "This paper addresses the issue that the commonsense inference model COMET was trained on English data based on Western culture, which limits its capability to perform inference accurately for other cultures. To overcome this, the authors propose to finetune the underlying model of COMET (BART large) on the CANDLE dataset before finetuning it on the ATOMIC dataset, resulting in their GD-COMET model that can leverage geo-diverse knowledge. They performed a human evaluation on GD-COMET-generated inference statements and also conducted experiments to show that GD-COMET could be helpful on a GD visual commonsense reasoning task, by incorporating GD-COMET-generated inference into the input of a VLC-BERT model. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide a well-written paper that addresses an important issue in NLP, the cultural-awareness of commonsense knowledge models. The proposed GD-COMET, a culturally-aware generative COMET model, appears a valuable contribution to the field, and the (for a short paper) comprehensive evaluation of its benefits is a strength of the paper. There are a few areas where the paper should be further improved, such as providing a more detailed analysis of the effect of pretraining on CANDLE, and comparing the baseline models' performance in the intrinsic evaluation. Additionally, the paper could benefit from evaluating on more datasets if possible. The authors have provided detailed and helpful responses to the reviewers' comments which would allow them to address most concerns raised for the final version. Overall, this paper appears a valuable contribution to the discourse in the field.
Decision: Accept-Main