Culturally Aware Natural Language Inference
Authors: Jing Huang
Diyi Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: cultural norms
natural language inference

Abstract: Humans produce and consume language in a particular cultural context, which includes knowledge about specific norms and practices. A listener's awareness of the cultural context is critical for interpreting the speaker's meaning. A simple expression like *I didn't leave a tip* implies a strong sense of dissatisfaction when tipping is assumed to be the norm. As NLP systems reach users from different cultures, achieving culturally aware language understanding becomes increasingly important. However, current research has focused on building cultural knowledge bases without studying how such knowledge leads to contextualized interpretations of texts. In this work, we operationalize cultural variations in language understanding through a natural language inference (NLI) task that surfaces cultural variations as label disagreement between annotators from different cultural groups. We introduce the first Culturally Aware Natural Language Inference (CALI) dataset with 2.7K premise-hypothesis pairs annotated by two cultural groups located in the U.S. and India. With CALI, we categorize how cultural norms affect language understanding and present an evaluation framework to assess at which levels large language models are culturally aware. Our dataset is available at https://github.com/SALT-NLP/CulturallyAwareNLI.

Culturally Aware Natural Language Inference
Jing Huang, Diyi Yang Stanford University Department of Computer Science [hij, diyiy]@cs. . stanford. edu
Abstract
P: "Did he leave tip for the waiter at the restaurant?" His friends started laughing quietly as they asked.
Humans produce and consume language in a particular cultural context, which includes knowledge about specific norms and practices. A listener's awareness of the cultural context is critical for interpreting the speaker's mean- ing. A simple expression like "/ didn't I leave a tip" implies a strong sense of dissatisfaction when tipping assumed be the norm. As NLP systems reach users from different cul- tures, achieving cultrally aware language un- derstanding becomes increasingly important. However, current research has focused on build- ing cultural knowledge bases without study- ing how such knowledge leads to contextu- alized interpretations of texts. this work, we operationalize cultural variations in lan- guage understanding through natural lan- guage inference (NLI) task that surfaces cul- tural variations label disagreement between annotators from different cultural groups. We introduce the first Culturally Aware Natural Language Inference (CALI) dataset with 2.7K premise-hypothesis pairs annotated by two cul- tural groups located the U.S. and India. With CALL, categorize how cultural norms affect language understanding and present an evalua- tion framework to assess which levels large language models are culturally aware. Our dataset is available at https://g thub. com/ SALT-NLP/Cul tural MurallyAwareN.I.
H2: His friends thought he did not leave for the waiter,
H1: friends did not know whether left tip not.
Listeners from culture where tipping is optional
Listeners from culture where tipping is customary
An example of how cultural differences in tipping lead different interpretations. We operationalize cultural variations in language understanding with NLI task, where cultural variations measured by label disagreement.
Figure 1:
As NLP systems reach billions of users across cultural boundaries, building culturally aware lan- guage models is an emerging requirement (Hovy and Yang, 2021; Hershcovich et al., 2022). By the term "culturally aware", we specifically fer to three levels of awareness in language un- derstanding: (I) having knowledge of specific cul- tural norms; (2) recognizing linguistic context that invokes cultural norms; (3) accommodating cul- tural variations by making a culture -sstufff- infer- ence. A lack of cultural awareness can result in models with poor accuracy and robustness (Liu et al., 2021), social biases and stereotypes (van Miltenburg, 2016; Rudinger et al., 2017; Arora et al., 2023), and discrimination against annota- tors (Miceli and Posada, 2022). To incorporate cultural factors in NLP systems, recent work has built knowledge bases cultural symbols, prac- tices (Yin et al., 2022; Nguyen et al., 2023; Fung et al., 2022; Ziems et al., 2023; CH-Wang et al., 2023), and values (Johnson et al., 2022; Cao et al., 2023; Arora et al., 2023), along with methods to probe elicit such knowledge from LLMs.
1 Introduction
Language, as a tool of social interaction, is used in a cultural context that involves specific norms and practices. Cultural norms are behavioral rules and conventions shared within specific groups, connect- ing cultural symbols and values (Hofstede et al., 2010). They provide contextual knowledge to inter- pret the meanings behind actions and words. For example, when tipping is customary, simple ex- pression like "I didn't leave a rip" implies strong sense of dissatisfaction, while in a culture where tipping is optional, the implicature no longer holds.
In this work, we take step towards culturally aware language understanding. Unlike previous work focusing on building knowledge bases of cul-

---

tural norms and practices-level (1), we focus on how people make different inferences based on cultural knowledge, i.e., level (2) and (3). We operationalize cultural variations in language un- derstanding through an NLI task that measures cultural variations as label disagreement between annotators from different cultural groups. We in- troduce CALL, the first Culturally Aware Natural Language Inference dataset!, with 2.7K premise- hypotheses pairs. Each premise centered around a type of normative behavior. We ask annotators from two different cultural groups, one based in the United States and the other based in India, to label the entailment relationships in the context of their culture. The label variations between the two groups capture the cultural variations in language understanding. With the new dataset, categorize how cultural norms affect language inference (Sec- tion 4) and evaluate at which levels large language models are culturally aware (Section 5). We find despite LLMs having knowledge of many cultural norms, especially ones associated with the U.S. cul- ture, they lack the ability to recognize such norms in the NLI task, i.e., level (2), and adjust inference based on cultural norms, i.e., level (3). Our work highlights that cultural factors contribute label variations in NLI and cultural variations should be considered language understanding tasks.
1991). Recent work in NLP has studied differences in time expressions (Vilares and Gódmez-Rodriguez, 2018; Shwartz, 2022), perspectives over news top- ics (Gutiérrez et al., 2016), pragmatic reference of nouns (Shaikh et al., 2023), culture-specific entities (Peskov et al., 2021; Yao et al., 2023), figurative language (Kabra et al., 2023; Liu et al., 2023b). Our work connects the two lines of research by inves- tigating how cultural knowledge affects language understanding. Instead of studying a specific type of expression, we measure language understanding through generic NLI task.
Our work is also related to reasoning about cial norms. Social norm reasoning tasks involve social judgement (Forbes et al., 2020; Shen et al., 2022; Fung et al., 2022: Ziems et al., 2023), intents and consequences (Emelin et al., 2021; Rashkin et al., 2018), 8), and reference resolution (Abrams and Scheutz, 2022). A recent line of work has focused on the situational and defeasible nature of social norm reasoning (Rudinger et al., 2020; Ziems al., 2023; Pyatkin al., 2023). Similar to these prior works, we emphasize the culture-contineent nature of normative reasoning.
2.2 Natural Language Inference
Natural language inference (NLI) is one of the most fundamental language understanding tasks (Bow- et al., 2015). The task is determine whether the given hypothesis logically follows from the premise. Our work revisits NLI through a cultural lens. Cultural variations are first manifested in cross-lingual NLI (Conneau et al., 2018), moti- vated curating language-specific datasets (Artetxe et al., 2020; Hu et al., 2020). Our work focuses on the cross-cultural aspect. Unlike cross-lingual NLI captures culture task- our task measures cultural variations on the instance level.
2
Related Work
2.1 Cultural Factors and Norms
Modeling cultural factors in language has received increasing attention in the NLP community (Hovy and Yang. 2021: Hershcovich et al., 2022). Cul- ture serves as common ground in communica- tion, motivating the line of work to build cultural knowledge bases and probe cultural knowledge in LLMs, which includes commonsense knowledge (Yin et al., 2022; Nguyen et al., 2023; Keleg and Magdy, 2023; Palta and Rudinger, 2023), norms (Fung et al., 2022; Ziems et al., 2023; CH-Wang et al., 2023), values (Johnson et al., 2022; Cao et al., 2023; Arora et al., 2023; Ramezani and Xu, 2023), and knowledge cross modalities (Liu et al., 2021). More importantly, culture interacts with lan- guage, shaping what we convey and how we convey it In cross-cultural communication, cultural differ- ences cause misunderstandings of speakers' inten- tions (Thomas, 1983; Tannen, 1985; Wierzbicka,
The cultural-specific inference is related to the discussion on speaker meaning and implicature (Manning, 2006; Pavlick and Kwiatkowski, 2019). Implicatures are inferences likely true in given context. They are useful for commonsense reason- ing and pragmatic inferences (Gordon et al., 2012: Rudinger et al., 2020), both involve cultural norms. source of "human
Lastly, cultural variations label variations" (Plank, 2022). Along with work on uncertainty (Pavlick and Kwiatkowski, 2019; Chen al., 2020; Nie et al., 2020), ambiguity (Liu et al., 2023a), we challenge the assumption that there is a unique label per premise-hypothesis pair.
We de- de-emphasize "natural" as the majority of the exam- plas in the dataset are generated by large language models.

---

3 A Culturally Aware NLI Dataset
Besides norms that are likely differ between the two cultures, we also sample set of social norms from the NormBank (Ziems et al., 2023) that are likely consistent between the two cultures. This set allows us to measure label variations when cultural variations are not presented. We describe how to convert each norm into premise in Section 3.4.
We introduce a new NLI dataset that measures cul- tural variations in language understanding through label variations of the NLI task. We choose NLI as a starting point, as it is one the most fundamental and widely used natural language understanding tasks. The dataset generation framework is shown in Figure 2.
3.3 Representing Cultural Norms
Existing work on the computational representation of social norms converges the rule of thumb (RoT). However, for cultural norms, RoT does not provide a structured way to specify the context where the norm applies. For example, having soup for dinner is normal, but whether one should have soup before or after the main dish that cap- tures the cultural variation (Palta and Rudinger, 2023). Following carlier work on behavioral norms, we adopt the script representation (Schank and Abelson, 1977: Bicchieri, 2000), where behav- iors conforming achieve a goal. The context of an action hence can be represented by structured elements of the script, such as the goal of the actor, previous or next ac- tions, identities of the actors, objects involved, and changes in states. These structured elements cap- ture fine-nrained variations of norms, which allows us to generate premises and hypotheses targeting at these differences.
3.1 Task Definition
Given a cultural norm, we generate a narrative or an opinion related to the normative behavior as the premise and three statements about the goal or consequences of the behavior as hypotheses. The task is to infer the entailment relationship between the premise and the hypothesis under a particular cultural context.
Due to the culture-con calture-contingent nature of the in- ference, the entailment relationship is inherently an implicature that is always cancellable, i.e., true in some cultural context but false in others. Fol- lowing strict definition of entailment may lead to most examples being labeled as "neutral". Hence, in addition to an absolute entailment relationship, we also want to capture which hypothesis more likely to be truc, i.c. more plausible, in a given cultural context. We present multiple hypotheses per premise and measure the interpretation of a premise as ranking of entailment relationships over the set of hypotheses.
script form causal chain
3.4 Model-in-the-loop Generation
With the script representation, describe how to embed cultural norm into sentence context to generate a premise and hypothesis pair. The structured representation of cultural norm allows us to leverage LLMs to generate premises and hy- potheses, a promising approach to construct NI.I datasets (Liu et al., 2022; Chakrabarty et al., 2022). Specifically, we first prompt ChatGPT/GPT-3.5- turbo to generate premises and hypotheses from the script, followed by human editing on the gen- erated content. We discuss the generation pro- cess in details below and list all prompts in Ap- pendix A. For about 20% of norms, LLMs failed to generate suitable context. We discuss these cases in Appendix A.2. In these cases, We fall back to scarching for sentences containing the behav- ior from existing corpus, mainly through Google Books Ngrams Viewer As last resort, authors manually write about 5% premises and hypotheses.
3.2 Collecting Cultural Norms
A major challenge in measuring the influence of cultural norms on language understanding is to identify norms that have strong cultural variations. We survey the rich literature on cultural differences around social norms, especially the ones that study norm variations between the U.S. and India, such as tipping norms (Lynn and Lynn, 2004), dining etiquette (Hegde et al., 2018), wedding customs (Buckley, 2006), bargaining practices (Druckman et al., 1976), politeness (Valentine, 1996), etc. We also collect cultural norms from the Wikipedia corpus² and online forums where people from different cultures interact, such the Subreddit A Note that we only extract norms mentioned without directly including the post or comment from users our dataset.
hnttps://boos..got con/ngrams/

---

Context
Premises
AAAHAA
Action
Actor
Obtained Norm Descriptions
Human Annotations
Object
State
Prompt LLMs or Search in corpus
Hypotheses
Figure 2: An overview the dataset generation framework. Our framework takes in a normative behavior, represented as a behavioral script, and generates a premise that describes or comments on the behavior by prompting LLMs or searching in the existing corpus. Then, we prompt LLMs to generate hypotheses to infer different elements in the script. After light post-editing. we ask two cultural groups to annotate the entailment relationship between the premise and hypothesis where cultural differences surface as label variations.
Premise Generation Given a behavior the form a verb phrase, we prompt LLMs to generate naratives or opinions related to the behavior, for example, "Describe scene of tipping the waitress at a restaurant." In the post editing, we remove the target of the inference. For example, if the target of inference (hypothesis) is the cause of reaction, remove the previous action but keep the reaction. We also remove cultural indicators, e.g., names and countries, and irrelevant content.
do not affect the entailment relationships. We also do not observe any toxic content.
Generation / Awareness It is tempting to think that if model generate the hypotheses, the same model can also solve the inference task. How- ever, there is a key difference between the genera- tion task and the NLI task the generation process is agnostic of the actual human label. For exam- ple, a model generates a hypothesis with a prompt asking for highly plausible hypotheses, but human annotators may label it as contradicting. More- over, cultural variations are also not involved in the generation process, i.e., the prompt does not specify that the hypothesis should be plausible for particular cultural group.
Hypothesis Generation Hypotheses ideally should cover a diverse set of interpretations. We encourage diversity by specifying different condi- tions in the prompt: 1) specifying diffferent levels of certainty, such "write statements that are likely true" or "write statements that are definitely false": specifying different inference targets, such as "what the intention the speaker" or "what are different interpretations of the reaction". We again prompt LLMs with a set of instructions and manu- ally select three hypotheses per premise.
3.5 Collecting Annotations
We collect human labels under cultural con- texts using MTurk. The annotation effort has re- ccived an IRB exemption. We use two MTurk crowdsource worker pools as proxies to two differ- ent cultural groups. To ensure intergroup cultural differences and worker availability, we choose one group to be workers based in the United States (US) and the other group to be workers based in India (IN) using the Locale Qualification function on MTurk. We recruit in total 125 US and 35 IN workers. We follow the standard NLI task instruc- tions with three modifications: 1) Each annotator labels three hypotheses at once. Presenting multi- ple hypotheses encourages annotators to compare which hypothesis is more plausible. 2) Replacing the two extremes "definitely truc/false" with "very likely true/false" for reasons discussed in Section 3.1. 3) Using finer-grained five-scale rating. The full annotation template in Appendix B.2. To reduce randommess, we collect at least labels per hypothe-s pair. We also apply extensive
Artifacts in Generated Examples LLMs gener- ated contents may contain artifacts, societal biases, and toxic texts (Lucy and Bamman, 2021; Bender al., 2021: Sheng al., 2021; Yuan et al., 2021; Borchers et al., 2022). For our dateset, we observe that underspecified prompts with only the behav- ior information occasionally lead to contents with particular styles and gender/cultural stereotypes. See failure cases in Appendix A.2. We mitigate these biases in two ways (1) control the generation by specifying more content (c.g., behaviors, situ- ations, reactions) and style (e.g., writing style the structure of particular sentence); (2) au- thors manually review all generated contents and remove gender/cultural indicators (e.g., people's names, country names). Moreover, such artifacts

---

filtering, The Fleiss Kappa for US, IN is 0.58, 0.51 for entailment, 0.37, 0.29 for neutral, and 0.58, 0.46 for contradiction. The level of agreement is lower than the range of 0.6-0.8 reported in SNLI, which uses a stricter sense of entailment and contra- diction, but comparable with the range of 0.4-0.6 reported by Liu et al. (2023a) where ambiguity and implicatures are also involved.
Contradict
500
400
IN Label
300
Entail
12
127
563
200
100
Analyzing Cultural Variations
Diverge
68
140
114
72
With CALI, we empirically investigate (1) how cul- tural knowledge leads to label variations in the NLI task and (2) what are other factors that contribute to label variations between the two cultural groups.
Contradict
Neutral
Entail
Diverge
US Label
Figure 3: Distribution of premise-hyn hypothhsss pair labels standard NLI label space, after label mapping. aggregating by majority vote, and filtering out "Diverge" examples. About 30% pairs, ie.. off diagoaa entries, are assigned different labels by the two cultural groups with at least one label being "Entail" or "Contradict".
4.1 Categorizing How Cultural Norms Affect Language Understanding
Understanding how cultural knowledge leads to label variations in the NLI task a missing link between having cultural knowledge and achieving culturally aware language understanding. To an- swer this question, we follow the categorization approach from the NLI literature, which is used to characterize lexical and world knowledge needed for recognizing textual entailment (Clark al.. 2007; Sammons al., 2010; LoBue and Yates, 2011). For involve cultural norms and label variations, we manually annotate what linguistic phenomena and what types of cultural knowledge are involved in these examples. Examples are shown in Table 1.
quality control to make sure workers can achieve high accuracy on the standard NLI task. We de- scribe the details in Appendix B.3.
3.6 CALI: Culturally Aware NLI Dataset
total, we collect annotations for 900 premises and 2.7K hypotheses. We preserve labels from all annotators to reflect differentt cultural perspectives within a cultural group. However, to compare with existing NLI datasets, we map our 5-scale rating to the standard NLI label space and use majority vote for analysis. The mapping is determined by calibrating annotator labels against set of 300 MultiNLI labels. Specifically, "1 (Very Likely False)" is mapped to "Contradict", "3 (Neutral)" is mapped to "Neutral", "5 (Very Likely True)" mapped to "Entail". The mapping of "2" "2" and "4" is annotator dependent, with the majority maps to "Neutral". Details of label mapping can be found in Appendix B.4. We then take the majority vote of the mapped labels. If majority vote does not exist, we assign label of Diverge. The label dis- tribution is shown Figure 3. To focus on cases where least one cultural group reach a strong agrcement, i.c., at least 75% annotators agreed on the same label, we discard 30% pairs and use the rest for analysis.
pairs in CALI that
The Linguistic Dimension We first consider what kinds of linguistic phenomena may invoke cultural knowledge. We divide the linguistic di- mension into semantic and pragmatic categories. Semantic categories involve lexicon items with context-dependent meaning. including 1) lexical ambiguity due to multiple word senses, e.g., "bat" as "baseball bat" or "cricket bat' (Example #1 in Table 1); 2) context-dependen definitions, e.g.- "formal dressing" as "wearing long dress": 3) re- ferring expressions, c.g., "it" in Example #2 and "she" in Example #8. Context-dependent expres- sions recruit common sense knowledge to fill in unspecified details. As result, annotators with different cultural backgrounds resolve expressions to different entities, causing label variations.
As culture- deeteded inference is implicature by nature, annotator disagreement is expected to be more common than dataset focusing on textual entailment. Following the setup of SNLI (Bowman et al., 2015), we compute Fleiss kappa for binary classification task per class with 5 randomly sam- pled raters per question over all examples passed
Pragmatic categories are more common and com- plicated. We list two major subcategories: 1) con- versational implicatures, including connotations of phrases describing normative or transgressive

---

Premise and Hypotheses
Lingustic Knowledge Category
Label Variation
Example #1
Linguistic: Lexical ambiguity
P: A boy hits a ball, background watch him.
bat, outside, while others the
Knowledge: Object, US: The most com- bat-and-hall game baschall. INC most common bat-and-bol game cricket.
50%
H: The kid playing baseball game. (An example from SNI.I)
(SNLE Entail)
Linguistic: Referring expression Knowledge: Object: US: common to eat rice dishes with utensils such as spoon. IN: is common to rice dishes with hands.
Example #2
60%
P: It's important to wash your hand and the spoon, as eat the rice dish with it. H: The last word "it" refers to your hand.
Example #3
Linguistic: Implicature Knowledge: State; US: Tipping is custom- IN: Tipping optional.
P: The customer left tip for the waiter the restaurant. H: The customer satisfied with the excellent service.
Example #4
Linguistic: Flouting
P. Nothing says "I trust this medication" like a commercial that lists off all the possible side effects. H: Listing all possible side effects in commercial a sign of transparency and honesty that builds trust.
Knowledge: Goal; US: It is normal have advertising for medications with mandatory risk disclosure.
10%
Example #5
Linguistic: Politeness
P: When asked if the violence issue is common, she replied "No, it's common."
Knowledge: Goal; US: "No" signals dis- agreement. IN: Avoid face threatening at the expense of contradictory statement.
HI: She agreed with the person who asked the question.
H1: She disagreed with the person who asked the question.
Example #6
P: He had to admit that the house was taking shape. Most of the furniture was either hers or what they'd been given for their wedding.
Linguistic: Conversational implicature Knowledge: Previous action; US: Cou- ples communicate wedding gift prefer- ences to guests through registry. IN:
HI: The couple asked their guests to buy furniture as gifts.
des
H2: The couple used monetary gifts from their wedding to purchase furniture.
Money is the traditional wedding gift.
Example #7
P: "Did he leave for the waiter the restaurant?" His friends started laughing quietly they asked. H1: His friends did not know whether he left not.
Linguistic: Conversational implicature Knowledge: Previous action: US: Tipping customary. IN: Tipping optional.
H2: His friends thought did leave for the waiter.
Example #8
Linguistic: Referring expression Knowledge: Actor; US: White the tradi- tional color for wedding dresses. IN: It okay white wedding dress, but traditionally bride wearing vihrant colors.
P: Because Mona was the bride and Pia was her bridesmaid, she did not dress up in white for the wedding.
HI: Mona did not dress up in white for the wedding.
H2: Pia did not dress up in white for the wedding.
12%
(An example adopted from WinoGrande)
Table 1: Examples of cultural variations in the entailment label distribution between the two annotator groups US and IN. For entailment task, "P"; Premise, "H": Hypothesis, "E": Entails, "N"; Neutral, "C": Contradict. For plausibility task, "HI (H2)": Hypothesis 1 (2), labels are "HI (H2)": Hypothesis (2) is more plausible and "N": Neutral, two hypotheses are equally plausible.
actions, e.g., "not leaving a tip" implies dissatis- faction (Example #3). Following the cooperative maxim, listeners compelled to make pragmatic inferences by assuming speakers share the same set of norms. 2) violation of Gricean maxims or clash with politeness, either due to flouting, e.g.. Example #4 where the violation quality can be
interpreted as a touch of sarcasm or a strong asser- tion depending on the behavior and Example #5 where contradictory statements used to avoid face threatening situation (Valentine, 1996).
The Knowledge Dimension The knowledge di-
mension consists the cultural norm used to gen- pair and the script
erate each

---

elements associated with the norm. These script elements provide definitions, e.g., "object" in Ex- ample #1 and #2, speaker intentions, e.g., "goal" in Example #4 and #5, causes, e.g., "state" in Exam- ple #3 and "previous action" in Example #6 and #7, and effects of actions, c.g., "actor" in Example #8.
Model
Entailment Classification Task
FI macro
S/MNLI
0.70
0.71
0.70
S/M/F/ANLI
0.69
0.70
0.70
WANLI
0.69
0,68
0.67
XNLI
0.61
0.60
0.61
X/ANLI
0.66
0.67
0.65
GPT-3.5
0.74
0.74
0.72
GPT-3.54US
0.70
0.75
0.71
4.2 Ambiguity and Subjectivity
GPT-3.5+IN
0.70
0.68
0.73
GPT-4
0.76
0.80
0.75
We then examine other factors identified by previ- research label variations: ambiguity and sub- jectivity (Pavlick and Kwiatkowski, 2019; Zhang and de Marneffe, 2021; Jiang and Marneffe, 2022: Liu et al., 2023a).
GPT-4+US
0.74
0.80
0.71
GPT-4+IN
0.71
0.74
0.72
Choice of Plausihle Alternatiees Task
Accuracy
GPT-3.5
54.45
57.59
58.63
GPT-3.5+US
48.69
52.36
53.93
GPT-3.5+IN
43.16
45.26
46.84
GPT-4
58.12
67.54
59.16
GPT-4+US
56.54
67.54
58.64
Ambiguity To evaluate whether linguistic ambi- guity alone can explain cultural variations observed, we annotate 50 premises from Ambient (Liu et al., 2023a) and compare with label variations observed on premises involved cultural norms. Unlike the coneext-dependent cases Section 4. 1, ambiguity in these examples cannot be resolved by culturally specific knowledge. We observed that only 10% of the premises have label variations, which is lower than cultural norm related premises. In other words, linguistic ambiguity does not fully account for the cultural variations observed in CALI.
GPT-4+IN
50.79
56.54
52.88
Table Fl macro for entailment classification accuracy for choice of plausible alternatives w.r.. three sets of labels. Fine-tuned models are referred as the name of fine-tuning dataset.
5.1 Entailment Classification
We first evaluate models on two NLI tasks. For each task. we consider three sets of labels: "cul- turally generic" labels, i.e., "All", which is the ma- jority vote from both groups and "culturally aware" labels, i.., US (IN) labels, which is the majority vote from US (IN) annotators.
Annotator Subjectivity Lastly, we examine whether annotators are simply inferring based on culture-specific knowledge, but ignore the lin- guistic context. We found annotators do assign non-neutral labels to hypotheses irrelevant to the premise when strong culture conventions pre- sented in the hypotheses only, however, aggregat- ing by majority vote largely mitigates the subjec- tivity. We include these examples in Appendix C.
Tasks We cluster premise-hypeeee hypothesis pairs by the normative behavior they covered and select the 40 largest clusters, which yields 500 pairs in total. The label distribution is roughly 25%, 50%, 25% for contradiction, neutral, entailment, with 30% labels differ between the two groups. For the entailment classification task, we perform a binary classifica- tion that treats only "entail" as positive. We do not use the 3-way classification setup, due to a distri- bution shift in labels discussed in Section 3.6. For each model, we take the probability of the "entail" class as the prediction and whether the majority vote from annotators is "entail" the ground-truth.
5 Evaluating Language Models
We first investigate how well models fine-tuned on existing NLI datasets perform on CALL Specif- ically, whether entailment features learned from other datasets are transferable to CALI and whether these features are biased towards a particular cul- tural context. Then, we use the three-level cultural awareness defined in Section and the categories developed in Section 4.1 work assess GPT-3.5/4 cultural awareness on ILI tasks, where we ask models to make culture- specific inference by prompting models with ex- plicit cultural indicators.
The choice the plausible alternative task takes in a premise and two hypotheses. We use a 3-way classification to predict which hypothesis is more likely to be true or if there is no clear preference between the two. The task is suitable for norm- based inference, there always exceptions the norm that can cancel out the entailment.
evaluation frame-
We consider five state-of-the-an mod- els fine-tuned on different NLI dataset (with ac-
Models

---

curacy on the standard NLI test MultiNLI dev mismatched): SNLI MultiNLI (90.49)3.
Behavior
Knowledge
Con- text
Infer-
US
ence
SNLI + MultiNLI FEVER NLI + ANLI (90.6)). WANLI (80.46)², MultiNLI + XNLI (87.58)", and MultiNLI + XNLI + ANLI (86.11)². 1)". We also considered zero-shot prompting GPT-3.5/4 with two setup: one using NLI tasks prompt, the other adding locale information culture indicators, e.g.. "Let's think someone who lives in the United States." "Remind yourself of American culture". We select prompts with the highest accuracy on a validation set (details in Appendix D.I).
GPr3s
1.0
1.0
1.0
0.50
Leave a tip
GPT4
1.0
1.0
1.0
0.70
Touch
GPE3S
1.0
1.0
0.6
0.69
one's feet
Grra
1.0
1.0
1,0
0.61
Have
GPE33
1.0
0.0
0.0
1.0
gift registry
GPT4
1.0
0.0
1.0
0.5
Receive first paycheck
arr33
1.0
1.0
0.0
0.73
GPr4
1.0
1.0
0.83
0.64
Share food
GFT33
1.0
0.86
GPT4
1.0
1.0
1.0
0.82
Results are shown in Table 2. We observe that (I) models fine-tuned on existing NLI datasets have similar accuracy on all three sets of labels, i.e., not culturally aware but also not culturally biased. (2) GPT-335/4 in general outperform fine-tuned mod- els, but do not show consistent improvements on cultural-specifi labels when prompted with cul- ture indicators. Among models fine-tuned on the existing NLI datasets, the two models with the highest accuracy on MultiNLI still have the highest F1 on our test sets, suggesting entailment features learned from the existing datasets are transferable our dataset. Fine-tuning unfortunately does not help- X/ANLI model has the highest false positive rate, while XNLI has the highest false negative rate.
All 40 behaviors
GPT:335
0.93
0.79
0.62
0.68
GPT- 0.98
0.93
0.88
0.71
Table 3:
Results of the three cultural awareness diagnostic tasks. report accuracy on the knowicdge/inferenec task and recall on the context task. We show per per-behavior accuracies for five behaviors and averaged accuracy all 40 behaviors.
inference by specifying the cultural context.
We evaluate GPT-3.5/4 on the three tasks through prompting. The knowledge task consists of 40 questions on cultural norms, such as "What amount is considered normal when giving tip in [country) ) We compare model outputs against sources of norms in Section 3.2 to determine the correctness of the answer. The context task asks models to list cultural norms involved in a premise. We use recall as the metric to check whether models can retrieve the norm used to generate the premise. Lastly, the inference task is formulated as defea- sible inference (Rudinger et al., 2020), where the norm is present as update either strengthen weaken the entailment (details in Appendix D.2).
cross-lingual NLT
5.2 Diagnosing Cultural Awareness
We perform error analysis to investigate at which levels the LLMs lack cultural awareness. We fo- cus on evaluating GPT-3.5/4, which allows us to specify the cultural context through prompting.
Three-level Diagnostic Tasks We decompose the culturally aware NLI into three diagnostic tasks: (1) Knowledge task: Whether errors are due to a lack of cultural norm knowledge; (2) Context task: Whether models fail to recognize contexts that invoke the norm; (3) Inference task: Whether models can reason about the effect of norms on entailment. Together, these tasks explore whether we steer the model towards culture specific
The results are shown in Table 3. On the knowl- edge level, there is indeed an accuracy gap between US and IN. However, this knowledge gap is reduced from 14% to 5% when switching from GPT-3.5 GPT-4, leading to over 90% accuracy for both cul- tural groups. These results suggest that GPT-3.5/4 have the knowledge of most cultural norm tested, in the sense that the models can pass at least one set of behavioral tests in a QA format that requires such knowledge. With the improvement in cultural norm knowledge task accuracy, we also observe 30% improvement in recognizing norm in the sentence context. Lastly, for the inference task. we find that even when models are provided with cul- tural norm knowledge, there is still a 30% error rate in updating the inference with the cultural norms.
ali-debbrta--v--larg
"nttps::/huugiingaace wewefeeerraee _anli_nnn _1____1_________
/nttps://huggiingffcc.
roberta-large-wanlii
nts-large-innttun---mm--xxxmmm
"nttps://huggingface

---

Calegory
GPT-4
GPT-4
Limitations
Lexical Ambiguity (7%)
1.00
1.00
Referring Expressions (5%)
0.73
0.61
Our work only studies two cultural groups. While we have already observed evidence that cultural variations contribute to NLI label variations, ex- panding the study to multiple cultural groups would be highly valuable. We expect the proposed model- in-the-loop loop data collection framework to be helpful for studying NLI label variations in other cultures.
Violation of Maxims (9%)
0.74
0.82
Other Implicatures (79%)
0.81
0.72
Table A breakdown of macro for entailment clas- sification per linguistic dimensions.
Which linguistic phenomena make cultural norms hard to recognize? Now we take closer look at the gap between the knowledge task and the inference task, using the linguistic dimensions laid out in Section 4.1. We evaluate on the same set of examples used in Table 2, which has about 12% semantic and 88% pragmatic cases. We focus our analysis on the GPT-4 results with the culture indicator prompting.
We use two locale-based crowdsource work pools as proxies for two different cultural groups. However, culture can vary greatly within a locale due to social class, age, and gender (Tamnen, 1985). With label aggregation such as taking the majority vote, norms of larger or privileged groups might be overrepresented. Hence, we do not claim that par- ticular cultural variations identified in dataset are generalizable different populations within the United States and India. The collected dataset is best used for analyzing how cultural norms interact with language understanding, yet it can be limited as a knowledge base of cultural norms in the United States and India. With our new framework to study cultural variations in language understanding, we hope that future work can investigate finer-grained cultural variations.
Results are shown in Table 4. GPT-4 performs the best on the lexical ambiguity category and per- forms the worst the Winograd schema challenge- style reference resolution task. For the reference resolution task, despite having the necessary cul- tural knowledge (as shown the knowledge prob- ing task), GPT-4 is less sensitive to cultural indi- cators presented in the prompt, but biases towards using syntactic cues. These results highlight that the challenges of culturally aware language under- standing lie in not only the knowledge level but also the interactions between knowledge and language.
Ethics Statement
Using LLMs for Data Generation To generate premises and hypotheses at scale, we use model- in-the-loop paradigm for data collection. Model- generated premises and hypotheses may contain artifacts and stereotyped gender or culture asso- ciations (Lucy and Bamman, 2021; Bender et al., 2021; Yuan et al., 2021). We discuss these arti- facts and mitigation strategics in Section 3.4 and Appendix A.2.
6 Conclusion
We present a framework to study cultural aware- ness in natural language understanding. We de- fine cultural awareness at three levels: (1) having knowledge of cultural norms, (2) recognizing cul- tural norms in context, and (3) making cultural- specific inferences. Our work focuses on level (2) and (3). We operationalize cultural variations in language understanding through an NLI task and contribute the first culturally aware NLI dataset, CALL. With our dataset, we categorize how cul- tural norms influence language understanding and present an evaluation framework to assess at which levels LLMs are culturally aware. We show despite having knowledge of cultural norms, LLMs still lack awareness levels (2) and (3). Our work highlights that cultural factors contribute label variations in NLI and cultural variations should be considered in language understanding tasks. We advocate for the community to incorporate cultural awareness in NLP systems in future research.
Annotation Considerations This work has re- ceived an IRB exemption at the authors' institution. For annotator compensation, we aimed for a pay rate of at least $12/hour, which is compliant with the federal minimum wage in the U.S. and India.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable feedback. We are also grateful of members of SALT lab, especially Raj Sanjay Shah for help on dataset collection, Caleb Ziems, Camille Harris, Ella Li, Myra Cheng, Will Held, and Weicheng Ma for feedback on the work. This

---

work was partially sponsored by the Defense Ad- vanced Rescarch Project Agency (DARPA) grant HIR0011220100/HHBR 1260656, and NSF grant IIS-2247357 and 2308994.
Yong Cao, Li Zhou, Scolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and hu- man societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53-67, Dubrovnik, Croatia. Association for Computational Linguisties.
References
Sky CH-Wang. Arkadiy Saakyan, Oliver Li, Zhou Yu, and Smaranda Muresan. 2023. Sociocultural norm similarities and differences via situational alignment and explainable textual entailment.
Mitchell Abrams and Matthias Scheutz. 2022. Social norms guide reference resolution. In Proceedings of the 2022 Conference of the North American Chap- ler of the Association for Compulational Linguisties:
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7139-7159, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Human Language Technologies, pages 1-11. Seattle, United States. Association for Computational Lin- guistics.
Arnay Arora, Lucie-aimée Kaffee, and Isabelle Augen- stein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cias Cultural Considera- fions in NLP (C3NLP), pages 1 114-130, Dubrovnik, Croatia. Association for Computational Linguistics.
Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Un- certain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772-877 8779, On- line. Association for Computational Linguisties.
Mikel Artetxe. Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computa- tional Linguistics.
Peter Clark, Phil Harrison, John Thompson, William Murray, Jerry Hobbs, and Christiane Fellbaum. 2007, On the role of lexical and world knowledge in RTE3. In Proceedings of the ACI-PASCAL, Workshop on Textual Entailment and Paraphrasing, pages 54-59 59, Prague. Association for Computational Linguistics.
Emily M. Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Trans- parency, FAccT 221. page 610-6 (20,,23, New York, NY, USA. Association for Computing Machinery.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman. Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guisties.
Proceedings of
Cristina Bicchieri. 2000. Words and Deeds: A Focus Theory of Norms, pages 153- 184. Springer Nether- lands, Dordrecht.
Daniel Druckman, Alan A. Benton, Faizunisa Ali, and J. Susana Bagur. 1976. Cultural differences in bar- gaining behavior: India, argentina, and the united states. Journal of Conflict Resolution, 20(3):413- 452.
Conrad Borchers, Dalia Gala, Benjamin Gilburt, Eduard Oravkin, Wilfried Bounsi, Yuki M Asano, and Han- nah Kirk. 2022. Looking for a handsome carpenter! debiasing GPT-3 job advertisements. In Proceedings of the 4th Workshop on Gender Bias in Natural Lan- guage Processing (GeBNLP), pages 212-224 Seattle, Washington. Association for Computational Linguis- tics.
Denis Emelin, Ronan Le Bras, Jena D. Hwang. Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situ- ated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Con- ference on Empiricul Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap. and Yejin Choi. 2020. Social chem- istry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online, Association for Computational Linguisties.
Erika Buckley. 2006. A cross-cultural study of wed- dings through media and ritual: Analyzing indian and north american weddings. McNair Scholars Journal, 10:3.

---

Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Ram- bow, Smaranda Muresan, and Heng Ji. 2022. Norm- sage:
Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3.
Multi-lingual multi-cultural norm discovery from conversations on-the-fly.
Anubha Kabra, Emmy Liu, Simran Khanuja, Al- ham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neu- big. 2023. Multi-lingual and multi-cultural figurative language understanding. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 2269-8284, Toronto, Canada. Association for Computational Linguistics.
Andrew Gordon, Zoritsaa Kozareva, and Melissa Roem- mele. 2012. SemEval- 2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In "SEM 2012: The First Joint Conference on Lexical and Computational Seman- lics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394-398, Montréal, Canada, Association for Computational Linguisties.
Amr Keleg and Walid Magdy, 2023. DLAMA: A frame- work for curating culturally diverse facts for prob- ing the knowledge In Findings of the Association for Computational Linguisties: ACL 2023. pages 6245-6266, Toronto. Canada. Association for Computational Linguistics.
pretrained language models.
E.D. Gutiérrz,, Ekaterina Shutova, Patricia Lichtenstein, Gerard de Melo, and Luca Gilardi. 2016. Detecting cross-cultural differences using a multilingual topic model. Transactions of the Association for Computa- tional Linguisties, 4:47-60.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Shweta Hegde, Leena Nair. Haritha Chandran, and Ha- roon Irshad. 2018. Traditional indian way of eating- an overview. Journal of Ethnic Foods, 5.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, This Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross- cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 6997-7013, Dublin, Ireland. Association for Computational Lin- guistics.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023a. We're afraid language models aren't 't modeling ambiguity.
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023b. Are multilingual 1lms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.
G. Hofstede, G.J. Hofstede, and M. Minkov, 2010, Cul- tures and Organizations: Software of the Mind, Third Edition. McGraw-Hill Education.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond El- liott. 2021. Visually grounded reasoning across lan- guages and cultures. Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588 602, Online. Association for Computational Linguistics.
Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 329-334, Portland, Oregon, USA. Association for Computational Linguistics.
Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. OCNLI: Orig- inal Chinese Natural Language Inference. In Find- ings of the Association for Computational Linguistics: EMNLP 2020., pages 3512-3526, Online. Association for Computational Linguistics.
Li Lucy and David Bamman. 2021. Gender and rep- resentation bias in GPT-3 generated stories. In Pro- ceedings of the Third Workshop on Narrative Un- derstanding, pages 48-55. Virtual. Association for Computational Linguistics.
Nan-Jiang Jiang and 2022. Investigating Reasons for Disagreement in Natural Language Inference. Transactions of the Association for Computational Linguistics, 10:1357- 1374.
de Marneffe,
Michael Lynn and Ann Lynn. 2004. National values and tipping customs: A replication and extension. Jour- nal of Hospitality & Tourism Research, 28(3):356- 364.
Rebecca Johnson, Giada Pistilli, Natalia Menédez- González, Leslye Denisse Dias Duran, Enrico Panai,

---

Christopher D. Manning, 2006, Local textual inference: Its hard to circumscribe, but you know it when you see it-and nip needs it.
Long Papers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics.
Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Process- ing, pages 74-79, Valencia, Spain. Association for Computational Linguistics.
Milagros Miceli and Julian Posada. 2022. The data- production dispositif. Proc. ACM Hum, Comp-Ct In- teract. 6(CSCW2).
Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023, Extracting cultural com- monsense knowledge scale. In Proceedings of the ACM Web Conference 2023, WWW '23, page 1907-1917, New York, NY, USA. Association for Computing Machinery.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chan- dra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4661-4675, Online. Association for Computational Linguistics.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective hurman opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNIPP) pages 9131-9143, Online. Association for Computational Linguistics.
Mark Sammons, V.G. Vinod Vydiswaran, and Dan Roth. 2010. "ask not what textual entailment can do for you..... In Proceedings of the 48th Annual Meet- ing of the Association for Computational Linguistics, pages 1199-1208, 1208, Uppsala, Sweden. Association for Computational Linguistics.
Shramay Palta and Rachel Rudinger. 2023. FORK: A
bite-sized test set for probing culinary cultural biases commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACI.
Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures. I.. Erlbaum, Hillsdale, NJ.
2023, pages 9952-99929 9962, Toronto, Canada. Associa- tion for Computational Linguisties.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677 7:677-694.
Omar Shaikh, Caleb Ziems, William Held, Aryan Pari- ani, Fred Morstatter, and Diyi Yang. 2023. Modeling cross-cultura pragmatic inference with codenames duet. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 6550-6569, Toronto, Canada. Association for Computational Lin- guistics.
Denis Peskov, Viktor Hangya, Jordan Boyd-Graber and Alexander Fraser. 2021. Adapting entities across languages and cultures. In Findings of the Associa- tion for Computational Linguisties: EMNLP 2021, pages 3725-3750) Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tao Shen, Xiubo Geng, and Daxin Jiang. 2022. So- cial norms-grounded machine ethics in complex nar- rative situation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 1333-1343, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671- Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing ( (Volume I: Long Papers). pages 4293,, Online. Association for Computational Linguistics.
Valentina Pyatkin, Jena D. Hwang. Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. 2023. Clarifydelphi: Reinforced clarifi- cation questions with defcasibility rewards for social and moral situations.
Aida Ramezani and Yang Xu. 2023. Knowledge of
cultural moral norms in large language models. In
Vered Shwartz. 2022. Good night at 4 pm?! time pressions in different cultures. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 1842-2853 Dublin, Ireland. Association for Computational Linguistics.
Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 428 446, Toronto, Canada. As- sociation for Computational Linguistics.
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah Smith, and Yejin Choi. 2018. Event2Mind: Com- monsense inference events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I:
Deborah Tannen. 1985. Cross-culuraal communication, volume 4. Academic Press.
Jenny Thomas. 1983. Cross-Cultural Pragmatic Failure. Applied Linguistics, 4(2):91-112.

---

Tamara M. Valentine. 1996. Politeness models in indian english.
Emiel van Miltenburg. 2016. Stereotyping and bias in the flickr30k dataset.
David Vilares and Carlos Gómez-Rodrrguees 2018. Grounding the semantics of part-of-day nouns world- wide using Twitter. In Proceedings of the Second Workshop Computational Modeling of People's s Opinions, Personality, and Emotions Social Me- dia, pages 123-128, New Orleans, Louisiana, USA. Association for Computational Linguistics.
Anna Wierzbicka. 1991. Cross-Cultune Pragmatics- Semantics of Human Interaction. De Gruyter Mouton, Berlin, Boston.
Binwei Yao. Ming Jiang, Diyi Yang. and Junjie Hu. 2023. Empowering llm-based machine translation with cultural awareness.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai- Wei Chang. 2022. GeoM- LAMA: Geo-diverse commonsense probing multi- lingual pre-trained language models. Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 2039-205S Abu Dhahi, United Arab Emirates. Association Com- putational Linguistics.
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021. Synthbio: A case study faster curation of text datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagree- ment natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908-4915, Online. Association for Computational Linguistics.
Caleb Ziems, Jane Dwivedi-Yuu Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. NormBank: knowl- edge bank of situational social norms. In Proceed- ings of the 61 st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7756-7776, Toronto, Canada. Association for Computational Linguistics.

---

Supplementary Materials
A Dataset Generation Details
A.1 Prompts Used for Premise and Hypothesis Generation
We list the full set of prompt templates that we have experimented with in Figure 4. For premises, given a normative behavior, we randomly sample two premise prompt templates. For hypotheses, we randomly sample one of the "goal" prompt and two others prompts. For prompt selection, we observe that prompts with more information specified generally produce higher quality outputs.
Prompts for premise generation
In 1-2 sentences, describe scene of (behavior) at [location].
In sentence, write (sentence_ type) of (behavior) at (location). Write (sentence_type) implying (behavior) (location) is (judgement). In sentence, write narrative Write sentence with (style) implying (behavior) (location) (judgement).
comment implying (behavior) at (location) is (judgemen L).
The value of each placeholder is as following
(sentence_ type]: One "a sentenee", "a short sentenee", "a narrative", "a comment", "a sarcastic comment". (behavior 3; verb phrase describing an action, such as "leaving a tip". (location): noun phrase deseribing a place, such as "restaurant". (judgement):
of "normal", "expected", "unexpected", "taboo", Norms sampled from NormBank come with this annotation, however, for data generation, we simply random sample two opposite judgement per behavior.
(style): One of writing styles, such as "colloquial style", "Mark Twain style". "Emest Hemingway style". and "Reddit style" or an example sentence, such as "the style and structure
thought working on Liddy's 's campaign would be better than working on Bob's." "." where the example sentence is taken from an existing NI.I datasct. We use style to improve the diversity generated text, in particular, diversify sentence structures and choice words.
Prompts for hypothesis generation, listed by inference type.
Actor: In a single sentence, infer what is the relation between the speakers/people mentioned
the scenario.
Scenario: [premise)
Actor: Write three sentences with different hypotheses on the identity of people mentioned in the scenario.
Scenario: [premise)
Goal: Write three sentences with different interpretations of the intention of the person or the
speaker in the following scenario.
Scenario: {premise)
Goal: Write three sentences with different interpretations of the reactions in the following scenario?
Scenario: [premise)
Previous Action: Given the following scenario, can you infer what was the person doing or what happens to the person before this?
Scenario: (premise)
Next Action; Given the following scenario, can you infer what would the person do after this?
Scenario: (premise)
Object: Given the following scenario, can you infer what objects have been involved?
Scenario: (premise)
State: Write single sentence inferring what the person may feel in the following scenario: (prenise)
Entailment: Write few statements that definitely true given the following premise: (premise)
Neutral:
Write three sentences with different implications of the following premise: (premise)
Paraphrase: Wr wite three paraphrases of the sentence with different interpretations: (premise)
Figure 4: Prompts used for premise and hypothesis generation.
A.2 Premise Generation Failure Cases
In general, we do not observe that ChatGPT/GPT-3.5 --tttu generate toxic content on the prompts used, however, we observe degraded generation quality when given social norms are non-stereotypical. These cases can be viewed as -iit-f--iistrbbttion" inputs for the model. We list four major failure cases below.
Repeating content in the prompt Generated texts are simply repeats of the behavior and judgement specified in the prompt, without any new information.

---

Prompt In 1-2 sentences, describe scene of saying "please" and "thank you" at restaurant when doing so is unexpected.
Output: A customer at busy restaurant unexpectedly says "please" and "thank you" to the waitstaff, causing them to pause and smile with appreciation.
Prompt: In 1-2 sentences, describe scene of calling strangers "Uncle" and "Aunty".
Output: In sone cultures, it is customary to show respect addressing strangers who are older as "Uncle" or "Aunty," " even if they are not.
Wrong cultural associations Generated texts contain normative behaviors associated with the wrong culture. As we remove the culture indicator in the post-editing phase, this type of failure is less of a concern for the data generation process. However, this type of failure shows the need to improve cultural awareness LLMs.
Prompt: In 1-2 sentences, describe scene of greeting with three kisses on alternating cheeks at airport when doing so is unexpected. Output: As she stepped off the plane, her American friends greeted her with three kisses alternating cheeks, custonary greeting in their culture, but unexpected for her.
Stereotyped responses Generated texts contain stereotyped responses, include gender and social role associations. For stereotyped gender association, we try to rewrite the premise with gender-neutral pronouns in the post-editing stage if possible. For stereotyped social role, we discard the generated output, the stereotyped associations likely have cultural variations.
Prompt: sentences, describe scene of knitting a scarf bank.
Output: woman sits in the waiting area the bank, calmly knitting a scarf with a ball of yarn knitting needles her lap, while waiting for her number.
Prompt: 1-2 sentences, describe scene wearing red dress on a wedding.
Output: The bride wore a stunning white gown while her bridesmaids wore matching red dresses, standing out against the greenery of the outdoor wedding ceremony.
Refuus to generate ChatGPT/GPT-3.5 refuses to generate texts for certain norms, despite such norms are commonly practiced in some countries.
Prompt: In sentence, write narrative or comment implying threatening to leave wi thout raise at work is normal.
Output: I'm sorry, as an intelligent assistant, cannot provide statement that implies threatening to leave wi thout raise at work is normal acceptable behavior.
B Annotation Details
B.1 Annotator Compensation
We aimed for a pay rate of at least S12 per hour. Workers were paid $0.1 for each example (50 words) annotated.
B.2
Annotation Template
We show the annotation template for qualification and actual task in Figure 5a and Figure 5b.
B.3 Annotation Quality Control
We apply extensive quality control make sure the annotations are valid responses from annotators who can follow the NLI task instruction. First, annotators were required to have HIT approval rate of 95%. a total of 50 approved HIT' to participate the research. We also set up qualification test with four standard NLI questions and two localc-specific commonsense questions. About 50% of Amazon MTurk workers who have completed the qualification test passed the test.

---

Plausibiiity Qualification Test
You givn a coman abscavennts beitadoo
tos mway rnttmmn
arfomsstinn
federr alon Rrceator cortooo comk conmon coroo Moo mulloni Rrcoom
Therati soct fow Tonyy ouch lot fucr Mich loee.
Examoless
Eatereent Ai nan Wosk aooluutu
Rating: Jvery deviy trewet
satig: pright de RTA.
maiing bary meep swe.
Cake Iely atoe ment the bared Infrmation conted you commer teate
stahaep 10-8 to
(a) MTurk qualification test template.
house ivic meco
Staterennt
Beaky Eayly
focsel ty pailing open windee
Axmall wnims wich oua bribe and che rawes
O
Asbrogg aszund
Contest: the atting
Ho so oce cown the Nos
O O
O
O O
(b) MTurk annotation task template, with the same instruction as the qualification test shown in Figure 5a.
Figure 5: MTurk qualification test and annotation task templates.
Apart from standard worker qualifications, dedicate 15% of examples to quality control questions. These examples are sampled from either MultiNLI dataset where all five annotators agreed the same label or from our norm-related examples where the entailment relationship is mainly logical. On control questions, the accuracy for US and 87% and 84%. We discard all annotations from workers with an accuracy below 60%, which about 16% annotations. Besides quality control questions, the three hypotheses also allow us to apply some consistency checks to detect nonsensical ratings, such as rating two contradictory hypotheses as "5 - very likely true" at the same time.
B.4 Annotator Label Distribution
The full label distribution is shown in Figure 6a. The mapping between our 5-scale rating and MultiNLI label space shown in Figure 6b.
Examples of Cultural Variations
We include additional cultural variation examples in Table 5.

---

4030
Group
2600
2500
US
Lorbad
7000
Keutny
500
(Ver Libely
(Very) Liblly
[Very Likely Faise!
(Neutrall Label
(very Likely Iruat
Mappings between 5-scale rating and MultiNL classes. For each MultiNLI class, show the median and quartiles under 5-scale rating scheme both groups. "Contradict" roughly corresponds "1-2", "Neu- tral" roughly corresponds to "2-4", and "Entail" roughly corresponds to "S"
Distribution of all annotator labels from cultural groups. Overall, there significant distribution shift between the two groups, there either cultural variation correlation between the label and cultural variation.
Figure 6: Label distributions.
Category
Premise and Hypothesis
Explanation
Label Variation
P: In spite the recession, many people are still going out to eat at restaurants.
The first example shows cultural varia- tion due ambiguity. However, the variation does not persist once introduce cultural into the context The last two exam- ples generated by prompting ChatGPT follow the structure of spite of, still doing Y"..
H: In spite of the recession, many people are still going out to cat at restaurants relative to hefore. (An example from Ambient)
20%
P: In spite the pandemic, many people are shaking hands to greet others.
Ambiguity
I: In spite of the pandemic, many people are still shaking hands relative to before.
In spite of the new fashion dresses in bold colors, many people are still choosing to wear white color dress.
prefers wedding
H: spite of the new fashion trend, many people are still choosing to wear white dresses compared to before.
easy to overdose on many national parks re- gional parks United States. H: Visiting many national and regional parks the country is free charge.
This example shows annotators from both cultural groups are prone to subjective inference when content is related to their cultural background.
Subjectivity
P: It's easy overdose the many temples, palaces, and muscums India.
H: Many temples and palaces free visit in country. (An example from MultiNLI)
Table 5: Cultural variation examples due to ambiguity and subjectivity, as discussed in Section 4.2.
D Model Evaluation Details
D.1
Prompt Selection
For each task, we use a dev set 60 premise-hypothesis pairs from MultiNLI and CALI to select the prompt with the highest accuracy from a set of candidate prompts. The selected prompts are shown Figure
D.2 Diagnostic Task Details
We describe the setup of the three diagnostic tasks, with examples show in Figure 8.

---

Entailment Classification Task
To what extent does the given premise entail the hypothesis? Your answer should be a percentage indicating the probability of entailment. Premise:
Hypothesis:
Prompt with Culture Indicator: Let's think as someone who lives in the United States. To what extent does the given premise entail the hypothesis? Remind yourself of common sense knowledge and American cure. Your answer should be a percentage indicating the probability of entailment. Premise: Hypothesis:
Choice of Plausible Alternatives
Given the premise, which of the following two hypotheses is more likely to be true? Your answer should be one of "Hypothesis 1", "Hypothesis 2", or "Same". Premise:
Hypothesis 1:
Hypothesis 2:
Prompt with Culture Indicator: Let's think which of the two hypotheses more likely to be true? Remind yourself of common sense knowledge and American curt.re. Your answer should be of "Hypothesis "Hypothesis 2", "Same". Premise: Hypothesis Hypothesis 2:
who lives in the United States. Given the premise,
A list of candidate prompts that lower accuracy on the dev Please identify whether the premise entails hypothesis. answer should be exact "entail" or "not entail", followed by percentage indicating how confident you are.
Premise:
Hypothesis:
Let's think step by step. To what extent does the given premise entail the hypothesis? Your answer should be a percentage indicating the probability of entailment.
Premise:
Hypothesis: Let's think step by step, What is the likelihood of the premise entails the hypothesis? Your answer should be percentage indicating the probability of entailment.
Premise:
Hypothesis: Does the following premise entail the hypothesis, i.e. the hypothesis is true given the premise. You answer should be "Yes" or "No", followed by a percentage indicating how confident you are.
Premise:
Hypothesis: Given the premise, which of the following two hypotheses is more plausible? Your answer should be one of "Hypathesis 1", "Hypothesis 2", or "Same".
Premise:
Hypothesis 1:
Hypothesis 2:
Figure 7: Prompts used for entailment classification and choice of plausible alternatives evaluation.

---

Task 1: Probe cultural norm knowledge
PROMPT: What is the cul tural practice in (country) regarding tipping wai waiters? PROMPT: What amount is considered normal when giving tip in [country)?
PROMPT: Is it okay to tip waiters in (country)?
Task 2: Recognize cultural norms in context
PROMPT: Does the following context involves cultural norms and conventions? If 1ist each norm convention "X is expected/normallttabo in culture Y", where X is behavior and Y is the of the culture.
Context: "Did he leave tip that
than percent?" chuckled despite myself.
Task 3: Inference with cultural
PROMPT: Given premise and hypothesis, does additional information make the hypothesis more less likely
true? Your answer should be "more likely true", "less likely true", *does
not change".
Premise: "Did leave a tip that was more than percent?" chuckled despite myself. Hypothesis: left generous tip for the waiter even though he shouldn't. Additional Information: typical tip is usually between 15% and the pre-tax total of the bill.
Figure 8: We decompose the culturally aware NLI task into three diagnostic tasks: the knowledge task, the context task, and the inference task.
Knowledge Task We generate norm-related questions by prompting GPT-3.5-turke convert set of norms into questions, with the prompt "Write question about the given information for cultural knowledge quiz. Information: (information]" For example, "Giving a tip that more than ten percent is normal in American culture." is converted into "What considered normal when giving a tip in American culture?". For evaluation, we compute accuracy as the #correct answer #total answer. An answer is correct if it matches the actual norm we collected from cross-cultural studies or internet sources. We ask GPT-4 to compare the output against the actual norm, followed by manual review of the results.
Context Task
We ask models to list cultural norms involved in premise, with the prompt shown in Figure 8. As there can be multiple norms involved in given context, we choose recall as the metric output counted as correct the norm used for generating the example is covered by the output.
Inference Task We formulate the cultural norm inference as a defeasible inference problem: given a cultural norm as "extra information". how does it change the entailment relationship. This formulation allows us to decouple the inference process from the previous two levels. We use the set of norms from the previous two tasks as "extra information" and rating differences between the two groups as the label for strengthening weakening. Prompts used for this task shown in Figure 8. We compute the 3-way classification accuracy over "more likely true", "less likely true". and "does not change".


Reviews:
[{"review_id": "hNGFKke0Bu", "reviewer": "Reviewer_61Gu", "summary": "", "questions": "How do annotator judgements vary across these two cultures? What does this say about these two societies? Any social insights that we can gain from this dataset?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe would like to clarify the motivation and goal of this work.\n\n* Our goal is to highlight that **cultural factors contribute to label variations in NLI** and **cultural variations should be considered in language understanding tasks**.\n\n* Our work is largely motivated by two lines of research\n  * Social norms differ across cultures (To recap: Cultural differences have been noted in social behaviors and practices. See Geertz 1973; Triandis 1994; Hofstede et al., 2010. Recent work in NLP has computationally identified differences in social norms across cultures. See Yin et al., 2022; Nguyen et al., 2023; Fung et al., 2022; Ziems et al., 2023; CH Wang et al., 2023)\n  * Social norms are involved in language inference (To recap: Social norms are involved in reasoning about social judgment, intents and consequences, default assumptions, and ambiguity in expressions. See Forbes et al., 2020; Shen et al., 2022; Fung et al., 2022; Ziems et al., 2023, Emelin et al., 2021; Rashkin et al.; 2018, Abrams and Scheutz, 2022; Rudinger et al., 2020; Ziems et al.; Pyatkin et al., 2023) \n\n  These two lines of research suggest that people from different cultures likely make different inferences about social behaviors. Yet, language inference tasks such as NLI have been largely assumed to have a single label across cultures.\n\nThus, our goal is **not** using the NLI task as a tool to study how norms differ across cultures. Instead, we revisit the assumptions and setup of the NLI task itself through a culture lens, showing cultural factors lead to label variations. With this goal in mind, we believe some concerns could be addressed.\n\n### Response per Question\n\n1. *\"no analysis of how these norms may differ across cultures\"*: Our work builds on previous work that performs extensive analysis on how certain norms differ across cultures, including work that studies tipping, dining, wedding customs, chronemics, waiting in line, politeness, etc [1-6], recent work in NLP that aims to curate knowledge bases of cultural norms (line 101-105), and online resources [7-8] (also line 239-246). In light of prior work, we focus our analysis on how cultural norms interact with language understanding, instead of cultural norm variations themselves. We choose NLI as a starting point, as it is one of the widely used and fundamental tasks in NLP in terms of language understanding.\n\n2. *\"key evaluation here should not be on how existing NLI models ... would perform on this task\"*: Given the goal of this paper is to show cultural factors should be considered in NLI, it is natural and necessary to study how existing models that do not consider cultural factors fail on examples that involve cultural variations. Our results show that existing NLI models indeed do not transfer well to CALI examples involving cultural variations (Section 5.1).\n\n3. *\"cross-cultural insights\"*, such as *“How do these cultures fundamentally differ? What does this say about these two societies? Any social insights that we can gain from this dataset?”*: While these are intriguing questions, existing cultural studies can already provide some insights. For example, Hofstede's Cultural Dimensions Theory shows that the U.S. and India differ significantly on Power Distance, Individualism, and Indulgence (Hofstede et al., 2010). Our work builds on top of these cultural analyses to provide insights into how cultural factors affect language understanding.\n\n4. *\"How do annotator judgements vary across these two cultures?\"*: We provide a quantitative analysis in terms of statistical distribution of ratings (To recap: Annotators from the two cultural groups disagree on 30% of the labels; See Section 3.6 and Figure 7 in Appendix) and qualitative analysis on what types of inferences lead to variations in annotator judgment (To recap: We observe annotation disagreement on examples that involve lexical ambiguity, referring expressions, implicatures, and violations of Gricean maxims; See Section 4).\n\n\nReferences:\n1. Lynn, M., & Lynn, A. (2004). National Values and Tipping Customs: A Replication and Extension. Journal of Hospitality & Tourism Research, 28(3), 356–364. https://doi.org/10.1177/1096348003256605\n2. Rozin P. The meaning of food in our lives: a cross-cultural perspective on eating and well-being. J Nutr Educ Behav. 2005 Nov-Dec;37 Suppl 2:S107-12. doi: 10.1016/s1499-4046(06)60209-1. PMID: 16246277.\n3. Buckley, E. (2006). A Cross-Cultural Study of Weddings through Media and Ritual: Analyzing Indian and North American Weddings. McNair Scholars Journal, 10, 3.\n4. Macduff, I. (2006). Your Pace or Mine? Culture, Time, and Negotiation. Negotiation Journal, 22(1), 31–45. https://doi.org/10.1111/j.1571-9979.2006.00084.x\n5. Fagundes, D. (2017), The Social Norms of Waiting in Line. Law and Social Inquiry, 42: 1179-1207. https://doi.org/10.1111/lsi.12256\n6. Valentine, T.M. (1996). Politeness models in indian english.\n7. Incluude. (2023). The Cultural Atlas. Retrieved from https://culturalatlas.sbs.com.au/\n8. Wikipedia entries such as: https://en.wikipedia.org/wiki/Gratuity, https://en.wikipedia.org/wiki/Table_manners, https://en.wikipedia.org/wiki/Wedding_customs_by_country", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel dataset on cross-cultural judgments of normative behavior in a relatively unstudied setting in this area (US/India). The paper uses thorough human annotation, with each annotation labeled by 5 annotators. This dataset could prove useful for future studies in the area, towards computational studies of cultural norms across cultures.", "reasons_to_reject": "The key concern I have with this paper is in evaluation. Given that the paper's framing and motivation are in \"culturally aware\" NLI, the paper does not go into, in any detail, what *cross-cultural insights* this dataset provides. Yes, the authors have collected a dataset of annotator judgments in different cultures. However, no analysis of how these norms may differ across cultures, in what qualitative or qualitative way, is given. Thus, it would seem to me that the key evaluation here should not be on how existing NLI models tuned on other NLI datasets would perform on this task, but rather on how these *cultures* fundamentally differ. Without this analysis, it presents several questions on whether such a dataset is ultimately useful. I would implore the authors to further evaluate their annotations in this way, to see if computational studies of cross-cultural norms provides any cultural insights.\n\nEDIT: Thank you to the authors for providing more detail on what they see to be the specific contribution of this work. Nonetheless, my key gripe with this paper stands. To highlight that cultural factors contribute to label variations in NLI (and that they should indeed be considered) is, while entirely and accurately poised, not surprising—it would be hard to find researchers that disagree on the point that cultural factors and variations should be considered for many, if not all, traditional NLP tasks. Beyond this acknowledgment, it is precisely *how* we should take into account these cross-cultural differences that matters and informs future work; and on this front, the paper, in its current state, does not do a rigorous evaluation on the points listed above. Improving on these fronts would make this a much more rigorous paper.", "questions_to_authors": "How do annotator judgements vary across these two cultures? What does this say about these two societies? Any social insights that we can gain from this dataset?", "paper_topic_and_main_contributions": "The paper gathers a series of cultural norm premises using prompting on GPT with online (i.e. reddit) and existing sources (normbank), further generates hypotheses of these premises (i.e. given a situation, what is the intention of the speaker? --> \"the intention of the speaker is...\"), and collects MTurk annotations from workers based in the US and those based in India on whether these hypotheses are likely to be true or false using a Likert scale; dataset size is 900 premises and 2.7K hypotheses. The authors then formalize this as a NLI task, mapping \"very likely false\" as \"contradict\", and \"very likely true\" as \"entail\". \n\nThe authors then test how different NLI models perform on this task in both culturally specific (US/India) and overall (majority vote from all) versions. In general, existing models are shown to perform pretty mediocre on all 3 settings, with GPT-4 & culturally-specific prompting performing slightly better than others.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "Given the nature of this work, it should be expected that an IRB review is present. Specifically, \"Please indicate if your project was approved by an IRB (institutional review board).\" in (https://2023.eacl.org/ethics/faq/)"}}, {"review_id": "yaWok5mnO5", "reviewer": "Reviewer_QEVL", "summary": "", "questions": "See reasons to reject", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe thank the reviewer for their feedback. We are encouraged that the reviewer found our dataset and analysis are valuable for improving cultural awareness in NLP. We also appreciate that the reviewer touched on many important open questions in the area, such as dataset coverage and biases, artifacts and biases of using LLMs, majority and minority cultural perspectives, reproducibility of GPT results which we address below.\n\n### Response per Question\n1. *Differences between our work and Fung et al.*: Our work and Fung et al. have very different goals. Fung et al and also CH-Wang et al, in their own term, aim to **discover norms** from existing corpus or online resources. The discovery process might use NLI models as an automated data processing tool. Our work revisits the NLI task itself, specifically **how cultural norms can lead to label variations in NLI**. We take this position since cultural factors have been largely ignored in the NLI literature, yet are critical for building culturally-aware NLP systems.\n\n2. *\"The coverage of only two cultural groups is extremely limited\"*: We agree that having multiple cultural groups would be highly valuable. We will consider extending our dataset to cover more culture pairs in future work. For now, with two cultural groups, we can already observe how cultural variations contribute to NLI label variations. We also want to clarify that we do not expect the labels collected on a particular premise-hypothesis pair to generalize to other cultures, instead, we expect the proposed model-in-the-loop data collection framework to be reusable for studying NLI label variations in other cultures. We will emphasize this limitation of having only two cultural groups in our paper.\n\n3. *\"The impact of model-produced (GPT-3) biases is not sufficiently analyzed.\"* also *\"Biases related to gender, race, and other intersectional factors are not addressed sufficiently.\"*: We will move the bias analysis in the appendix A.2 to the main text and discuss the impact on our dataset as follows: (1) Biases in data generation: We acknowledge that GPT-3 generated outputs are susceptible to artifacts and bias, in particular factual errors such as wrong culture-norm associations and stereotyped gender and social role associations (Lucy and Bamman, 2021; Bender et al., 2021; Yuan et al., 2021). As a mitigation measure, authors have manually inspected all premises and hypotheses generated by GPT-3 before sending for annotation and discard all examples that promote stereotyping. However, we want to caution users of the dataset that the removal process itself can still be affected by the subjectivity of the authors. (2) Biases in annotations: Culture can vary greatly within a locale due to social class, age, and gender. Thus, the annotations are best used for analyzing cultural factors in NLI. In particular, the annotations should not be assumed to generalize to all populations in the United States and India. \n\n4. *\"Additional experiments are needed to justify why a certain prompt is chosen and whether authors explored other prompting methodologies\"*: We indeed conducted experiments on prompt selection. We construct a dev set for prompt selection \"For each task, we use a dev set of 60 premise-hypothesis pairs from MultiNLI and CALI to select the prompt with highest accuracy from a set of 5 candidate prompts.\" (Appendix D.1). We also disclosed the full set of prompts that we have explored in the Appendix A Figure 4 and Appendix D Figure 9.\n\n5. *\"The majority vote for labels risks marginalizing minority cultural perspectives.\"*: We understand the concern here. In fact, we discuss the limitations of majority vote and mitigation (line 583-588). However, we also want to clarify labels in the dataset vs. labels used in modeling. Our dataset preserves the full label distribution and supports modeling techniques like multi-annotator. We also present the full label distribution whenever we show examples from the dataset (e.g. Table 1 and 4). The only two places we use the majority vote are when we compare with existing NLI dataset (Section 3.6) and when we evaluate existing NLI models (Section 5.1). We unfortunately have to use the majority vote in these cases for a fair comparison, as existing dataset, i.e. SNLI and MultiNLI, use majority vote as the label.\nWe will include the paper suggested as a related work. Although the current work did not involve training new models, we think the modeling approach can be useful for future work that trains models on CALI.\n\n6. *\"Can you not use some situated, socially-grounded dialogue dataset that already exists?\"*: We agree that “socially-grounded dialogue” is another interesting data source to explore. We view “narratives or opinions” and “socially-grounded dialogue” as **complementary** in studying how cultural factors lead to variations in language inference. Interpreting narratives and opinions allows us to measure direct reasoning about cultural norms. Narratives and opinions are also closer to the format of existing NLI datasets, i.e. SNLI and MultiNLI.\n\n7. *\"pragmatic insincerity of sarcasm comprehension\"*: Why recognizing sarcasm involves social norms -- under Gricean pragmatics, sarcasm results from flouting; flouting involves violation of the Gricean maxims, such as quality (i.e. being truthful). Whether the listener perceives the speaker as being truthful can depend on the normative behavior of the situation.\n\n8. *Presentation of the task and data creation pipeline*: We are happy to revise Figure 2 and task definition in Section 3.2 to make the task clear. To clarify here, Figure 2 shows the entire dataset creation pipeline, which include premises (task inputs), hypotheses (task inputs), and human annotations (i.e. task labels). Our task follows the standard NLI task, where input is a pair of premise and hypothesis, label is the entailment relationship between the premise and hypothesis. The difference is that we condition the label on the culture group of the listener (annotator), i.e. having more than one label per input. Concrete examples of the task are shown in Table 1. \n\n9. *Presentation of linguistic dimension*: We (authors) annotated these dimensions ourselves by categorizing the label differences we observed on our dataset. We included additional results in response to review PaX1 on the cluster of examples we found per category and how we can use these categories for evaluation. Our additional results show that the linguistic dimension provides insights into which types of expressions models are good or bad at: For GPT-4, we found the model is perfect at resolving lexical ambiguity given culture indicators, yet struggles the most with referring expressions.\n\n10. *Presentation of r/AskEurope*: The name r/AskEurope might be a bit misleading. r/AskEurope is not just about European cultural norms -- it is a community where people from different cultures (including immigrants and tourists) discuss the cultural differences between Europe and their own cultures, which include people from US and India. For example: norms around the structure of the name exist in many cultures https://www.reddit.com/r/AskEurope/comments/hapxks/whats_the_structure_of_names_in_your_country.\n\n11. *\"Specific checks around whether someone has a significant presence in either country\"*: Yes, we did include a local-specific common sense question in the qualification test. About 50% of Amazon Turk workers passed the qualification test. (Appendix B.2 Annotation Quality Control).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) It tackles an important problem of improving cultural awareness in NLP systems, which is becoming increasingly relevant as systems interact with diverse users globally. The framework and definitions around cultural awareness are useful contributions.\n2) The new dataset CALI could be valuable - as it provides a concrete way to evaluate cultural awareness in inference tasks, something that has been lacking. Manually collecting premise-hypothesis pairs labeled by different cultural groups enables rigorous analysis.\n3) The categorizations of how cultural norms influence language understanding is somewhat insightful. Considering lexical, semantic, pragmatic factors and links to behavioral scripts provides nuance.\n4) The experiments on current LLMs highlight limitations in their cultural awareness at the knowledge vs context recognition vs inference levels. The analysis sheds light on where the gaps are.", "reasons_to_reject": "1) The coverage of only two cultural groups (US and Indian) is extremely limited. The variations observed may not generalize to other cultures, reducing the usefulness of the dataset and analyses. While prior work has done this, such as CH-Wang et al. 2023 (https://arxiv.org/pdf/2305.14492.pdf) across English and Chinese, it is not particularly trying to create an NLI data set, but rather understanding differences in cultures via NLI and explanation. One of the limitations of this paper is that CALI is probably only relevant to study cultural NLI across two countries. I also feel like there are at least several works that have done similar things in the past on the same theme, such as one by Fung et al. (2022) (https://arxiv.org/pdf/2210.08604.pdf), who also basically treat <dialogue,socialnorm> as an NLI task, and the data is a lot bigger and more diverse. Therefore, I am unsure what is new here. Using a model in the loop technique to study cross-cultural differences, framing it as an NLI task, has already been done. This work seems heavily inspired, limiting at least some of the novelty aspects.\n\n2) Relying heavily on language models like GPT-3 for data generation raises concerns about artifacts and bias in the data. The impact of model-produced biases is not sufficiently analyzed. We see some stuff in the Appendix, but this should be part of the main paper. The same holds for prompts. Too much of the results, and data depend on prompt sensitivity, rather than robust cultural reasoning. Additional experiments are needed to justify why a certain prompt is chosen and whether authors explored other prompting methodologies.\n\n3) The majority vote for labels risks marginalizing minority cultural perspectives. Better aggregation methods should be explored. Authors should read and cite [https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00449/109286/Dealing-with-Disagreements-Looking-Beyond-the]. Annotator disagreements may capture important nuances in subjective tasks like this, and their judgment is often ignored while aggregating annotations to a single ground truth. In order to address prior work, research has investigated the efficacy of multi-annotator models. This might be something to look into.\n\n4) Biases related to gender, race, and other intersectional factors are not addressed sufficiently.\n\n5) I am still not convinced why the authors are using sarcasm here. Is there any connection of the pragmatic insincerity of sarcasm comprehension with social norms? Additionally, prompting LLMs to generate narratives or opinions related to behavior, for example, “Describe a scene of tipping the waitress at a restaurant,” seems disconnected. Can you not use some situated, socially-grounded dialogue dataset that already exists?\n\n6) I find the writing extremely hasty. The task is unclear. There needs to be a clear diagram that shows the entire pipeline from data creation to NLI. What is your task? What are the input, output, and label, etc.? The introduction of the Linguistic Dimension seems rather abrupt and ad hoc. Did the authors annotate for a linguistic dimension? If so, who is annotating these? MTurks cannot be qualified for such a task. Finally, why are the authors using r/AskEurope when their annotators are either from the US or India? Additionally, the US has lots of immigrants who might not be familiar with US-specific norms. Authors should have specific checks around whether someone has a significant presence in either country to be able to perform this task.", "questions_to_authors": "See reasons to reject", "paper_topic_and_main_contributions": "The paper proposes a new framework for cultural awareness in language understanding, consisting of knowledge of cultural norms, recognizing relevant norms for a context, and making appropriate culture-specific inferences. It introduces a new dataset CALI to evaluate this, with entailment judgments from two cultural groups, revealing differences due to cultural factors. The paper categorizes how norms influence understanding across lexical, semantic, pragmatic and script elements. Experiments show current models have norm knowledge but lack full awareness - they struggle to identify relevant norms for a context and adapt inferences. The paper advocates improving cultural awareness in models through task and model design. It also examines ambiguity and subjectivity as factors influencing cultural variations in entailment labels, providing a nuanced analysis of what drives differences.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "@inproceedings{liu-etal-2022-wanli,\n    title = \"{WANLI}: Worker and {AI} Collaboration for Natural Language Inference Dataset Creation\",\n    author = \"Liu, Alisa  and\n      Swayamdipta, Swabha  and\n      Smith, Noah A.  and\n      Choi, Yejin\",\n    booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2022\",\n    month = dec,\n    year = \"2022\",\n    address = \"Abu Dhabi, United Arab Emirates\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.findings-emnlp.508\",\n    pages = \"6826--6847\"}\n\n@inproceedings{chakrabarty-etal-2022-flute,\n    title = \"{FLUTE}: Figurative Language Understanding through Textual Explanations\",\n    author = \"Chakrabarty, Tuhin  and\n      Saakyan, Arkadiy  and\n      Ghosh, Debanjan  and\n      Muresan, Smaranda\",\n    booktitle = \"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing\",\n    month = dec,\n    year = \"2022\",\n    address = \"Abu Dhabi, United Arab Emirates\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.emnlp-main.481\",\n    pages = \"7139--7159\"}"}}, {"review_id": "we8kzja9Dk", "reviewer": "Reviewer_PaX1", "summary": "", "questions": "* A: Explain clearly what Table 2 shows us.  Specifically, what two numbers\n  are you comparing (cite them) and what conclusions do you draw?  Consider\n  my comments below.\n\n* B: How do you think the LLM performances in Table 3 relate to the LLM\n  performances in Table 2?\n\n* C: Will you release the corpus?  How?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe thank the reviewer for their detailed feedback on experiment analysis and presentation. Following the suggestions, we performed additional analyses to help address the concerns below.\n\n### Response per Question\n\n1. *\"The categories developed in Section 4 are interesting but they are not used in a quantitative analysis.\"*: Thanks for the suggestion! We add an analysis below to show how the categories developed in Section 4 can be used to perform error analysis of models.\n\nWe use the semantic vs. pragmatic categorization to investigate which categories the model performs the best/worst. We evaluate on the same set of examples used in Table 2, which has about 12% semantic and 88% pragmatic cases. We focus our analysis on GPT-4 results in Table 2 (with the country indicator prompting).\n\nExample distribution per categories (semantic, pragmatic)\n| | Lexical Ambiguity (7%) | Referring Expressions (5%) | Violation of Maxims (9%) | Other Implicatures (79%) | \n|:---:|:---:|:---:|:---:|:---:|\n|GPT-4 US | 1.00 | 0.73 | 0.74 | 0.81|\n|GPT-4 IN | 1.00 | 0.61 | 0.82 | 0.72|\n\nWe indeed find some interesting patterns: GPT-4 performs the best on the lexical ambiguity task and performs the worst on the Winograd schema style reference resolution task. For the reference resolution task, despite having the necessary cultural knowledge (as shown in the knowledge probing task), GPT-4 is less sensitive to cultural indicators presented in the prompt, but biases towards using syntactic cues.\n\nWe show examples of GPT-4 predictions on the two categories below. The number is GPT-4 predicted probability of entailment.\n\n* Lexical Ambiguity\n  * P: I had some ladyfingers for dinner.\n  * H: I had some desserts for dinner.  US: 0.80  IN: 0.05\n  * H: I had some vegetables for dinner. US: 0.20 IN: 0.90\n* Referring Expressions\n  * P: She picked the white rose bouquet instead of the red rose bouquet, because it contrasts the color of traditional wedding attire.\n  * H: She picked the white rose bouquet instead of the red rose bouquet, because the white rose bouquet contrasts the color of traditional wedding attire. US: 1.00 IN: 1.00\n  * H: She picked the white rose bouquet instead of the red rose bouquet, because the red rose bouquet contrasts the color of traditional wedding attire. US: 0.00 IN: 0.00\n\n2. *\"takeaway from the evaluation of the LMs\"*:\n(1) GPT-3.5/4 can achieve a higher accuracy on the US-label (culturally aware view) than All-label (culturally generic view), i.e. GPT-3.5 0.74 vs. 0.74 (+0), GPT-3.5+US 0.70 vs. 0.75 (+0.05), GPT-4 0.76 vs. 0.80 (+0.04), GPT-4+US 0.74 vs. 0.80 +0.06). \n(2) All models evaluated, including GPT-3.5/4, do not perform well on the IN label (a max F1 of 0.74 among all models for IN vs. a max F1 of 0.80 among all models for US), i.e. there is an accuracy gap between cultures.\n\n3. *\"How do you think the LLM performances in Table 3 relate to the LLM performances in Table 2?\"*\nWe discuss the connections in Section 5.2 and to recap:\n   * Table 3 serves as an error analysis for the GPT-3.5 and GPT-4 results in Table 2.\n   * We show that (1) higher accuracy on NLI tasks (Table 2 comparing GPT-3.5 vs. GPT-4: 0.74 vs. 0.80 on US and 0.72 vs. 0.75 on IN) corresponds to higher accuracy on all three diagnostic tasks, with the largest improvement observed on “context task” (Table 3 “all 40 behaviors” row: comparing 0.62 vs. 0.88). (2) Comparing across the three tasks in Table 3, the knowledge task is mostly saturated for the best model (GPT-4 0.98 on US and 0.93 on IN) context and inference tasks have larger room for improvement.\n   * The two observations suggest, apart from adding cultural knowledge to the model, we should also improve the model's ability to recognize when the cultural norm knowledge is relevant to the context and how to update inference with respect to normative behaviors. \n\n    Regarding the concern on whether these probing tasks are faithful explanations of the model performance on NLI task, we think this is an open question for prompting-based explanations (see Atanasova et al. 2023). However, if we define a model having certain knowledge as the model can pass at least one set of behavioral tasks that requires such knowledge, then we can make a claim about whether the model has certain knowledge, i.e. in our case, diagnosing whether a model knows a particular cultural norm or not.\n\n4. *\"The use of the term \"implicature\" is confusing.\"*\n    * In the context of \"*culture-dependent inference is implicature by nature*\" (and other places where we discuss inference): We use “Implicature” to refers to inferences that are cancellable, as opposed to textual entailment that are not (Line 203-205). Similar usage of “implicature” can be found throughout the NLI literature. See Manning, 2006, Gordon et al., 2012, Pavlick and Kwiatkowski, 2019, Rudinger et al., 2020 for a discussion on “implicature” vs. “textual entailment” and why “implicature” should be included in NLI (Line 151-154). \n\n       Culture-dependent inferences are implicatures, as culture-dependent inferences can always be canceled by adding an norm that counters the original one used in the inference. For example, “I didn't leave a tip.” implies “I am not happy with the service.” can be canceled by “because tipping is not practiced in this culture”. \n    * In the pragmatic dimension, i.e.\"*implicatures, including connotations of phrases*\": We agree that the usage here is a bit confusing. We use it as a label to loosely refer to conversational implicatures. We will choose a better label to refer to this category. \n\n5. *\"The authors do not actually promise to release the resource.\"*: We are planning to release the corpus. Also see our response to ethics review (b).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper addresses an interesting issue in a novel way.\n\n* The paper presents an interesting resource.", "reasons_to_reject": "* Some important parts of the paper are hard to understand (Sections 3,\n  5.1), despite mostly grammatical English and quite  abit of detail (maybe\n  too much?).\n\n* The categories developed in Section 4 are interesting but they are not\n  used in a quantitative analysis.\n\n* It is unclear what the takeaway from the evaluation of the LMs is, other\n  than that GPTn is amazing.\n\n* REMOVED. The use of the term \"implicature\" is confusing. [This is not a reason to reject after all.]\n\n* The authors do not actually promise to release the resource.", "questions_to_authors": "* A: Explain clearly what Table 2 shows us.  Specifically, what two numbers\n  are you comparing (cite them) and what conclusions do you draw?  Consider\n  my comments below.\n\n* B: How do you think the LLM performances in Table 3 relate to the LLM\n  performances in Table 2?\n\n* C: Will you release the corpus?  How?", "paper_topic_and_main_contributions": "This paper addresses the problem that writers produce and readers consume\nlanguage in a particular cultural context, which includes culturally\nspecific knowledge about social norms.  This kind of knowledge is assumed\nto be shared.  NLP systems are not always sensitive to this kind of\nknowledge.  This paper introduces a culturally aware NLI dataset based on\nEnglish texts from the US and India, and uses it to investigate whether and\nhow current models are sensitive to cultural context.  The authors provide\nan analysis of the dataset and of the performance of off-the-shelf systems\nand zero-shot prompting approaches.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Looks good"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "003: A simple expression like “I didn’t leave a tip” can imply a strong sense of dissatisfaction, only if tipping is the norm.\n\nNo comma\n\n\n029: Language, as a tool of social interaction, is grounded in cultural norms.\n\nNot sure what this means.  Maybe \"cultural norms affect language usage\"?\nGrounding has a specific meaning.\n\n\n145: Unlike cross-lingual NLI captures culture on task-level, we measure cultural variations on the instance level.\n\nEditing error.\n\n\n\n265: “Write a sarcasm over waiting in line.” \n\nWhat does this mean? A “sarcasm” is not something one can write, and the\nword is a count noun in English.\n\n\n\n315: We recruit in total 125 US and 35 IN workers, which matches the prior\ndistribution of MTurk workers from the U.S. (75%) and India (16%) reported\nin previous studies (Difallah et al., 2018).\n\nWhy use this distribution?\n\n\n\n\n354: As culture-dependent inference is implicature by nature, annotator disagreement is expected to be more common than dataset focusing on textual entailment.\n\nREVIEWER REMOVES THIS COMMENT -- I misread the quote above.\n\nOriginal comment:\nWhat do you mean? Conversational implicatures are not\nculture-specific. “Implicature” has various specific meaning in semantics,\ndistinct from “entailment” and \"implication\". It is not simply the noun\nderived from “imply” (which is “implication”). You need to be more\nspecific. Do you mean Gricean conventional implicatures? Do you mean\nimplicature in the sense of Relevance Theory? Are you inventing a new\nmeaning? Be precise.\n\n369: Deep Dive of Cultural Variations\n\nA little unclear what this section is about: it lays out dimensions of\nanalysis, but no serious analysis is performed with theses dimensions.\n\n\n399: 1) implicatures, including connotations of phrases de- scribing\nnormative or transgressive actions, e.g.\n\nSee above comment on your usage of the word “implicature”.\n\n\n406: 2) violation of Gricean maxims, either due flouting, e.g.\n\nDue to\n\n\n409: or a clash with politeness, e.g.\n\nPoliteness is not a Gricean maxim.\n\n\n411: The Knowledge Dimension\n\nWas this annotated?  How was this annotated? Was there agreement?\n\n\n415: speaker intentions, e.g. “context” in Example #4 and #5, causes, e.g.\n\nI don’t understand this. What does “context” mean here and how is it\nrelated to speaker intention?\n\n\n460: We select a subset of 40 behaviors,\n\nHow?\n\n\n465: For the entailment classification task, we perform a binary classification.\n\nI don’t understand what you do with the “neutral” label in the evaluation.\n\n\n493: Results are shown in Table 2.\n\nWhy does one task get 2 digits and the other 4 digits?\n\n\n\n494: there is a larger gap between the culturally generic view\nv.s. culturally aware view, up to 5% and 13% on the two tasks for GPT-4.\n\nI actually don’t understand what numbers you are comparing in Table 2.\n\n\n498: More- over, when prompted with locale as cultural indi- cators, model\naccuracy mostly improves with the U.S. locale on US labels.\n\nNo they don’t (unless I am misreading the Table). I take it you are\ncomparing the X and X+Y lines, where X is a model, and Y is either US or\nIN. For the two tasks and the two possible models (GPT3.5 and GPT4), ONLY\nGPT3.5 helps with X+Y evaluated on Y data, and only by 0.01 which is\nprobably not statistically signifying given your small data pool.  \n\n\n\n509: Fine-tuning on cross-lingual NLI unfortunately does not help…\n\nWhere is this experiment described? Where are the results?\n\n\n514: We investigate at which levels the LLMs lack cul- tural awareness.\n\nThe LLM is just a language model. It is unclear whether the performance in\nthe three tasks presented here actually relate to the performance on the\nNLI tasks. What evidence is there for a correlation?"}}]

Author rebuttal:

Meta-review: This paper presents a large dataset of NLI annotated with cultural awareness. The resulting CALI dataset would enable more research into how cultural differences affect the task of NLI. This work also opens future research directions with other languages and cultural differences. While, as pointed out by one reviewer, the dataset is limited to just one other culture of India, it is nevertheless challenging and important to build this cross-cultural dataset, and I encourage the authors (and our NLP research community) to broaden the set of cultures considered in future work.

There was some concern that this research collected human annotations but does not present IRB approval. Please describe how the research ethics were considered and adhered when collecting the annotations and go through IRB if such process exists in your research institution.
Decision: Accept-Findings