NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation
Authors: Oliver Li
Mallika Subramanian
Arkadiy Saakyan
Sky CH-Wang
Smaranda Muresan
Venue: EMNLP 2023 Main (2023)
Keywords: social norms
resources and evaluation
large language models

Abstract: Social norms fundamentally shape interpersonal communication. We present NormDial, a high-quality dyadic dialogue dataset with turn-by-turn annotations of social norm adherences and violations for Chinese and American cultures. Introducing the task of social norm observance detection, our dataset is synthetically generated in both Chinese and English using a human-in-the-loop pipeline by prompting large language models with a small collection of expert-annotated social norms. We show that our generated dialogues are of high quality through human evaluation and further evaluate the performance of existing large language models on this task. Our findings point towards new directions for understanding the nuances of social norms as they manifest in conversational contexts that span across languages and cultures.

NORMDIAL:
A Comparable Bilingual Synthetic Dialogue Dataset for Modeling Social Norm Adherence and Violation
Oliver Li Mallika Subramanian
Arkadiy Saakyan Sky CH-Wang' Smaranda Muresan
Department of Computer Science, Columbia University
Data Science Institute, Columbia University
(a14143, ms6544, smara]ecolumbil edu, {a. saakyan, skywang] cowumbia. edu
Abstract
Chinese Norm Adherence Dialogue
#7. 38 WERE, 3895 2A58 (Undle LJ, this dish is delitious and your cooking skills are amozing)
Social norms fundamentally shape interper- sonal communication. We present NORMDIAL, a high-quality dyadic dialogue dataset with tum-by-turn annotations of social norm adher- ences and violations for Chinese and American cultures. Introducing the task social norm observance detection, cally generated in both Chinese and English us- ing human-in-ther tho-loop pipeline by prompting large language models with small collection of expert-annotated social norms. We show that our generated dialogues of high quality through human evaluation and further evalu- ate performance existing large language models on this task. Our findings point towards new directions for understanding the nuances social norms as they manifest in conversa- tional contexts that span across languages and cultures.
members
(Not at all. thanks for the compliment, it's nothing, just follow the recipe)
Undle
Norm Adherence
Family mombers
N B. (t-t (Really? We al think this already deliciuss)
dataset is syntheti-
FFUi My # #: still have lot to learn, and hope to bring you better dishes next time)
2: n # 3
Unde
Norm Adherence
Chinese Norm: reeponding
compliment, can
respond with suphemism dowmplyy their achievement.
American Norm Violation Dialogue
Excuse me. could you please your voice? disrupling the other gym--oors
Sarah
(Continuas phone call the same volume)
James
(Sighs) Hey. tumn off your phone and quiet down. You're dienupling training session and the other gym-goers. respect, man.
Sarah
1 Introduction
Social normss-ipllicity learned notions of ac- ceptable behavior both develop from and guide our everyday interactions (Sherif, 1936). As with the value systems that underlie these notions, the acceptability and deemed typicality of behaviors varies across cultures (Triandis et al., 1994). For example, due to strong emphasis individual- ism, open and direct expression of opinions and disagreement often encouraged and valued in Western cultures (Arieli, 1964), while such acts may often be viewed as disruptive to social order in Eastern Asian cultures that value collectivism (Triandis, 1993). Understanding these cultural nu- ances is key to empower computational systems to reason across cultural contexts (Liu et al., 2021).
Norm Violation
American Norm:
Whan an equal, lower status, or unfamiliar person gives a direct command, it considered unaccaptable to give a command rude impolite language [..]
Figure 1: Examples generated dialogues with ad- herences (top, in Chinese) and violations (bottom, in English) to social norms about responding to compli- ments and giving requests, respectively.
context of identifying social norm adherences and violations across multiple cultural contexts. This paucity of data hinders progress towards develop- ing cross-cultural communication tools.
As a small step towards addressing this gap, leveraging recent successes in utilizing large lan- guage models (LLMs) for social data generation and augmentation (Kim et al., 2022a: Chen et al., 2023), we propose a human-in-the -n-tth--oooo framework to synthesize realistic conversational data under expert prompting for modeling social norm adher-
We introduce NORMDIAL, bilingual synthet- ically generated dyadic dialogue dataset of social norms as they appear within the context of con- versational interactions. Gathering realistic data at scale in this domain presents challenging and potentially cost-prohibitive task, particularly in the

---

ence and violation. Using this human-Al collabo- ration framework, we generate a series of 4231 dyadic dialogues totaling 29550 conversational turns grounded in theoretical norm categories (Lin- guistic Data Consortium, 2022) across Chinese and American cultures, and demonstrate that our syn- thetic bilingual conversations are comparable to exceed the quality of existing, naturally occur- ring datasets under interactive human evaluation and automatic metries; examples of our dialogues are shown in Figure Our dataset presents so- cial norm adherences and violations labeled on a dialogue- turn basis; with this labeling task decou- pled from dialogue generation, we further evalu- ate the capability of existing LLMs in reasoning about norm adherences and violations in a con- versational setting and show that existing mod- els often fail reason correctly about these con- texts. We hope that this resource will further mo- tivate research towards designing better systems able to promote more fluid cross-cultural conversa- tional interactions. We make NirmDial available at
(Ziems et al., 2023), spanning commonsense rea- soning (Sap al., 2019; Rashkin al., 2018), the determination of appropriate and morally ethical behavior (Emelin et al., 2021; Jiang et al., 2022), and the further grounding of this behavior in areas like dialogue systems and situated question answer- ing (Kim et al., 2022b; Ziems et al., 2022; Gu et al.. 2022) more specifically
underlying knowledge of social norms. While most work on computa- tional models of social norms has been focused on the American context (Forbes et al., 2020), recent work has begun to bridge this gap cross-culturally to enable a comparison of descriptive nuances in norms across cultures (CH-Wang et al.. 2023; Fung et al., 2022). Here, our work builds dialog dataset around conversational social norms for both Amer- ican and Chinese cultures.
3 The NORMDIAL Dataset
NORMDIAL is a human-in-the-loww synthetically generated bilingual (Chinese & English) dyadic di- alogue dataset for studying social norms as they ap- pear in different conversational contexts. Dialogue turns are further labeled whether they adhere or violate given social norm with textual explana- tions. Our human-Al collaboration framework for creating NORMDIAL shown in Figure 2.
2 Background and Related Work
LLMs for Synthetic Data Generation. Prompt- ing LLMs to synthesize and augment language data for existing tasks (Li et al., 2022; Møller et al., 2023; Chen et al., 2023) has emerged as a viable, cost-effective alternative in lieu of crowd-sourced annotation at scale or alternative strategies such as fine-tuning language generators (Papangelis et al., 2021: Zhang et al., 2020) in the dialogue domain. LLMs, trained on massive amounts of web text, suffer from representational and allocational harms (Blodgett et al., 2020; Weidinger et al., 2021). Yet, such models often also possess high algorithmic fidelity in the realm of representing latent social variables (Argyle et al., 2023), in that these sources of bias may often be finely controlled for to accu- rately emulate responses from a variety of human demographic sub-popue cabb-opulations in areas such as pre- dicting historically missing survey responses in so- cial research (Kim and Lec, 2023). Here, under this vein, we employ a human-in-the-loop framework to both finely condition and validate the generation of dialogues for modeling social norms.
Social Norm Augmentation (Stage 0 in Figure 2). The Linguistic Data Consortium (LDC) (Linguistic Data Consortium, 2022) taxonomizes 10 catego- rizations of social norms in its quidelines-apolgee compliment, condolence, criticism, greeting, leave, persuasion, request, response to compliment, giv- ing thanks and provides a detailed set of associ- ated norms (5 for each category) for Chinese cul- ture. Examples of verbal evidence of adherence to norm in conversational context, alongside the relationship data of each hypothetical interlocutor, are provided details for norm definitions.
We augment this starting set of validated so- cial norms by in-context prompting CharGPT (Wei et al., 2022), making use of LDC norm descriptions and examples in our prompt as few-shot in-context examples, to generate greater set of social norms for Chinese culture, which are then conditioned on and further prompted to generate corresponding norms for American culture. To verify the correct- ness and applicability of generated norms for each cultural context, we task annotators who identify as native speakers of each respective language and who have significant (10+ years) lived experiences
Computational Social Norms. Our work is situ- ated in the broader push towards empowering com- putational systems of interaction with the capabil- ity to reason in coutolcul situated contexts

---

Prompt:
Prompt:
LDC Expert-Arnatited
Given social nom, you are taeked magine 10 ecenaries that conversation that entalls the Norm
Given stuation norm, you create and indiudo details to the siuution
Norm Semple
Aomnuugnnnaatonn
desirable to give light apology ...
Norm:
Stage Scenaria Generation
Scenarto: student lbberiant collage lirwar)
Norm:
Stualiin Elsbaraiion
Stage 2:
Situation:
Naaoring calling ltrrar ..)
Labeled Dialogue:
(Xeoming What's Not Relevant| not spology
(--0 M. ÆR #rS 7F
Stage Bym Norm Labeling
dh d #t ttss
Dialogue:
Dellguue Generation
Stago 3:
(Xiammig: Hey, Ets mo. What's up?)
Prompt: Given dilogue and norm
Prompt
Stop 1. Summarize the norm
Given real-ffe stuation around a Chinase social norm topic, creatiely generate dialogue
Step
performs norm
Stop 3. For each turn, label Adhered, Violated or Not Relevant
Stap 4, Aftor each labal, provide short oxplanation
Our human- Al collaboration framework for creating NORMDIAL, through (0) norm augmentation with a small set of expert-labeled social norms under the LDC taxonomy, (1) scenario generation, (2) situation elaboration, (3) dialogue generation, and (4) turn turn-by-tur by-turn norm labeling with human verification at every stage. We show a Chinese dialogue generation example here for illustration; the same pipeline is adopted for English dialogues.
Figure 2:
in each culture to manually evaluate and rectify if each generated norm is (1) factually correct accord- ing their own lived experiences, (2) in line with the defined norm category, (3) specific the cul- ture, and (4) detailed its description, removing those that did not satisfy these criteria. This pro- cess enables us to collect a dataset of 133 and 134 Chinese and American social norms, respectively (see Table 1). Additional norm examples alongside the prompts used are shown in Appendix B.
ChatGPT using one-shot learning to generate list of 10 short scenarios in the form of social relation; location where given norms are most likely take place in real life. In the second stage, we combine each scenario with given social norm to enable CharGPT to elaborate on and expand each scenario description into ones that are more detailed and realistic. In total, we obtained 4231 unique situ- ations from this process; the topics of elaborated situations as captured by 30-topic Latent Dirichlet Allocation (LDA) model are presented in Appendix E. To ensure that situations are elaborated faithfully from the given norm, we collected a sample of 218 situations along with their norms for three anno- tators to verify if cach situation entails the norm. The results in Appendix D show high faithfulness scores, with the lowest for American norm viola- tions. For the final version of NORMDIAL, we manually verify and remove situations that deviate from the norm descriptions (releasing both raw and cleaned datasets).
Chinese Norm (Respond to Compliments) When person of lower status respond the compliments of one high status, one can say "2 Fi 52 fi ft & fs E. ( Bi 48 RK 3433" (1 still have many short- comings, I'll seek advice and learn from my seniors.)
American Norm (Respond to Compliments); When person of lower status responds to compliment from someone of higher status, it is common to express grati- tude and acknowledge the compliment gracefully. [...]
Table I:
Examples of manually verified Chinese and American norms generated by ChatGPT.
Scenario Imagination and Situation Elaboration (Stages and 2 in Figure 2) Social norms mani- fest in different ways depending on the conversa- tional context (Lockshin and Williams, 2020). An issue in dialogue generation from small amount of hand-written data is its lack of diversity, as in- context examples have large impact on prompting results (Min et al., 2022). To tackle this issue, with our augmented set of social norms, we first prompt
Dialogue Generation (Stage 3 in Figure 2). By prompting ChatGPT with pairs of norms and their elaborated situations along with an in-context ex- ample, generate turn-by-turn dyadic dialogues that either adhere norm. Shown in Figure find that CoT prompt- ing with Scenario Generation (Stage 1) and Situ- ation Elaboration (Stage 2) greatly improves di- alogue lexical diversity as compared to directly
violate the given social

---

Distinct N-Grams: CoT vs. Simple Prompting
dialogue-turn norm adherence and violation labels with a pool of 12 annotators, each possessing sig- nificant lived experiences in both cultures.
1.0
CaT (Chinese)
Simpie (Chinese)
CoT (US)
0.8
Simple (US)
Synthetic Dialogue Quality. To determine the quality of our Chinese and American synthetic dia- logues, we perform a human evaluation against two types of baselines: (1) specifically curated domain- relevant dialogues pre-existing and naturally oc- curring dialogue datasets, and (2) human-written dialogues specifically for our task.
Distinct-N
0.6
0.4
For the former, we compare our Chinese dia- logues against a collection of 200 dialogues ran- domly sampled from the Linguistic Data Consor- tium (LDC), which contains 413 Chinese conversa- tions with expert-annotatee turn-by-tumn Chinese norm observations from video transcripts. These conversations contain dialogues and chitchat from sources such as TV shows, Vlogs, and other types of videos from Bilibili, Chinese video-sharing platform. For English dialogues, as no existing domain-comparable dialogue is available, we com- pare against set of 200 realistic, human-writter dialogues reflecting daily communication that cov- various topics about daily life, DailyDialog (Li et al., 2017); comparison that has been used for previous evaluations of synthetic dialogue quality (Chen al., 2023).
0.2
0.0
1-gram
2-grams
3-grams
4-grams
N-Grams
Figure 3: Distinct N-grams for both US and Chinese generated dialogues under Simple and CoT prompting. CoT prompting with situation elaboration improves dia- logue diversity compared to Simple prompting without using situations. Chinese texts are tokenized by jieba.'
generating dialogues from norms alone (Simple), measured by distinct N-grams. Prompts used at each stage are provided in Appendix C.
Automatic Turn-level Annotation of Norm Ad- herence and Violation (Stage 4 in Figure 2). With this set of situationally grounded dyadic di- alogues, we then further prompt ChatGPT using Chain-of-Thought (CoT) (Wei et al., 2022) reason- ing to label whether each dialogue turn (1) adheres to, (2) violates, or (3) is not relevant to the given norm. Providing the social norm, situation, and di- alogue, prompt ChatGPT to (1) summarize the norm into a short description of its rules and actions the norm action; (2) respond with the names of the characters who are mostly involved with acting alignment with the norm as the norm actors; and (3), for each dialogue turn, with information about the norm action and norm actors, predict the label (Adhered, Violated, or Not Relevant) together with short textual explanation of each decision.
For the latter, to evaluate our dialogues against Human-written counterparts specific to our task, we asked three native speakers of Chinese and English to creatively write set of 20 dialogues for each language, based sample of 20 norms and situa- tions, which we selected from each of our 10 norm categories by randomly selecting an adherence and violation (norm, situation) pair from each category. We ask sets of 3 annotators to rate each con- versation, formatted consistently, on their (1) nat- uralness, or how natural each dialogue sounded, (2) nativeness, if they thought the dialogue came from a native speaker, (3) coherence, and (4) in- terestingness, each on a 5-point Likert scale, with final scores for each aspect being the average of the scores received. On a separate task, we ask annotators to rate if synthetic dialogues were faith- ful and on-topic to their provided social norm, i.e. does the main topic of the dialogue match the pro- vided social norm, yes/no?, of which final labels are obtained via majority-voting. A comparison of quality evaluation scores is shown in Table 2 and details about evaluation metrics are shown Appendix Section A.
In the next Section, discuss the evaluation of generated dialogue quality and automatic turn-level annotations of adherences and violations.
4 Evaluation
We conduct a human evaluation on the quality of dialogues and the correctness of model-predicted
hhaps///gitubb.oom//sssjiiiia

---

Sources
Natural
Native
Coherent
Interesting
ing disagreements via discussion to produce set of final gold standard labels for 1281 Chinese and 1503 English dialogue rounds. Table 3 shows the precision, recall, and Fl scores of ChatGPT predic- tions against ground truth labels, stratified across dialogue language and label categories.
Ours (ZH)
3.78
3.78
3.94
3.66
LDC
3.69
3.81
3.28
2.53
Human (ZH)
4.50
4.60
4,55
4.25
Ours (EN)
4.41
4.78
4.85
4.22
DailyDialog
4.39
4.40
4.72
4.34
Human (EN)
4.15
4.95
4.20
3,95
Table Dialogue quality evaluation across NORMDIAL synthetic dialogues (Ours), established and domain- specific baselines (LDC and DailyDialog). and human- written baselines (Human). Chinese language data marked as ZH; English as EN.
Chinese
Norm Labels
Precision
Recall
F1-Score
Adhered Not Relevant
78.4%
84.3%
0.81
94.4% 53.6%
80.7%
0.87
Violated
85.6%
0.66
For baseline (1), Annotators rate NORMDIAL dialogues higher in almost all dialogue quality as- pects than their pre-existing curated and naturally occurring dialogue baseline counterparts, with only DailyDialog outperforming NORMDIAL in inter- estingness and LDC outperforming our synthetic dialogues in nativeness; synthetic dialogues were rated higher statistically significant degree in coherence and interestingness for Chinese and na- tiveness and coherence for the American side. As for benchmark (2). NORMDIAL. dialogues were found to be of higher quality than their specificially tasked human written counterparts for English and lower for Chinese, in line with language perfor- mance differences for ChatGPT. Despite this per- formance difference in Chinese dialogues, it took the average annotator more than an hour to finish writing 20 dialogues; as this is a highly creative task, tasking annotators in our task can prove to be a challenge in scalability, given emerging evidence of annotator fatigue (Derczynski et al., 2020), espe- cially for creative tasks. On the other hand, taking the majority vote of dialogue on-topic labels from annotators showed that 92.5% and 86.5% of dia- logues for Chinese and English, respectively, were faithful to their prompted (norm, situation) pairs.
English
Adhered 77.0%
89.7%
0.83
Not Relevant
95.9%
68.6%
0.80
Violated 51.2%
98.8%
0.68
Table 3: ChatGPT norm adherence and violation label prediction performance against annotator-corrected gold labels.
Shown in Table 3, empirically, ChatGPT achieved higher FI score correctly predicting if dialogue round adhered to or was not relevant to given norm, but performed significantly worse in predicting norm violations for both languages. Given violation's s high recall, conversational turns that are not violations of the norm were falsely la- beled as so, even with few-shot expert prompting. Under qualitative examination, we found that many of the turns that were falsely labeled violations served to instead provide context before the actual violations rather than the violation behavior itself, suggesting the potential for further future improve- ment in this area.
Conclusion
We presented NORMDIAL, synthetic, validated, high-quality dyadic dialogue dataset with turn-y-- turn annotations of social norm adherences and violations for Chinese and American cultures Chinese and English. Our evaluation of synthetic dialogue quality reveals that our dataset is com- parable to and/or exceeds the quality of naturally occurring and Nomain-specifii dialogue datasets. Furthermore,
Automatic Turn-level Annotation. As our auto-
matic dialogue-turn norm adherenceeviolation la- beling via prompting is separate from dialogue gen- eration, a natural question arises as to how well existing L.Mss are able to perform this task, i.e.,
how well can LLMs accurately detect if a given social norm is adhered to or violated for conver- sational round? Here, collecting sample of 200 dialogues for each language, two annotators manu- ally validated the correctness of ChatGPT labeled dialogue rounds to check for correctness, resolv-
analysis of LLM predictions of norm observance reveals areas for existing models to improve in this domain. Our resource points to- wards new directions for understanding the nuances of social norms as they manifest conversational contexts that span across languages and cultures.

---

Limitations
are by nature aggregated judgments of acceptabil- ity that are subject to variation across longitudinal scales, Nonetheless, central contribution of this work a framework to create faithful dialogues that are themselves based on any given set of social norms, which allows for a Aplug-and-play dia- logue generation for any additional/given set of social norms.
Annotation Bias. While have augmented our synthetic data generation pipeline with human vali- dation at every stage from individuals who possess significant lived experiences Chinese and Amer- ican cultural contexts to ensure correctness, it is important to acknowledge that our ultimate repre- sentation of the views and values of these cultures is limited to these lived experiences. In working towards more culturally representative studies, it is important broaden the views and values of those who are represented in experimental data and acknowledge the presence of further intra-cultural variations (Plank, 2022).
Acknowledgements
We thank Maximillian Chen and Bo Feng for their helpful comments, thoughts, and discussions; Win- ston Wu, Zoie Zhao, Shiyu Zhang, Gechen Shen, Anxin Yi, Feiyang Zhu, Christopher Lee, Aniv Ray, Batool Taraif, and other anonymous crowdworkers for their help in annotations and evaluation; and the anonymous reviewers for their helpful feedback. This research being developed with funding from the Defense Advanced Research Projects Agency (DARPA) CCU Program No. HR001122C003A
Language Model Bias. As with that which has been aforementioned, language models also pos- sess sources of bias arising from the fundamental trained behavior of the tendency mimic patterns important to
in their training data. As such, critically question and challenge the viewpoints of those who are represented and reproduced within and which may seep into our dataset as a result, even under significant human validation.
The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Sky CH-Wang is supported by National Science Foundation Graduate Research Fellowship under Grant No. DGE-2036197.
Ethical Considerations
Names as Sources of Bias. Within our human evaluation and annotation, deliberate measure was implemented to address the potential introduc- tion of biases by excluding character names during dialogue rounds. The purpose of this approach was to minimize the potential impact of personal biases or preconceived notions that may arise from spe- cific names, ethnic backgrounds, or genders. As a result, annotators were solely guided by the dia- logue's is content and the cultural norms under discus- sion. In our data release, need for future work to undertake similar measures mitigate such sources of bias in annotation.
References
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language mod- els to simulate human samples. Political Analysis. 3133)3337-351.
Yehoshua Arieli. 1964. Individualism and nationalism in American ideology. Harvard University Press.
emphasize the same
David M Blei, Andrew Y Ng, and Michael Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jnn)993-1002..
Social Norms. Cultural nuances are complex and multifaceted and do not remain static across time. Here, we have collaborated with social science re- searchers with significant established expertise in Chinese and American cultures to ensure the accu- racy and validity of our set of social norms under rigorous verification, further consulting and adapt the taxonomy provided by the LDC in defining and characterizing social norms. is important to note here that it is impossible to have a "ground truth" set of social norms for every culture, as they
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: critical survey "bias" NLP. Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguisties, pages 5454- 5476, Online, Association for Computational Lin- guistics.
Sky CH- Wang. Arkadiy Saakyan, Oliver Li, Zhou Yu. and Smaranda Muresan. 2023. Sociocultural norm similarities and differences via situational alignment and explainable textual entailment. arXiv preprint 2iiv:2305.14499

---

Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tui Tur. 2023. PLACES: Prompting language models for social conversation synthesis. In Findings of the Association for Com- putational Linguistics: EACI. 2023, pages 844-868, Dubrovnik, Croatia. Association for Computational Linguistics.
Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Junsol Kim and Byungkyu Lec. 2023. Ai-augmented surveys: Leveraging large language models for opin- ion prediction in nationally representative surveys. arXiv preprint arXiv:2305.000 0062000
Junlong Li, Zhuosheng Zhang, and Hai Zhao. 2022. Self- prommponn large language models for open- domain qa. arXiv preprint arXiv:2212.08633.
Leon Derezynski, Julie Binau, and Henri Schulte. 2020. Maintaining quality in fever annotation. In Proceed- ings of the Third Workshop Fact Extraction and VERification (FEVER), pages 12.46.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-tur dialogue dataset. In Proceedings of the Eighth International Joint Conference Nat- ural Language Processing (Volume I: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Denis Emelin, Ronan Le Bras, Jena D. Hwang. Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situ- ated reasoning about norms, intents. actions, and their consequences. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Linguistic Data Consortium. 2022. Mandarin/Chinese Development Annotation LDC2022E18..
CCU TAI
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chem- istry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653 Online. Association for Computational Linguistics.
Fangyu Liu, Emanuele Bugliarello. Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond EI- liott. 2021. Visually grounded reasoning across lan- guages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yi R Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2022. Normsage: Multi-lingual multi-cultural norm discov- cry from conversations on-the-fly. arXiv preprint arXiv:2210.08604.
Jane Lockshin and Tom Williams. 2020. "we need to start thinking ahead": the impact of social context on linguistic norm adherence. In Annual meeting of the cognitive science society.
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022. DREAM: Improving situational QA by first elab- orating the situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages I 1115-1 127, Seattle, United States. Association for Computational Lin- guistics.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in- in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048- 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ro- nan Le Bras, Jenna Liang, Jesse Dodge, Keisuke Sakaguchi, Jon Borchardt, Saadia Gabriel, Yu- lia Tsvetkov, et al. 2022. Can machines learn morality? the delphi experiment. arXiv preprint 10.05:210000754..
Anders Giovanni Møller, Jacob Aarup Dalsgaard, Ari- anna Pera, and Luca Maria Aiello. 2023. Is a prompt and few samples all you need? using GPT-4 for data augmentation in low-resource classification tasks. arXiv preprint arXiv:2304.1380
Alexandros Papangelis, Karthik Gopalakrishnan, Aish- warya Padmakumar, Seokhwan Kim, Gokhan Tur, and Dilek Hakkani-Tur. 2021. Generative conversa- tional networks. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 111-120, Singapore and Online. Association for Computational Linguistics.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2022a. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization. ArXiv, abs/22 10.665
Hyunwoo Kim, Youngjac Yu. Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022b. ProsocialDialog: A prosocial backbone for conversational agents. In Proceedings of the 2022 Conference on Empirical Methods in Nut- ural Language Processing, pages 4005-4029, Abu
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing, pages 10611-1682,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Hannah Rashkin, Maarten Sap, Emily Allaway, Noah Smith, and Yejin Choi. 2018. Event2Mind: Com- monsense inference events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics.
A Dialogue Quality Evaluation Metries
Annotators were asked to rate conversations from both synthetic sources and actual dialogue sources according to the following dimensions and instruc- tions;
Maarten Sap, Ronan Le Bras, Emily Allaway. Chan dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi, 2019. Atomic: An atlas of machine commonsense for if- then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01)3307730335
On- Does the dialogue demonstrate the stated social norm?
Option: 1 (Yes) or 0 (No)
Naturalness: How natural is the overall dia- logue? How realistic does the content sound? Scale: (completely unnatural) to 5 (as natu- ral as conversations in real life)
Muzafer Sherif. 1936. The psychology of social norms. Harper.
Harry C Triandis. 1993. Collectivism and individualism as cultural syndromes. Cross-cultural research, 27(3- 1):
Nativeness:
Does the dialogue sound native in its language? (If it uses any idioms, for instance, it might sound more native.)
Harry Charalambos Triandis al. 1994. Culture and social behavior. McGraw-Hill New York.
Scale: (not native at all) to 5 (as native as what native Chinese speaker would say)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, al. 2022. Chain-of-thought prompting elicits soning in large language models. Advances Neural Information Processing Systems, 35:24824-24833
Coherent: How coherent is the overall dia- logue? (e.g., If it contains illogical responses, it might not be coherent.)
Scale: (completely incoherent) to 5 (as co- herent as native speaker)
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint
Interesting: How interesting is the overall dialogue? Is the dialogue full of content? Scale: 1 (generic and dull) to 5 (full of content and very engaging)
Rongsheng Zhang, Yinhe Zheng, Jianzhi Shao, Xiaoxi Mao, Yadong Xi, and Minlie Huang. 2020. Dialogue distillation: Open-domain dialogue augmentation us- ing unpaired data. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNL.P), pages 3449-3460, Online. As- sociation for Computational Linguistics.
Each dialogue is evaluated by 3 crowd workers. Final scores for On-Topic are determined by the majority vote of all scores; for the other four di- mensions, final scores are the unweighted average of all three annotator ratings.
Caleb Ziems, Jane Dwivedi-Yoe Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. NormBank: A knowl- edge bank of situational social norms. In Proceed- ings of the s1 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7756-7776, Toronto, Canada. Association for Computational Linguistics.
B Social Norm Generation Prompts
B.1 Prompts for Generating Chinese Norms
Social norms are informal rules that govern behav- iors in groups and societies. Different conversa- tional social norms are applicable to different con- versation types. Imagine you are a culture-aware system that understands social norms in Chinese society. Using some examples provided, you are tasked to list down and describe 10 new conver- sational social norms that are related and specific to the conversation type given. While generating additional norms keep the following 3 instructions in mind:
Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755-3773.
1. Ensure that the norms generated are specific to the Chinese culture and are not generic moral or

---

conversational norms.
C.2 Prompt for Situation Expansion
2. Specify the context where the norm is to be fol- lowed, wherever required.
Social norms are informal rules that govern be- haviors in groups and societies. You are given a situation and social norm, and you are tasked to create and include details to the real-life situation which takes place in Chinese society.
3. Mention verbal pieces of evidence in Chinese that should be used in conversation for the accom- panying norms.
Format: start with "New Situation."
B.2 Prompts for Generating American Norms
Norm: It is socially preferred to apologize im- mediately if you disturb another person and give the affected person chance to identify and speciffy if they are hurt Situation: On the street; Chinese young man and a woman
Social norms are informal rules that govern behav- iors in groups and socicties. Different conversa- tional social norms are applicable to different con- versation types. Imagine you are a culture-aware system that understands social norms in American society. You are tasked to check whether given set of norms for the Chinese culture aligned to the American culture as well, or if they differ.
New Situation: A Chinese young man, A fti, on his way back home, bumped into a stranger named Susan on the street. Susan is from New York, America, and it is her first time coming to China looking for her friend, she doesn't 't speak fluent Chinese and is lost the street.
For each of the Chinese norms, if there exists an aligned social norm in American culture, gen- erate the equivalent norm. If the American norm differs from the Chinese norm, then, generate the difference in the norm. In the norm descriptions generated, also include verbal phrases of evidence from an American culture that support the norm, if any. Do not list these down separately, include them in the norm description itself.
C.3 Prompt for Chinese Dialogue Generation
1. R
Conversation Type: <Norm Category> Chinese Culture Norm: <Chinese Norm American Culture Norm:
2. X
3. """ 31 -I
* バタークン M
C Dialog Generation Pipeline Prompts
f -10
C.1 Prompt for Scenario Generation
Social norms are informal rules that govern behav- iors in groups and societies. You are given social norm, and you are tasked to imagine 10 scenarios that conversation that entails the Norm can take place in a real-life setting of Chinese society.
Нे T # XAA.HHHGG
JJ # ス
Format: Start your response with "Scenario:" Norm: It is socially preferred to apologize im- mediately if you disturb another person and give
*BB: 1811), X/ 3
the affected person chance to identify and specify if they are hurt
*T:: W 
3.: 1K1J
kff: {
Scenario: 1. in a university: college students 2. on the street; strangers 3. in a company's 's office; colleagues 4. in a hospital; patient and doctors 5. in restaurant; waiter and customers 6. in cafe; two customers 7. in shopping mall; sales associates and customers 8. in a park; a morning jogger and lady 9. in a suburb neighborhood; two neighbors who know each other 10. in a family gathering; two cousins
:. R # MRE
*FF: # # #
C.4 Prompt for Norm Adherence Violation Labeling
Given a dialogue and norm on which the dialogue is based upon, the task has 4 steps:
1. Summarize the Norm in 5 words as Norm

---

Action
( AR : Adhered B
2. Indicate which character in the dialogue per- forms the Norm Action
t.
offers a mild criticism by saying " JM M
3. Repeat every turn and only indicate 'Adhered' when the sentence closely aligns with the Norm. Otherwise, indicate 'Not Relevant'.
Bf: 2 Not Relevant / is not an actor of criticism norm (Kk T
4. After each label, provide short explanation for why the norm is strongly entailed or not relevant at sentence level.
##3
11 ): Not Relevant H does not criticize
here
(sbB:: , .
- X- ): Not Relevant / is not a criticizer
Format:
Repeat each turn in bracket Append Adhered or Not Relevant label for each turn
o श E ): Not Relevant not
criticism statement
Use "I" to separate role, label and explanation needed
D Faithfulness Evaluation of Elaborated Situations
Norm: professional setting with higher sta- tus speaking to lower status, it is permitted to use direct language, strong tone of voice, and display emotions when criticizing one's behavior, ideas, and work.
Chinese Situations
US Situations
Adherence Violation
98% (57/58)
90% (45/50)
100% (56/56)
74% (40/54)
Dialogue:
# A # Иイ
Faithfulness of the situation elaboration stage for both Chinese and American norm adherences and violations, measured by the percentage of situations that entail the corresponding norms upon which they were generated.
Table 4:
RR EEL thi
N #M
: T # 3 /
m
To validate the faithfulness of situation elabo- rations by ChatGPT, we collect sample of 218 (norm, elaborated situation) pairs. Specifically, for each language (Chinese and English), for each of 10 norm categories, and under norm adherence and violation settings, we randomly select 3 norms along with 2 situations for each norm. Three anno- tators are then asked to label, for each (norm, elab- orated situation) pair, if the situation entails the corresponding norm, and to resolve disagreements via majority vote. The final faithfulness score is measured by the percentage of situations that entail the norms each pair for each language and under norm adherence or violation settings.
&. 7
NR: +
#4 3
M 18
XX
1.T.
Norm Action: offer criticism
Actor of the Norm:
H #h: coach, higher status, criticizer
Dialogue:
# M - T
3: NR.
th オ I ): Adhered I 5k क criticizes his player's 's performance by using direct wordings including "Ai ""5.""" and "it #
Table 4 shows that elaborated situations for Chi- nese norms achieve high faithfulness scores for both norm adherences and violations. On the other hand, most of the situations elaborated for Ameri- can norm adherence are also faithful, while those for American norm violation achieve relatively low faithfulness. In addition, to measure inter- annotator agreements among three annoators, we compute Fleiss Kappa score on the sample of 218 (norm, elaborated situation) pairs and individually for norm adherence and violation settings. The
(1T TR T ): Not Relevant I 1/N
is not acting the criticism norm
IT ): Adhered #
Sh criticizes /i 's idcas of how to play basketball by questioning him

---

overall Fleiss Kappa score on the entire sample among 3 annotators is 0.51. For 1 114 norm adher- ence subsample, Fleiss Kappa score is 0.59; for the remaining 104 norm violation pairs, the Fleiss Kappa score 0.44.

---

E Topic Models for Generated Situations
To provide greater clarity into the topics present in the situations behind our dialogues, we train 30-topic LDA model (Blei et al., 2003) and label each topic manually with its most prominent theme. Details are shown in Table 5 and Table 6, below.
Topic Theme
Top Tokens
0. Compliments
compliments named situation compliment decides insincere feels hair
1. Respects
respect language show chinese formal showing greeting respectful young respected
2. Friend & Social Gathering
friends group friend party situation chinese close social conversa- tion restaurant
3. Gym & Workout
chinese named gym situation china workout american cultural foreigner
4. Name
ming xiao ming's ' situation david miss named expresses day words compliments chinese work compliment hard responds success humility situation response
S.Compliment & Humility
6. Religion Temple
temple service situation master restaurant religious waiter church customer buddhist
Apology & Responsibility
jack apology apologize accidentally situation mistake immediately apologizes chinese responsibility
8. Gratitude & Appreciation
chinese situation gratitude china group norm zhou express culture grateful
Community & Social Service
event felt situation charity china volunteers community volunteer
didn't noticed
10. Family
family parents gathering situation dinner members mother uncle
younger father
11. Funeral
family condolences offer deceased situation support chinese mem-
bers loss show
12. Office
liu colleagues team project meeting work colleague situation chi-
nese company
13. School & Classroom
teacher students student chinese class school professor classroom university classmates
14. Gift Giving
chinese mrs gratitude gift appreciation norm situation show ex-
press host
Table 5: Manually labeled generated situation topics and their top tokens, captured from 30-topic LDA model trained on situations generated by ChatGPT.

---

Topic Theme
Top Tokens
15. Office & Professional Setting
company manager employees senior situation status employee junior
16. Office Professional Setting
business chinese meeting company potential american situation conference clients card
17. Criticism & Feedback
chinese direct situation request language norm hesitant status ap- proach judge
18. Criticism & Support
feedback criticism situation improve work chinese norm construc- tive giving
19. Community
group members community situation member approach meeting concerns discussing suggests
20. Name
wei wei's lin named jing store customer norm xia asks
21. Sports & Teamwork
team coach game xiaoming situation players baskethall teammates sports performance
22. Taking Leave
leave leaving situation norm social goodbye carly chinese attend time
23. Name
zhang zhang's s named situation feels lei notices li's decides china woman young situation elderly chinese named bus seat susan
24. Etiquette & Public Trans- portation
liang government status situation official named office yang offi- cials question
25. Goververmennt & Officials
26. Music & Concert
audience chinese ling music performance concert situation fans beijing named
27. Wedding
guests wedding bride situation groom ceremony dress reception traditional chinese
28. Argument & Debate
situation chinese behavior argument tom discussion china dis-
cussing opinions aggressive
29. Social Events
behavior situation feels uncomfortable norm china starts phone feel social
Table 6: Manually labeled generated situation topics and their top tokens, captured from 30-topic LDA model trained on situations generated by ChatGPT.


Reviews:
[{"review_id": "FdF1nDPq1y", "reviewer": "Reviewer_nowG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and recognizing novelty of our work! We also note that we will be releasing all the data and codebase to encourage reproducibility and further research.\n\n* Reason to Reject 1: \nWe thank the reviewers for pointing out some papers that are very relevant to ours. To clarify on the first related work listed as missing (Chen et al., 2023), we cite this paper multiple times throughout our paper, for example on the Introduction (49) and Background and Related Work (76-77) sections. We will emphasize more on the cross-cultural and bilingual nature to highlight the novelty and contributions of our work, as the reviewer correctly points out! And as the reviewer advised, we will include the other two papers in the background section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I appreciate the fact that the dataset is in a language other than English, and that a non-Western culture is being included.\n- The machine-and-human creation of the dataset is reasonable and effective, and I particularly appreciated the careful manual verification of the dialogues.\n- The experiments on ChatGPT (§4) are well-executed and yield some interesting insights.", "reasons_to_reject": "This work is sound and reasonable. However, the paper missed some important related work that was not covered, making some claims of novelty not well supported. Specifically, Kim et al. (2022a) introduced the ProSocial Dialogues corpus which contains English conversations with social norms annotated at the utterance level. Additionally, the machine-generated-human-verified pipeline to generate social situations which are fleshed out into dialogues was introduced by Kim et al (2022b) and Chen et al. (2023). I suggest the authors edit their introduction to better scope their claims of novelty, and include these works into the related work section. That being said, the paper's main novelty lies in the cross-cultural nature of the dataset, which is a huge asset for the NLP community!", "questions_to_authors": "", "paper_topic_and_main_contributions": "This short paper presents NormDial, a new dataset of English and Chinese dialogues annotated with relevant social norms along with labels on whether the norm is being followed or violated. Authors use a machine generation with human verification pipeline to create the dataset, which contains 4k dialogues and 30k conversational utterances: authors first generate a set of social norms with ChatGPT, then flesh out the social norms into dyadic social situations, from which a dialogue is then generated. The English and Chinese conversations are then verified for naturalness, interestingness, and coherence, as well as whether the speakers follow the norms of the respective cultures. Finally, authors perform a small investigation of whether ChatGPT can accurately predict whether a conversation follows/violates a social norm, finding that it struggles to detect norm-violating utterances.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Chen, Maximillian, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023. “PLACES: Prompting Language Models for Social Conversation Synthesis.” In *Findings of EACL 2023*. http://arxiv.org/abs/2302.03269.\n- Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap & Yejin Choi (2022) **SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization**. *arXiv*. https://arxiv.org/abs/2212.10465\n- Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi & Maarten Sap (2022) **ProsocialDialog: A Prosocial Backbone for Conversational Agents**. *EMNLP*. https://arxiv.org/abs/2205.12688"}}, {"review_id": "nFcHs3sMMB", "reviewer": "Reviewer_q6Mj", "summary": "", "questions": "A. How did the Chain-of-Thought prompting improve the labeling process?\n\nB. How effective was ChatGPT for scenario generation and situation elaboration?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Reasons to reject:**\nAbsolutely, social norms are constantly evolving! This longitudinal aspect must be contended with for any computational social science work around social norms. As our focus is on current social norms, we mitigated the potential possibility of having significantly outdated norms in our data by using a human-AI collaboration framework where experts with significant, contemporary lived experiences in both cultures (10+ years) manually validated the norms for each culture (lines 142-151), before using them to prompt the generation of dialogues that violate or adhere to the given norm. We hope that this framework will allow us and other researchers to continuously generate datasets and capture the dynamic nature of social norms, as they evolve and change over time. We will ensure the release of a data sheet that explains the timeframe of when the data was generated & evaluated, and will add a discussion of this in the limitations section. As a result of this verification, we believe that our bilingual dataset is a valuable resource for current work on cross-cultural norms.  \n\n**Question A: How did the Chain-of-Thought prompting improve the labeling process?**\n\nPrevious literature shows that Chain of Thought prompting significantly improves reasoning and task performance of language models including classification tasks  (Wei et al., 2022; Kojima et al., 2022, inter alia). Thus, we decided to use CoT to improve our labeling accuracy.\nWe have included metrics (precision, recall, F1 score) to evaluate the labeling process in Table 3 of our paper. Moreover, we will also include the results of comparing the labeling accuracy of a sample of 50 dialogues generated with and without the CoT prompting approach in the appendix to address this concern.\n\nThe task of labeling dialogue turns with norm adherence/violation is complicated because \n\na. Norm descriptions tend to be very lengthy. We observed that ChatGPT, without CoT, often fails to focus on the norm action but rather uses information weakly related to the norm itself to label dialogues.\n\nb. Most of our dialogues are multi party. Without using CoT, ChatGPT is often observed to hallucinate turns where a character actually performs the norm versus utterances that serve as context for the norm demonstration, as we discussed in the paper (256 - 264).\n\nWe use CoT to address problem a. because by summarizing the lengthy norm description into a short action, ChatGPT is observed to be less prone to use irrelevant information to label dialogues, thus better understanding the task. Similarly, for problem b., because by first identifying characters who perform the norm, ChatGPT is observed to better distinguish utterances where norms actually are adhered/violated versus the context that is not related to the norm.\nYet, as we mentioned in our paper (260-264), these phenomena remain an exciting field for future research to tackle and address.\n\n\n**Question B: How effective was ChatGPT for scenario generation and situation elaboration?**\n\nWe thank the reviewer for this question! On effectiveness, we base our analysis on two main fronts: 1) faithfulness of the situation elaboration against the original situation and 2)  diversity of the generated scenarios (for the scenario generation stage)\n\nOur qualitative analysis showed that ChatGPT was effective across these fronts. \n\n1. On the faithfulness of the situations elaborated by ChatGPT, our qualitative analysis conducted during human validation shows that the elaborated situations are faithful to the norms as they don’t deviate from the gist of the norm. We will add details of this analysis to the appendix of the final camera-ready version of the paper.\n\n2. With our Chain-of-Thought prompt and in-context examples, ChatGPT is able to produce diverse scenarios and detailed situations in stage 1 (scenario generation) and 2 (situation elaboration). In Appendix D of our work, we included the results of a 30-topic LDA model trained on situations generated by ChatGPT, which covers a variety of topics from family, office, to funeral. \n\n**References**\n\nWei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in Neural Information Processing Systems 35 (2022): 24824-24837.\n\nKojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. \"Large language models are zero-shot reasoners.\" Advances in neural information processing systems 35 (2022): 22199-22213.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduced a dialogue dataset that can improve AI's conversational skills by considering social norms. The scenarios presented in the paper could encourage research and facilitate the development of language technology that is culturally aware and sensitive.", "reasons_to_reject": "The annotations for norm adherence and violations in the generated situations may become misleading as social norms evolve with the progress of society. The author did not present the result to support the claims that Chain-of-Thought prompting improves the labeling process.", "questions_to_authors": "A. How did the Chain-of-Thought prompting improve the labeling process?\n\nB. How effective was ChatGPT for scenario generation and situation elaboration?", "paper_topic_and_main_contributions": "The authors utilized ChatGPT to create dialogues that follow either American or Chinese social norms, with the option to either conform to or defy these norms. Evaluation results indicate that the dialogues generated are more fluent and natural compared to those generated by DailyDialog and the Linguistic Data Consortium.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "38thCgtMi5", "reviewer": "Reviewer_Gicu", "summary": "", "questions": "1. What are some use cases for this dataset?\n2. How is the diversity of the dataset? You mentioned that diversity is a problem for dialogue generation using prompting. That's why you first prompted ChatGPT for 10 scenarios and then prompt ChatGPT again using these scenarios to generate dialogue. Do you have quantitaive measure of the diversity to justify this approach?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Reason To Reject 1:**\n\nWe would like to clarify that the dialogues in both DailyDialogue and LDC are naturally occurring, not expert-annotated. The only part that is expert annotated are social norms from LDC. Hence, our claim that the dialogues generated through prompting exceed quality of naturally occuring datasets is valid, as we show by our detailed human evaluation following strict criteria (see response to Reason to reject 2 and Appendix A). We also note that similar conclusions with respect to synthetic dialogues have been observed in prior work (Chen et. al, 2023).\n\n Chen, Maximillian, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023. “PLACES: Prompting Language Models for Social Conversation Synthesis.” In Findings of EACL 2023. http://arxiv.org/abs/2302.03269.\n\n**Reason To Reject 2:**\n\nIn our work, we mention in line 142-151 that social norms were validated by interpersonal communication experts with knowledge of the culture so they are of high quality. \n\nTo ensure both the correctness and detailedness of norm description, our experts used the following detailed criteria to qualitatively evaluate \n* if the model-generated norm aligns with annator’s experience in the society (to make sure the norm is factual)\n* if the norm falls into the desired norm category \n* if the norm describes specific rules/action (instead of being generic)\n* if the norm is specific to Chinese/ American culture (instead of being universal across cultures)\n\nAnnotators resolved any disagreements between each other by discussing with each other their labels, so we do not report quantitative IAA metrics.\n\nWe will put the details of this criteria that our experts used for evaluating norms in the appendix to address this concern. Our norms are also sufficiently diverse (see lines 120-151), covering 10 distinct categories, and we will also put the results of a 20-topic LDA trained on norms in the appendix D.\n\n**Question 1** What are some use cases for this dataset?\n\nIn our Introduction (46 - 52) and Background section (95), we discussed in detail the utility of a LLM- generated dialogue dataset of social norms. We align with prior work in this realm (Chen et al.,  2023; Kim et al., 2022; Kim et al., 2022) but also point a new direction in cross-cultural and cross-lingual dialogues of social norm adherence and violation.\n\nTo summarize, our dataset \n1. Can be explicitly used for model pre-training and fine tuning under supervision to make LLM aware of social norms in different cultures or be able to detect and avoid violation of social norms in different cultures.\n\n2. Can be used to analyze cross-cultural commonalities and differences in conversations in a variety of life situations for future social science research. \n\n3. Can be useful for researchers to analyze linguistic characteristics that, for example, might cause norm violation in one language but often observed in another.\n\n\n**Question 2** How is the diversity of the dataset?\n\nTo quantify the diversity of the dialogues generated by our CoT prompting method, we randomly selected 5 Chinese and American norms from different norm categories. We compared the distinct n-gram results of dialogues that are generated by going through Scenario Generation, Situation Elaboration, and Dialogue Generation stages versus simply taking the norm to generate dialogues without using situations.\n\nCoT includes dialogues that are generated by going through Scenario Generation, Situation Elaboration, and Dialogue Generation stages, while Simple only takes the norm and directly generates dialogues without using situations. We control the same hyperparameters and the same model (gpt3.5-turbo) for these two prompting methods. The lexical diversity of the dialogues generated by CoT has shown a significant  improvement compared to Simple prompting method. This shows that our approach is effective to improve the diversity of synthetic dialogues and we will include all the details in the Appendix.\n\nMore details on the Distinct N-grams results:\n\n1-gram: 0.24 (CoT)\t        0.17 (Simple)\n\n2-grams: 0.74 (CoT)\t0.59 (Simple)\n\n3-grams: 0.94 (CoT)\t0.80 (Simple)\n\n4-grams: 0.98 (CoT)\t0.89 (Simple)\n\n\n**References**\n\nChen, Maximillian, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023. “PLACES: Prompting Language Models for Social Conversation Synthesis.” In Findings of EACL 2023. http://arxiv.org/abs/2302.03269.\n\nHyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap & Yejin Choi (2022) SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization. arXiv. https://arxiv.org/abs/2212.10465\n\nHyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi & Maarten Sap (2022) ProsocialDialog: A Prosocial Backbone for Conversational Agents. EMNLP. https://arxiv.org/abs/2205.12688", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Social norm-based dialogue datasets are relatively under-explored.", "reasons_to_reject": "1. The quality of the model-generated conversations is debatable. The authors claim that the quality of this dataset \"exceed the quality of naturally occurring datasets\" through human evaluation (Table 2). However, this is questionable because as the authors mentioned, those are expert-annotated datasets, whereas their dataset is generated through prompting.\n\n2. The quality of the model-generated social norms is not analyzed. Although the authors mention that invalid norms are thrown away by the three human annotators, there isn't an analysis of, for example, the inter-annotator agreement for this part.", "questions_to_authors": "1. What are some use cases for this dataset?\n2. How is the diversity of the dataset? You mentioned that diversity is a problem for dialogue generation using prompting. That's why you first prompted ChatGPT for 10 scenarios and then prompt ChatGPT again using these scenarios to generate dialogue. Do you have quantitaive measure of the diversity to justify this approach?", "paper_topic_and_main_contributions": "The authors propose a synthetic dialogue dataset based on adhering/violating social norms. They first augment an existing set of social norms by prompting. Then, they further prompt ChatGPT to generate dialogues based on some pre-determined social norms.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This short paper introduces a dataset of dialogues which are annotated as to whether they adhere to social norms. The dataset contains English and Chinese dialogues. The dialogues are synthetic - they were generated using ChatGPT and then annotated. The authors also evaluate whether ChatGPT can recognize norm violations and find that it does not do very well on that task. 
The reviewers appreciate the cross-cultural social norms challenge that this dataset can help to address. 
The reviewers also point out some shortcomings with respect to missing citations, unsupported claims on CoT reasoning effects and unsupported claims regarding the quality of the proposed dataset. However, the authors convincingly address these points in their rebuttal, which should flow into a final version of this paper.
Decision: Accept-Main