EtiCor: Corpus for Analyzing LLMs for Etiquettes
Authors: Ashutosh Dwivedi
Pradhyumna Lavania
Ashutosh Modi
Venue: EMNLP 2023 Main (2023)
Keywords: Social Norms
Etiquette
LLMs
TL;DR: We anlayze how well LLMs are able to understand culture specific social norms and etiquette

Abstract: Etiquettes are an essential ingredient of day-to-day interactions among people. Moreover, etiquettes are region-specific, and etiquettes in one region might contradict those in other regions. In this paper, we propose EtiCor, an Etiquettes Corpus, having texts about social norms from five different regions across the globe. The corpus provides a test bed for evaluating LLMs for knowledge and understanding of region-specific etiquettes. Additionally, we propose the task of Etiquette Sensitivity. We experiment with state-of-the-art LLMs (Delphi, Falcon40B, and GPT-3.5). Initial results indicate that LLMs, mostly fail to understand etiquettes from regions from non-Western world.

EtiCor: Corpus for Analyzing LLMs for Etiquettes
Ashutosh Dwivedi*
Pradhyumna Lavania" Indian Institute of Technology Kanpur (ITT Kanpur) (ashut
Ashutosh Modi
p pradhyumna20 0att200@iiti Biitkkkaaeee ac.in
ashutoshmê cse iitk. ac.in
Abstract
Wikipedia Data
Percentage
English Articles
Etiquettes essential ingredient of day- to-day interactions among people. Moreover, etiquettes are region-specific, and etiquettes in region might contradict those in other re- gions. In this paper, we propose EtiCor, an Etiquettes Corpus, having texts about social norms from five different regions across the globe. The corpus provides a test bed for evalu- ating LLMs for knowledge and understanding of region-specific etiquettes. Additionally, we propose the task of Etiquette Sensitivity. We experiment with state-of-the-art LLMs (Delphi, Falcon4OB, and GPT-3.5). Initial results indi- cate that LLMs, mostly fail understand eti- quettes from regions from non-Western world.
among all articles
76
Active Editors
(for English Articles) having Western Origin
93
Table I:
Wikipedia statistics for English articles
technology. nowadays people usually refer to web- search and PDAs to inquire about social norms of the region they plan to visit. However, are LLMs the underlying foundation of the tech- nological mediums (e.g.. search engines) used glean social norms-specifie information sensi- tive to culture-specific etiquettes? Many LLM mod- els have been developed using data (scrapped from the internet) heavily skewed toward Western cul- tures; for example, language models like BART (Lewis et al., 2019) are trained using Wikipedia text as the primary data source. However, an in- vestigation of statistics of Wikipedia pages (Table 1) authored by various writers reveals that the En- glish content is mainly created by people belonging to western (North America and Europe) socicties (Yasseri et al., 2012). When authoring an article, people tend to be implicitly influenced by the cul- ture they live in, and hence, this tends to introduce some culture/society-speeiff biases in the content. To promote research towards understanding how LLMs generalize concerning the understanding of knowledge about etiquettes of different cultures and regions and with the aim to analyze the responses generated by the generative language models are skewed towards or against certain cul- tural norms and values, we introduce new corpus Et corrrr (Etiquettes Corpus). The corpus consists of texts social norms followed in different world regions. The corpus is in English, but we plan make it multi-lingual in the future. One of the fo- cuses of EtiCor is to investigate potential biases LLMs exhibit when addressing regional etiquettes. The long-term goal of our research is to pave the
1 Introduction
Etiquettes define system of rules and conventions that regulate social and professional behavior. Eti- quettes are an integral part of every culture and society. Consequently, etiquettes carry significant cultural and regional implications every region of the world. While some of the social norms' are common across many cultures, every region has some society-specific norms that may be contradic- tory to norms in other societies. When person is visiting another region/culture, they need to be aware of social norms in that region avoid unde- sirable situations that may hamper the business and sometimes lead to conflicts.
In recent times, generative models and Large Lan- guage Models (LLMs) have become part of many digital technologies (such web search and per- sonal digital assistants (PDA)). In the past, people have referred various books written to teach and explain etiquettes for a region (Hitchings, 2013; Martin, 2011; Vanderbilt, 1958; Post et al., 2011; Foster, 2002a,b). However, given the advances in
Equual Contributions
Inn this paper, we use the term etiquette and social norm inter-changeably

---

way for the development of Al systems that are more inclusive and sensitive to cultural differences. This paper introduces a corpus to enable the re- search community to advance toward the goal. We also propose the task of Eriquette Sensitivity to eval- uate LLM's sensitivity towards different etiquettes. In nutshell, we make the following contributions:
Region
#Social Norms
East Asia (FA)
7432
India (IN)
4556
Middle East and Africa (MEA)
10031
North America- America-Burope Europe (NB)
6156
Latin America (LA)
8172
Total
36347
We introduce new corpus EtiCor. It is corpus of texts about etiquettes of five major regions of the world. The corpus consists of 36k social norms in the form of labeled En- glish sentences.
Table 2: EtiCor distribution.
called Unicorn (based on the T5 model) (Lourie et al., 2021) to develop a system to judge whether situation and corresponding action are ethically correct and acceptable. An interesting study in grasping the moral context of text is done via the Moral Integrity Corpus (MIC) (Ziems et al., 2022). Authors compiled prompt-reply pairs (prompts are questions about social norms) crowdsourced via websites like Ask-Reddit (ht.tss: /www. re ddit com/ r //t.//kkeedii/.// These prompt- reply pairs were further used to create a distinct 99k Rule of Thumb (RoT) having hard-coded rules based on social norms. LLM models (GPT-Neo and Dialo-GPT) are trained using these annotated RoTs to judge the sentiment and morality of the text. In another work, Abrams and Scheutz (2022) examines how pragmatic cues (in the form of so- cial norms) affect the resolution of mentions in the task of conference resolution. In the study, dif- ferent daily life situations are textually described, and the machine response is compared the hu- response, giving idea about how much cial context is utilized by the model. A human- annotated database of social norms that apply to ev- eryday tasks under various constraints presented in NORMBANK (Ziems et al., 2023). The authors propose a SCENE framework that links a particular norm with surrounding attributes and constraints like (who/what/where) to include contextual infor- mation.
As a use case for the corpus, we propose the task of Eliquette Sensitivity, Given statement about social norm about region, the task of Etiquette Sensitivity is to classify if the social norm is appropriate concerning that region. We experiment with some of the existing state- of-the-art LLMs in zero-she shot settings. Results indicate gaps in the knowledge of LLMs. Fur- ther, we experiment with a supervised setting, train a smaller transformer-basee based model (like BERT) on the corpus, and test its performance on the task. Fine-tumed BERT model performs well on the task of Etiquette Sensitivity. We release the dataset and code via the GitHub repository: pettts://githhe com/ Exp Coration-Lab/Bt.e.
2
Related Work
In recent years, there has been works in the NLP and ML communities to develop algorithms un- derstand human ethics and social commonsense (Jiang et al., 2021; Louric et al., 2021; Forbes et al., 2020; Tolmeijer et al., 2020). We describe some of the major works in here. To develop an ethically sound MI. model Jiang et al. (2021) have devel- oped the Delphi system. The model is trained to perform three different tasks: classify (binary: yes/no) if a given social situation is as per social norms, b) open text judgment: whether the model agrees with the action taken in the particular situa- tion or not c) relative selection: given a context and two actions, the task is to select the action that is more appropriate. The authors create new dataset, Commonsense Norm Bank, that incorporates ex- isting datasets such as Social Chemistry (Forbes al., 2020), Scruples (Lourie et al., 2020), Ethics (Hendrycks al., 2020), Commonsense Moral- ity (Frederick, 2009), Moral Stories (Emelin et al., 2021), and Social Bias Inference Corpus (Sap et al., 2020). Delphi uses prr-traan trained language model
EtiCor: Corpus for Etiquettes
We create EtiCor (Etiquettes Corpus), a corpus of regions- etiquettes. In this work, we use a standard dictionary definition of etiquette and define it as set of social norms/conventions or rules that tell how to behave in particular social situation. Etiquettes social and professional behavior, and are subjec- tive. In the following text, we sometimes refer each etiquette as a social norm or norm for short. The corpus contains specific etiquettes (as English sentences) relevant to particular region, chosen
region-specific, dictate

---

Travel
Business
Visits Dining
Sentences
Labels
Regions
DON'T pour soy sauce directly over sushi or sashimi
30.0%
12.9%
EA
Pointing chopstick towards someone while having dinner
-1
EA
30. 1
Women should not appear in front of non-family men without covering themselves
1
MEA
27.0%
Eating with hare hands is common practice here
IN
Figure 1: Etiquette Type Distribution
sure to put any advanced educational degree and your title or position on both sides your business card.
from relevant sources to ensure authenticity. In EtiCor, we cover five regions across the globe: East Asia, India, the Middle East and Africa, North America and Europe, and Latin America. Table 2 shows the number of social norms for each re- gion. The scope of norms took care that culture-specific practices relevant to a particular region were aptly included in the cor- pus, for example, seseivi norms for India and Latin America, procedures related to religion in the Middle East and Africa. We also tried to collect data for generic day-to-day practices for each region to cover any cultural pattern engraved in daily routines like cating food, kitchen manners, etc. EtiCor covers four major types of etiquettes: 1. Dining and Festivals, 2. Visits and Social Inter- actions, Travel, and Business. Fig. shows the distribution of etiquettes across types. App. Ta- ble 6 shows region- wise distribution norms. A norm's ' average length (in terms of word count) is 18; App. Fig. 3 shows the word length histogram. Table 3 shows few examples from EtiCor.
LA
Table 3: Few examples from EtiCor
is act that is frowned upon by people of that culture. About 20% of the norms were self-labeled these started with words like "Do" and "Do not." The remaining norms were manually labeled after careful examination. App. Table 7 provides the distribution of labels across different regions.
expansive
Choice of Regions: We selected regions based on diversity and data availability to cover much of the world as possible. We could not include Russia, China, and other Southeast Asian countries due to the unavailability of sizable number of social norms in the English language. We did not aim for automatic translations (to English) of social norms as these could not be verified for correctness. 1) East Asia (EA): We included the follow- ing countries: Japan, Korea, Taiwan, Vietnam, Malaysia, and the Philippines. These countries have several customs in common, especially dining and traveling; however, country-specifie etiquettes are labeled with the country tag. This helped to pre- vent any conflict among the samples in the corpus. In this case, the corpus focuses mainly on dining, greeting, public transportation, and taboos.
EtiCor Creation: To avoid conflicting informa- tion, we created the Etiquettes Corpus by collect- ing texts from authentic and publicly available sources (e.g., government-aided websites). The sources of information included websites providing information about regional etiquettes, tour guide points and pamphlets, etiquette information chan- nels, and tweets and magazines on etiquettes (de- tails about sources on GitHub repo). The collected (and scrapped) data was noisy and sometimes repet- itive, so was pre-processed. We manually re- moved texts that described the same norm; sen- tences were filtered if the length was less than to- kens, and large paragraphs were summarized. The pre-processed data were manually labeled.
2) Middle East and Africa (MEA): Since this is a large area with diverse culture, we studied it care- fully to capture various variations. We excluded some countries, such as Israel, Congo, and South Africa, where culture differed significantly from other regions. We also focused on common aspects of etiquette, such as dining, greetings, and shared religious norms. In cases of minor conflict, we in- cluded the country name or the cultural group with which it was associated.
Labeling: A norm in the corpus is labeled ac- ceptable (with label +1, positive class) if it general etiquette the region, and it is labeled non-acceptable (with label -1, negative class)
3) India (IN): India diverse and culturally rich country where different societies co-exist. More- over, India is the most populated country in the

---

Delphi
Falcon-40B
GPT-3.5
Region
# of
of Abstentions
of
Accuracy
Fl score
Accuracy
Fl score
Accuracy
Fl score
Abstentions
Abstentions
EA
0.63
0.69
809
0.67
0.70
480
0.55
0.67
IN
0.71
0.77
154
0.71
0.73
806
0.52
0.64
14
MEA
0.65
0.71
1149
0.69
0.73
123
0.55
0.68
46
NE
0.78
0.85
220
0.74
0.78
453
0.62
0.75
36
LA 0.69
0.71
1654
0.59
0.61
731
0.54
0.67
88
Overall
0.68
0.74
3986
0.67
0.71
2593
0.55
0.68
Table 4: Results different models on EtiCor. The models are evaluated after removing the abstentions. Overall scores are calculated by weighted sum of region- wwie scores. Please refer to the text more details.
world. It was essential prepare common culture-oriented corpus. In India, several religious and social norms are practiced by Indians, as well as gift-giving and nome-visiting-relat practices. We also included some etiquette related to popular festivals.
parameters) (Brown al., 2020). We analyzed the overall and etiquette type-wise performance of each of the models. We measured the model performance using standard accuracy and the Fi-score metric. We also calculated overall scores by taking weighted sum of the scores of each of the regions, where the weight for each region is the normalized count of number of instances for that region (more details in App. B). Each of the models was queried for etiquette appropriateness; majority times, model replied in binary yes or no answer; in some instances, provided an explanation that was manually mapped to labels +1 or -1. However, in some cases, the model did not provide an answer gave an unclear response that could not be mapped to either of the labels. Such samples were not included in the evaluation and are referred as abstentions. For checking Etiquette Sensitivityof Falcon-40B and GPT-3.5 models, we prompted each of the models to provide a single word (yes or no) answer. The prompt used was: "In a single word, say whether this is a good etiquette or socially acceptable norm in <region>
5) North America and Europe (NE): We added cultural eustoms and practices from North America and Europe the corpus. We grouped the two regions due their strong correlation. Also, most the ctiquettes have their country tags attached to them to avoid any conflict.
4) Latin America (LA): With extensive area cover- age in this region, we faced similar problems as in the case of the MEA region, and it was tackled sim- ilarly. Nevertheless, covered several countries, and the only exceptions were Cuba and Colombia, which were removed because of the availability of a very small amount of data.
4 Etiquette Sensitivity
Task Definition: Existing LLMs are trained on huge corpora of texts, which has been shown to result in an improved overall general understand- ing of language. However, we are also interested in testing if the LLMs understand region-specific societal etiquettes. EtiCor serves this purpose. We propose the task of Etiquette Sensitivity to test our hypothesis. Given a statement about eti- quette, the task is to predict whether the statement is appropriate for region.
Results and Analysis: Table 4 shows results for the three models. We also provide some example oulputs in the App. Table 11. The models are evaluated after removing abstentions. We also cal- culated accuracy by including the abstentions; the results are shown in the App. Table 8. As can be observed from Table 4, scores for the North America- America-Europe (NE) region are much higher than for other regions. Also, the number of abstentions in the case of NE regions is much lower than in other regions. This is possibly due to the fact that these LLMs are primarily trained on data from the NE regions and, hence, are oblivious to etiquettes from other regions, pointing towards model bias for NE region etiquettes. Another observation from
Experiments: We evaluated various LLMs in zero-shot setting for the task. We ex- perimented with Delphi (a pre-trained lan- guage model with 1IB parameters) (Jiang al., 2021), Falcon-40B (an open-source LLM with 40B parameters) (h totss. / /huggingffcce co/blog/falllaocooooooooooooeaaa falcon) and GPT-3.5 Turbo (a closed LLM with 175B
Phttps: htttps::/tinyurlll com cow./ayam/772222

---

Travel Business Visits Dining
Region
Test Accuracy
Test F1
EA
0.872 (=0.034)
0.913 (=0,038)
15.4%
15.5%
0.878 (+0.028)
0.882 (+0.028)
MEA
0.938 (=0.027)
0.946 (=0.023)
NR
0.919 (=0.032)
0.934 (#0.082)
LA
0.869 (+0.015) 0.913 (a0.023)
9.8%
Overall (Weighted)
0.899
0.922
34.5%
Regions Combined 0.871 (=0.015) 0.902 (=0.021)
26.6%
Table 5-fold Cross validation performance fine- tuned BERT model for various regions and over the entire dataset (all regions combined).
GPT-3.5
28.
40.3%
29.2%
FaIcon-40B
may be related to changes in greeting styles and business format.
Delphi
Fine-tuned Models: We also experimented with fine-tuned models for the task of Etiquette Sensi- tivity. In particular, we tried the BERT (Devlin et al., 2018) model. We munet BERT model on Stor (Train-Val-Tess, split=70:10:20 for each fold, ADAM Btarminggratc=O00000 batch=8 and 5 epochs). A separate model was trained for each region. The model was tuned to predict whether given input is considered appropriate etiquette in a particular region. The class imbalance was overcome by generating negative values of posi- tive labels (Lee et al., 2021) and climination of ex- cess positive class by merging smaller sentences to- gether. Furthermore, we did 5-fold cross-vaas volsdvaali for each region as initial experiments showed vari- ance in BERT's performance across different runs. We also ensured the proper distribution of each type of etiquette in all the folds so that any form of bias could be avoided easily. Table shows the results averaged across 5-folds (more details in App. E). Overall results are calculated by a weighted sum of the mean scores for each region. We also combined the data from all the regions and performed 5-fold cross validation using BERT. The BERT model has fairly high performance for all regions, pointing out that region-specific training should be done when dealing with subjective and culture-specific norms.
Figure 2: Percentage of Wrong Predictions for each etiquette type by a model. The innermost circle results are for GPT-3.5, the middle circle is for Falcon-4OB, and the outermost circle is for Delphi.
the results is that GPT-3.5 (despite being the largest model among the three models) has the worst per- formance for all regions. However, the number of abstentions for each region in the case of GPT-3.5 is the least among the three models, possibly because of its larger size and wider coverage. The results of the Falcon model slightly worse than the Delphi model, possibly because the Delphi model is specifically trained with social norms the end goal, and Falcon a general-purposs LLM. Falcon also had a high abstention rate for the India (IN) region, where stated a lack of regional knowledge as the primary reason for not answering. Further examination of the prediction outputs revealed that women-related etiquettes in the MEA region were misclassified the most. Overall, saw similar trends across all three models.
We also analyzed the percentage of wrong predic- tions for each etiquette type for each model. Fig. 2 shows the percentage of wrong predictions for Delphi, Falcon-4OB, and GPT-3.5. App. Table provides detailed results. More details about the percentage of wrong prediction calculations are provided in the App. C. Region-wise percentage wrong predictions for each model are provided in
Conclusion
the App. Table 10. As can be observed in Fig. 2, Travel and Business etiquette are predicted well by all the models due to the global nature of these eti- quette types. Etiquettes related to Dining and Visits are usually region-specific; hence, the performance of models on these is poor. In the case of GPT-3.5, we also observed an increase in the contribution wrong predictions in Business etiquette, which
In this paper, we presented EtiCor. corpus of etiquettes covering major world regions. We further evaluated the performance of LLMs the task of Etiquette Sensitivity and the results indicate a significant gap in knowledge and understanding of L.Mss. In the future, we plan to develop region- specific Adapters and integrate them into an LLM via a mixture of experts (Wang et al., 2022).

---

Limitations
Dean Foster, 2002a. The Global Etiquette Guide to Africa and the Middle East: everything you need to know for business and travel success. John Wiley & Sons.
In this paper, we proposed new corpus and ex- perimented on the task of Etiquette Sensitivity in a limited set of few LLMs. We do not develop any new model and leave it for future work. This resource paper aims to introduce the corpus and the task and show the limitations of LLMs when it comes to region-specific etiquettes. The work is a first step towards making more sophisticated etiquelte-sensttive models.
Dean Foster. 2002b. Global eliquette guide to Mexico and Latin America. John Wiley & Sons.
Robert Frederick. 2009. What is commonsense moral- ity? Think, 8(23):7-20.
Dan Hendrycks. Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning Al with shared human values. CoRR, 0222200
Ethical Considerations
Corpus was created from publicly available infor- mation, and no copyright was violated. During corpus creation, we ensured that any personal infor- mation was removed to prevent a model from de- veloping any biases. To the best of our knowledge, we took all steps to keep the corpus as bias-free possible and are unaware of any direct ethical consequences.
Henry Hitchings. 2013. Sorry!: the English and their manners. Macmillan.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ro- nan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi, 2021. Delphi: Towards machine ethics and norms. CoRR, abs/2110.07574.
Minwoo Lee, Scungpil Won, Juae Kim, Hwanhee Lee, Cheoneum Park, and Kyomin Jung. 2021. Crossaug: A contrastive data augmentation method for debias- ing fact verification models. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages
References
Mitchell Abrams and Matthias Scheutz. 2022. Social
norms guide reference resolution, In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1-11, Seattle, United States. Association for Computational Lin- guistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zeulemoyer. 2019. BART: denoising sequence- suneenc-to-sequeee to-seueencee pre-training for natural language generation, translation, and com- prehension. CoRR, abs/1910.13346
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. UNICORN on RAINBOW: A universal commonsense reasoning model on new multitask benchmark. CORR, abs/2103.13009,
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2020. Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes. CoRR. abs/2008.00 09094..
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:18100.00
Judith Martin. 2011. Miss Manners' Guide to Exeru- ciatingly Correct Behavior (Freshly Updated). WW Norton & Company.
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situ- ated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peggy Post, Anna Post, Lizzie Post, and Daniel Post Senning. 2011. Emily Post's Etiquette, 18. Harper Collins.
Maarten Sap, Saadia Gabriel, Lianhui Qin. Dan Juraf- sky. Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. ACL.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap. and Yejin Choi. 2020. Social chem- istry 101: Learning reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
Suzanne Tolmeijer, Markus Kneer, Cristina Sarasua, Markus Christen, and Abraham Bernstein. 2020. Im- plementations machine ethics: A survey. ACM Computing Surveys (CSUR), 53(6):1-38.

---

Amy Vanderbilt. 1958. Complete book of etiquette. Doubleday.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jian- feng Gao. 2022. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205. 12410.
Taha Yasseri, Robert Sumi, and János Kertész. 2012. Cireadian patterns of wikipedia editorial activity: demographic analysis. PLOS ONE, 7(1):1-8.
Caleb Ziems, Jane Dwivedi-Yis Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023, NormBank: A knowl- edge bank situational social norms. Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7756-7776, Toronto, Canada. Association for Computational Linguistics.
Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems.

---

Appendix
TP TN
Accuracy with Abstention
Total Data Points
A EtiCor Details
where:
TP is the number of positives (correct predictions of positive cases)
Table 6, Figure 3, and Table 7 provide more details about the corpus.
TN is the number of true negatives (correct predictions negative cases)
Region
T'rvvll
Dining # Visits
Buisness
FP is the number of false positives (incorrect predictions of positive cases)
EA
1198
2605
1439
2190
MEA
1041
2907
2861
3222
IN
690
1305
1504
1057
FN is the number of false negatives (incorrect predictions of negative cases)
LA
1107
2184
2494
2387
NE
642
1921
1508
2085
Total
4678
10922
9806
10941
N Overall Score Σ
wy Score,,
Table 6: Distribution of different etiquette types
where,
count, Wr countrotat where, county is the number of samples in region and countrotal is total number of samples in the dataset.
12000
Etiquette Count
Score, is the accuracy/Fi score for region r.
8000
C Results Details
6000
Table 8 provides results when abstained outputs are considered negative labels. this case, we could not calculate the Fl score, as it is difficult to automatically classify an abstained output as a false positive or false negative. Table shows the percentage of wrong predictions for each eti- quette type for each model. These are calculated by taking all the incorrect predictions model makes and then manually classifying them into various etiquette types. Subsequently, the ratio of wrong predictions for each type out of all the incorrect predictions gives the percentage of wrong predic- tions (for a model) for that type. Table 10 shows
4000
2000
20
Sentence Length
Figure
Word Length Distribution for EtiCor
# Positive Labels
# Negative Labels
Region
Total
(+1)
(-1)
EA
4218
3214
7432
MEA
5733
4298
10031
2356
2200
4556
LA
4544
3628
8172
the percentage of wrong predictions for each model for each region.
NE
4230
1926
6156
Total
21081
15266
36347
D Model Output Examples
Table 7: EtiCor's label distribution
Table 11 provides some example outputs of the models.
B Evaluation Metrics
TP
E BERT Results
Precision TP + FP
Table 12, 13, 14, 15, and Table 16 show 5-fold cross validation results for each of the region. We also combined the data from each of the region and performed 5-fold cross validation using the BERT model. The results are given in the last row Table 5. Detailed results are shown in Table 17.
TP
Recall TP + FN
2. Precision Recall
F1 Precision Recall
TP TN
Accuracy without Abstention
TP+ TN FP FN

---

Delphi
Falcon
GPT3.5
Region
Accuracy with Abstentions
f of Abstentions
Accuracy with
# of Abstentions
Accuracy with
of
Abstentions
Abstentions
Abstentions
EA
0,56
809
0.621
480
0.54
32
IN
0.69
154
0.603
806
0.52
14
MEA
0,57
1149
0.681
123
0.55
46
NE
0,75
220
0.697
453
0,62
36
LA
0.55
1654
0.541
731
0.53
88
Overall
0.61
3986
0.63
2593
0.55
216
Table 8: Results of models after including abstained predictions as negatives
Etiquette Type Delphi Falcon-40B GPT-3.5
Travel
15.46
19.70
Business
28.87
26.56
24.65
Visits
40.27
29.17
23.74
Dining
15.40
34.45
31.91
Overall
100.00
100.00
100.00
Table 9: Percentage of Wrong Predictions for each model for each etiquette type. Note that the numbers in the table are percentages.
Delphi
Falcon-40B
GPT-3.5
Etiquette Type
EA
IN
MEA
NA
LA
EA
IN
MEA
NA
LA
EA
IN
MEA
NA
LA
Travel
37.2
21.2
20.7
16.6
36.5
47.8
57.6
30.5
65.0
28.8
22.3
22.0
Business
12.4
35.3
27.7
42.3
11.0
23.3
13.8
19.5
19.3
11.3
18.9
46.3
12.6
14.8
40.5
Visits
12.2
12.2
24.1
25.6
56.2
26.8
11.3
10.8
30.3
10.5
36.7
21.0
26.7
21.1
10.1
Dining
10.0
15.3
27.0
11.4
16.2
13.4
27.1
12.1
19.9
13.2
15.6
10.4
48.9
42.1
12.7
Table 10: Percentage of Wrong Predictions for each etiquette type for each region by various models, Note that the numbers in the table are percentages.

---

Gold
Region Sentence
Delphi Falcon-40B GPT-3.5
Label
English is widely spoken in business and government contexts though there
EA
exceptions such trading.
Pointing chopstick towards someone while having dinner
Public display affection of opposite genders fine
1
Extended office hours widcly accepted throughout Japan.
-1
In rural India, it is common practice to guests and invite them to sit the floor
IN
abstain
-1
Yawning in public is considered impolite you must cover your mouth when you yawn
IN
IN Eating with bare hands is common practice here
-1
IN
okay drink alcohol parks and public gardens
-1
-1
Indians often ask intrusive questions out of curiosity about foreign people.
IN
-1
India is still very conservative nation and hugging and kissing are not common practices especially with newly made acquaintance
IN
unethical to eat beef India with religious context added it.
IN
Women should be sensitive the fact that they may be seated only with other
MEA
-1
-1
-1
Typically, conversation while eating is extensive, and most Nigerians avoid eye contact when dining
MEA
abstain
Follow the guidelines and procedures for performing Tawaf Allfadah and the symbolic stoning of the devil Ramy al Jamarat during Hajj
MEA
Use of Perfumes Fragrances play crucial part in Kuwaiti society not uncommon offered
MEA
perfume at social gatherings
MEA
Mawtayn marhabtayn alan is less formal greeting
-1
abstain
In some of the countries in the Arab world video taping is illegal and traditionally, in Islam images of people and the human form considered sacrilegious
MEA
-1
-1
Ethiopians shake hands slowly when greeting each other
MEA
In Hungary: Close friends one another lightly both cheeks, starting with the left cheek.
NE
-1
-1
Avoid excessive physical contact, such as hugging, unless is culturally inappropriate
NE
Keep a fire blanket or baking soda quite far away to prevent small kitchen fires.
NE
Restaurants usually have percent tip already included the hill.
LA
-1
Children Colombia expected to be respectful and overly conversational when speaking with adults
LA
-1
There is rigid separation of the genders in Haiti, more in the less urbanised areas. Women mainly sell the crops and wares in the market while men work the fields and manufacturing jobs in small shops
LA
Table II: Some Examples of Etiquette's and their corresponding shot results

---

Validation Accuracy
Test Accuracy
Test
Fold
FI Score
Fold
0.869
0,836
0.873
Fold
0.879
0.859
0.900
Fold 3
0,880
0.869
0,908
Fold
0.902
0.869
0,907
Fold 0.950
0.928
0.978
Overall 0.896 (#0.032) 0.872 (=0.034) 0.913 (=0.038)
Table BERT Model: validation accuracy, test accu- racy, and test Fl score for each fold for EA region.
Validation Accuracy
Test Accuracy
Test FI Score
Fold
Fold
0.918
0.892
0.880
Fold
0.902
0.887
Fold
0.863
0.842
0.854
Fold 4
0,939
0.903
0.908
Fold
0,881
0.864
0.876
Overall 0.901 (=0.028) 0.878 (=0.028) 0.882 (=0.028)
Table 13: BERT Model: validation accuracy,
racy, and test Fl score for each fold for IN region.
Validation Accuracy
Test Accuracy
Test
Validation Accuracy
Test Accuracy
Test FI Score
Fold
Fl Score
Fold
Fold
0.911
0.890
0.915
Fold
0.939
0.918
0,928
Fold 2
0.904
0.883
0,921
Fold
0.979
0.965
0.970
Fold 3
0.881
0.847
0.864
Fold
0,980
0.969
0.973
Fold 4
0.897
0,873
0.914
Fold
0,927
0.919
0.928
0.866
0,895
Fold 5
0.939
0.916
0.931
Overall 0.894 (20.012) 0.871 (10.015) 0.902 (10.021)
Overall 0.953 (+0.025) 0.938 (=0.027) 0.946 (=0.023)
Table 17: BERT Model: validation accuracy, test ac- curacy, and test score for each fold for data of all regions combined.
Table 14: BERT Model: validation accuracy,
racy, and test Fl score for each fold for MEA region.
Validation Accuracy
Test Accuracy
Test Score
Fold
Fold
0.868
0.858
0.878
Fold 2
0.958
0.930
0.950
Fold
0.964
0.938
0,955
Fold
0.927
0.918
0.945
Fold 0,936
0.920
0,942
Overall 0.931 (#0.038) 0.919 (=0.032) 0.934 (=0.032)
Table 15: BERT Model: validation accuracy, test accu- racy, and test Fl score for each fold for NE region.
Validation Accuracy
Test Accuracy
Test F1 Score
Fold
Fold
0.935
0.880
0.909
Fold 2
0.941
0.858
0.878
Fold
0.943
0.875
0.919
Fold
0.917
0.848
0.918
Fold
0,914
0.882
0.942
Overall 0.930 (20.013) 0.869 (=0.015) 0.913 (=0.023)
Table 16: BERT Model: validation accuracy, test accu- racy, and test Fl score for each fold for LA region.


Reviews:
[{"review_id": "T3OUgGeUkL", "reviewer": "Reviewer_vwjx", "summary": "", "questions": "- What is the theoretical underpinnings for definition of \"Etiquette\" and how do you ensure that dataset isn't subjective?\n- What was the criteria for \n     - annotating the data? or weak labelling?\n     - selecting websites?\n- What was the prompt used to query LLMs?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your insightful comments and we are glad you appreciate the corpus. We provide detailed answers to your comments below:\n\n---\n**Comparison to Existing Resources:** \n\nTo the best of our knowledge there is no existing dataset that focuses on Etiquettes. We take a general dictionary definition as a set of social norms/conventions or set of rules which tell about how to behave in a particular social situation.\nThe motivation behind EtiCor is to promote development of systems (e.g., LLMs) that are sensitive to etiquettes (social norms) of a region (as discussed in lines 70-74). Corpora created with regard to Ethics are usually based on ethical theories and do not necessarily address etiquettes and social conventions.  In fact, we tried an existing model (Delphi) trained on a ethics corpus (Rainbow corpus) and we found that its performance on EtiCor (Etiquette corpus) is not good (Table 4), pointing towards the gap in Delphi’s knowledge and hence there is a need for a corpus focusing on etiquettes. \nWe discuss and compare with existing corpora in Section 2 (Related Work). Due to the space constraint of 4 pages we could go into more details but in the final version we can provide more details about differences between our corpus and corpora based on Ethics. \n\n **Missing References:**\nThanks for pointing missing references\n1. We were not aware of NormBank work as it was released in July 2023 and we submitted the paper in June 2023. NormsBank corpus is about situational commonsense and norms and it comes close to our corpus. We will include details about this in our updated version of the paper. \n2. Paper by Abrams & Scheutz, 2022 focuses on general etiquettes and not region specific etiquettes. Our corpus focuses on region specific etiquettes, as social norms vary across regions. Moreover, their study is mostly about the context of a sentence, and they have tried out different models to see how well they understood the context of a sentence, which helped them understand social norms. Our study on the other hand does not dwell much into the underlying linguistic context of the sentence and aims to include regional context by sensitizing about regional etiquettes. \n\n---\n**Data Creation Methodology**\n\n1. As described in the paper (lines 157-164), we scraped data from authentic, trusted and publicly available official travel and government websites which are trusted (e.g., by travelers), so each etiquette (sentence) is an acceptable social norm in that region. \n2. Criteria for selection of website is already mentioned in lines 160-165. \nWe didn’t involve annotators but ourselves validated the data. We also manually filtered out noisy sentences (line 167-170). We took each of the scraped sentences as it is and verified that it made sense. \n3. Criteria for combining sub-geographic regions was based on geographical proximity of regions and variance in the etiquette of each of the sub-region. We also provide details in the paper (lines 182-225). We manually examined etiquette for each sub-region and then merged the regions that had similar etiquettes into a bigger region. We can provide more details about this in the updated version of the paper. \n---\n\nWe provided details about the prompt used for LLMs in lines 280-285. In the updated version of the paper, we can provide more details. \n\n---\n\n**Questions**\n1. We use the following definition of etiquette: set of social norms/conventions or set of rules which tell about how to behave in a particular social situation. By definition, since etiquette is region specific, it will be subjective.\n2. We provide details about the criteria, labeling and website selection above. \n3. We provide details about the prompt above. \n\n---\n\nThanks for your suggestions about typos, grammar and presentation style, we will fix these in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors present a new corpus about etiquette of five regions of the world, contianing 35k labels. \n- Based on the dataset, author propose a new task - Etiquette Sensitivity - to test if a social norm is appropriate or not given a geographical region.", "reasons_to_reject": "- If there are existing resources to test model's response are ethical or stick to moral integrity - it is not clear the research gap a new etiquette corpus is filling. Including a discussion on the same would be immensely beneficial to the paper. \n- Data creation methodology is presented without necessary details - \n    - what is criteria that makes a sentence acceptable social norm or not?\n    - what is the criteria for selecting those specific websites listed in appendix?\n    - verification/validation of the scrapped data and if the labels were infact correct? How many annotators, guidelines for annotation?\n    - Social norm can be very subjective - within a sub-geographic region as well - what is the rationale for combining continents/countries. \n- Manner of querying models for their knowledge of social etiquette is not presented in the paper.", "questions_to_authors": "- What is the theoretical underpinnings for definition of \"Etiquette\" and how do you ensure that dataset isn't subjective?\n- What was the criteria for \n     - annotating the data? or weak labelling?\n     - selecting websites?\n- What was the prompt used to query LLMs?", "paper_topic_and_main_contributions": "Authors present a dataset, EtiCor, containing text samples labelled for weather or not the text is an acceptable etiquette. Samples are sourced from diverse geographical locations. Author use the dataset as a test bed for evaluating LLMs for ability to be sensitive for to acceptable and unacceptable social norms (etiquettes).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [NormBank: A Knowledge Bank of Situational Social Norms](https://aclanthology.org/2023.acl-long.429) (Ziems et al., ACL 2023)\n- [Social Norms Guide Reference Resolution](https://aclanthology.org/2022.naacl-main.1) (Abrams & Scheutz, NAACL 2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "- Appendix D can be much more readable if it can be condensed into a table. \n- Appendix figures, tables and subsections can be referenced better. In its current form they are hard to follow. \n- Some of the assumptions made in the paper - either should be supported with citations or shown emperically - for instance the claim that the model is trained on \"data from North American and European region\" (L259).\n- Maybe short paper is not suitable for this work - as paper lacks details in most of the sections of the paper."}}, {"review_id": "aK8EMwUnuv", "reviewer": "Reviewer_j5Th", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed and insightful review. We find your review very encouraging! Thanks! We are glad that you appreciate the simplicity of the paper and find experimental results interesting. Due to space constraint we had to move the examples table in Appendix, if accepted, we will move a part of it into the main paper. \n\nThanks for pointing out confusion about Falcon-40B results, we will re-write and make it more clear. \n\nRegarding BERT, we ran it only once. Yes, we agree with you that we should report averaged out results over multiple runs. If accepted, we will include these results in the paper. \n\nThanks for pointing out the typos, we will fix these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written, easy-to-follow and straightforwardly explained paper, which I personally appreciate. \n\nThe authors use norms extracted from verified sources, such as guides, government web pages, etc. They only use information in English. I believe it's a good point to avoid automatic translation systems as these can have the same biases regarding social etiquette as LLM, so the translation could not be accurate. I don't know if the decision was taken considering that, but it's a strong point to back it. \n\nThe annotation is by region, but in specific cases, the label can point to a single country or even a specific cultural group. I consider this dataset very interesting and useful, not only for NLP purposes but also to better understand others and create a more equal, understanding and responsible society in general, I will be very interested in having a look at it once it is made available.\n\nThe GPT3.5 results about MEA social norms, which the model refuses to process and/or qualifies as gender-specific hate are very interesting and could motivate an extensive multidisciplinary qualitative analysis.\n\nTable 9 examples of outputs are very interesting, one of the most interesting tables in the paper, but it's not referred to in the whole paper (it is in app.C, but many readers might not arrive there! Consider either including some examples in the main body of the paper, or at least letting the reader know that that table is in the Appendix because it's really interesting.)", "reasons_to_reject": "Falcon40B results are confusing, consider re-writing.\n\nExperiments with BERT, are the results coming from a single run? There is much difference between validation and test accuracy, which points to either a different distribution in both chunks or a high degree of randomness in the results. Running several runs and calculating the average would make the results much stronger.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces an interesting topic, by analyzing the (gap of) knowledge of LLM about localized etiquettes. It also reveals how LLM are biased towards Western societies, missing many aspects of other cultures, such as etiquette. \n\nThey propose EtiCOR, a corpus of 35k annotated social norms/ etiquettes followed in 5 big world regions, namely India, Latin America,\nJapan-Korea-Taiwan, Middle-East-Africa, North America, and Europe. The corpus is in English, but the objective is to make it multilingual in the future.\n\nThe authors also propose the task of Etiquette Sensitivity to test several models. Given a statement about a social norm pertaining to a region, the objective is to classify if the social norm is appropriate or not relative to that region. They experiment with zero-shot learning with LLM (i.e. Delphi, Falcon-40B and GPT3.5, and then compared it with a finetuned BERT model. \nThe paper addresses the analysis of the potential biases regarding social etiquettes included in LLM, with the final goal of contributing towards more responsible and inclusive AI systems.\n\n ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "lines 150-151 - Dining and Festivals, Visits, and Social, Travel, and Business. --> The distribution of the commas makes it a bit confusing to understand the 4 groups. I assume it is 1) Dining and Festivals, 2) Visits and Social (?), 3) Travel and 4) Business, is that correct?\n\nlines 152,155, 179, 256, 272 (at least) - You mention App. without saying in which appendix we can find the information, the appendix name / letter is missing.\n\nlines 184 -187: Sentence \"We could include China and other South-East Asian countries due unavailability of a large number of social norms in the English language\" --> We could NOT include China [...] due TO unavailability [...]\n\nIt would be useful if you include the acronym for each region, so it's easier to read the tables.\n\nline 261 --> You say scores are low for each region, but they don't seem that low, consider rephrasing.\n\nlines 270-271 -->  \"The results were quite close to Delphi but not good\". --> consider rephrasing \n\nIn the results, you mention GPT3.5 first and ChatGPT later. Maybe it's better to unify the way of naming the model or explain that you use ChatGPT with GPT3.5. "}}, {"review_id": "PSNV4AnWkf", "reviewer": "Reviewer_FAuQ", "summary": "", "questions": "1. The related work section focuses on ethically sound Machine Learning (ML) models, is there any work related to corpus on social norms?\n2. In the discussion of the preprocessing step (line 168), two filters on sentence length have been mentioned but never clarified. What is the range of sentence lengths accepted?\n3. In the discussion of results achieved by LLMs the authors mention the fact that \"scores are low for each region\". May the authors argue on this point? In my opinion the F1 scores are not that low, especially when compared with the fine-tuned BERT model on the dataset. Although the numbers are not comparable, BERT scores little better than the three LLMs.\n\t3.A. Did the authors consider framing the task as multi-label? Reducing the problem to a binary task is a strong simplification; it might be interesting to study the performance of the models in a case where they have to decide for which region a norm is acceptable.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and thoughtful suggestions. \n\n---\n\nThanks for pointing out the missing definition of etiquette. We use a standard dictionary definition of an etiquette and define it as a set of social norms/conventions or set of rules which tell about how to behave in a particular social situation. If accepted, we will update the paper to include the definition of etiquette. \n\n---\n\nThanks for pointing out that zero-shot results of LLMs and BERT are not comparable. Yes, we agree with this idea and we will perform BERT experiments using ten-fold cross-validation. \n\n---\n\n**Questions**\n1. Related Work: Yes, to the best of our knowledge, there are two corpora related to social norms (SocialChem and Moral Integrity Corpus). We have written about this in lines 118-130. Due to space limitations we could not include more details but if accepted we can include more details. \n2. Range of sentence length: minimum length was 50 characters, any sentence having length below 50 characters was removed..\n3. The accuracy and F1 score reported in Table-4 are for cases where models always reported a definite answer (i.e., without abstention cases). If we had considered abstention as incorrect prediction then the overall accuracy and F1 scores comes out much lower (by approx. 15-20%). If accepted, we include results with abstention in the paper as well. \n4. Thanks for pointing out the idea of posing a problem as multi-label. However, there are some etiquette which are contradictory to each other in different regions ( lines 193-104 and lines 204-206), so this can create problems in the multi-label setting. Nevertheless, we could remove such conflicting information and then we can try out the multi-label setting and provide results in the updated version of the paper.\n\n---\n\nThanks for pointing out typos and minor mistakes, we will fix these in the updated version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is interesting, the paper is well-written and easy to follow. The paper introduces a novel corpus with intense manual curation.", "reasons_to_reject": "The paper is not always self-contained: for example no general definition of etiquette has been provided, although it can be guessed from the text, I suggest the authors provide a clear definition from the introduction. The experimentation on the Etiquette Sensitivity task should be reviewed: the results achieved by the BERT model are not comparable to those reported on Table 4 since they have been obtained on different partitions of the same set. In order to make these numbers comparable a ten-fold cross validation approach should be used for the fine-tuned system.", "questions_to_authors": "1. The related work section focuses on ethically sound Machine Learning (ML) models, is there any work related to corpus on social norms?\n2. In the discussion of the preprocessing step (line 168), two filters on sentence length have been mentioned but never clarified. What is the range of sentence lengths accepted?\n3. In the discussion of results achieved by LLMs the authors mention the fact that \"scores are low for each region\". May the authors argue on this point? In my opinion the F1 scores are not that low, especially when compared with the fine-tuned BERT model on the dataset. Although the numbers are not comparable, BERT scores little better than the three LLMs.\n\t3.A. Did the authors consider framing the task as multi-label? Reducing the problem to a binary task is a strong simplification; it might be interesting to study the performance of the models in a case where they have to decide for which region a norm is acceptable.", "paper_topic_and_main_contributions": "The paper presents EtiCor, a novel English corpus of etiquettes covering five major world regions. More precisely, the corpus includes data for five major regions: India, Latin America, Japan-Korea-Taiwan, Middle-East-Africa, North America, and Europe. The corpus has been constructed by scraping websites providing information about regional etiquettes, tour guide points, pamphlets, etiquette information channels tweets and books on etiquettes. The scraped norms have been manually labeled as pertaining to a specific region (with label +1) or not belonging to the specific region (with label -1).\nFinally, the authors introduced a novel Etiquette Sensitivity task: given a statement about etiquette, the task is to predict whether the statement is appropriate for a region or not. Three Large Language Models (LLMs) have been assessed on the novel task and the results show no particular difference among the assessed models (Delphi, Falcon and GPT3.5). Additionally, the authors fine-tuned four different BERT binary classifiers reporting small improvement over the LLMs assessed in a zero-shot setting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please find some minor comments below:\n- please consider providing a definition for personal digital assistants on line 032;\n- please consider referencing Table 2 before Table 6 on rows 152-153, this may help the reader to understand the abbreviations in tables that have been introduced in Table 2;\n- please consider removing the two \"and\" on line 163, rewriting the list as regional etiquettes, tour guide points, pamphlets, etiquette information channels tweets and books on etiquettes;\n- please consider introducing vertical lines on Table 4 separating the three different LLMs and introducing the bold notation for highest model for each region;\n- please consider rewriting Sensitivityby on line 279."}}]

Author rebuttal:

Meta-review: **Summary:**
The authors introduce the EtiCor dataset, comprising 35,000 manually annotated text samples labeled for determining whether the text adheres to acceptable localized etiquette or not. Specifically, this corpus encompasses etiquettes from five major world regions plus Europe: India, Latin America, Japan-Korea-Taiwan, Middle-East-Africa, North America, and Europe. The dataset was constructed by scraping websites that provide information about regional etiquettes, tourist guide points, pamphlets, etiquette information channels, tweets, and books on etiquettes. While the corpus is currently in English, the long-term goal is to make it multilingual. The authors employ this dataset as a testbed for evaluating Large Language Models (LLMs) such as Delphi, Falcon-40B, and GPT3.5, introducing a novel Etiquette Sensitivity task. In this task, given a statement about etiquette, the goal is to predict whether the statement is appropriate for a specific region. This research reveals how LLMs exhibit biases toward Western societies while overlooking many aspects of other cultures. Additionally, the authors fine-tune four distinct BERT binary classifiers, reporting modest improvements over the LLMs evaluated in a zero-shot setting.

**Strengths:**
The reviewers are in agreement regarding the strengths of the paper:
1. The authors introduce a novel corpus focusing on etiquettes from five distinct global regions, featuring 35,000 meticulously curated labels.
2. Building upon this dataset, the authors propose a new task, 'Etiquette Sensitivity,' aimed at assessing the appropriateness of a social norm within a given geographical context.
3. The paper delves into an analysis of potential biases within Large Language Models (LLMs) concerning social etiquettes, with the overarching aim of promoting responsible and inclusive AI systems.
4. The dataset is not only valuable for NLP applications but also for fostering cross-cultural understanding and promoting a more equitable, empathetic, and responsible society.
5. Additionally, one reviewer highlights the interesting findings related to GPT3.5's responses to Middle East and Africa (MEA) social norms, where the model either refuses to process or categorizes them as gender-specific hate. This outcome could motivate an extensive multidisciplinary qualitative analysis.

**Weaknesses:**
Reviewers have identified the following weaknesses in the paper:
1. The paper lacks clarity in articulating the specific research gap addressed by the new etiquette corpus, as it may not be evident considering the existence of other resources for evaluating model responses in terms of ethical and moral integrity.
2. The methodology used to create the data is presented without the necessary level of detail.
3. The paper does not adequately describe the approach taken to query models for their knowledge of social etiquette.
4. Some of the results reported by the authors are not well-described, and the paper is not consistently self-contained. For instance, the results related to Falcon40B lack sufficient clarity, and there are discrepancies in the reported accuracy of BERT experiments between the validation and test sets. Furthermore, the BERT experiments do not employ a ten-fold cross-validation approach for the fine-tuned system.

**Author-Reviewer discussion and acknowledgment:**
The authors gave clarifications in response to the concerns raised by the reviewers and have outlined the planned improvements to be made during the rebuttal response and discussion phase. All reviewers have responded and acknowledged the authors' arguments

**Conclusion:** 
The topic presented in this work is interesting, and the paper is well-written and easy to follow. However, reviewers suggest that the authors address the identified typos. Furthermore, reviewers recommend that the authors incorporate additional references and enhance the paper based on the points raised during the discussion phase.
Decision: Accept-Main