Comparing Styles across Languages
Authors: Shreya Havaldar
Matthew Pressimone
Eric Wong
Lyle Ungar
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
Style
Cross-Cultural
Multilingual
Explainability
Lexica
TL;DR: We introduce an explanation framework to extract stylistic differences from multilingual LMs and compare style across languages.

Abstract: Understanding how styles differ across languages is advantageous for training both humans and computers to generate culturally appropriate text. We introduce an explanation framework to extract stylistic differences from multilingual LMs and compare styles across languages. Our framework (1) generates comprehensive style lexica in any language and (2) consolidates feature importances from LMs into comparable lexical categories. We apply this framework to compare politeness, creating the first holistic multilingual politeness dataset and exploring how politeness varies across four languages. Our approach enables an effective evaluation of how distinct linguistic categories contribute to stylistic variations and provides interpretable insights into how people communicate differently around the world.

Comparing Styles across Languages
Shreya Havaldar, Matthew Pressimone, Eric Wong, & Lyle Ungar University Pennsylvania
(shreyah,mpressi,, exwong, ungar) upenn.ee edu
Abstract
English Utterance
Politemess 1.02
Thanks, but do you really think spiders are insects? Pleuse
Understanding how styles differ across lan- guages is advantageous for training both hu- mans and computers to generate culturally ap- propriate text. We introduce an explanation framework extract stylistic differences from multilingual LMs and compare styles across languages. Our framework (1) generates com- prehensive style lexica in any language and (2) consolidates feature importances from LMs into comparable lexical categories. We apply this framework to compare politeness, creating the first holistic multilingual politeness dataset and exploring how politeness varies across four languages. Our approach enables an effective evaluation how distinct linguistic categories contribute to stylistic variations and provides interpretable insights into how people commu- nicate differently around the world.
0.83
0.03
stop editing as possible, I'd really appreciate
0.44
1.09
Parallel Chinese Utterance
Politemess 00.26
0.09 00.28
0.59
Comparing Multilingual Politeness via Lexicul Calegories
Chinese: 0.61
Gratitudo
Rade
Polite
Englshh 0.96
Chineees 0.09
Please
Rude
Polite
English: 00.08
Chimese: 0.28
Emergency
Rode
Polite
English: 0.44
1 Introduction
Figure 1: Explaining how politeness differs between par- allel sentences English and Chinese. Though these two sentences are identical in contenr, they differ in style. Our framework provides a way to quantitatively compare politeness in English and Chinese by compar- ing the importance of interpretable lexical categories.
Communication practices vary across cultures. In- herent differences in how people think and behave (Lehmaa et al., 2004) influence cultural norms, which in turn, have significant impact commu- nication (Moorjani and Field, 1988). One key way that cultural variation influences communication is through linguistic style. In this work, we introduce an explanation framework to extract stylistic differ- ences from multilingual language models (LMs), enabling cross-cultural style comparison.
interactions in workplace setting. (Khatri, 2009). For example, consider the two conversation snip- pets in Figure 1. Though these two utterances are direct translations of each other, multilingual LM fine-tuned for politeness classification outputs dif- ferent labels, with the Chinese utterance labeled as impolite. We ask a bilingual English/Chinese speaker provide further insights, and she ob- serves that the same request "stop editing as soon as possible" sounds more aggressive (and thus, impolite) in Chinese than in English, as cultural norms in China do not typically condone giving such harsh, direct instructions to stranger. Stylistically appropriate language is crucial successful communication within a certain culture. However, multilingual LMs sometimes struggle to
Style is a complex and nuanced construction. In communication, style is heavily used to con- vey certain personal or social goals depending on the speakers' culture (Kang and Hovy, 2021). For example, cultures that are high-context tend to use more indirect language than those that are low- context (Kim et al., 1998), and cultures that have a high power-distance" tend to have more formal
'Conversations high-context cultures have lot llety require more collective understanding. (Hall, 1976) Power distance refers to the strength of socicty's social hicrarchy. (Hofstede, 2005)

---

SHAP
Token-level importances
(2) Feature Set
Aggregation
Multilingual
.14
12
hola .21
1u X -.18
Category-level Importances
yes
.25
U
.19
sí -.06
Style LMs
Lexical Categories
Fn
(I) Multilingual
Ja
Zh A
Lexica Creation
please
favor
En Ja Es Zh Languages
En Style Lexica
Ja/ Zh/ Style Lexica
Figure 2: Our two-part explanation framework. Note that we use standard ISO language codes: En, Ja, Es, and 7h for English. Japanesc, Spanish, and Chinese respectively.
generate language that is stylistically appropriate in non-English languages (Hershcovich et al., 2022; Ersoy et al., 2023; Zhang et al., 2022). Standard training methods for multilingual models lead to little stylistic variation in generated text across lan- guages (Pircs et al., 2019; Libovický et al., 2020; Muller et al., 2021), and multilingual systems rarely address these socially-driven factors of language (Hovy and Yang. 2021). As a result, downstream applications of these systems, like chatbots, are not as usable or beneficial to a non-American audience (Bawa et al., 2020; Havaldar et al., 2023a).
way. Our framework consists of two components:
Multilingual Lexica Creation: We utilize embedding- basedd methods to translate and ex- pand style lexica in any language.
2. Feature Set Aggregation: We extract feature importances from LMs and consolidate them into comparable lexical categories.
To study how styles differ across languages, we create a holistic politeness dataset that encom- passes rich set of linguistic variations in four lan- guages. Politeness varies greatly across languages due to cultural influences, and it is necessary understand this variation in order build multilin- gual systems with culturally-appropriaa politeness. Trustworthy and successful conversational agents for therapy, teaching, customer service, etc. must be able to adapt levels and expressions of politeness to properly reflect the cultural norms of user. et al.
One step towards correcting this is to understand how styles differ across languages. Though modern multilingual LMs struggle with generating stylisti- cally appropriate language, they are generally quite successful at classifying stylistic language (Briakou et al., 2021; Plaza-del Arco et al., 2020; El-Alami et al., 2022; Srinivasan and Choi, 2022). Psycho- logical or linguistic studies to analyze language are resource-heat heavy and uime-intensive. alternatively. we can utilize these trained LMs to computationally capture both the overt and subtle ways language reflects certain style.
Previous work by (2013) and Srinivasan and Choi (2022) uses NLP to study politeness, but their datasets reflect only small subset of language the conversational utterances they analyze consist of only questions on either extremes of the impolite/polite spectrum. Our dataset the first holistic multilingual po- liteness dataset that includes all types of sentences (i.e. dialogue acts) across the full impolite/polite spectrum. We include English, Spanish, Japanese, and Chinese select these languages as they all high-resource, (and therefore well-supported by modern LMs) and each have unique way of expressing politeness. For instance, politeness in Japan frequently arises from acknowledging the place of others (Spencer- Oatey and Kádár, 2016),
Most current methods extract feature impor- tances from multilingual LMs are at the token-level (Lundberg and Lee, 2017; Ribeiro et al., 2016; Sun- dararajan et al., 2017) and specific to each lan- guage; as a result, there is no "common language" for comparison. In addition, most explanations are not easily human-interpretablee and it is difficult to extract useful takeaways from them.
In this work, we present a framework to extract differences in style from multilingual LMs and ex- plain these differences in a functional, interpretable

---

while politeness in Spanish-speaking countries of- ten relies on expressing mutual respect (Placencia and Fercaae-e,,,addz, 2017). This global variation in politeness (Leech, 2007; Pishghadam and Navari, 2012) makes an important style to understand for effective cross-cultural communication.
Synonym Expansion
Apologetic
Apologetic siento perdón perdona
Purification
Apologetic to sienio perdón perdona discudpa
apulogy
English Style Lexica
disculpa
Spunish Style Lexica
Our contributions
follows:
Concept Expansion
Hedge evidente
1. We present an explanation framework ex- tract differences in styles from multilingual LMs and meaningfully compare styles across languages.
Hedge typically
Hedge evidente
apurente seride aveguro
suggest tends
apareuse sehala asrguró
2. We provide the first holistic politeness dataset that reflects realistic distribution of conver- sational data across four languages.
Figure 3: Multilingual Lexica Creation (MLC). We use word embeddings to expand and purify a style lexica translated from one language into another. This allows for maximum coverage per lexical category and corrects issues with standard 1:1 machine translation.
3. We use this framework and dataset to show differences in how politeness is expressed (e.g. words like "bro" and "mate" are rude in En- glish, but polite in Japanese, and yes/no ques- tions are rude in Chinese but not in English.)
Much work has already been done in curating theory-grounde lexica that classify style. Danescu- Niculescu-Mizz
al. (2013) curate lexical strate- gies that inform politeness in English, and Li et al. (2020) extend these strategies to Chinese.
Figure 1 shows an example comparison of En- glish and Chinese politeness using this framework. We explain differences in politeness using both lex- ical categories and dialogue acts, with Figure highlighting three chosen lexical categories. We make all code and data available publicly.
Though they provide insight into how politeness is expressed, lexica have limited predictive power; lexica-based models are drastically outperformed by modern LMs. Rather than relying on lexica to classify style, we instead use lexica to curate a common language for interpretable multilingual comparison. We present our method, Multilingual Lexica Creation (MLC), to expand curated style lexica into multiple languages.
2
A Framework for Multilingual Style Comparison
this section, we detail our two-part framework for multilingual style comparison. (1) Multilin- gual Lexica Creation takes a curated style lexica and uses umbeddingg-assed methods to refine lex- ica translation into another language, creating set of parallel lexical categories. (2) Feature Set Aggregation maps extracted feature importances from trained LMs to these parallel lexical cate- gories across languages. This framework helps us to interpret what multilingual LMs learn about style during training, and allows us to meaningfully compare how style is expressed across languages.
Motivation. When using standard 1:1 machine translation to translate style lexica, number of issues arise. Sometimes, there is no 1:1 mapping between words have O words or 2+ words that express it in another. Additionally, context and culture influence how linguistic style expressed (Moorjani and Field, 1988) - a word that reflects politeness one lan- guage may not reflect politeness the same way in another. To combat these issues, MLC uses word embeddings to improve the translation process.
word in one language may
2.1 Multilingual Lexica Creation (MLC)
Lexica provide an interpretable grouping of words into meaningful categories. The traditional use of lexica in NLP relies on simple bfg-of-words representation, but allows humans to casily visual- ize which lexical categories appear in dataset.
Step 1: Expansion. In the expansion step, we tackle the flawed assumption that there always ists 1:1 mapping between words in different lan- guages, First, we machine translate curated style lexica into the target language. We refer to the words in this machine translated lexica as our set of seed words. Next, we embed each seed word us-
hhttps://github.. htwpr/hhw..comm/hheyaaaallaaaaaa haltilingual_politeee

---

ing FastText (Bojanowski al., 2017) and perform synonym expansion on each seed word and concept expansion on each lexical category. Figure details this two-stage expansion process.
cal category C, we correlate the style scores of all utterances containing w against the style scores of all utterances containing any word in C. We then remove all words that do not correlate positively with their category (product-moment correlation < 0.15). For example, "señala" does not have similar role to other Hedge words when indicat- ing politeness in Spanish (r -0.08), and so, we remove it from the final lexica. This guarantees that all words within lexical category play a sim- ilar role in determining an utterance's style score, ensuring internal correlation within each category.
For synonym expansion, we find the nearest neighbors of each seed word in embedding space (within tunable distance threshold) and append them to the corresponding lexical category. This adds any synonyms of seed words that may have been bypassed during machine translation. For instance, "sorry" in English is expanded to "lo siento", "perdón", and "perdona" in Spanish.
For concept expansion, we average the embed- dings of all seed words within lexical category and find the centroid embedding of the category. We then append the nearest neighbors of this cen- troid (within a tunable distance threshold) to the overall category. This adds any additional words conceptually similar to the lexical category that were not included via machine translation. For ex- ample, the Hedge category in Figure 3 is expanded to additionally include "evidente", "aparente". and "señala" in Spanish.
MLC takes a curated style lexica and creates a parallel style lexica in a larget language, correcting issues with 1:1 machine translation. Though the lexical categories are parallel, the words within each category are selected to best reflect how a style is expressed in the target language.
2.2 Feature Set Aggregation
We now seek to leverage the fact that trained multi- lingual LMS do successfully learn to encode how multiple languages reflect certain style.
We choose FastText over other embedding mod- els as it has a fixed vocabulary size, and thus, effi- cient nearest neighbors functionality. Additionally, FastText performs well in uncontextualized settings (Laville et ul., 2020) and supports 157 languages. However, embeddings from any model can be used,
Motivation. However, traditional feature attribu- tion methods to explain what LMs learn cannot be used for multilingual comparison, as extracted features always are specific single language. We bypass this limitation by aggregating extracted attributions into directly comparable categories.
Step 2: Purification. In the purification step, we tackle the issue that word reflecting style in one language may not reflect that style in the same way when translated to another language. So, after combining the words returned from synonym and concept expansion, we ensure that each category of the expanded lexica contains words that are both pertinent and internally correlated.
Specifically, we extract feature attributions from trained multilingual style LMs and aggregate them into the parallel lexical categories from MLC. This enables interpretable common language for com- parison. We detail aggregation with token-level Shapley values (Lundberg and Lee, 2017), but any additive feature attribution method can be used.
We first filter out rare words (i.e. any words below given usage frequency). This addresses issues where machine translation results in words not commonly used in day-to-day conversation.
Token-to grouping. Given word con- sisting of tokens (U1, v 2k] be the corresponding teken-level Shapley values. Equation (1) first aggregates token- level Shapley values into word-level importance scores.
[m] 22, ... xx,, RK), let
Next, we ensure that the words in each lexical category reflect a given style similarly in the target language. Style is highly influenced by culture word that might indicate rudeness in English (e.g. "stubborn", "bossy", etc.) may not necessarily do so in other languages. To remove uncorrelated words, we first apply our lexica on any large cor- pus, and use a pre-trained LM to calculate style score (e.g. politeness level) for each utterance in the corpus. Then, for each word w within a lexi-
Imp(wr)
Uk
Category-level importances. Next, we derive category-level importance scores for each lexical category by aggregating local word-level impor- tance scores. For each category C, we iterate over all utterances and sum the werd-level importance

---

scores for all words in C. Finally, we divide by N. or the total number of times word in C appears in the dataset. This process is detailed in Equation (2), Let wij denote the jth word in the ith utterance.
Each utterance is 2-3 sentences long, and randomly sampled from all 41,000 scraped talk pages in each language. Note that we only scrape talk pages of articles that exist in all four languages, ensuring a similar distribution of topics across our dataset.
Imp(C)=
I = N 5
Σ
[m.ccylmp(uu)
(2)
To label the dataset, to source annotators. Respondents are required be native speakers of the language they annotate, as well as indicate that it is their primary spoken lan- guage day-to-day. We include attention and fluency checks to ensure our annotations are of high quality. Annotators use 5-point scale for labeling: "Rude", "Slightly Rude", "Neutral", "Slightly Polite", and "Polite", with three annotators labeling each utter- ance. We observe average annotator agreement (Fleiss' kappa) of 0.186 across languages.
Annotation.
Prolific
This gives us an importance score for each lexi- cal category across languages. Now, we can easily compare how important certain categories are at lin- guistically reflecting a style in different languages.
3 A Holistic Politeness Dataset
Politeness, like all linguistic styles, is a complex, nuanced construction. In order to compare how politeness differs across languages, it is necessary to analyze the full distribution of conversational data, without any oversimplifying assumptions. We follow the process of the Stanford Politeness Cor- pus Cannescu-Nicuescuu Mizil et al., 2013) and TyDiP(Srinivasan and Choi, 2022) when creating and evaluating our politeness dataset, but with three key differences:
Given the highly subjective nature of politeness, we expect to see a score in this range. Addition- ally, we convert the Stanford Politeness Corpus DannesuuNicullseu-Miii et al., 2013) to a 5 5-point scale and observe a Fleiss' kappa of 0.153, indicat- ing our agreement aligns with past work.
A key distinction between our dataset and the Stanford Politeness Corpus/TYDiPis that we do not normalize the scores of each annotator to be cen- tered at neutral. Levels of politeness vary culturally (Leech, 2007: Pishghadam and Navari, 2012); we therefore do not make any assumptions about the politeness level of an average utterance.
We include all dialogue acts to replicate the real distribution of conversational data. (Both previous datasets only include questions.)
We include all annotated data in model train- ing and evaluation, as we want to compare language along the full politeness spectrum. (Both previous evaluations only consider the highest and lowest 25% of politeness utter- ances, eliminating all neutral utterances.)
Figure A6 provides a visualization of score distri- butions in our final annotated dataset. We observe that the average utterance from English, Spanish, and Chinese closest neutral, while the average Japanese utterance is closer to slightly polite.
We treat politeness as a regression task rather than a binary classification task. (Both previ- ous evaluations classify an utterance as "po- lite" or "impolite", destroying the nuance be- tween slight and strong politeness.)
Comparing Politeness via PoliteLex
PoliteLex (Li ct al., 2020) consists of curated Iex- ica to measure politeness in English and Chinese, based on the twenty politeness strategies intro- duced by Danescu-Nicalescu-Miiii et al. (2)13). We use MLC to expand PoliteLex to cover all four of our languages.
3.1 Data Collection & Annotation Overview We provide high-level overview of our dataset construction and annotation process, and give spe- cific details in Appendix A.
Chinese PoliteLex has some additional lexical categories, such as taboo words and honorifics, to account for cultural and linguistic norms in Chinese that do not have English equivalent. As polite- ness is expressed more similarly within Eastern and Western cultures than between them (Spencer- Oatey and Kádár, 2016), we use English PoliteLex as the seed for Spanish and Chinese PoliteLex as
Dataset. Our dataset contains 22,800 conver- sation snippets, or utterances, scraped from Wikipedia Talk Pages* in English, Spanish, Chi- nese, and Japanese (5,700 utterances per language).
'Wikipedia Talk Pages are used by editors of the platform to communicate about proposed edits to articles. This is the same domain as the Stanford Politeness Corpus and TYDIP.

---

PoliteLex: Average Category Importance & Category Frequency
0.20
Gratitude-
0.128 (0.88 (0.8%)
0.078 (1.1%)
0.192 (5))%)
0.073 (0.7%)
Greeting
0.006 (1.4%)
0.030 (1.7%)
0.119 (18.9%)
0.035 (0.3%)
0.15
Apologetic-
0.148 (0.5%)
0.209 (0.4%)
0.154 (1.6%)
0.065 (0.5%)
0.10
Please
0.013 (2.4%)
0.014
(1.0%)
0.208 (2.0%)
-0.010 (3. 1%)
0.05
Emergeecyy
-0.003 (0.3%)
-0.024 (0.2%)
0.111 (0.5%)
-0.063 (0. 1%)
0.00
Ingroup Identity
-0.049 (0.1%)
-0.005 (0.5%)
0.163 (0.8%)
-0.049 (0. (0.7%)
-0.05
Direct "You"-
0.003 (9.5%)
-0.033 (0.8%)
0.082 (0.2%)
0.017 (8.0%)
-0.10
Taboo-
NA
NA
0.026 (0.7%)
-0.018 (4.0%)
-0.15
Honorific "You"
0.145 (7.0%)
0.076 (5.0%)
-0.20
English
Spanish
Japanese
Chinese
Figure 4: PoliteLex category-level importances across languages. Each importance indicates the category's s average numerical contribution to an utterance's 's politeness label, where -2 = Rude, O = Neutral, and 2 = Polite. We additionally show the frequency each category (% total sentences that contain word from the category.)
the seed for Japanese, to create set of four lexica with parallel, comparable categories.
% of Dataset Covered
Language
Lexica
When purifying our expanded lexica in Spanish and Japanese, we use the full set 41,000 scraped talk pages to calculate internal correlation and re- move uncorrelated words from each category.
English
PoliteLex PoliteLex
98.0%
Chinese
83.7%
MT
94.4%
Spanish
Next, we fine-tune XoM-RRaERTa models (Conneau al., 2020) on our holistic politeness dataset (sce Appendix A.4 for training details) and use the SHAP library (Lundberg and Lee, 2017) to extract Shapley values for each utterance. Finally, we apply Feature Set Aggregation to calculate im- portance scores for each PoliteLex category.
MLC
96.9% %
MT
57.0%
Japanese
MLC
90.2% %
Table 1: Lexical coverage of our holistic politeness dataset. We define dataset coverage as the percent of utterances containing word in at least one lexical cate- gory. MLC produces lexica with better coverage than MT (machine translation).
Dataset coverage. Table analyzes our gener- ated politeness lexica in Spanish and Japanese. We measure dataset coverage for each language an utterance is "covered" if it contains at least one word in lexical category, and we define dataset coverage the percent of covered utterances. In both cases, the lexica generated by MLC has bet- ter coverage than 1:1 machine translation using Google Translate.
Shapley values highest for words within those categories. Similarly, categories with negative im- portance scores indicate rudeness.
For certain categories, ity across languages. Apologetic expressions (c.g. "I m sorry", "my bad") and expressions of grat- itude (e.g. "thank you", "I appreciate it") tend to universally indicate politeness. Interest- ingly, we see Japanese speakers using expressions of gratitude and apology with the highest frequency across languages.
strong similar-
Results. Figure 4 details the resulting category- level importances from select PoliteLex categories, with the frequency words each category given in parentheses. Categories with positive impor- tance are indicators of politeness, as the extracted
We also notice interesting differences. The word

---

Dialogue Acts: Average Act Importance & Act Frequency
Conventional Opening
0.663 (0.5%)
0.583 (1.2%)
0.344 (1.2%)
0.782 (0.2%)
1.0
Declarative Statement
0.080 (59.0%)
0.066
(56.3%)
0.136
(53.5%)
-0.008 (53.9%)
0.5
Opinion Statement
-0.052
(21.3%)
-0.062
(25.2%)
0.076
(19.0%)
-0.074 (23.5%)
Forward Function: Thanking
0.912 (0.4%)
0.706
(0.5%)
1.090 (1.9%)
1.328 (0.9%)
Yes/No Question
0.042 (4.6%)
-0.157 (3.4%)
-0.054 (4.4%)
-0.307 (5.2%)
Conventional Closing
-1.0
0.602 (0.7%)
0.437 (0.7%)
0.909 (3.9%)
0.948 (0.8%)
English
Spanish
Japanese
Chinese
Figure 5: Dialogue act importances across languages. Each dialogue act importance score indicates the act's average numerical contribution utterance's politeness label, where -2 Rude, 0 Neutral, and 2 Polite. We additionally show the frequency of each dialogue act (% of total sentences classified as that act.)
"please" in English, Spanish, and Chinese does not indicate politeness, despite being used with sim- ilar frequency in all four languages. For example, the following English and Spanish utterances
acts), additionally apply the second part of our framework to dialogue act groupings as categories, In the previous section, we compared how linguis- tic expressions politeness differ across languages. In this section, we seek to compare how the linguis- tic form of politeness differs as well.
"This has been debated to death; please read the archives."
To classify the dialogue acts of each utterance, machine translate our dataset English. We then run trained English dialogue act classifier provided by Omitaomu et al. (2022) on the trans- lated dataset and label each sentence of an utterance with one of 42 dialogue acts (Stolcke et al.. 2000). Table 2 shows examples for select dialogue acts. are sentence-level, modify Equation (1) to aggregate over all tokens in given sentence, as opposed to all tokens a given word. Finally, we treat each dialogue act as a unique cate- gory (analogous to a lexical category) and use our feature set aggregation method to map sentence- level SHAP values to their corresponding dialogue acts across our four languages.
"Antes de cuestionar si lo que digo es verdad, por favor trate de corroborarlo usted mi smo." ("Refore you question whether what I say is true, please try to verify it yourself.")
are both labeled by annotators to be quite rude, despite containing the word "please". In Japanese however, "please" strongly indicates politeness. the findings of Li al. (2020), expressions of in-group identity (e.g. "bro", "mate") are indicators of rudeness En- glish and Chinese, but indicators of politeness in Japanese. This may be because these terms are uncomfortably familiar, and so taken as rude, or due to sarcastic uses of these terms in English and Chinese. This phenomenon does not appear to be paralleled in Japanese, terms of in-group identity are very polite. We give examples of top words in each Politelex category in Table A6 and our full set of results for all categories in Figure A7.
As dialogue
Additionally, contrary
Figure shows the average importance of cach dialogue act, with the frequency of cach dialogue act given in parentheses. Once again, we observe some similarities across languages: con- ventional openings, conventional closings, and sen- tences of thanks are strong indicators of politeness across languages.
Results.
5 Comparing Politeness via Dialogue Acts
Given our dataset is the first politeness dataset to include multiple types of sentences (i.e. dialogue
However, statements appear to have differing roles across languages. Declarative statements

---

Dialogue Act
Example
Conventional Opening
"Hi all, this article in desperate need of attention."
Declarative Statement
"As it stands, we do not know the place or date of Shapiro's 's birth."
Opinion Statement
"There is no need to be combative and attack over typos."
F Thanking
"Looks good, thanks for the help!"
Yes/No Question
"Is the second picture an actual representation of the BCG vaccine?"
Conventional Closing
"Happy to discuss anything further."
Table 2: Examples for selected dialogue acts from our holistic politeness dataset.
mostly lean polite across languages, while state- ments expressing an opinion lean slightly rude in English, Spanish, and Chinese. Surprisingly. yes/no questions only indicate politeness in En- glish, and are viewed as mildly rude in all other languages, particularly Chinese. Consider the fol- lowing yes/no questions in English and Chinese:
from XLM-RoBERTa, we notice stability in the direction of importance score (i.e. positive, nega- tive, and neutral lexical categories/dialog acts are stable across both LMs). Interestingly, we observe differences in the magnitude of importance score (e.g. Llama-2-7b sees the "Greeting" category in Chinese be larger indicator of politeness than XLM-RoBERTa does).
"To be pedantic, are we sure that was born in Milton, West Dunbartonshire?"
Effect language-speccfic pretraining, To in- vestigate the role language-specific pretraining, we fully fine-tune four RoBERTa models trained on only their respective languages. Similar to Llama- 2-7b, we observe high similarity in the direction of the importance score compared to XLM- LLM-RoBERTa. We also notice much more similarity in the mag- nitude of importance scores. This may be due inherent similarities between all RoBERTa mod- els (parameter size, training methods, training data, etc.), which do not exist between XLM-RoBERTa and Llama-2-7b.
"RGAー- N AB 4 &
A ("In the last application section, is necessary to add so many pictures?"')
To an English speaker, both the English sentence and the Chinese translation appear to be similar levels of politeness. However, American annota- tors label the English question as "Neutral" while Chinese annotators label the Chinese question as "Slightly Rude," highlighting the ways in which cultural norms influence perceptions of politeness. Interestingly, we do not observe any major dif- ferences in the frequency of dialogue acts across languages; conversations in all four languages ap- pear to have similar distribution of dialogue acts, though the average politeness of each dialogue act often varies based on language. Results for all dialogue acts are shown in Figure A8.
Our ablation analysis reveals that different lan- guage models pay attention to the same markers when learning to predict politeness, but learn to weigh these markers in different ways. Overall, notice stability in which lexical categories and dialog acts indicate politeness vs. rudeness. This suggests that our groupings for feature set aggre- gation are both stable and successful. Section contains additional details.
6 Ablation Analysis
The PoliteLex category-level importances in Fig- ure 4 and dialogue act importances in Figure 5 are dependent on the Shapley values extracted from fine-tuned XLBMRoBERTa models. In this section, we analyze the effect of using alternate models and training paradigms.
7 Related Work
Multilingual style. Previous work on multilin- gual style predominantly focuses on training LMs to perform cross-lingus and multilingual style clas- sification and style transfer. Key styles studied in- clude formality (Briakou al., 2021; Krishna al., 2022: Rippeth et al., 2022) and emotion (Öhman et al., 2018; Lamprinidis et al., 2021; Öhman et al., 2020), with another body of work focusing on style-
Effect of model size. To investigate the role of LM size and architecture, we fine-tune Llama-2-7b (Touvron et al., 2023) to analyze politeness. Com- paring the results from Llama-2-7h to the results

---

aware multilingual generation with any subset of chosen styles (Niu et al., 2018; Garcia et al., 2021).
At higher level, our framework provides a gen- eral methodology for explaining LMs; using fea- ture attributions such as Shapley values to provide explanations in terms of human-interpretable cate- gories (e.g. lexica and dialogue acts) gives expla- nations that are both grounded in the model and useful to humans.
Explaining style. One line of work builds on ex- isting techniques (Lundberg and Lee, 2017; Ribeiro et al., 2016) to explain style within single LM (Aubakirova and Bansal, 2016; Wang et al., 2021). Another line of work interprets style LMs by com- paring learned features to those humans would con- sider important (Hayati et al., 2021), mapping fea- ture attributions to topics (Havaldar et al., 2023b), and training models to output relevant features alongside their predictions (Hayati et al., 2023).
Limitations
When detailing our framework, we used FastText for Multilingual Lexica Generation and Partition SHAP (Lundberg and Lee, 2017) for Feature Set Aggregation. Our results are dependent on these two choices. Because FastText only supports word embeddings, we could only apply our MLC frame- work to words in the lexica. A contextual embed- ding could be used to additionally expand phrases, Additionally, we did not use native speakers to fur- ther purify our final lexica, is the case in most past work curating multilingual lexica.
Politeness. Danesau-Niculesu-MMall (2013) presents one of the earliest quantitative analyses of linguistic politeness, with Srinivasan and Choi (2022) following in a multilingual setting. Other computational work focusing on politeness uses LMs to generate or modify text with specified politeness level (Niu and Bansal, 2018; Fu et al., 2020; Mishra et al., 2022; Silva al., 2022).
al.
Individuals view politeness differently even within cultures, and as such, it a highly sub- jective task. The subjectivity of politeness has been studied previously (Leech, 2007; Pishghadam and Navari, 2012; Spencer-Oatey and Kádár, 2016; Pla- cencia and Farcia-Fernandez, 2017) and as a result, there is no "correct" label for utterance; this subjectivity contributes heavily to our annotator agreement scores and fine-tuned LM accuracy.
Previous work focused on multilingual style has little emphasis on investigating how style differs amongst languages. Additionally, most work on explaining style LMs English-scoped, and thus, developed methods do not easily allow for cultural comparison. We are the first to present a method to compare styles across languages and provide quantitative, human-interpretable insights into how communication differs globally.
We also draw conclusions about politeness var- ious cultures from a single context. We only study Wikipedia Talk Page data, which reflects polite- ness in workplace communication, but expressions of politeness are likely to differ in other settings: non- work conversations, communication between friends or family, social media, etc.
8 Conclusion
this work, we present a framework to extract the knowledge implicit in trained multilingual LMs. This knowledge enables comparison of styles across languages via a human- iumaa-interprettbbe com- mon language for explanation. We also provide the first holistic multilingual politeness dataset, which we hope will encourage future research exploring cultural differences in style.
Ethies Statement
When comparing styles across languages in this study, we treat language and culture as monolithic. However, we recognize different communities of people who speak the same language will use it differently. For example, politeness is likely ex- pressed differently in Spain vs. Mexico, even though they are both Spanish-speaking countries. recognize that it is highly contextual calling stranger "bro" can be perceived an insult, while calling a close friend "bro" can be viewed as an expression of bonding and closeness. Age, gender, and other demograph- ics also play an important role in perceived po-
Our framework provides insights into how style is expressed differently across languages. These insights can improve multilingual LMs - un- derstanding how and why an LM generation is not stylistically appropriate can inform culturally- adaptable models. These insights can also help people learning a second language become more aware of the language's culturally-infe tic nuances.
In studying politeness,
stylis-

---

liteness in a conversation. Politeness is a deeply interpersonal act, but it is decontextualized when studied computationally; NLP studies of politeness destroy key components of it.
Cristian Daneseu-Niculeseu-Mzzz Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application social factors. arXiv preprint arXiv:
We additionally recognize the implications, both good and bad, of working towards culturally- appropriate stylistic language generation. Our method can be used to inform more culturally at- tuned LMs multilingual polite agents are impor- tant for beneficial uses like teaching and therapy, but these polite agents could potentially also be used for more effective manipulation or misinfor- mation tactics.
Javier De la Rosa, Eduardo G. Ponferrada, Manu Romero, Paulo Villegas. Pablo González de Prado Salas, and María Grandury. 2022.
Bertin: Elfficient pre-training of spanish language model using perplexity sampling. Procesamiento del Lenguaje Natural, 68(0):13-23.
Fatima-zahn El-Alami. Said Ouatik EI Alaoui, and Noureddine En Nahnahi. 2022. A multilingual of- fensive language detection method based on transfer learning from transformer fine-tuning model. Jour- nal of King Saud University-omppute and Informa- tion Sciences, 34(8):6048-60)
References
Malika Aubakirova and Mohit Bansal. 2016. Interpret- ing neural networks to improve politeness compre- hension. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages
Asum Ersoy, Gerson Vizearra, Tasmiah Tahsin Mayee- sha, and Benjamin Muller. 2023. In what languages are generative language models the most formal? an- alyzing formality distribution across languages.
Austin, Texas. Association for Computational I Linguistics.
Liye Fu, Susan Fussell, and Cristian Danescu-Niculesee Mizil. 2020. Facilitating the communication of politeness through fine-grained paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 5127-5140, Online. Association for Computa- tional Linguistics.
Anshul Bawa, Pranav Khadpe, Pratik Joshi, Kalika Bali, and Monojit Choudhury. 2020. Do multilingual users prefer chat-bots that code-mix? let's nudge and find out! Proc. ACM Hum.-Comper Interact., 4(CSCW1).
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
Xavier Garcia, Noah Constant, Mandy Guo, and Orhan Firat. 2021. Towards universality in multilingual text rewriting. arXiv preprint arXiv:2107.144 14749.
subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Edward T Hall. 1976. Beyond culture. Anchor.
Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021. Olá, bonjour, salve! XFORMAL: A bench- mark for multilingual formality style transfer. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3199-3216, Online. Association for Computa- tional Linguistics.
Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023a. Multilingual language models are not multicultural: A case study in emotion. In Pro- ceedings of the 13th Workshop Computational Approaches to Subjectiviry, Sentiment, & Social Me- dia Analysis, pages 202-214.
Tianyu Cho and Kei Sawada. 2021. Pre-learning model for japanese natural language processing. Japanese Society for Artificial Intelligence Research Group Material Language/Speech Understanding and Dia- logue Processing, 93:169-170.
Shreya Havaldar, Adam Stein, Eric Wong, and Lyle Ungar. 2023b. Topex: Topic-based explanations for model comparison.
Shirley Anugrah Hayati, Dongyeop Kang, and Lyle Ungar. 2021. Does bert learn as humans perceive? understanding linguistic styles through lexica.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- cooss-lingual representation learning scale.
Shirley Anugrah Hayati. Kyumin Park. Dheeraj Ra- jagopal, Lyle Ungar, and Dongyeop Kang. 2023. Stylex: Explaining style using human lexical annota- tions.
Yiming Cui, Wamxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre- traierd models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668, Online. Association for Computa- tional Linguistics.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello. Laura Cabello Pi- queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders

---

Søgaard. 2022. Challenges and strategies in cross- cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6997-7013, Dublin, Ireland. Association for Computational Lin- guistics.
Mingyang Li, Louis Hickman, Louis Tay, Lyle Ungar, and Sharath Chandra Guntuku. 2020. Studying po- liteness across cultures using english twitter and man- darin weibo. Proc. ACM Hum.-Com -ompput. Interact., 4(CSCW2).
Jindřích Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained mul- tilingual representations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020. pages 1663-1674, Online. Association for Computa- tional Linguisties.
Geert Hofstede. 2005. Cultures and organizations: Soft- ware of the mind, volume 2.
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588-602, Online. Association for Computational I Linguisties.
Haokun Liu, Derek Tam, Mohammed Mugeeth. Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-effene efffiittreeficientt fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, pages 1950-1965. Curran Associates, Inc.
Dongyeop Kang and Eduard Hovy. 2021. Style is not a single variable: Case studies for cross-stylistic lan- guage understanding, In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 2376-2387.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs//997.11192
Naresh Khatri. 2009. Consequences of power distance orientation in orgamisations. Vision, 13(1):1-9.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in neural information processing systems, 30.
Donghoon Kim, Yigang Pan, and Heung Soo Park. 1998. High-versus low-context culture: A comparison of chinese, korean, and american cultures. Psychology & Marketing, 156))5077-52.
Sourab Mangrulkar, Sylvain Gugger, Lysandre De- but, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter- efficient fine-tuning methods. https://gi thub. com//huggingface/pefff
Kalpesh Krishna, Deepak Nathani, Xavier Garcia, Bidisha Samanta, and Partha Talukdar. 2022. Few- shot controllable style transfer for low-resource mul- tilingual settings. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7439-7468.
Kshitij Mishra, Mauajama Firdaus, and Asif Ekbal. 2022. Please be polite: Towards building a politeness adaptive dialogue system for goal-oriented conversa- tions. Neurocomputing. 494:242-254.
Sotiris Lamprinidis, Federico Bianchi, Daniel Hardt, and Dirk Hovy. 2021. Universal joy a data set and results for classifying emotions across languages. In Proceedings of the Eleventh Workshop on Compu- tational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 62-75.
Angela Moorjani and Thomas T Field. 1988. Semiotic and sociolinguistic paths to understanding culture. In Toward a new integration of language and culture. Reports of the Northeatt Conference on the Teaching of Foreign Languages, pages 45.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Un- derstanding the cross- mansullnggul ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 2214-2231, Online. Association for Computational Linguistics.
Martin Laville, Mérième Bouhandi, Emmanuel Morin, and Philippe Langlais. 2020. Word representations, seed lexicons, mapping procedures, and reference lists: What matters in bilingual lexicon induction from comparable corpora? In Advances in Artificial Intelliience: 33rd Canadian Conference on Artifi- cial Intelligence, Canadian Al 2020, Ottawa, ON, Canada, May 13-15, 2020, Proceedings 33, pages 349-355. Springer.
Tong Niu and Mohit Bansal. 2018 Polite Dialogue Generation Without Parallel Data. Transactions of the Association for Computational Linguistics, 6:373 389.
Geoffrey Leech. 2007. Politeness: is there an east-west divide?
Xing Niu, Sudha Rao, and Marine Carpuat. 2018 Multi-task neural models for translating between styles within and across languages. arXiv preprint 1rXiv:1806.04533 04357.
Darrin R Lehman, Chi-yue Chiu, and Mark Schaller. 2004. Psychology and culture. Annu, Rev. Psychol. 55:689-714.

---

Emily Öhman, Kaisla Kajava, Jörg Tiedemann, and Timo Honkela. 2018. Creating a dataset for
Anirudh Srinivasan and Eunsol Choi. 2022. Ty- dip: A dataset for politeness classification nine typologically diverse languages. arXiv preprint urXiv:2211.66490
multilingual fine-grained emotion-detection using gamification- annotation. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 24-30.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Eliza- beth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
Emily Öhman, Mare Pamies, Kaisla Kajava, and Jörg Tiedemann. 2020. Xed: A multilingual dataset for sentiment analysis and emotion detection. arXiv preprint arXiv:2011.01612
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. Interna- tional conference on machine learning, pages 33 191 3328. PMLR.
Damilola Omitaomu, Shabnam Tafreshi, Tingting Liu, Sven Buechel, Chris Callison- Burch, Johannes Eich- staedt, Lyle Ungar, and João Sedoc. 2022. Empathic conversations: multi-level dataset of contextual- ized conversations.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1333
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- the 57th Annual Meeting the Association for
ings the Computational Linguistics, pages 4996-5001, Flo- rence, Italy, Association for Computational Linguis- tics.
Qinqin Wang, Hambing Yan, and Zhihui Han. 2021. Ex- plainable apt attribution for malware using alp tech- niques. 2021 IEEE 21 International Conference on Software Quality, Reliability and Security (QRS), pages 70 80.
Reza Pishghadam and Safoora Navari. 2012. A study into politeness strategies and politeness markers in advertisements as persuasive tools. Mediterranean Journal of Social Sciences, 3(2)1611-711
Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge. and Pengke Chen. 2022. Mdia: benchmark for multilingual dialogue generation in 46 languages. arXiv preprint arXiv:2208. 13078.
María Elena Placencia and Carmen 2017. Research on politeness in the Spanish- speaking world. Routledge.
Flor Miriam Plaza-del Arco, Carlo Strapparava, AI- fonso Urena Lopez, and M Teresa Valdiva.. 2020. Emoevent: multilingual emotion corpus based on different events. In Proceedings of the 12th Language Resourcey and Evaluation Confer- ence, pages 1492-1498.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1355- 1144.
Elijah Rippeth, Sweta Agrawal, and Marine Carpuat. 2022. Controlling translation formality using pre- trained multilingual language models. arXiv preprint 2756v22205006644.
Diogo Silva, David Semedo, and João Magalhães. 2022. Polite task-oriented dialog agents: To generate or to rewrite? In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis, pages 304-314.
Helen Spencer-Oatey and Dániel 7 Kádár, 2016. The bases of (im) politeness evaluations: Culture, the moral order and the east-west debate. East Asian Pragmatics, 11):73--106

---

A Dataset Construction and Annotation:
Politeness Distribution per Language
Additional Details
Mean
Annotated Poltteness Score
Mestlan
A.1 Data Collection
2.0
We scrape Wikipedia Editor Talk Pages in English, Spanish, Japanese, and Chinese using Wikimedia and keep the Talk Pages discussing articles that are present all four languages. Next, we extract utter- ances, or two-three sentence snippets from a single user, such that the length of single utterance does not exceed 512 tokens. We anonymize utterances to remove names, replacing mentions with "@ @user" and hyperlinks, replacing hyperlinks with "<url>". We then randomly sample 5,700 utterances from each language.
-0.5
-1.0
-1.5
-2.0
English
Spanish Japanese Chinese
Figure A6: Distribution of annotated politeness scores per language.
We select 5,700 as the size of the dataset for each language based on a power calculation to ob- serve a discernible difference in politeness level between four languages. We use the results from a pre-trained English-only politeness classifier taken from Hayati et al. (2021) to get the parameters for the power calculation (a- 0.05, 0.2.)
Learning Rate
Test RMSE
Language
Test
English
le 5
.667
.662
Spanish
1e 5
.738
.642
Japanese
le 5
.659
.650
Chinese
5e 5 .612
.646
A.2 Annotation
Table A3: Details model training. We fine-tune four XLM-RoBERTa models on our holistic politeness dataset. We show Test RMSE (root mean squared error) and product-moment correlation for the final model.
To annotate our dataset, we up surveys hosted using Prolific. We source annotators who are na- tive speakers of the target language and additionally indicate that the target language is their primary spoken language day-to-day. We have each an- notator label 100 utterances on five-point scale: "Rude", "Slightly Rude", "Neutral"" "Slightly Po- lite", and "Polite". Every 25 utterances, we give attention check to each annotator. The attention check is in the form "To ensure you are still pay- ing attention, please select [indicated option]" and translated the target language. If the annotator fails any one of the four total attention checks, we discard all of their annotations. We additionally use this as fluency check, as the attention check is fully written in the target language. We set up the survey such that each utterance is labeled by three different annotators.
Table A4 shows agreement for each language. We observe slightly higher agreement in all lan- guages than that of the Stanford Politeness Corpus (Fleiss' kappa = et al., 2013), indicating the highly subjective nature of politeness.
Next, we convert the annotations to numeric scale: "Rude"= 22 "Slightly Rude"= -1, "Polite"= 2. Finally, age numeric rating its final label.
each utterance's 's aver-
A.4 Model Fine- Tuning
To create LMs capable of detecting politeness multiple languages, we fine-tune RoBERTa models on our holistic politeness dataset.
Each annotator is paid $12 an hour, and each survey took, on average, 20 minutes complete. We do not notice any differences in survey time between languages.
We train four XLM-RoBERTa (Conneau et al., 2020) models, one for each language. For training, we randomly split each language's utterances into 80/10/10 train/validation/te split. We use the Huggingface Trainer library fine-tune our mod- els. All our models are trained for 50 epochs, and we select the epoch with the lowest validation loss as our final model. The results for the best itcration of each model can be found in Table A3.
A.3 Data Processing
Upon the completion of all four Prolific studies, we calculate annotator agreement using Fleiss' Kappa.
'https://dumps. wikimedia. wrwpaackpobbbbbanbnagee index. html

---

Language
Num Utterances
Fleiss's Kappa
Mean Politeness
Standard Deviation
English
5700
0.186
0.035
0.88
Spanish
5700
0.188
0.053
0.93
Japanese 5700
0.162
0.332
0,82
Chinese
5700
0.206
-0.123
0.80
Table A4; Dataset Statistics; annotator agreement and details on annotation distribution.
Note that choose train separate models for each language (as opposed single one for all languages) as we want our trained models to encode how politeness is expressed in each target language uniquely, without being influenced by utterances from other languages.
Dialogue acts. Figure A8 shows act-level importance scores across all dialogue acts, or types of sentences. We only show and compare dialogue acts that appear over ten times in each language, to ensure we have enough data for a meaningful comparison.
B Comparing Politeness Across Languages: Full Results
Interesting findings include:
Rhetorical questions are seen as rude in En- glish, Spanish, and Chinese, but closer to neu- tral in Japanese.
PoliteLex Figure A7 shows category-level
importance scores across all PoliteLex categories. Note that "NA" indicates that the category is not present that language. We use English PoliteLex as the seed for Spanish Politelex, and Chinese PoliteLex as the seed for Japanese PoliteLex; the categories present the expanded lexica parallel those present in the seed lexica.
"Wh-" questions (e.g. "Why did you do that?" or "Where are you going?") are seen as uni- versally rude.
Apology statements seen polite in all languages, but most polite in Chinese, despite the fact that Chinese speakers use apology statements with the lowest frequency.
Interesting findings include:
Unlike PoliteLex categories, certain dialogue acts do reflect rudeness in Japanese, namely "Wh-" questions and statements that summarize refor- mulate as a response.
Deference is seen as polite in English, Span- ish, and Japanese, but is seen slightly rude in Chinese.
Contrary to the findings of Danescu- Niculescu-Mizil et al. (2013), we observe no concrete difference in politeness between utterances that use the subjunctive form of a word ("could", "would"") vs. the indicative form ("can", "will").
C Ablation Analysis: Additional Details
To fine-tune Llama-2-7b with the computational resources available to we use parameter efficient fine-tuning (Liu et al., 2022). Specifically, we use the PEFT library (Mangrulkar al., 2022) to fine- tune subset of the weights. We use a learning rate of 1e 5 and default LORA parameters. We achieve similar performance to the metrics shown in Table A3.
Indirectness ("by the way"") is seen polite in English and Japanese, but rude in Spanish and Chinese.
To fine-tune the monolingual models (shown in Table A5), we use an identical set-up as Table A3, as the architecture of monolingual RoBERTa mod- cls and multilingual XLM-RoBERTa models is identical. Again, we observe similar performance to the metries shown in Table A3.
An overall analysis of our results reveals that all Japanese Politeleex strategies indicate either neu- trality politeness. This is not the case in other languages; certain strategies do indicate rudeness. These findings suggest that politeness expressed in unique way in Japan, and perhaps rudeness is more subtle and cannot be fully captured by the lexical categories used in Chinese PoliteLex.
We provide the resulting heatmaps in our repos- itory: multiiingual_pollie itness

---

Setting
Model Name
Monolingual English
roberta-base (Liu ct al., 2019)
Monolingual Spanish
bertiin-roberta-bbas--pnii (De la Rosa et al., 2022)
Monolingual Chinese
chinese-robbrrt--wwwee (Cui et al., 2020)
Monolingual Japanese
japanese-robbrrta-bbae (Cho and Sawada, 2021)
Multilingual
Llama-2-7b (Touvron et al., 2023)
Table A5: LMs used in our ablation studies. We experiment with different multilingual model to investigate the role of model size and architecture, well four separate monolingual models investigate the role of language-specific pretraining.
PoliteLex Category
Three Most Frequent Words (English)
Gratitude
thanks, thank you, i appreciate good, great, interesting hello, hi, hey sorry, apologize, apologies please, pls, plse please by the way, btw what, when, how but, and, so could you, would you you, will you i, my, mine we, us, our i, my, me you, your, u you, your should, think, seems actually, really, in fact right now, immediately, at once mate, homie, dude great, excellent, super sure, must, certainly together
Deference
Greeting
Apologetic
Please
Please Start
Indirect (btw)
Direct Question
Direct Start
Subjunctive
Indicative
1st Person Start
1st Person Plural
1st Person
2nd Person
2nd Person Start
Hedges
Factuality
Emergency
Ingroup Identity
Praise
Promise
Together
Direct "You"
you, u like, well, good issue, wrong, problem
Positive
Negative
Table A6: For each PoliteLex category, we show the three most frequently occurring words in our holistic politeness dataset. Note that this table only shows words in English PoliteLex; SpanishhCCineese/Jpppneee PoliteLex contains different words in each category.

---

PoliteLex: Average Category Importance & Category Frequency
Gratitude
0.128 (0.8%)
0.078 (1.1%)
0.192 (15.7%)
0.073 (0.7%)
0.20
Deference
0.040 (2.7%)
0.024 (6.3%)
0.147
(7.4%)
-0.032 (3.6%)
Greeting
0.006 (1.4%)
0.030 (1.7%)
0.119
(18.9%)
0.035 (0.3%)
Apologetic-
0.148 (0.5%)
0.209 (0.4%)
0.154 (1.6%)
0.065 (0.5%)
0.15
Please-
0.013 (2.4%)
0.014 (1.0%)
0.208 (2.0%)
-0.010 (3.1%)
Please Start-
-0.032
(1.5%)
-0.028 (0.4%)
NA
NA
Indirect (btw)-
0.053 (0.1%)
-0.028
(0.3%)
0.094 (20.6%)
-0.054 (0.1%)
Direct Question -0.006
(14.7%)
-0.040 (6.6%)
0.091 (8.5%)
-0.049 (6.3%)
0.10
Direct Start- -0.005
(3.0%)
-0.020 (4.5%)
NA
NA
Subjunctive
0.106 (0.1%)
0.025 (1.5%)
NA
NA
Indicative
0.114 (0.1%)
0.000 (9.9%)
NA
NA
0.05
1st Person Start-
0.005 (16.6%)
0.002 (3.4%)
NA
1st Person Plural 0.016 (5.2%)
-0.021 (1.4%)
0.091 (0.0%)
-0.018 (0.6%)
1st Person 0.012 (41.0%)
-0.008
(16.6%)
0.124 (8.8%)
-0.041 (23.5%)
0.00
2nd Person 0.005 (12.0%)
-0.003 (9.7%)
0.066 (1.1%)
0.017 (8.0%)
2nd Person Start- -0.025 (1.5%)
-0.031
(0.3%)
NA
Hedges 0.003 (29.2%)
0.012
(41.5%)
0.098
(45.4%)
-0.026 (48.4%)
Factuality -0.036 (3.8%)
-0.000 (7.4%)
0.080 (0.7%)
-0.013 (0.6%)
-0.05
Emergency- -0.003 (0.3%)
-0.024 (0.2%)
0.111 (0.5%)
-0.063 (0.1 %)
Ingroup Identity -0.049 (0.1%)
-0.005 (0.5%)
0.163 (0.8%)
-0.049 (0.7%)
Praise
0.016 (0.8%)
0.011 (2.5%)
0.139 (9.5%)
-0.032 (3.6%)
-0.10
Promise
-0.002 (2.6%)
0.013 (5.5%)
0.134 (2.7%)
-0.026 (3.8%)
Together- 0.034 (0.3%)
-0.005 (0.1%)
0.077 (0.1%) %)
-0.043 (0.3%)
Direct "You".
0.003 (9.5%)
-0.033 (0.8%)
0.082 (0.2%)
0.017 (8.0%)
-0.15
Positive
0.017 (39.8%)
0.011
(39.4%)
NA
Negative
-0.036 (34.6%)
-0.033 (39.6%)
NA
NA
Taboo
NA
NA
0.026 (0.7%)
-0.018 (4.0%)
Honorific "You".
NA
NA
0.145 (7.0%)
0.076 (5.0%)
-020
English
Spanish
Japanese
Chinese
Figure A7: The full set of PoliteLex category-level importances across languages. Each importance score indicates the category's average numerical contribution an utterance's politeness label, where -2 Rude, 0 Neutral. and = Polite. We additionally show the frequency each category (% of total sentences contain a word from the category.)

---

Dialogue Acts: Average Act Importance & Act Frequency
Declarative Statement
0.080 (59.0%)
0.066
(56.3%)
0.136 (53.5%)
-0.008 (53.9%)
Yes/No
0.042 (4.6%)
-0.157 (3.4%)
-0.054
(4.4%)
-0.307 (5.2%)
Question
1.0
Conventional Closing
0.602 (0.7%)
0.437 (0.7%)
0.909 (3.9%)
0.948 (0.8%)
Acceptance-
0.206 (2.0%)
0.199
(1.7%)
0.227 (2.3%)
0.149 (0.9%)
Quoted material
-0.021 (0.7%)
-0.044
(1 (1.2%)
0.059 (2.0%)
-0.107 (3.3%)
0.5
Directive Action
0.095
(2.1%)
-0.043
(1 (1.2%)
0.204 (2.2%)
-0.016 (2.2%)
"Wh-" " Question
-0.334
(1.8%)
-0.398
(1.7%)
-0.138 (1.3%)
-0.441
(2.1 %)
Opinion Statement
-0.052
(21.3%)
-0.062
(25.2%)
0.076
(19.0%)
-0.074 (23.5%)
0.0
Open Question
-0.309 (0.5%)
-0.344
(0.5%)
0.163 (0.6%)
-0.408 (0.7%)
Declarative Yes/No Question
-0.121 (0.8%)
-0.158 (0.6%)
-0.048 (0.4%)
-0.267 (0.6%)
Conventional Opening
0. 0.663 (0.5%)
0.583
(1.2%)
0.344 (1.2%)
0.782 (0.2%)
-0.5
Forward-Function Apology
0.445 (0.4%)
0.285 (0.3%)
0.800 (1.9%)
1.068 (0.3%)
Completion
-0.093 (0.2%)
-0.146 (0.2%)
0.004 (0.5%)
-0.040 (0.3%)
Forward Function: Thanking
0.912
(0.4%)
0.706
(0.5%)
1.090 (1.9%)
1.328 (0.9%)
-1.0
Rhetorical Question
-0.315 (0.6%)
-0.375 (0.5%)
-0.065 (0.3%)
-0.514 (0.8%)
Summarize/
0.236 (0.3%)
0.091 (0.3%)
0.106 (0.4%)
0.023 (0.4%)
Reformulate
English
Spanish
Japanese
Chinese
Figure A8: The full set of dialogue act importances across languages. Each dialogue act importance score indicates the acts' average numerical contribution an utterance's politeness label, where Rude, 0 = Neutral, and 2 Polite. We additionally show the frequency of each dialogue act (% of total sentences classified as that act.)


Reviews:
[{"review_id": "b8BSFrJPTt", "reviewer": "Reviewer_QT67", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your valuable feedback and for your excitement in our work! We look forward to improving our paper based on your comments:\n\n&nbsp;\n\n### Annotator agreement concerns\nTo better evaluate our annotation quality, we calculate additional agreement metrics beyond Fleiss' kappa and compare them to widely used datasets in the style space. In addition to comparing against the Stanford Politeness Corpus, we also compare against the following:\n\n-  TyDIP (multilingual politeness) (Srinivasan and Choi 2022)\n-  EmoEvent (multilingual emotion) (Plaza del Arco et al. 2020)\n-  GoEmotions (English emotion) (Demszky et al. 2020)\n-  Formality Corpus (English formality) (Pavlick and Tetreault 2016)\n-  XFormal (multilingual formality) (Briakou et al. 2021)\n \n\n|Agreement Statistic|Range in Above Datasets|Ours|\n|--|--|--|\n|Intraclass Correlation|0.39-0.83|0.66|\n|Pearson Correlation|0.16-0.64|0.545|\n|Cohen's Kappa|0.09 - 0.55|0.22|\n\n*Overall, our dataset's annotator agreement is better than the average agreements of existing human-annotated datasets in the style space.* Additionally, the highest agreements are for emotions such as Love and Sadness, which are much clearer than politeness.\n\nYour comment raises an extremely important question -- How to improve annotation quality of subjective style tasks? However, given our dataset aligns with existing agreements, we feel answering this question is outside the scope of this work.\n \n &nbsp;\n\n### Improved coherence across sections\nBased on your suggestions, we have edited the paper as follows to have a better storyline across sections:\n\n- We emphasize the overall motivation at the start of every new section\n- We tie back to our research questions when discussing our results\n\n&nbsp;\n\nPlease let us know if you have any other questions or concerns, and we welcome additional feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting study. Politeness variation across languages is, in general, very useful. They can help us to fine-tune chatbot replies appropriately to different segments of society. \n\nThe methods presented in this paper are very simple yet effective. So, it was a pleasure for me to go through it. \n", "reasons_to_reject": "The inter-annotator agreement score/ Fleiss’ kappa is 0.153, i.e. very low. Even though they have similar scores they got for the Stanford dataset, it raises serious questions on annotation quality. I understand that politeness is subjective and it depends on the individual. But in this case, we may not get very high score, but a moderate score is expected. Currently, it comes under ‘slight agreement’. \n\n\nI urge authors to improve the coherence across sections by using a simple storyline. It will help the readers to understand it better. \n\n\nI have reservations about the paper title. It is more generic, while the study done in this paper is more on the politeness dimension. So, I urge the authors to rephrase it. \n\n\nI feel the contributions made in this paper are not enough for a long paper. I would recommend it for a short paper now. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies how the expression of politeness is different across four languages, (i) English, (ii) Spanish, (iii) Japanese and (iv) Chinese. Authors propose a two-step approach, where they (i) Create multilingual lexica using existing multilingual embedding(MLM) and (ii) Do feature set aggregation to consolidate into appropriate lexical categories.  They extensively describe the motivation and steps they followed for creating multilingual lexica (Considering initial seed words, expanding to other languages by MLM, removing un-correlated words etc.), and feature set aggregation (getting token-level shapley values, getting category-level importance scores etc.), which clears overall set-up. They proposed a dataset (‘Holistic Politeness dataset’ (HPD hereafter)), which is similar to the Stanford politeness dataset and TYDIP, with differences like (i) HPD includes all dialogue acts, (ii) includes samples for borderline politeness annotations and (iii) considering the task as a regression one, unlike others where it is considered as a classification task. The data points are taken from Wikipedia Talk pages which I believe is a formal conversation dataset, and hence politeness variation in formal conversation can be easily studied. \n\nTo compare politeness, they used an existing curated lexica PoliteLex, which measures politeness using 20 politeness strategies in English and Chinese. They use the English one as a seed to construct the equivalent for Spanish and the Chinese one to construct the equivalent one for Japanese. So, they have four parallel lexica with comparable categories. They have finetuned XLM-RoBERTa models on the HPD and use the SHAP library to get shapley values. Feature set aggregation is applied to get categorical importance scores. \n\nThere have shown several interesting insights, (i) conversational openings, closings and thanking sentences are important indicators of politeness across all languages, (ii) Opinionated sentences are considered as slightly rude in English, Spanish and Chinese, (iii) Yes/No type questions are considered as mildly rude across all language except English, etc.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NVm6FAPxJN", "reviewer": "Reviewer_hCmg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your valuable feedback and for your excitement in our work! We look forward to improving our paper based on your comments:\n\n&nbsp;\n\n### Discussion about potential bias\nWe have added the following paragraph to the limitations section:\n\n**Bias and Error Analysis** Though our framework is general and can be applied to any LM + style lexicon, the specific results discussed in this work are dependent on many design choices. We use English and Chinese PoliteLex as our seed lexica, any knowledge gaps in these lexica may propagate to our expanded lexica too. We additionally train Roberta-XLM models on a human-annotated politeness dataset. Though we do our best to ensure sufficient annotation quality and use a standard training pipeline, any biases or uncertainties in the dataset or the LM may influence our downstream style comparisons. At a higher level, the results of our framework are highly specific to the dataset and LM chosen, and thus, any generalizations about politeness at the cultural level must be made with caution.\n\n&nbsp;\n\n### Annotator agreement concerns\nTo better evaluate our annotation quality, we calculate additional agreement metrics beyond Fleiss' kappa and compare them to widely used datasets in the style space. In addition to comparing against the Stanford Politeness Corpus, we also compare against the following:\n\n-  TyDIP (multilingual politeness) (Srinivasan and Choi 2022)\n-  EmoEvent (multilingual emotion) (Plaza del Arco et al. 2020)\n-  GoEmotions (English emotion) (Demszky et al. 2020)\n-  Formality Corpus (English formality) (Pavlick and Tetreault 2016)\n-  XFormal (multilingual formality) (Briakou et al. 2021)\n\n|Agreement Statistic|Range in Above Datasets|Ours|\n|--|--|--|\n|Intraclass Correlation|0.39-0.83|0.66|\n|Pearson Correlation|0.16-0.64|0.545|\n|Cohen's Kappa|0.09 - 0.55|0.22|\n\n*Overall, our dataset's annotator agreement is better than the average agreements of existing human-annotated datasets in the style space.* Note that the highest agreements are for emotions such as Love and Sadness, which are much clearer than politeness.\n\nYour comment raises an extremely important question -- How to improve annotation quality of subjective style tasks? However, given our dataset aligns with existing agreements, we feel answering this question is outside the scope of this work.\n\n&nbsp; \n\n### Scoping introduction\nBased on your suggestions, we have scoped and shortened the introduction as follows:\n- We introduce our contributions + the purpose of this work earlier in the introduction\n- We make the discussion of Figure 1 shorter and more high-level, focusing on the overall motivation for our study\n- We remove repetitive sentences from consecutive paragraphs to better draw readers' attention to the key points\n\n&nbsp;\n\nPlease let us know if you have any other questions or concerns, and we welcome additional feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The suggested framework is simple and can be easily applied to other theoretically grounded style lexicons, provided that classifiers and datasets for comparison are available. \n- Filtering the lexicon based on style ensures that the translations maintain linguistic coherence and stylistic consistency, making them more reliable to be used for downstream tasks as well. \n- In the context of politeness the method provides an understanding of how the concept varies across languages, both at word level and utterance level.\n- They create a relatively large corpus for politeness in 4 languages with open access. ", "reasons_to_reject": "- There is no discussion regarding the inherent bias and errors that can be introduced at different stages of the process, which could have propagated to the final analysis. For instance, the style scores are highly dependent on the annotated data which has a low agreement between annotators. Using these annotations to train the model could enhance any biases and uncertainties learned by the model.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a new method for generating reliable translations of lexicons categorized into meaningful groups. The approach involves expanding the lexicon with synonyms and conceptually related terms using word embeddings and nearest neighbor estimation. The expanded lexicon is then filtered based on style scores assigned by pretrained language models. The authors use this framework to study politeness across multiple languages, exploring the variations in the importance of lexical categories due to cultural differences. For studying politeness, they also create a corpus annotated with level of politeness on a 5 point scale.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Currently the introduction is a bit too long and lacks focus. It would be easier to follow the text if the introduction is made shorter focusing on the motivation, challenges and the contributions, giving a general overview rather than going into details. "}}, {"review_id": "oMnbvmSYMi", "reviewer": "Reviewer_a7sm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your valuable feedback and for your excitement in our work! We look forward to improving our paper based on your comments:\n\n&nbsp;\n\n### Using additional LMs\nGiven our chosen LMs must be multilingual and open-source*, we additionally experiment on a series of language-specific RoBERTa models and LLAMA-2-7b. We report an interesting subset of our results below, as we are unable to link heatmaps as per the EMNLP rebuttal guidelines. \n\n**Getting SHAP values for closed-sourced models requires a significant compute cost that is out of scope of our paper. Running our same set of feature extraction experiments on GPT-4, for example, results in a proposed cost of over \\\\$34,900, hence we only use open-source LMs.*\n\n**Results for LLAMA-2-7b**\nTo investigate the role of LM architecture, we fine-tune Llama-2-7b using PEFT (Mangrulkar et al. 2022) and report categories of interest. \n\n**Takeaways:** Compared to RoBERTA-XLM, we notice stability in *direction* of importance score (i.e. positive/negative/neutral lexical categories and dialog acts are stable across both LMs). Interestingly, we notice differences in *magnitude* (e.g. Llama-2-7b sees the \"Greeting\" category in Chinese to be a larger indicator of politeness than RoBERTa-XLM does).\n\n| Language | PoliteLex Category: *Greeting* | PoliteLex Category: *Emergency* | Dialog Act: *Conventional Opening* | Dialog Act: *Opinion Statement* |\n|--|--|--|--|--|\n| English |0.068|0.066|0.36|-0.056|\n| Spanish |0.001|0.02|0.765|0.087|\n| Japanese |0.189|0.089|0.612|0.033|\n| Chinese |0.173|0.148|0.519|-0.001|\n\n**Results for monolingual RoBERTa models**\n\nTo investigate the role of language-specific pretraining, we fully fine-tune four separate RoBERTa models that have been trained on ONLY their respective languages. \n\n**Takeaways:** Similar to Llama-2-7b, we observe a very high similarity in *direction* of importance score compared to RoBERTa-XLM. We also notice more similarity in *magnitude* of importance scores. This may be due to inherent similarities between all RoBERTa models (parameter size, training methods, etc.), which do not exist between RoBERTa-XLM and Llama-2-7b.\n\n| RoBERTa Model | PoliteLex Category: *Greeting* | PoliteLex Category: *Emergency* | Dialog Act: *Conventional Opening* | Dialog Act: *Opinion Statement* |\n|--|--|--|--|--|\n| English (Liu et al., 2019) |0.012|-0.032|0.741|-0.083|\n| Spanish (De la Rosa et al., 2022) |0.011|0.02|0.626|-0.057|\n| Japanese (Cho and Sawada, 2021) |0.055|0.051|0.450|0.150|\n| Chinese (Cui et al., 2020) |0.053|0.016|0.265|-0.039|\n\nWe have added an \"LM Ablation Study\" section to our appendix, where we will discuss the differences in results from different language models. Due to time constraints, these results use SHAP values from 20% of our dataset; we will include the full results in the camera-ready paper.\n\n\n&nbsp;\n\n### Figure Size\nBased on your suggestion, we have shrunk Figures 4 and 5.\n\n&nbsp;\n\nPlease let us know if you have any other questions or concerns, and we welcome additional feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well motivated.\n2. Clear writing.\n3. Release of code and data.\n4. Provides a detailed insight on LLMs explainability on different languages.", "reasons_to_reject": "1. Experiments could be more extensive.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present a framework to compare politeness knowledge included in LMs in order to compare how politeness style differ between four different languages: English, Spanish, Chinese and Japanese. This not only serves as a comparative linguistic study, but contribute in understanding how LMs handle these issues in different languages and if their style is appropiate for languages other than English. They also contribute with a multilingual politeness dataset.\n\nOverall, this work provides a detailed study of politeness in multilingual LLMs and existing differences between languages in use of politeness that contribute to increase the explainability of LLM. Comparing the results on more LLMs would've been interesting and not really costly.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figures 4 and 5 feel too big."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new method for generating reliable translations of lexicons categorized into meaningful groups. The approach involves expanding the lexicon with synonyms and conceptually related terms using word embeddings and nearest neighbor estimation. The expanded lexicon is then filtered based on style scores assigned by pre-trained language models. The authors use this framework to study politeness across multiple languages, exploring the variations in the importance of lexical categories due to cultural differences. For studying politeness, they also create a corpus annotated with level of politeness on a 5 point scale.

Based on the reviews and the subsequent discussion during the discussion period, the significant aspects shared by reviewers include:

* The task is interesting and eventual applications (e.g. fine-tuning chatbots appropriately for different segments of society) is a reason to accept
* The paper is well motivated and generally well written. Issues that have been raised wrt coherence and missing discussion points have been addressed in the author response
* The presented framework is simple yet effective, which is a strong aspect of the paper
* A nice contribution to the community with the code and the multilingual politeness dataset

A shared issue brought up was that the experiments were not extensive enough to warrant a long paper. It should be noted that the author response addressed this well by providing new/additional analyses and results which should be included in the camera-ready, and make the paper stronger.
Decision: Accept-Main