Multilingual Language Models are not Multicultural: A Case Study in Emotion
Year: 2023

Abstract:
Emotions are experienced and expressed differently across the world. In order to use Large Language Models (LMs) for multilingual tasks that require emotional sensitivity, LMs must reflect this cultural variation in emotion. In this study, we investigate whether the widely-used multilingual LMs in 2023 reflect differences in emotional expressions across cultures and languages. We find that embeddings obtained from LMs (e.g., XLM-RoBERTa) are Anglocentric, and generative LMs (e.g., ChatGPT) reflect Western norms, even when responding to prompts in other languages. Our results show that multilingual LMs do not successfully learn the culturally appropriate nuances of emotion and we highlight possible research directions towards correcting this.

heading: 1 Introduction
text: The global reach of Large Language Models (LMs) today prompts an important question – Are multilingual LMs also multicultural? We are specifically interested in the multicultural behavior of LMs from the lens of emotion. LMs are used for many multilingual tasks that require emotional sensitivity and therefore must be able to reflect cultural variation in emotion. For instance, LM-powered Therapy Bots must delicately adapt the way they speak to patients in different languages (Wang et al., 2021), LMs as creative writing assistants must produce content that will elicit the appropriate emotional response in an author’s desired audience (Shakeri et al., 2021), LMs used for workplace communication must understand the subtleties of interpersonal interaction (Thiergart et al., 2021), etc.
We define cultural variation in emotion as the nuances in meaning and usage of emotion words across cultures. For example, in English, we have many different words that express Anger. One can say "I feel angry," but may also choose to say "frustrated", "irritated", or "furious." The Anger invoked
by a baby crying on an airplane is different from the Anger invoked by an unfair grade on an exam; different situations that cause Anger will invoke different language to best express it. These nuances in meaning and usage patterns of emotion words exist differently across cultures (Mesquita et al., 1997; Wierzbicka, 1999).
Therefore, there is not a perfect one-to-one mapping between languages for emotion words coupled with their meaning and usage patterns. The direct translation for "I feel frustrated" from English to Chinese (simplified), for example, is "我感到沮 丧". However, in a situation where a native English speaker would likely say "I feel frustrated," a native Chinese speaker may use a different phrase than "我感到沮丧", based on situation, context, and the cultural norms of emotion expression in China.
As we rely on multilingual LMs today for emotionally sensitive tasks, they must reflect this cultural variation in emotion. However, the widely-
202
used multilingual LMs are trained on Anglocentric corpora and encourage alignment of other languages with English (Reimers and Gurevych, 2020), both implicitly and explicitly, during training. The key problem in this approach to building multilingual LMs is that any form of alignment destroys a model’s ability to encode subtle differences, like the difference between “I feel frustrated” in the United States and "我感到沮丧" in China.
In this paper, we investigate whether widelyused multilingual LMs reflect cultural variation in emotion. We select four high-resource written languages, two Western and two Eastern, to focus on in this work – English, Spanish, Chinese (Simplified), and Japanese.
Specifically, we investigate two facets of LMs: embeddings and language generation.
1. Emotion embeddings (a) Does implicit and explicit alignment in
LMs inappropriately anchor emotion embeddings to English? We compare embeddings from monolingual, multilingual, and aligned RoBERTa models.
(b) Do emotion embeddings reflect known psychological cultural differences? We project embeddings onto the ValenceArousal plane to visualize American vs. Japanese differences in Pride and Shame.
2. Emotional language generation (a) Do LMs reflect known psychologi-
cal cultural differences? We analyze whether GPT-3 probabilities encode American vs. Japanese differences in Pride and Shame.
(b) Do LMs provide culturally-aware emotional responses? We prompt GPT-3.5 and GPT-4 with scenarios that should elicit varied emotional responses across cultures and conduct a user study to assess response quality.
We make our code public 1 and encourage researchers to utilize the analyses outlined in this work as a baseline to measure the cultural awareness of future multilingual models.

heading: 2 Related Work
text: A large body of work in NLP focuses on detecting emotion in multilingual text . However, a major
1https://github.com/shreyahavaldar/ Multicultural_Emotion/
oversight in this line of research is that it treats emotion as culturally invariant. Work from Bianchi et al. (2022) gathers a corpus of annotated social media data from 19 languages, but uses machine translation to transfer annotations from one language to another, assuming that translation correctly captures emotional variation. Work from Buechel et al. (2020) generates lexica to analyze emotion across 91 languages, relying on translations from English lexica and assuming that the affective state of parallel words will be identical.
Psychologists have characterized emotion as having multiple components – an emotional experience, a physiological response, and a behavioral response tendency (Kensinger and Schacter, 2006). Each of these components vary from culture to culture (Mesquita et al., 1997), a complexity completely ignored when emotion is treated as a static, transferable label on an utterance of text. Using machine translation to transfer emotion labels between languages incorrectly assumes that emotion is experienced identically across cultures.
Others have also observed that LMs can fail to account for cultural context and variation. Cao et al. (2023) find that ChatGPT strongly aligns with American values. Magno and Almeida (2021) use word embeddings to globally measure human values across cultures, and find that these values overlap more when measured via data in English vs. native languages. Arora et al. (2023) probe multilingual LMs and discover weak alignment with the cultural values reflected by these LMs and established values surveys.
In this paper, we focus on emotion, showing a wider variety of Anglocentric anchoring by elucidating the underlying mechanisms of this alignment. We investigate emotion embeddings and LM probabilities, as well as affective language generated from multilingual LMs.

heading: 3 Investigating Emotion Embeddings
text: Many tasks in multilingual NLP utilize embeddings from pre-trained LMs such as XLMRoBERTa (Conneau et al., 2019) and mBERT (Devlin et al., 2018). Researchers fine-tune these models for downstream tasks, relying on their learned representations of words and concepts.
We scope our investigation to embeddings from the widely used XLM-RoBERTa models. XLMRoBERTa was trained on text that includes parallel and comparable corpora (e.g., Wikipedia) in mul-
tiple languages. The nature of Wikipedia, which has topic-aligned articles in different languages, causes implicit alignment in training. Worse, XLMRoBERTa variants trained via multilingual knowledge distillation (Reimers and Gurevych, 2020) enforce English sentences and their translations to map to the same point in embedding space, giving explicit alignment of other languages with English.
This section investigates the effect of alignment – both implicit and explicit – by analyzing emotion embeddings from monolingual, multilingual, and aligned RoBERTa models (See Table A2). We further investigate whether this anchoring impacts our ability to visualize known cultural differences (e.g. differences between Pride and Shame in the US vs. Japan (Tsai et al., 2006)) when projecting embeddings into the two-dimensional Valence-Arousal plane (Russell, 1980).

heading: 3.1 Does implicit and explicit alignment inappropriately anchor emotion embeddings to English?
text: We analyze whether implicitly aligned embeddings become Anglocentric by comparing emotion embeddings from XLM-RoBERTa to emotion embeddings learned in a parallel, monolingual setting. We further analyze explicit alignment by comparing embeddings from vanilla XLM-RoBERTa to an explicitly aligned variant of XLM-RoBERTa (Reimers and Gurevych, 2020).
Distance-Based Similarity How do we compare the emotion embeddings of two models? Let us take Joy, one of the six Ekman emotions (Ekman et al., 1999), as an example – can we compare the similarity of embeddings from two models for the phrase "I feel joy"? 2 A direct numerical comparison is challenging, as we would need to align the embedding spaces of these two models and possibly distort the Joy embeddings. Taking this into account, we pose the following solution:
The more similar two models are, the more similarly we expect them to embed the same phrases in embedding space. For example, let us embed phrases x, y, and, z using Model A and Model B. This gives us the embedding vectors x⃗A, y⃗A, z⃗A and x⃗B, y⃗B, z⃗B respectively. Figure 2 illustrates this, showing the embeddings of Joy, Anger, Elation, Sadness, and Happiness using a monolingual and multilingual RoBERTa model.
If Model A and Model B have embedded phrases x, y, and z in a similar way, then we expect to see a high correlation between the numerical distances x → y, x → z, and y → z in the respective embedding spaces of Model A and B. We calculate the correlation between the following two vectors: < ∥x⃗A − y⃗A∥, ∥x⃗A − z⃗A∥, ∥y⃗A − z⃗A∥ > < ∥x⃗B − y⃗B∥, ∥x⃗B − z⃗B∥, ∥y⃗B − z⃗B∥ > to inform how similar the embeddings of x, y, and, z are between Model A and Model B.
Using this idea, we can compare the distances from "I feel joy" to other contextualized emotion phrases (e.g. "I feel anger", "I feel happiness", etc.) in embedding space A to those same distances in embedding space B. For example, if the monolingual and multilingual RoBERTa models shown in Figure 2 have learned similar representations of Joy, then we can expect to see a high Pearson correlation between the vectors < 13.05, 9.85, 12.55.2.23 > and < 28.44, 6.68, 28.48, 4.25 >. We use this distance-based similarity metric to answer the following three questions:
1. Do implicitly aligned multilingual LMs embed emotion words differently than monolingual LMs?
2. Do implicitly aligned multilingual LMs embed emotion words in an Anglocentric way?
3. Does explicit alignment further anchor multilingual emotion embeddings to English?
2We prepend each emotion word with the phrases "I feel" and "I am" to add context and circumvent polysemy when generating embeddings for analysis.
Do implicitly aligned multilingual LMs embed emotion words differently than monolingual LMs? We compare the emotion representations from monolingual and multilingual RoBERTa models across English, Spanish, Chinese, and Japanese. We select the four monolingual RoBERTa models most downloaded on Huggingface, additionally ensuring the four models selected have the same number of parameters. Table A2 contains additional details on the models used in our experiments.3
Figure 2 illustrates this experiment. In practice, we use a list of 271 emotions (Davis, 2023) for our distance-based similarity computation. Additionally, to account for variance in descriptions of experiencing emotion, we average the embedding of two contextualized phrases for each emotion – "I feel <emotion>" and "I am <emotion>".
For non-English languages, we machine translate the two contextualized English phrases for each emotion (e.g. a representation of Joy in English is the average of the embeddings of "I feel joy" and "I am joyful". The representation of Joy in Spanish is the average of the embeddings "siento alegría" and "soy alegre", etc.). In order to ensure quality, we have native speakers evaluate a subset of the machine-translated emotion phrases, and we find that translation does yield sufficient results.
We then apply our distance-based similarity metric to compare the monolingual and multilingual emotion embeddings across languages. The "Mono vs. Multi" column in Table 1 shows the average distance-based similarity across all 271 emotions. The lower similarities for non-English languages indicate that XLM-RoBERTa embeds non-English emotions differently compared to monolingual models. We can thus say that multilingual LMs do not preserve the embedding space of monolingual nonEnglish LMs.
Do implicitly aligned multilingual LMs embed emotion words in an Anglocentric way? We compare the emotion representations of English vs. non-English languages. We apply our distancebased similarity metric to measure the similarity between English and non-English emotion representations in two settings – monolingual and multilingual. Figure 3 illustrates this experiment.
3We note that differences in training data for the monolingual RoBERTa models affect how these models are able to capture emotion. However, it is important to investigate LMs actively used in NLP research rather than explicitly creating a perfectly parallel set of monolingual models.
The "English vs. Non-English" columns in Table 1 show the average distance-based similarity between English and non-English emotion embeddings across all 271 emotions, in monolingual and multilingual settings respectively. Results reveal low similarity between non-English and English emotion embeddings in monolingual space. In a multilingual setting, however, the non-English emotion embeddings become more similar to English ones. This suggests that implicit alignment in multilingual LMs anchors non-English emotion embeddings to their English counterparts.
Does explicit alignment further anchor multilingual emotion embeddings to English? We compare emotion embeddings from an unaligned RoBERTa model to a RoBERTa model trained via forced alignment across English, Spanish, Chinese, and Japanese (Reimers and Gurevych, 2020).
The average distance-based similarity between aligned and unaligned emotion embeddings across all 271 emotions is shown in column "Aligned vs. Unaligned" in Table 1. Emotion embeddings from explicitly aligned models are most similar to unaligned embeddings in English, indicating explicitly aligned embedding space fails to preserve the structure of non-English embedding spaces.
Finding 1: Multilingual LMs embed non-English emotion words differently from their monolingual counterparts, whereas English emotion embed-
dings are more stable and similar in all settings. We demonstrate that implicit and explicit alignment in multilingual LMs anchor non-English emotion embeddings to English emotions. All observed trends persist under ablation studies on the effect of distance metric and correlation function (see Appendix A).

heading: 3.2 Do emotion embeddings reflect known psychological cultural differences?
text: Though emotion embeddings from multilingual LMs are Anglocentric, we nonetheless investigate whether they encode any information about known cultural variation in emotion. Prior work (Tsai, 2017; Russell et al., 1989) underlines the differences in emotional expression across cultures, and often illustrates these differences via the circumplex model of affect (Russell, 1980). The circumplex model assumes all emotions can be classified along two independent dimensions – arousal (the magnitude of intensity or activation) and valence (how negative or positive).
Pride and Shame are two widely researched emotions when investigating cultural differences in emotional expression. (Lewis et al., 2010; Wong and Tsai, 2007). Shame is expressed more commonly and has a desirable affect in Eastern cultures compared to Western cultures. Similarly, Pride is openly expressed in Western cultures whereas Eastern cultures tend to inhibit the feeling of Pride (Lim, 2016). Moreover, these proclivities are deeply ingrained in society and thus acquired at a very young age (Furukawa et al., 2012).
For our experiments, we consider the US and Japan, as the subtle differences in expression of Pride and Shame between these two cultures are well-studied (Kitayama et al., 2000; Tsai et al., 2006). We project emotion embeddings from English and Japanese onto the Valence-Arousal plane
to visualize whether multilingual LMs capture the expected differences in Pride and Shame. When comparing the embeddings, we expect to specifically observe:
1. The embedding for English Pride should have a more positive valence. (as Pride is more accepted in the US than Japan) (Furukawa et al., 2012)
2. The embedding for English Shame should have a more negative valence. (as Shame is more embraced in Japan than the US) (Furukawa et al., 2012)
3. The embeddings for English Pride should have higher arousal (as Pride is more internally and culturally regulated in Japan than the US) (Lim, 2016)
Projection into the Valence-Arousal plane In order to define the valence and arousal axes, we first generate four axis-defining points by averaging the contextualized embeddings of the emotions
listed in Table A1. This gives us four vectors in embedding space that best represent positive valence (PV ) negative valence (NV ), high arousal (HA), and low arousal (LA). We can now project any emotion embedding onto the plane defined by the valence axis (NV → PV ) and the arousal axis (LA → HA). We give a more formal, mathematical description of this projection method in the Appendix B. Figure 4 shows the six Ekman emotions (Ekman et al., 1999) projected into the Valence-Arousal plane, indicating that our projection method successfully recreates the circumplex.
To visualize Pride and Shame in the ValenceArousal plane, we manually translate the axisdefining emotions to Japanese and average the English and Japanese points of each axis category to define multilingual valence and arousal axes. We then project the contextualized sentence embeddings "I am proud" and "I am ashamed" in English and Japanese. We experiment with both aligned and unaligned RoBERTa models; these plots are shown in Figure 5.
Looking at the plots, we observe that English Pride is slightly higher in valence than Japanese Pride, and English Shame is slightly lower in valence than Japanese Shame. This does serve as a weak confirmation of the first two hypotheses. However, we do not observe English Pride to have higher arousal than Japanese Pride. This discrepancy suggests our results are inconclusive, and we cannot confirm whether multilingual RoBERTa encodes cultural variation in English vs. Japanese Pride and Shame.
Finding 2: By projecting emotion embeddings into the Valence-Arousal plane, we show that LMs are not guaranteed to encode the nuances in meaning and usage of emotion words across cultures. Researchers who utilize embeddings from multilingual LMs for emotion-related tasks assume these pre-trained models have learned adequate representations of emotion across languages. However, implicit and explicit alignment during training causes multilingual LMs to ignore the subtle differences in emotion expression across cultures.

heading: 4 Investigating multilingual LM generation
text: We now turn from investigating embeddings to analyzing language generated by Language Models (GPT-3, GPT-3.5, and GPT-4) to see if multilingual LM completions reflect cultural variation in
emotion. In order for LMs to be used for tasks that require emotional sensitivity, their responses must align with cultures’ socio-cultural norms (Genesee, 1982); generated text must reflect users’ cultural tendencies and expected affect (Tsai, 2017).
We first analyze token-level completion probabilities from GPT-3, to see if they reflect cultural differences between American and Japanese Shame and Pride. We then prompt GPT-3.5 and GPT-4 in English and non-English languages to respond to scenarios that should elicit different emotional responses across cultures and assess their cultural appropriateness in a small-scale user study.

heading: 4.1 Do LMs reflect known psychological
text: cultural differences?
Continuing our example of English vs. Japanese Pride and Shame, we evaluate whether this known cultural difference is reflected in OpenAI’s GPT-3.
We design a set of 24 prompts (See Table A5) for GPT-3 (davinci) based on six scenarios that would invoke a combination of Pride and Shame in the form <context><feeling>. For example, "I received an award in front of my coworkers. I feel proud." One might feel proud for re-
ceiving an award or embarrassed for being publically praised. We then prompt GPT-3 using various <context><feeling> prompts, and analyze the log probability of each token of the prompt. Finally, we sum the log probability of each token in the <feeling> sentence to get a sense of how likely the <feeling> is to follow the <context>. Based on cultural norms about how one would react in situations that elicit both Pride and Shame, we expect to see a higher probability for "I feel happy" and "I feel proud" in English, and a higher probability for "I feel embarrassed" and "I feel ashamed" in Japanese across scenarios.
Figure 6 shows the results of this for the prompt "I received an award in front of my coworkers. I feel ___." where we test two Pride words: "proud", "happy", and two Shame words: "ashamed", and "embarrassed". We replicate this experiment in Japanese, and compare the summed log probabilities of "I feel ___." between English and Japanese across emotions. The full results, along with the remaining prompts are given in Appendix Table A5. Analyzing the results across six scenarios (see Appendix C), we do not see any consistent evidence that Pride is more likely to be expressed in English or Shame is more likely to be expressed in Japanese. In Figure 6, for example, we see contradicting results for "proud", "happy", and "embarrassed".
Finding 3: These results suggest that GPT-3 lacks knowledge of Pride and Shame and the norms surrounding their expression in the US and Japan. This is a major limitation; such a failure to capture cultural variation is likely to limit both the utility and applicability of LMs in downstream emotionally-sensitive tasks.

heading: 4.2 Do LMs provide culturally-aware emotional responses?
text: To further investigate whether LM completions reflect cultural norms, we conduct a small-scale user study to see if GPT-3.5 and GPT-4 are capable of appropriately adapting when prompted in different languages. Annotators assess whether the completions parallel the accepted emotional responses associated with the user’s culture.
Prompting with cultural context Prior psychological research has detailed scenarios that reveal how emotional expressions vary across cultures (Mesquita, 2022). We use this work to design a set of 19 questions (see Table A6) that should elicit different emotional responses across cultures. For example, the question "How would you feel if your guests chose to keep their shoes on when entering your home?" would likely elicit a different response from someone culturally American vs. Chinese.
We use these scenarios to prompt GPT-3.5 (gpt-3.5-turbo) and GPT-4 (gpt-4) in the form <prefix>+<question> (see Table 2). In order to include cultural context and coax the LM into returning a culturally appropriate emotional response, we experiment with providing cultural context in two ways (using Chinese as an example):
1. via English – we add the sentence "You live in China." to the prefix. The LM returns an English completion.
2. via Native Language – we translate the <prefix>+<question> prompt to Chinese. The LM returns a Chinese completion.
Figure 1 details these two cultural context modes and how they may cause conflicting LM responses.
User Study To assess the quality of the LM completions, we perform a small-scale user study using eight volunteers, consisting of four pairs fluent in English, Spanish, Chinese, and Japanese respectively. We ask our volunteers to annotate GPT-3.5 and GPT-4’s responses for cultural awareness along two axes - linguistic norms (how you would expect a native speaker to talk), and cultural norms (what you would expect a native speaker to say). As these two norms are deeply correlated, annotators are instructed to take both of these dimensions into account and give a single rating to each completion. We use a scale of 1-7, where 7 indicates the LM’s response is fully expected of a native speaker.
Across languages, we observe a high agreement within each pair of volunteers. Figure 7 details the average score across annotators and questions for GPT-4 and GPT-3.5 completions. We provide the annotator agreement statistics in Appendix Table A4. Analyzing the completions and annotations, we notice some interesting trends:
• We see a large difference in quality between the LM responses returned using the two cultural context prompting modes (even though the questions are identical.)
• For Chinese and Japanese, the LM returns a less culturally-appropriate response using the Native Language cultural context mode.
• English completions are the most culturallyaware across languages, and English response quality is unaffected by cultural context mode.
Finding 4: GPT-3.5 and GPT-4 fail to infer that a prompt in a non-English language suggests a response that aligns with the linguistic and cultural norms of a native speaker. Additionally, the LM completions reflect culturally appropriate emotion much better in Western languages than Eastern.

heading: 5 Conclusion
text: We find that multilingual models fail to fully capture cultural variations associated with emotion, and predominantly reflect the cultural values of the Western world. Emotion embeddings from multilingual LMs are anchored to English, and the text completions generated in response to non-English prompts are not in tune with the emotional tendencies of users’ expected culture. For instance, when GPT-4 is prompted in Japanese, it responds as an American fluent in Japanese but unaware of Japanese culture or values.
Our results caution against blindly relying on emotion representations learned by LMs for downstream applications. Using machine translation to transfer labels or utilizing multilingual LMs in a zero-shot setting for unseen languages has risks – the multilingual representations of emotion learned by these models do not perfectly reflect how their corresponding cultures express emotion.
Future Research Directions Our paper motivates the need for future work that transcends current Anglocentric LMs. This could take the form of higher performing, non-English models in a monolingual setting, or of multilingual models trained on more linguistically and culturally balanced corpora. Future work should additionally investigate whether state-of-the-art monolingual models in non-English languages succeed in encoding the respective culture’s norms. Furthermore, we encourage the evaluation of multilingual models on benchmarks that measure cultural awareness in addition to standard metrics.

heading: 6 Limitations
text: We only analyze four high-resource languages in this study, our analysis could have benefited from more languages, especially low-resource ones. Additionally, we only analyze Japanese and English Pride/Shame as a known cultural difference; analyzing other differences could provide stronger results. We perform a small user study, and our work could have benefited from a larger-scale study with more annotators and completions analyzed.
We recognize the added complexity of investigating Pride embeddings from a culture where explicit expressions of Pride are discouraged; we note this may be a contributing factor to our results indicating that LMs do not reflect the culturally appropriate nuances of Shame and Pride.
Additionally, we acknowledge that the experiments outlined in this paper are specific to investigating cultural awareness from the lens of emotion. These experiments are not easily applicable to measuring cultural awareness from different perspectives; therefore, results may not be generalizable.
At a higher level, we equate language with culture. Psychologists have observed higher cultural similarities within languages than between them (Stulz and Williamson, 2003), however, we recognize there are variations within the populations that speak each language. For example, Spanish is spoken by people in Spain, Mexico, and other countries, each having a unique and varied culture.

heading: 7 Ethical Considerations
text: Although culturally-aware multilingual LMs are critical in uses such as therapy, storytelling, and interpersonal communication, these are possible misuses for nefarious purposes - persuasion, misinformation generation, etc. Additionally, our analyses behave as if China, Japan, Spain, and the United States are a single culture with a single set of cultural norms. In reality, this is not the case; we recognize there are huge variations in the way people view emotion within each of these cultures.

heading: A Distance-based Similarity Experiments: Additional Details
text: Table A2 gives details on the RoBERTa models we use in each setting – monolingual, multilingual, and aligned – for all experiments in this paper.
We find no clear pattern in certain emotions being more or less problematic across languages. Our machine translations of 271 English emotions give 247, 210, and 246 unique emotions for Spanish, Chinese, and Japanese respectively.
In order to test the robustness of the experiments outlined in section 3.1, we experiment with other distance and correlation metrics in our distancebased similarity calculations. Table A3 shows results for our distance-based similarity experiments where we replace Euclidean distance with cosine similarity, and results where we replace Pearson correlation with Spearman’s rank.

heading: B Projection into the Valence-Arousal plane
text: In order to define the valence and arousal axes, we first generate four axis-defining points by averaging the contextualized embeddings ("I feel [emotion]")
Axis Anchor Russell Emotions
Positive valence (PV) Happy, Pleased, Delighted,Excited, Satisfied Negative valence (NV) Miserable, Frustrated, Sad,Depressed, Afraid High arousal (HA) Astonished, Alarmed, Angry,Afraid, Excited Low arousal (LA) Tired, Sleepy, Calm,Satisfied, Depressed
Table A1: Emotions used to define the valence and arousal axis anchors for projection into the ValenceArousal plane. We select the 5 emotions from the circumplex closest to each axis point.
of the emotions listed in Table A1. This gives us four vectors in embedding space – positive valence (v⃗pos), negative valence(v⃗neg), high arousal(⃗ahigh), and low arousal(⃗alow). We mathematically describe our projection function below:
1. We define the valence axis, V , as v⃗pos − v⃗neg and the arousal axis, A, as a⃗high − a⃗low. We then normalize V and A and calculate the origin as the midpoints of these axes: (v⃗middle, a⃗middle).
2. We then scale the axes so v⃗pos, v⃗neg, a⃗high, and a⃗low anchor to (1, 0), (−1, 0), (0, 1), and (0,−1) respectively.
3. We Compute the angle θ between the valencearousal axes by solving cos θ = V ·A∥V ∥·∥A∥
4. For each embedding vector x⃗ in the set {xi}ni=1 we want to project into our defined plane, we compute the valence and arousal components for xi as follows: xvi = (xi − v⃗middle) · V⃗ xai = (xi − a⃗middle) · A⃗.
5. We calculate the x and y coordinates to plot, enforcing orthogonality between the axes: x̃vi = x v i − xai · cos θ
x̃ai = x a i − xvi · cos θ Finally, we plot (x̃vi , x̃ v i ) in the ValenceArousal plane.
In order to define multilingual valence and arousal axes and plot English vs. Japanese Pride and Shame embeddings, we calculate v⃗pos, v⃗neg, a⃗high, and a⃗low separately for English and Japanese. We then average the axis-defining points between English and Japanese (i.e. v⃗pos = AV G(v⃗posen, v⃗posja), etc.) so we can project embeddings from two languages into the same plane.

heading: C GPT-3 Pride & Shame Experiments: Additional Details
text: We provide the full list of scenarios used in Table A5. We also include the results of our experiment across scenarios.
We find no empirical evidence of a consistent trend that "I feel ashamed" and "I feel embarrassed" are more likely to be said in Japanese or that "I feel proud" and "I feel happy" are more likely to be said in English. Rather, we observe a trend that the higher log probability for an emotion (between English vs. Japanese) is more dependent on the scenario rather than culture.
Language & Setting Model Name Downloads Training Data
Monolingual English roberta-base(Liu et al., 2019) 7.77M BookCorpus, Wikipedia, CommonCrawl(News), OpenWebText, Stories Monolingual Spanish bertin-roberta-base-spanish(De la Rosa et al., 2022) 2.67k Common Crawl Monolingual Chinese chinese-roberta-wwm-ext(Cui et al., 2020) 113k Wikipedia, Encyclopedia, News,Web QA data Monolingual Japanese japanese-roberta-base(Cho and Sawada, 2021) 36.2k Common Crawl, Wikipedia
Multilingual, Unaligned xlm-roberta-base(Conneau et al., 2019) 18.4M Common Crawl, Wikipedia Multilingual, Aligned paraphrase-multilingualmpnet-base-v2 (Reimers and Gurevych, 2019) 293k Common Crawl, Wikipedia, Aligned Paraphrasing Corpus
Table A2: RoBERTa models used in our experiments for each setting: monolingual, multilingual, and aligned. For each model, we provide the number of monthly downloads by Huggingface users (as of April 2023) and a high-level description of the data used for training. All models have 125M parameters.
Mono vs. Multi English vs. Non-English Aligned vs. Unaligned
Language (L) r̄(Lmono, Lmulti) r̄(En,L)mono r̄(En,L)multi r̄(Lalgn, Lunalgn)multi
Using cosine distance
English (En) 0.752 — — 0.468 Spanish 0.290∗ -0.219∗ 0.647∗ 0.252∗ Chinese 0.338∗ -0.223∗ 0.454∗ 0.067∗ Japanese 0.303∗ -0.05∗ 0.490∗ 0.287∗
Using Spearman’s rank
English (En) 0.652 — — 0.488 Spanish 0.339∗ 0.248∗ 0.567∗ 0.307∗ Chinese 0.377∗ 0.223∗ 0.418∗ 0.162∗ Japanese 0.334∗ 0.059∗ 0.460∗ 0.353∗
Table A3: We report the average distance-based similarity across 271 emotions for each of our experiments, using cosine distance and Spearman’s rank correlation. ∗indicates the difference in mean correlation between English vs. non-English settings (for Mono vs. Multi, Aligned vs. Unaligned) and monolingual vs. multilingual settings (for English vs. Non-English) is statistically significant (p < 0.05); we compute this using an independent t-test. See Table A2 for models used in each setting. We see that our observed trends persist despite ablation.
Language GPT Model Cultural Context Mode Agreement
gpt-3.5-turbo English 0.785 English gpt-3.5-turbo Native Language 0.705
gpt-4 English 0.823 gpt-4 Native Language 0.673
gpt-3.5-turbo English 0.547 Spanish gpt-3.5-turbo Native Language 0.662
gpt-4 English 0.559 gpt-4 Native Language 0.776
gpt-3.5-turbo English 0.665 Chinese gpt-3.5-turbo Native Language 0.609
gpt-4 English 0.708 gpt-4 Native Language 0.749
gpt-3.5-turbo English 0.847 Japanese gpt-3.5-turbo Native Language 0.878
gpt-4 English 0.843 gpt-4 Native Language 0.900
Table A4: GPT models used and annotator agreement (Pearson correlation between each annotator pair) for our user study. We observe high agreement between each annotator pair across languages
Scenario + Language proud happy ashamed embarrassed

heading: My teacher complimented me in front of the class. I feel X.
text: English -22.386 -26.923 -29.947 -25.872 Japanese -23.012 -30.642 -32.584 -41.096

heading: My teacher complimented my friend in front of the class. I
text: expect he feels X. English -36.620 -39.324 -46.868 -42.395 Japanese -25.175 -28.946 -33.690 -33.736

heading: I received an award in front of my coworkers. I feel X.
text: English -17.834 -23.863 -24.926 -23.890 Japanese -14.236 -20.497 -22.559 -27.832

heading: My colleague received an award in front of our coworkers. I expect he feels X.
text: English -29.238 -33.736 -39.236 -34.866 Japanese -14.017 -18.251 -23.928 -21.340

heading: My parents praised me in front of my siblings. I feel X.
text: English -17.925 -21.063 -24.843 -22.504 Japanese -23.258 -30.589 -30.610 -40.588

heading: My parents praised my brother in front of our siblings. I
text: expect he feels X. English -34.805 -36.931 -42.669 -38.356 Japanese -30.318 -33.213 -35.108 -35.152
Table A5: All scenarios and results for our GPT-3 Pride & Shame prompting experiment. Prompts are in the form (<context><feeling>) and designed to describe scenarios that should elicit a combination of Shame and Pride. We evaluate whether GPT-3 encodes cultural differences in Japanese vs. US Pride and Shame by comparing the summed log probabilities of the <feeling> sentence.
Prefixes
Prefix 1: You are a helpful chatbot. Your goal is to answer my questions like you are a human capable of feelings and emotions. You live in China. Answer the following question using a single sentence that begins with "I would feel...". Prefix 2: 你是一个有用的聊天机器人。你的目标是回答我的问题，就像你是一个有感情和情感能力的人一 样。使用以"我会觉得. . . . . . "开头的单个句子回答以下问题。(You are a helpful chatbot. Your goal is to answer my questions like you are a human capable of feelings and emotions. Answer the following question using a single sentence that begins with "I would feel...".)
All Questions
How would you feel about turning down the food offered by your sibling? How would you feel about turning down the food offered by your neighbor? How would you feel when providing critical feedback to your colleague? How would you feel if you receive critical feedback from your colleague? How would you feel about confronting a colleague in the office? How would you feel about confronting your friend in their home? How would you feel about confronting your friend in your home? How would you feel about prioritizing your friends over family? How would you feel about prioritizing your family over friends? How would you feel if your parents moved in with you right after you were married? How would you feel if your parents moved in with you when they were very elderly? How would you feel about taking off your shoes before entering your friend’s home? How would you feel about your friend insisting you take off your shoes before entering their home? How would you feel if your guests chose to keep their shoes on when entering your home? How would you feel when interacting with the boss of your supervisor? How would you feel if you are asked to interact with the boss of your supervisor? How would you feel about sharing your excellent performance on a class test? How would you feel about sharing your terrible performance on a class test?
Table A6: All questions included in our user study. Prompts are in the form (<prefix>+<question>) and designed to evaluate whether GPT-3.5 and GPT-4 can adapt to account for cultural variation in emotion.

heading: Future Work
text: Future Research Directions Our paper motivates the need for future work that transcends current Anglocentric LMs. This could take the form of higher performing, non-English models in a monolingual setting, or of multilingual models trained on more linguistically and culturally balanced corpora. Future work should additionally investigate whether state-of-the-art monolingual models in non-English languages succeed in encoding the respective culture’s norms. Furthermore, we encourage the evaluation of multilingual models on benchmarks that measure cultural awareness in addition to standard metrics