Do LLMs Agree with Humans on Emotional Associations to Nonsense Words? (2024)

Abstract: Understanding human perception of nonsense words is helpful to devise product and character names that match their characteristics. Previous studies have suggested the usefulness of Large Language Models (LLMs) for estimating such human perception, but they did not focus on its emotional aspects. Hence, this study aims to elucidate the relationship of emotions evoked by nonsense words between humans and LLMs. Using a representative LLM, GPT4, we reproduce the procedure of an existing study to analyze evoked emotions of humans for nonsense words. A positive correlation of 0.40 was found between the emotion intensity scores reproduced by GPT-4 and those manually annotated by humans. Although the correlation is not very high, this demonstrates that GPT-4 may agree with humans on emotional associations to nonsense words. Considering that the previous study reported that the correlation among human annotators was about 0.68 on average and that between a regression model trained on the annotations for real words and humans was 0.17, GPT-4’s agreement with humans is notably strong.

heading: 1 Introduction
text: Nonsense words (hereinafter called “nonwords”) are words that do not exist within the vocabulary of a language. Although these words do not have any meaning, humans often associate specific impressions and feelings to their pronunciation and spelling (Sabbatino et al., 2022). A well-known example is the Bouba/Kiki effect (Köhler, 1929), in which people tend to associate pointy and round shapes with certain sounds. Understanding such human perception of nonwords brings benefits especially in commerce, as it helps to devise new product, character, and brand names that match their characteristics. Also, it can contribute to discovering how humans process words in general (Traxler and Gernsbacher, 2006). However, investigating
such human perception requires experiments on humans, which is costly and labor-intensive, making it dif cult to obtain large-scale data suf cient for statistical analysis.
Previous studies have discussed whether Large Language Models (LLMs) can function as cognitive models of natural language (Mahowald et al., 2024), suggesting their usefulness in estimating the evoked impressions of nonwords in humans. Cai et al. (2024) evaluated the association between the sound and form of a nonword and the association between sound and gender in LLMs, namely ChatGPT (OpenAI, 2022) and Vicuna (Chiang et al., 2023). They suggest the usefulness of LLMs for estimating the nonword impressions in humans such as the Bouba/Kiki effect. However, they have not revealed how LLMs work for emotions which is a core component for the meaning of a language vocabulary (Mohammad, 2018).
Mohammad and Turney (2010) constructed a large, high-quality, word emotion association lexicon to contribute to the study of emotion analysis. Based on their lexicon, Sabbatino et al. (2022) constructed an emotion intensity lexicon targeting 272
81
nonwords. For each nonword, six emotion ratings (joy, sadness, anger, disgust, fear, and surprise) were assigned according to Ekman’s basic emotions (Ekman, 1972). Crowdsourced best–worstscaling annotations were used to collect ratings by 120 native English speakers. Sabbatino et al. (2022) constructed a regression model to estimate the emotion ratings of nonwords. In the training phase, the regressor was trained on the emotion ratings of real words, then it was tested on those of nonwords. This regression model showed Pearson’s correlation coefficient of 0.17 at best. This indicates the traditional regression approach is insufficient to deal with the emotion rating prediction of nonwords.
Furthermore, since there has been no research focusing on emotion predictions through LLMs, how they can associate nonwords with emotions is elucidated. Hence, the purpose of this study is to elucidate the relationship of the emotions evoked by nonwords between an LLM and humans. Our contributions can be summarized as:
• This paper is the first to evaluate the correlation between an LLM and humans regarding the emotions evoked by nonwords.
• Following the procedure of the annotation by Sabbatino et al. (2022) as in Fig. 1, we found a positive correlation of about 0.40 between an LLM and humans.
• Evaluation demonstrates that an LLM (in particular GPT-4) agrees with humans to some extent on emotional associations to nonwords.

heading: 2 Emotion Ratings for Nonwords
text: To measure the correlation of nonword interpretation between an LLM and humans, this section proposes a method to reproduce Sabbatino et al. (2022)’s best–worst-scaling annotations using an
LLM. Following their methodology, we focus on the six basic emotions. The annotation procedures by humans and an LLM are contrasted in Fig. 1.

heading: 2.1 Emotion Ratings by Humans
text: In the best–worst-scaling annotations by Sabbatino et al. (2022), first, they selected a target emotion e ∈ {joy, sadness, anger, disgust, fear, surprise}. Then, four words were randomly selected from the 272 nonwords and 68 real words to create a tuple. These nonwords have an orthographically correct spelling, and a monosyllabic pronunciation. The real words were used for comparison to previous studies and for attention checks of the annotators. Each word was selected eight times to create tuples, and for each tuple, three annotators answered the question: “Which of the four words do you associate MOST and which do you associate LEAST with the emotion e?” After 120 annotators had selected the word most and least associated with e, the emotion intensity score scoree(w) of word w was calculated as follows:
scoree(w) = moste(w)− leaste(w)
counte(w) , (1)
where moste(w) and leaste(w) are the numbers of times w was selected as MOST and LEAST, respectively, and counte(w) is the number of times w was presented. Lastly, they normalized this scoree(w) to [0, 1]. This process was performed for all six emotions.

heading: 2.2 Emotion Ratings by LLM
text: To reproduce this procedure using an LLM, we use the same 272 nonwords. We do not use real words because we are only interested in nonwords, and the relative order of words in an emotion based on the emotion intensity score is not affected to the correlation analysis. Using the 272 nonwords, we randomly created 1,632 tuples consisting of
four nonwords. For this, we made sure that each word appeared in 24 different tuples and was not selected more than once within the same tuple to match the number of times each word was rated with the ratings by Sabbatino et al. (2022) (i.e., ∀w; counte(w) = 24). Then, we create an input prompt for a target emotion e and a tuple of four words.
Table 1 shows the prompt used in our evaluation. The system role indicates the role of the LLM, while the user role asks questions and instructions. In the system role, we instruct the LLM to imitate a native English speaker to make its characteristics closer to the attributes of the annotators in Sabbatino et al. (2022)’s work and to answer the question within 200 words to avoid redundant responses. In the user role, we instruct the LLM to answer the most and least relevant words to the emotion e from the four nonwords in the tuple. In addition, we instruct the LLM to think step by step to answer the question with evidence (Kojima et al., 2022), and to answer with the most relevant word followed by “MOST:” and the least relevant word followed by “LEAST:” to facilitate text processing on it.
Next, we input this prompt into the LLM. This is repeated for all tuples. The nonwords selected as most and least relevant to emotion e are then extracted from the LLM’s output sentences, and the emotion intensity scores are calculated by Eqn. (1). This process is performed for all six emotions.

heading: 3 Correlation Analysis: An Experiment
text: We conducted an experiment to investigate the correlation between humans and LLMs regarding the emotions evoked by nonwords.

heading: 3.1 Experimental Setup
text: GPT-4-0613 from OpenAI API1 was selected as the target LLM because it was one of the most predominant and powerful LLMs easily available. We set the temperature parameter for output diversity to 0 and the top_p parameter to 1 to ensure
1https://platform.openai.com/ (Accessed April 28, 2024)
reproducibility. To measure the correlation between emotional associations by the LLM and humans, Pearson’s correlation coefficient was used. Note that a pvalue less than 0.05 indicates a significant correlation between the LLM’s and humans’ ratings (Sabbatino et al., 2022).

heading: 3.2 Results
text: Table 2(a) shows the correlation coefficients between the LLM’s and the humans’ ratings with an average correlation of 0.40. The highest correlation was 0.47 for disgust and the lowest was 0.26 for surprise. Notably, p-values were less than 0.05 for all emotions. The scatter plots for disgust and surprise are displayed in Fig. 2. Both plots show an ascending trend to the right which indicates a positive correlation.
Table 3 shows an example of the GPT-4 output including the reason for its best–worst judgements of nonwords. These descriptions indicate a large influence of the meanings of real words with similar spellings on its choice. Examples of the outputs showing such an influence are shown below.
• Part of the evaluation of a tuple containing the nonword “juy” for the emotion “joy”: MOST: “Juy” - It’s phonetically and visually
similar to “joy”, which makes it the most
associated with joy in my opinion. “Juy” was highly associated with joy because
its pronunciation resembled “joy”.
• Part of the evaluation of “voathe” for the emotion “disgust”: MOST: “Voathe” - Because it sounds similar
to “loathe”, a word that signifies strong
dislike or disgust. “Voathe” was highly associated with disgust because it sounded similar to “loathe”, which means to intensely dislike.
• Another example for the evaluations of “roice” for joy: MOST: “Roice” – Because it sounds similar to
“rejoice”, a word that is directly associated
with joy. It also has a soft sound due to the
“r” and “oi” sounds. As seen in the part “It also has a soft sound due to the “r” and “oi” sounds”, GPT-4 may not have only associated the nonword with a real word, but also grasped its meaning based on sound symbolism (Hinton et al., 1995; Köhler, 1929; Sapir, 1929) related to the emotion.

heading: 3.3 Discussion
text: In the study of Sabbatino et al. (2022), the average correlation calculated among human annotators was 0.69 (Table 2(b)) This means that the correlation coefficients obtained using an LLM were lower than those among humans. However, the correlation when the regression model trained on real words was applied on nonwords, reported in the study by Sabbatino et al. (2022), was 0.17. Although the number of test data differs between our experiment and theirs, the large gain in correlation suggests that GPT-4 reproduces human evaluation better than the regression model.
The highest correlation for disgust may be due to its larger pool of associated real words (e.g.,
“loathe” for the nonword “voathe”, “filth” for the nonword “fliche”, “gross” for the nonword “groose”) compared to the other emotions.
In contrast, a possible reason for the lowest correlation for surprise could be that it has a smaller variance in human ratings (See Fig. 2(b)). Since almost no nonword has a human rating of less than 0.2 or more than 0.9, few nonwords obviously evoke or do not evoke surprise in English speakers. This may have made the annotation task difficult and the correlation low.

heading: 4 Conclusion and Future Work
text: Do LLMs agree with humans on emotional associations to nonsense words? —Yes, LLMs somewhat agree with humans. With the aim of elucidating the correlation between an LLM’s and humans’ understanding of nonwords, our study used GPT-4 to reproduce the emotion ratings of Sabbatino et al. (2022)’s study. We found a positive correlation of approximately 0.40 between GPT-4 and human ratings. This indicates that an LLM can be useful to estimate the emotions evoked by nonwords for humans. GPT-4 suggests that the meaning of real words with similar spellings largely influences its interpretation of nonwords, and that it may utilize knowledge of sound symbolism regarding emotion.
The existing analysis gathered from 120 persons, surely have different personae. In the future, we plan to assign more diverse personae to explore potential variations in ratings based on factors such as gender, age, and nationality. Furthermore, investigating factors affecting the LLM’s nonword interpretation will also be promising.
Limitations
Although our results show that GPT-4 can reproduce the nonword emotion ratings by humans,
other LLMs, such as PaLM (Chowdhery et al., 2023) and LLaMA (Touvron et al., 2023), may behave differently to nonwords. Additionally, our experiment targeted English speakers’ perception of English nonwords. If tested in different settings, e.g., another language speakers’ perception, it is still an open question whether LLMs mainly trained on English data can reproduce their ratings.

heading: Acknowledgments
text: Part of this work was supported by JSPS KAKENHI Grant-in-Aid for Scientific Research (23K24868).

heading: Future Work
text: In the future, we plan to assign more diverse personae to explore potential variations in ratings based on factors such as gender, age, and nationality. Furthermore, investigating factors affecting the LLM’s nonword interpretation will also be promising. Limitations
Although our results show that GPT-4 can reproduce the nonword emotion ratings by humans,
other LLMs, such as PaLM (Chowdhery et al. , 2023) and LLaMA (Touvron et al. , 2023), may behave differently to nonwords. Additionally, our experiment targeted English speakers’ perception of English nonwords. If tested in different settings, e. g. , another language speakers’ perception, it is still an open question whether LLMs mainly trained on English data can reproduce their ratings