How Do Moral Emotions Shape Political Participation? A Cross-Cultural Analysis of Online Petitions Using Language Models (2024)

Abstract: Understanding the interplay between emotions in language and user behaviors is critical. We study how moral emotions shape the political participation of users based on cross-cultural online petition data. To quantify moral emotions, we employ a context-aware NLP model that is designed to capture the subtle nuances of emotions across cultures. For model training, we construct and share a moral emotion dataset comprising nearly 50,000 petition sentences in Korean and English each, along with emotion labels annotated by a fine-tuned LLM.1 We examine two distinct types of user participation: general support (i.e., registered signatures of petitions) and active support (i.e., sharing petitions on social media). We discover that moral emotions like other-suffering increase both forms of participation and help petitions go viral, while self-conscious have the opposite effect. The most prominent moral emotion, other-condemning, led to polarizing responses among the audience. In contrast, other-praising was perceived differently by culture; it led to a rise in active support in Korea but a decline in the UK. Our findings suggest that both moral emotions embedded in language and cultural perceptions are critical to shaping the public’s political discourse.

heading: 1 Introduction
text: Moral emotions influence group judgments and behaviors on social issues and further impact political participation (Van Bavel et al., 2023). They drive individuals to collectively act on common issues, deter actions, polarize groups, and can lead to extremism (Inbar et al., 2012; Brady et al., 2021; Finkel et al., 2020). Their influence extends beyond the offline realm and affects political discussion on social media (Brady et al., 2020; Van Bavel et al.,
*Equal contribution. †Corresponding authors. 1https://github.com/Paul-scpark/Moral-Emotion
2023). As online platforms connect vast networks of people, understanding how different types of moral emotions expressed through language affect user actions has become crucial.
Online petitions are excellent data sources for studying the role of moral emotions on user response because they record the motivations, sentiments, and behaviors of individuals who engage in collective action. We use cross-cultural data to answer the following questions: (1) How can we systematically measure moral emotions? (2) Which moral emotions most influence users’ political participation? (3) Do moral emotions have similar effects across cultures? We collected data from two petition websites with similar designs and social media functions: South Korea’s Blue House National Petition and the UK’s Government and Parliament Petitions. By utilizing two datasets, we can test the cross-cultural generalizability of our findings on the role of moral emotions in political participation.
This paper presents a 5-step framework for analyzing moral emotions (Figure 1) and shares a dataset of nearly 50,000 sentences in Korean and English each. We consider broader emotion categories than previous works (Brady et al., 2017; Solovev and Pröllochs, 2022) and include: othercondemning, other-praising, other-suffering, selfconscious, neutral, and non-moral emotion (see Table 1). This dataset was labeled by GPT that has been fine-tuned to learn from human annotators. Because GPT models can be expensive for labeling large datasets, we trained open-source language models like BERT and ELECTRA for label predictions. We constructed language-specific versions of these models to reflect socio-cultural traits better, as suggested by Havaldar et al. (2023). Our framework, which labels data using a combination of GPT and human annotation to train lightweight language models, can be reused for other low-resource languages and previously unseen tasks.
16274
Two types of user actions are examined. The first is the number of signatures on petitions (called general support here); this represents a direct political action that grants the signatory the right to receive a government response or even have the petition discussed in a legislative setting. The second is the number of shares of petition information originating from the official government website through the social media share button (called active support). The latter form represents a more substantial commitment because it makes the sharer’s public ID visible over the network (Kim and Yang, 2017; Proskurnia et al., 2017). By these definitions, our work seeks to understand the impact of moral emotions on “direct” political participation, as opposed to studying general political discourse on social media and the likes or retweets of such postings.
Our results point to an exciting interplay between moral emotions and political participation. We discover that emotions like other-suffering that appeal to compassion and sympathy positively correlate with both forms of political actions. Emotions like self-conscious that emphasize feelings of public shame and guilt have the opposite effect, substantially reducing participation. Other-condemning, which blames others, polarized the audience; this emotion negatively correlates with actual signatures but positively correlates with social media sharing. In contrast, other-praising shows mixed patterns; while it negatively correlates with signatures in general, it led to a rise in social media sharing in Korea but a decline in the UK.
We discuss the role of specific moral emotions in shaping political dynamics: other-suffering in fostering consensus and other-condemning in driving polarization. This perspective has implications for the political persuasiveness of large language models (LLMs) that can generate texts infused with moral emotions, potentially influencing public opinion.

heading: 2 Related Work

heading: 2.1 Moral Emotion and Political Discourse
text: Moral emotions are key to spreading messages in political discourse on social media platforms (Brady et al., 2020; Van Bavel et al., 2023). Prior research identified their significance by using metrics like retweets. Brady et al. (2017) showed that including a single moral emotional word in a tweet on political topics can increase the retweet probability by 20%. Solovev and Pröllochs (2022) indicated the presence of emotions that “condemn” others amplified the spread of political rumors, regardless of whether they are true or false. While these studies offer fascinating insights, retweets alone cannot fully reflect the spectrum of political engagement, as they overlook the contributions of less vocal participants (Yang and Kim, 2017). Like shy supporters, such participants may engage in quiet, anonymous actions that are less visible. To address this gap, we study government-led online petitions, focusing on visible and subtle forms of political participation.

heading: 2.2 Moral Emotion Detection
text: Unlike general emotions such as happiness, moral emotions are prosocial, driven by the intention to protect and support the interests of others over self-interest, often stemming from social injustice (Haidt et al., 2003; Van Bavel et al., 2023). The theoretical framework categorizes these emotions into four distinct types: other-condemning, otherpraising, other-suffering, and self-conscious (Haidt et al., 2003). Moral emotion detection remains challenging due to the scarcity of datasets. Previous studies have used lexicon-based or word embedding approaches to identify moral emotions in texts. However, such efforts have been restricted to the English language and covered only a subset of the moral emotion categories, as detailed in Table 2.

heading: 2.3 Data Annotation Using LLMs
text: High-quality labeled data is crucial for machine learning. However, creating large-scale, highquality data requires extensive human labor, substantial cost, and time. As one way to assist the expensive task of data labeling, LLMs have been considered for their remarkable performance in various downstream tasks such as adaptation to unseen tasks (Brown et al., 2020). In particular, recent studies have investigated whether LLMs, such as GPT-3 or open-source LLMs, can reliably replace human annotation (Wang et al., 2022; Ding et al., 2023; Alizadeh et al., 2023). Some studies also propose innovative strategies to enhance the annotation quality using LLMs through data augmentation and developing effective prompt guidelines (Bansal and Sharma, 2023; He et al., 2023; Latif et al., 2023). In this paper, we contribute by constructing a new dataset labeled with LLMs, showcasing their ability to label and adapt to unseen tasks.

heading: 3 Moral Emotion Dataset

heading: 3.1 Data Preparation
text: Data Collection: We collected petition data from the Korean government archive2 and the UK Government and Parliament Petition website.3 The Korean archive recorded 459,447 petitions with 161,856,648 signatures from August 25, 2017, to
2http://webarchives.pa.go.kr/19th/www. president.go.kr/petitions/
3https://petition.parliament.uk/
May 9, 2022. The UK website logged 41,292 petitions with 47,554,399 signatures from March 2, 2020, to December 7, 2022. For comprehensive statistics, see Table 11 in the Appendix. The two platforms have common fields such as petition ID, URL, start date, end date, title, content, state (e.g., open, closed, or rejected), and the signature count. The UK platform was launched in 2015 but only displays petitions created since March 2, 2020.
Both platforms offer a ‘Share via Twitter’ function that generates tweets with the petition’s unique URL and the endorsement message (e.g., https://petition.parliament.uk/petitions/xxxxxx). By following the syntax and utilizing the Twitter API, we identified and collected all public tweets that were directly shared through the governmental petition websites. There were 251,245 tweets in Korean and 853,222 tweets in English. Data Preprocessing: Regular expressions were used to remove personal information from petition titles and contents, including email addresses, phone numbers, and special characters such as emojis. To segment text into sentences, sentence tokenization was performed using Kiwi (Lee, 2022) and PySBD library (Sadvilkar and Neumann, 2020). After removing short sentences with one or two words, we had 4,705,292 and 210,304 sentences from the Korean and UK petitions, respectively.

heading: 3.2 Human Annotation
text: We first collected human-annotated data to classify moral emotions following the method introduced in Field et al. (2022) to adapt human knowledge in the domain of moral emotions to language models. From both petition datasets, we selected approximately 700 sentences each. Annotations were received from five native speakers and citizens of each country who understand the political context. Annotators were given guidelines on the
moral emotion definitions, reflecting the theoretical framework established by Haidt et al. (2003). They were instructed to choose multiple responses if they detected more than one emotion category. If annotators captured an emotion that did not fit the predefined categories, they selected ‘Emotional but not equivalent to the above’. Additionally, we included a ‘Difficult to distinguish (Hard to tell)’ checkbox for ambiguous cases. Please see Figure 4 in the Appendix for the guideline. Only sentences with a majority vote (at least three out of five annotators) were considered ground truth labels, and sentences with no consensus were excluded.
The final human-annotated dataset consists of 640 sentences each for Korean and English, with the distribution presented in Appendix Figure 5. Table 4 shows the inter-annotator agreement score of the final human-annotated dataset. The observed discrepancy in agreement can be attributed to the differential propensity for multi-label responses between Korean (95.52% single-label) and English annotators (79.41% single-label).

heading: 3.3 LLM-based Annotation
text: Two primary methods are used to perform unseen tasks. First is in-context learning or few-shot learning that enables a model to learn based on a few training samples within prompts. Second is finetuning which updates the model’s weight parameters. Fine-tuning requires a large training dataset, whereas it can learn from more examples than can fit in the context window (Brown et al., 2020).
To determine a better-performing approach among the two options, we tested the in-context learning and fine-tuning prompts using the Chat Completions API (OpenAI, 2023). Each format includes 1) short definitions of moral emotion categories, 2) text instances from the human-annotated training dataset, and 3) labels of the corresponding sentences. We describe the detailed experimental setting in Section B.1 in the Appendix.
We split the human-annotated dataset into 300 and 340 samples for the training and testing sets. Table 5 summarizes the comparison of performance and costs. This table also shows the result of human annotation, comparing the accuracy of each human annotator’s responses against the majority vote. The cost column indicates the estimated expense for labeling the entire 50,000 samples. We estimated the cost of human annotation from the Google Cloud Platform and used the pricing model based on the number of words.4 Although Korean sentences typically contain fewer words, processing Korean texts with GPT approximately doubles the expense compared to English due to higher tokenization costs. In-context learning and fine-tuning methods cost substantially less than human annotation, which includes the combined cost of multiple annotators per task. The fine-tuning cost calculation includes both training and inference expenses.
The results of the GPT-3.5 in-context learning experiment indicate that performance improves with the training samples. However, the fine-tuned model consistently outperformed the in-context learning models in all settings, most likely because fine-tuning allows training on more examples than what can be accommodated within via prompting. Fine-tuned GPT-3.5 achieved comparable performance to that of human annotators. GPT-4 fewshot experiment also showed high performance, but
4https://cloud.google.com/ai-platform/ data-labeling/pricing#labeling_costs
the fine-tuned GPT-3.5 model yielded more costeffective and better-performing results. For this reason, we chose the fine-tuned GPT-3.5 version as our annotator model. Additional experiments are included in the Appendix Section B.

heading: 3.4 Dataset Description
text: Data Selection: In preparation for labeling with our fine-tuned GPT-3.5, we curated petition sentences using methods inspired by GoEmotion (Demszky et al., 2020). Our initial dataset consisted of approximately 4.7 million Korean and 210K UK petition. To ensure these sentences reflected societal engagement, we selected those with at least one signature and share. We also sought a balanced distribution in sentence length, choosing sentences between 3 and 30 tokens with the aid of the NLTK word tokenizer (Bird et al., 2009). To aim for a balanced representation of emotions in our dataset, we focused on reducing bias by trying to even out the distribution of sentences across emotion categories. We employed a pilot model trained on human-annotated examples to estimate the emotional content of petition sentences. This approach helped us identify and initially extract 5,000 sentences for each emotion label, creating a set of 30,000 sentences. Then, 20,000 sentences were randomly selected to achieve 50,000 samples. After removing duplicates and samples that were incorrectly labeled by the GPT’s inference process (e.g., Overall Condemning), our dataset was finalized with 49,930 Korean and 49,896 English sentences.
Data Statistics: Figure 2 shows the distribution of moral emotion labels. We make four observations. First, other-condemning is the most prevalent moral emotion, taking up one-third in English and
one-fourth in Korean. This moral emotion is more common than general emotions (i.e., non-moral emotions). Second, other-suffering is the next frequently used moral emotion, with UK petitions exhibiting nearly twice the exposure (19.0%) compared to Korea (10.1%). Third, other-praising and self-conscious make up a small proportion of moral emotions. Fourth, petitions contain a substantial proportion of neutral sentences, which may offer factual statements to support the petition.

heading: 4 Political Participation Analysis
text: We built a classifier from our fine-grained moral emotion dataset to analyze the entire collection of petitions and performed regression analysis to assess the impact of these emotions on political participation.

heading: 4.1 Moral Emotion Measurement
text: Moral Emotion Classifier: Our analysis, which aims to measure moral emotion in online petition data using a Transformer-based model, utilizes the approach suggested by Wang et al. (2021). This work has established that compact, open-source lan-
guage models, when trained with LLM-generated labels, can outperform the raw LLM while also reducing costs. Employing open-source language models like BERT, RoBERTa, and ELECTRA, pretrained on Korean (Park et al., 2021; Lee, 2021, 2020) and English corpora (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020), we analyzed petition data across these languages. Each model was added with a fully connected classification layer and fine-tuned on a GPT-labeled moral emotion dataset. For multi-label prediction, we applied binary cross-entropy loss, set the learning rate at 2e-5, and trained the models for up to four epochs with early stopping.
The models were evaluated on the humanannotated dataset, and the results are presented in Table 6. Here we also report ensemble models. On the macro F1 score, we observe that small models like ELECTRA perform well compared to the finetuned GPT. We chose the weighted ensemble with the best F1 score as our final model.
Moral Emotion Score: We employed weighted ensemble models for both countries to compute petition’s moral emotion score. For tokenized petition sentences, emotion scores were predicted as sixdimensional vectors with sigmoid outputs ranging from 0 to 1, and the average of these values determined the final score for each petition document.

heading: 4.2 Regression Model Specification
text: We separately estimated two count variables (i.e., signatures and social media shares) using negative binomial regressions against the emotion variables and control variables. All overdispersion parameters were significant at the α = .01 level. Here, only the four moral emotions and neutral were used as emotion categories; non-moral emotion was excluded due to its low F1 score for English (see Figure 7 in the Appendix). The control variables
port for emotions ( positive , negative ). For complete regression outcomes, refer to Appendix Table 15.
included information about the text length, URL usage, and time information. Temporal information was added in the regression to account for yearspecific and seasonal variability.

heading: 4.3 Regression Result
text: Table 7 summarizes the regression results demonstrating the effects of moral emotion on two count variables after adjusting for control variables. The color background represents the prominent direction of correlation, which shows both positive and negative directions. The two dependent variables of signatures (i.e., general support) and social media shares (i.e., active support) themselves have a positive correlation (Korea: Pearson’s r = 0.49, p < 0.001; UK: r = 0.67, p < 0.001). In both countries, other-suffering and self-conscious appear to drive these results. Other-suffering, positively correlated with both forms of support, effectively secured the number of signatures and shares. In contrast, selfconscious is negatively correlated, implying a reduction in political support for both types.
Although petitions with many signatures tend to be shared more frequently on social media, moral emotions could explain the subtle response patterns of users. For example, increase in the othercondemning emotion negatively correlates with general support and positively with active support in the two countries. Such an inverse trend may reflect nuanced divisions among the supporter groups, even for the same petition.
Figure 3 shows the trends of the count variables (y-axis) across the studied emotions (x-axis) for Korea (top row) and the UK (bottom). Patterns with opposing regression trends have a crossing sign, such as other-condemning, neutral (Korea), and other-praising (Korea). In these cases, active support correlates positively with the corresponding emotion, whereas general support correlates negatively. This finding suggests that active support
does not consistently translate into general support in the presence of a specific emotion and culture.
Other-praising shows different results by culture. The number of shares in Korea increases with this emotion, whereas it decreases in the UK. Conversely, the number of signatures decreases in both countries with emotion, indicating that the enhancement of universal support is not consistent. The correlation between social media shares and otherpraising in Korea mirrors the dynamics seen with other-condemning. In the UK, it aligns with selfconscious emotion, reducing support levels.

heading: 5 Discussion

heading: 5.1 Implications for Social Science
text: Political and moral psychology research has identified moral emotions as politically motivating through metrics like retweets (Brady et al., 2017; Solovev and Pröllochs, 2022). Our study expands the literature to encompass comprehensive moral emotion categories and two direct forms of political participation: general and active support seen in government-led online petitions. Additionally, our research extends cross-cultural insights by analyzing petitions from multiple countries, addressing the urgent need for broader cultural analysis in this domain (Van Bavel et al., 2023).
Prior studies have focused on the role of othercondemning emotions in social sharing (Brady et al., 2021; Solovev and Pröllochs, 2022). However, our findings highlight the significant role of other-suffering in amplifying both active and general support in two distinct countries. These results suggest that the expression of other-suffering harmonizes the perspectives of both dedicated and gen-
eral supporters, fostering widespread consensus on petitions addressing social issues (Sirin et al., 2016, 2017). This implies that policymakers and activists could cultivate a more engaged and unified public discourse by leveraging this emotional dynamic.
Aligning with past research, our study indicates that other-condemning may boost active support through social media sharing. Additionally, our analysis shows that other-condemning will likely diminish petition signatures, illustrating a polarization effect. This effect lowers the broader base of general supporters’ willingness to engage but also sharply increases participation among a more dedicated segment. This observation aligns with earlier research discussing other-condemning as a catalyst for political polarization (Crockett, 2017; Finkel et al., 2020; Brady et al., 2021). Thus, our study highlights the complex nature of online political participation: while other-condemning can enhance issue visibility on social media, it also poses deepening societal divisions. Interestingly, neutral, the absence of emotional engagement, also acts as a polarizing force, encouraging active support but not broadening general support. This indicates that political polarization can manifest in both emotional and rational forms (Singer et al., 2019).
Our cross-cultural data analysis reveals distinct cultural impacts on the role of other-praising in political participation between Korea and the UK. According to the WVS Inglehart-Welzel World Cultural Map 2023, Korea has lower self-expression values (-0.47) emphasizing in-group cohesion, while the UK with higher values (2.24) reflects a societal norm of more tolerance towards outgroups (Haerpfer et al., 2022). In political contexts, expressing other-praising often enhances the rep-
utation of one’s in-group and reinforces internal unity (Brady et al., 2020). Consequently, this emotion may create a clear division between in-groups and out-groups (Brady et al., 2020).
In cultures valuing in-group cohesion like Korea, promoting petitions with other-praising emotion on social media can emphasize the in-group’s prestige and strengthen belonging, delineating a clear divide between in-groups and out-groups. Considering cultural backgrounds and emotional attributes provides a compelling explanation for the role of other-praising expressions in Korean online petitions in contributing to polarized support tendencies. Conversely, in the UK emphasizing tolerance towards out-groups, such expressions might fail to garner support and provoke antipathy. This finding reiterates the call for research that considers cultural variation in moral emotions (Haidt et al., 2003; Malti and Keller, 2010; Van Bavel et al., 2023).

heading: 5.2 Implications for AI Community
text: We measure moral emotions cost-effectively, leveraging fine-tuned GPT-3.5 to inherit human knowledge, significantly reducing annotation costs. Further cost reductions in inference are achieved through the use of open-source language models. Our strategy led to the development of a classifier proficient in identifying moral emotions within both Korean and English texts. This is a notable achievement, considering the complexity of moral emotion classification and its application to the niche area of online political petitions. Applying this method opens up possibilities for reuse in languages with fewer resources and in tasks that previously faced challenges due to the high costs associated with developing domain-specific data.
Using real-world data, we also analyzed how moral emotions in language influence political actions. Our insights offer valuable implications for the AI community, which is increasingly interested in understanding the mechanics of political persuasion through content (OpenAI, 2024; Bai et al., 2023). We confirmed through our experiment that LLMs can understand and classify moral emotions, even from a limited sample of sentences. These discoveries prompt future work into the potential of generative AI in crafting content that may influence public opinion (Bai et al., 2023). For instance, the ability of generative AI to quickly generate content expressing other-condemning emotion presents a risk of polarizing public discourse and deepening societal divisions (Coeckelbergh, 2022).

heading: 6 Conclusion
text: This study proposed a 5-step framework for analyzing moral emotions and their effects on political participation, leveraging cross-cultural data from online petitions. Our framework addresses research gaps using a comprehensive Korean and English moral emotion dataset and is adaptable to lowresource languages and topic domains. The dataset, annotated with fine-tuned GPT-3.5, enables training language-specific transformer models, allowing for the precise quantification of moral emotions within the petitions. Our analysis reveals that othersuffering enhanced both general and active political participation. In contrast, other-condemning led to polarization in these cultural contexts. Patterns of other-praising by countries underscore the cultural difference of moral emotions’ influence on political engagement. The discussion of these findings, particularly the pronounced effects of petitions on specific moral emotions, provides valuable insights into the fields of social science and AI.

heading: 7 Limitation
text: Inherent Biases in LLMs: LLMs, known to reflect Western-centric biases (Atari et al., 2023; Tao et al., 2023), are mitigated by our methodology of fine-tuning GPT with a significant number of human-annotated datasets compared to in-context learning. This approach reduces the reliance of the LLM on its pre-trained biases, aligning it more closely with human judgment. This effort is reflected in the dramatic increase in F1 scores for Korean moral emotions, which soared from 0.5810 to 0.8678, exceeding the enhancements observed in English annotations (0.6111 to 0.7426), as reported in Table 5. Despite these advances, it is important to acknowledge that our method may not completely neutralize the Western-centric norms embedded within LLMs. Users of this dataset should remain aware of the potential cultural biases. Inter-Annotator Agreement: We acknowledge that the inter-annotator agreement for our English moral emotion dataset is at best fair to good (Fleiss’ kappa value ranging between 0.40 and 0.75), according to the guidelines outlined by Fleiss et al. (2013). This result suggests significant room for future improvement. In response, we have made the human-annotated dataset publicly available on GitHub and have conducted experiments to enhance agreement. By increasing the consensus threshold from three to four annotators, we raised
the Fleiss’ kappa to 0.55; however, this adjustment simultaneously reduced the dataset size from 640 to 409 samples. Future work can improve inter-annotator agreement by expanding the dataset or experimenting with alternative approaches to consensus-building.
Data Scope and Cultural Representation: Our dataset, aimed at capturing cross-cultural nuances, may not fully represent moral emotions beyond Korea and the UK. The data collected from government-led petitions might exhibit biases inherent to this context. Such biases manifest in the expression of moral emotions and in shaping public opinion to prompt government action.
Correlational Analysis: This observational study highlights correlations without confirming causality, suggesting the need for future experimental studies to explore the causal effects of moral emotions in language on political participation. Future research can seek to validate our findings through more rigorously controlled laboratory experiments.
Misuse Potential: Here, we designed prompts to train LLMs as annotators and constructed a dataset enriched with various moral emotion categories. However, we acknowledge the potential for modifying prompts to enable language models to generate texts infused with specific moral emotions. As discussed in Section 5.2, texts infused with moral emotions could potentially be misused to sway public opinion for specific political agendas. Therefore, we stipulate that the prompts and examples in the moral emotion dataset should be utilized solely for research purposes.

heading: Acknowledgements
text: The authors would like to thank all anonymous reviewers for their constructive feedback. We would also like to thank the annotators for their hard work. Kim, Park, and Lee were supported by funding from the BK21 Plus Postgraduate Organization for Content Science. Cha was supported by the National Research Foundation of Korea grant (RS2022-00165347).

heading: A Human Annotation
text: A.1 Human Annotation Guideline
Korean petition annotations were conducted via Google Surveys, while the UK petition utilized a POTATO data annotation tool web interface (Pei et al., 2022). Figure 4 displays our Korean and English guidelines given to the users. Regarding compensation for the annotation task, each of five Korean annotators was paid ₩70,000, leading to a total of ₩350,000. On the other hand, each of the five UK annotators received £37.5, along with an additional Prolific service fee of £62.5, resulting in an overall expenditure of £250.
A.2 Human Annotation Dataset Result
We obtained 640 human-annotated data for the Korean petition, comprising 619 single-label and 21 multi-label instances. The distribution of emotion labels was as follows: other-condemning (136), other-praising (113), other-suffering (110), selfconscious (87), neutral (97), and non-moral emotion (118). Similarly, we acquired 640 humanannotated examples for the UK petition, consisting of 590 single-label and 50 multi-label instances. The distribution of the labels is as follows: othercondemning (226), other-praising (111), othersuffering (141), self-conscious (25), neutral (68), and non-moral emotion (119).
A.3 Inter-Annotator Agreement Score
Table 8 presents the inter-annotator agreement score for each class in a human-annotated dataset, measured by Cohen’s kappa (C), Fleiss’ kappa (F), and Krippendorff’s alpha (K). The other-praising shows the highest score in both Korean and English, in contrast to non-moral emotion, which achieves lower score. The average scores across all emotion categories provide a comprehensive view of the annotation reliability, with the Korean dataset exhibiting a higher average agreement score than the English dataset.

heading: B LLM Annotation
text: B.1 Prompt Format
Each prompt and fine-tuning data is constructed in a chat format, consisting list of messages from the “system”, “user”, and “assistant”. Both formats start with short definitions of moral emotion categories. While the few-shot learning prompt contains N number of training samples and human-annotated labels, fine-tuning training examples include a single training sample and corresponding labels per chat object. Finally, the few-shot learning prompt provides a task to label an unlabeled sentence in a multi-label manner.
B.2 Monolingual vs Bilingual Fine-tuning
To compare the performance of monolingual and bilingual training approaches, we first fine-tuned GPT-3.5 models on separate Korean and English single-language datasets, creating two monolingual models. Subsequently, we fine-tuned another model on a combined bilingual dataset. Training on monolingual corpora may provide a deeper understanding of each language’s unique nuances and socio-cultural contexts. Conversely, joint training could enhance multilingual understanding and performance by facilitating knowledge transfer across languages. Table 9 presents the training dataset size for each experiment along with the models’ macro F1 scores and accuracy. The models achieved the best performance in Korean and English datasets under the monolingual condition when the training size was N = 300.
B.3 Fine-tuning Train Dataset Size
We conducted experiments with different training set sizes ranging from 50, 100, 150, · · ·, to 300 to optimize the train data size and improve fine-tuned model performance. Figure 6 depicts the model macro F1 score and accuracy as a function of finetuning dataset size. The performance tends to improve as the train dataset size grows. The Korean model shows the highest F1 score and accuracy when the training dataset size N = 300, while the English model performs best when N = 250.

heading: C Moral Emotion Dataset
text: C.1 Moral Emotion Dataset Statistics
The basic statistics and features are represented in Table 10. The moral emotion dataset comprises 49,930 Korean and 49,896 UK petition samples. Both datasets contain six classes: four pertaining to moral emotions (other-condemning, other-praising, other-suffering, self-conscious) and two to nonmoral emotions (neutral, non-moral emotion). Most of our data samples are single-labeled, accounting for 99.93% in the Korean and 99.99% in the UK.
C.2 Moral Emotion Classifier Performance
Figure 7 presents the per-class F1 scores for Korean and English weighted ensemble classifier models, noting that the F1 score for the non-moral emotion class in the English model is below 0.7.

heading: D Petition Dataset Statistics
text: The basic statistics of collected petitions are shown in Table 11.
Table 12 demonstrates additional details of dependent variables. Our data exhibit a significant positive correlation between the number of signatures and shares at a p-value below 0.001 for both countries (Korea: Pearson’s r = 0.49, UK: Pearson’s r = 0.67), supporting the intuitive assumption that petitions with more signatures tend to be shared more frequently.

heading: Future Work
text: This result suggests significant room for future improvement. In response, we have made the human-annotated dataset publicly available on GitHub and have conducted experiments to enhance agreement. By increasing the consensus threshold from three to four annotators, we raised
the Fleiss’ kappa to 0. 55; however, this adjustment simultaneously reduced the dataset size from 640 to 409 samples. Future work can improve inter-annotator agreement by expanding the dataset or experimenting with alternative approaches to consensus-building. Data Scope and Cultural Representation: Our dataset, aimed at capturing cross-cultural nuances, may not fully represent moral emotions beyond Korea and the UK. The data collected from government-led petitions might exhibit biases inherent to this context. Such biases manifest in the expression of moral emotions and in shaping public opinion to prompt government action. Correlational Analysis: This observational study highlights correlations without confirming causality, suggesting the need for future experimental studies to explore the causal effects of moral emotions in language on political participation. Future research can seek to validate our findings through more rigorously controlled laboratory experiments. Misuse Potential: Here, we designed prompts to train LLMs as annotators and constructed a dataset enriched with various moral emotion categories. However, we acknowledge the potential for modifying prompts to enable language models to generate texts infused with specific moral emotions. As discussed in Section 5. 2, texts infused with moral emotions could potentially be misused to sway public opinion for specific political agendas. Therefore, we stipulate that the prompts and examples in the moral emotion dataset should be utilized solely for research purposes