Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis
Authors: Daniela Teodorescu
Saif M. Mohammad
Venue: EMNLP 2023 Findings (2023)
Keywords: Emotion Arcs
Sentiment Analysis
Low-Resource NLP
Multilingual
Emotion Lexicons
TL;DR: We for the first time evaluate generated emotion arcs--which are widely used in industry and research--across languages (in English, middle- and low-resource languages), demonstrating that simple and interpretable methods achieve high correlations.

Abstract: Emotion arcs capture how an individual (or a population) feels over time. They are widely used in industry and research; however, there is little work on evaluating the automatically generated arcs. This is because of the difficulty of establishing the true (gold) emotion arc. Our work, for the first time, systematically and quantitatively evaluates automatically generated emotion arcs. We also compare two common ways of generating emotion arcs: Machine-Learning (ML) models and Lexicon-Only (LexO) methods. By running experiments on 18 diverse datasets in 9 languages, we show that despite being markedly poor at instance level emotion classification, LexO methods are highly accurate at generating emotion arcs when aggregating information from hundreds of instances. We also show, through experiments on six indigenous African languages, as well as Arabic, and Spanish, that automatic translations of English emotion lexicons can be used to generate high-quality emotion arcs in less-resource languages. This opens up avenues for work on emotions in languages from around the world; which is crucial for commerce, public policy, and health research in service of speakers often left behind. Code and resources: https://github.com/dteodore/EmotionArcs

Evaluating Emotion Ares Across Languages: Bridging the Global Divide in Sentiment Analysis
Daniela Teodorescu 1.2- and Saif M. Mohammad
MMaiNLP, Center for Information and Language Processing, LMU Munich, Germany DDepartment of Computing Science, University of Alberta National Research Council Canada
daniela@cis uni-muenchen. de, saif. mohammadenre-cnrc.ceo .c.ca
Abstract
health interventions such mask mandates and vaccine policies (Hu et al., 2021). Researchers in Digital Humanitics are interested in understand- ing basic components of stories such as plot struc- tures, how emotions are associated with compelling characters, categorizing stories based on emotion changes (Reagan et al., 2016), etc. In all of these applications, the goal is determine whether the degree of a chosen emotion has remained steady, increased, or decreased from one time step the next. The time steps consideration may be days, weeks, years, etc. This series of time step-emotion value pairs, which can be represented as time- series graph, is often referred to as an emotion are (Mohammad, 201 I; Reagan et al., 2016).
Emotion ares capture how an individual (or population) feels over time. They are widely used in industry and research; however, there is little work evaluating the automatically generated ares. This is because of the diffi- culty of establishing the true (gold) emotion Our work, for the first time, systemat- ically and quantitatively evaluates automati- cally generated emotion ares. also com- pare two common ways of generating emo- tion ares: Machine-Learning (ML) models and Lexicon-Only (LexO) methods. By running ex- periments on 18 diverse datasets in 9 languages, we show that despite being markedly poor at in- stance level emotion classification, LexO meth- ods are highly accurate generating emotion ares when aggregating information from hun- dreds of instances. We also show, through ex- periments on six indigenous African languages, well as Arabic, and Spanish, that automatic translations of English emotion lexicons can be used to generate high-quality emotion ares in less-resource languages. This opens up avenues for work on emotions in languages from around the world; which crucial for commerce, pub- lie policy, and health research in service of speakers often left behind. Code and resources: https://githul: http:///gthuu.cco//ttteeeeeee htmm//ttoooore/Emtioorreee
Automatic methods generate emotion ares using:
The text of interest where the individual sen- tences (or instances) are temporally ordered; possibly through available timestamps indicat- ing when the instances were posted/uttered: e.g., all the tweets relevant to (or mentioning) a government policy along with their meta- data that includes the date and time of posting.
The emotion dimension/categgry of interest: e.g., anxiety, anger, valence, arousal, etc.
The time step granularity of interest (e.g., days, weeks, months etc.)
1 Introduction
The ares are generated through two-step process:
Commercial applications well research projects often benefit from accurately tracking the emotions associated with an entity over time. Pub- lie health researchers are interested in analyzing social media posts to better understand population- level well well-being (Vishnubhotla and Mohammad, 2022), loneliness (Guntuku et al., 2019), depres- sion (De Choudhury et al., 2013), ctc. Government Policy makers benefit from tracking public opin- ion over time for developing effective interventions and laws. For example, tracking sentiment towards
Apply emotion labels to units of text. Two common approaches for labeling units of text are: (1) The Lexicon-Only (Lex0) Method: to label words using emotion lexicons and (2) The Muchine-Learning (ML) Method: to label whole sentences using supervised ML models (with without using emotion lexicons).
Aggregate the information compute time step-emotion value scores; e.g., if using the lexicon approach: for each time step, compute the percentage anger words in the target text pertaining to each time step, and if
Work done while at the University of Alberta and inter- ship with the National Research Council Canada.

---

using the ML approach: for each time step, compute the percentage of angry sentences for each time step.
have benefited from using translations of lexical resources (from say English) into the target lan- guage. Thus, high-aceuracy LexO methods for creating emotion arcs can open up avenues of work in various languages using (manual or automatic) translations of existing emotion lexicons.
Despite their wide-spread use in industry and research, there is little work evaluating the gen- erated emotion ares (in part due to the difficulty of establishing the true (gold) emotion arc). This is problematic because we should know how accurate the generated emotion ares are before drawing in- ferences from them or deploying these systems in applications. Further, different methods of generat- ing emotion arcs (Lex0 or ML) have different char- acteristics. For example, LexO methods are inter- pretable, accessible, compute-friendly and do not require annotated data (especially for each domain of application) (Mohammad, 2023, 2021; Öhman, 2021); whereas ML methods tend to consider con- text and longer-range dependencies, making them more accurate at labeling individual instances for emotions. (Table 5 in Appendix A depicts the pros and cons in more detail.) Thus, it is often assumed that ML methods markedly more accurate than Lex0 methods for all tasks, including the creation of emotion ares. However, this assumption may be false; and can only be tested for the task of gener- ating emotion arcs when there is a mechanism to evaluate the arcs.
Our work, for the first time. systematically and quantitatively evaluates automatically generated emotion arcs. We also compare the ML and LexO methods for generating ares. Note, that our aim is not to build more accurate emotion classification systems, but rather we apply existing methods to evaluate emotion ares generated across domains and languages. We conducted experiments on 18 datasets in nine languages from diverse domains, including: tweets, SMS messages, reviews, etc. (Two English datasets from movie reviews, four English datasets from Semlvval 2014, six datasets from SemEval 2018 (two each in English, Arabic, and Spanish), and six African languages datasets from SemEval 2023 (AfriSenti).)
Emotions can be characterized by various dimen- sions and categories (e.g., valence, arousal, dom- inance vs. anger, fear, joy, sadness). In this work we explore the valence (or sentiment), and there- fore generate valence (or sentiment) ares. Our work sets the foundation for future work which can extend analyses to various emotion categories.
A robust and simple setup for evaluating emo- tion arcs also allows one to test different design decisions in how the arcs are generated. For ex- ample, we know little about how best aggregate information when using emotion lexicons; e.g., is better to use coarse or fine-gras grained lexicons, should we ignore slightly emotional words, etc.
Related Work
Emotion ares have commonly been created from literary works and social media content. Alm and Sproat (2005) were the first to automatically clas- sify sentences from literary works for emotions using a machine learning paradigm. Mohammad (201 1) was the first to create emotion ares and anal- yse the flow of emotions across the narrative in various novels and books using emotion lexicons. Kim et al. (2017) built on this work by creating emotion arcs to determine emotion information for various genres using the NRC Emotion Lexicon (Mohammad and Turney, 2013).2 Reagan et al. (2016) and Del Vecchio et al. (2018) clustered emo- tion ares and found evidence for six prototypical are shapes in stories. Hipson and Mohammad (2021)
Work on emotions has largely focused on En- glish, excluding the voices, cultures, and perspec- tives of a majority of people from around the world. Such an exclusion is not only detrimental to those that are excluded, but also to the world whole. For example, western-centric science that only draws on data from western languages may falsely claim certain universals about language, emotions, and stories, that when tested on data from other cul- tures does not hold true anymore. Thus, working with more languages helps discover new paradigms that represent the world better.
However, generating emotion ares in most lan- guages other than English is stymied by the lack of emotion-labeled resources. Using translated la- beled texts with MI. methods is problematic (Mo- hammad et al., 2016; Hogenboom et al., 2014). However, for a number of NLP tasks, systems
'Terminoloyy: The positive-negand (or pleasure- displeasure) dimension has long been focus study, espe- cially psychology. They refer to this dimension as valence. However, early NLP work in the area made use product and movie review datasets, and referred the dimension as sentiment - a term that has stuck in subsequent NLP work. htt //sai fmohammad. . hom/WebPages/ MRC-Emotion Lew.ioonneeei..nnnoeem htm

---

analysed emotion ares for individual characters (in- stead of the whole narrative) in movie dialogues. Emotion arcs of literary works have been used to better understand plot and character development (Reagan et al., 2016; Kim et al., 2017; Hipson and Mohammad, 2021); and also for assisting writers develop and improve stories (Ashida et al., 2021; Somasundaran et al., 2020).
speaker, or sentiment towards product or entity. For a pre-chosen bin size of say 100 instances per bin, we compute the gold emotion score by taking the average of the human-labeled emotion scores of the instances in that bin (in-line with the commerce and social media use cases discussed earlier). We move the window forward by one instance, com- pule the average in that bin, and so (Using larger window sizes does not impact conclusions.) We created text streams for the experiments Sec- tion by ordering instances from dataset by in- creasing gold score before binning. This created ares with a more consistent emotion change rate and we first explore results in this scenario. In Section 7, we look at the impact of more dynamic emotion changes (e.g., varying peak heights and widths) on performance.
Recently, Hipson and Mohammad (2021) in- troduced how the patterns with which emotions change over time - emotion dynamics - can be inferred from text. Vishnubhotla and Mohammad (2022) computed UEDs from tweets and analysed how they have changed over the years. Teodorescu et al. (2023b) studied emotional development in children's is writing and found meaningful patterns across age. In the public health domain, Teodor- escu et al. (2023a) created emotion ares for tweeters who self-disclosed as having a mental health diag- nosis (on Twitter). They found that certain emotion dynamics patterns differed significantly between tweeters with mental health conditions (e.g., at- tention deficit hyperactivity disorder, depression, bipolar) compared the control group.
We automatically generate emotion ares using the Lex0 and ML methods discussed in the two sections ahead. We standardized ares (aka z- score normalization) so that the gold ares are com- parable to automatically generated arcs.? Finally, we evaluate the closeness of automatically gener- ated emotion arcs with gold ares using two met- rics: linear correlation and Root Mean Squared Error (RMSE). We use Spearman rank correlation (Spearman, 1987) (range: -1 to 1). High correla- tion implies greater fidelity: no matter what the overall shape of the gold emotion arc, when it goes up (or down), the predicted emotion are also goes up (or down). RMSE (range: 0-co) is a measure of the error between the true and predicted values. It penalizes bigger errors more, and thus is sensi- tive to outliers. Scores closer to O indicate better predictions. In our experiments, the correlation and RMSE scores are determined for thousands of points between the predicted and gold arcs.
However, there is surprisingly little work on are evaluation. A key reason for this is that is hard to determine the true emotion are of story from data annotation. One attempt to evaluate aspects of an emotion are can be seen in Bhyravajjula et al. (2022). They asked one volunteer to read mini- segments of 'The Lord of the Rings' novel to determine whether the protagonist's circumstance undergoes a positive or negative shift. They then de- termined the extent to which the automatic method captured the same shifts. Here we propose sim- pler alternative way to robustly evaluate automati- cally generated emotion arcs on wide variety of domains and multiple languages.
Table shows key details the English instance- labeled datasets. To determine whether using au- tomatic translations of English lexicons into rela- tively
3 Experimental Setup to Evaluate Automatically Generated Emotion Arcs
languages is viable option, we also experiment with instance-labeled datasets in Arabic (Ar). Spanish (Es), Amharic (Am), Hausa (Ha), Igbo (Ig), Kinyarwanda (Kr). Swahili (Sw), and Yoruba (Yo). Just as the English set, these con- tain original tweets (not translated) with emotion
We begin by describing the evaluation setup. This is a key contribution of this work since no prior work systematically evaluates automatically gener- ated emotion arcs. We construct gold emotion ares from existing datasets where individual instances are manually annotated for valence (sentiment). Here, an instance could be a tweet, sentence from a customer review, a sentence from a personal blog, etc. Depending on the dataset, manual annotations for an instance may represent the emotion of a
Subtract the mean from the senre and divide hy the stan- dard deviation (ares then have mean and unit variance). "Different orderings of given dataset produce different shapes; however, the Spearman correlation between the auto- matic and gold ares for that dataset remains the same. For example, in dataset with 3K instances, when using rolling window of bin size 300, the are consists of 2700 points.

---

Dataset
Source
Domain
Dimension
Label Type
Instances
Movie Reviews Categorical
Sucher et al. (2013)
movie
valence
categorical (0. 1)
11.272
Movie Reviews Continuous
Socher et al. (2013)
reviews
valence
continuous to 1)
11.272
SemEval 2014
Rosenthal et al. (2014)
multiple"
valence
categorical (-1.0.1) 1)
multiple'
SemFval 2018 (V-OC) SemEval 2018 (V-Reg)
Mohammad et al. (2018)
tweets
valence
categorical (+3,22....)
2,567
Mohammad et al. (2018)
tweets
valence
continuous (0 to 1)
2,567
Table 1: Key details of the English emotion-labeled datasets used. "The SemEval 2014 dataset is a collection of 18,611 instances, including LiveJournal posts (1141), SMS messages (2082), tweets (15302), sarcastic tweets (86).
Lexicon
Source
Categories Dimensions
Label Type
l'Terms
NRC VAD Mohammad (2018)
valence, arousal, dominance
continuous (-1 to 20,007
valence, arousal, dominance
categorical (-1, 0, I)
20,007
Table Lexicons used in this study. The subset of emotions explored in our experiments are marked in bold.
labels by native speakers. The datasets are of two kinds: those with categorical labels such as the Sc- mEval 2014, which has - (negative), O (neutral), and (positive), as well as those with continuous labels such as SemEval 2018 (V-Reg), which has real-valued sentiment intensity scores between O (lowest/no intensity) and (highest intensity).
markedly better emotion arcs.
The two OOV handling methods explored were: 1. Assign label NA (no score available) and disre- gard these words, and 2. Assign O score (neutral or not associated with emotion category), thereby, leading to a lower average score for the instance than if the word was disregarded completely.
In all, we conducted experiments with 18 emotion-labeled datasets from nine languages, with labels that are either categorical or continuous, for valence. The results also establish key benchmarks; useful to practitioners for estimating the quality of the ares under various settings.
We then evaluated how closely the ares respond to the gold valence ares. In Appendix B, we show the impact of using emotion lexi- cons more ceteetvvvellusiing only the entries that have an emotion association greater than some pre- determined threshold.
Results (Valence): Figure shows the correlations and RMSE values between predicted valence ares and the gold valence arcs.
4 LexO Ares: Emotion Ares Generated from Counting Emotion Words
Bin Size: Overall, across datasets and regardless of the type of lexicon used and how OOV words are handled, increasing the bin size dramatically improves correlation with the gold arcs. In fact, with bin sizes as small as 50, many of the generated ares have correlations above 0.9. With bin size 100 and above correlations approach high 0.90's. Similarly, RMSE starts approximately 1.20 1.10 bin size 1, and quickly drops bin size increases. At bin size 300, RMSE approaches quite low scores (0.3-0.1).
Recall that in the Introduction we described how emotion arcs are automatically generated. Key pa- rameters in that process include bin size, type of emotion lexicon used (e.g., categorical or contin- uous emotion scores), and how to handle terms in the text that are not in the lexicon (OOV terms).
Choice of bin size depends the application and available data. For example, if a company wants to know how the proportion of angry com- ments has changed from month to month in the last 10 years, then the bin size to use month. instead, they want to know how things changed on a day-by-des day basis over the last 45 days, then the bin size to use is a day. Depending on the volume of relevant data and the bin size chosen, the bins may include higher or lower number of instances, For our experiments explored various bin sizes (1, 10, 30, 50, 100, 200, and 300).
If we use a bin size of 1, we only get the ups and downs of the emotions associated with individual words, which will likely be quite far off from the true emotion arc. As bin size increases (more data per bin), the predicted are gets closer to the truc are probably because:
If bin x is expressing more of an emotion (higher emotion intensity) than bin y, then x has higher average emotion association score than y.
Table shows the English emotion lexicon we used: The NRC VAD Lexicon. 6 It has both cate- gorical and real-valued versions, which is useful to study whether using fine-grained lexicons leads to
Categorical vs. Real- Laad-Valeddla Lexicons: Using a real- valued lexicon obtains higher correlations across bin sizes, methods for processing OOV terms, and
"http://saiffooaat http://mmm...a//eeee/a/////rrrer html

---

Size
Bin Size
Data
Lexicon
Test Data
Emation
00v
1
10
30
50
100
300
10
30
50
100
300
Language
Scores
Movie
Eng
valence
assigned NA
categorical
0.286
0.848
0.863
1.20
0.84
0.53
0.42
0.28
Reviews
real-valued
0.329
0.724
0.842
0.859
0.854
0.866
1.17
0.38
Categorical
assigned
categorical
0.296
0.701
0.829
0 0.852 0.851
0.862
1.18
0.81
0.61
0.54
0.45
0.31
neutral score realivalued
0.335
0 0,750
0.847
0.859 0.863
0.864
1.15
0.76
0.56
0.50
0.42
0.30
Movie Reviews
Eng
valence
assigned NA
catagorical
0.326
0.733
0.881
0.917
0.951
0:980
1.16
0.74
0.51
0:42
0.52
0.20
roal-valued
0.766
0.912
0.941 0.965
0.986
1.12
0:27
0.16
Continuous
assigned
categorical
0.344
0.763
0.890
0.921
0.951
0.978
1.14
0.32
neutral score real-valued
0.386
0 0.805
0.915
0 0,942
0,965
0.985
1.11
0.64
0.43
0.35
0.28
0.20
SemEval
Eng
valence
assigned NA
cotegorical
0.333
0.742
0.872
0. 0,900
0:982
0.986
1.15
0.75
0.51
0:41
0.28
0.15
2014
realivalued
0.404
0.820
0:907
0.925 0.939
0.981
1.09
0.65
0.44
0.36
0.27
0.26
(LiveJournal)
assigned
catagorical
0.725
0.842
0.878 0.924
0.987
1.18
0.22
nautral score real-valued
0.378
0.811
0.896
0.923
0.939
0.988
1.11
0.29
SemEval 2014 (SMS)
Eng
valence
assigned NA
categorical
0.265
0.609
0.768
0.810
0.852
0.915
1.22
0.87
0.64
0.55
0:42
0.26
real-valued
0.343
0.716
0.879 0.858
0.895
0.980
1.15
0.72
0.47
0.38
0.28
0.16
assigned
categorical
0.274
0.656
0.819
0.864
0,882
0:907
1.21
0.82
0.56
0.44
0.30
0.27
neutral score
realivalued
0.738
0.685
0.916
1.15
0.18
0.09
SemEval
Eng
valence
assigned NA
categorical
0.175
0,738
0.948
0.927
1.26
0.33
2014
real-velued
0.234
0.814
0.964 0.952
1.21
0.54
0.18
0.23
(tweets
assigned
categorical
0.206
0.587
0.830
0.873
1.28
0.85
0.43
0.48
sarcasm)
neutrall score valued
0,270 0. 0.722
0.830 0.985
1.24
0.59
0.39
0.36
SemEval
Eng
valence
assigned NA
categorical
0.572
0.619
0.907
1.24
0.90
0.56
0.44
0.32
2014
real-valued
0.323
0.711
0.854
0.891 0.910
0.917
1.16
0.32
(tweets)
assigned
categorical
0.2M
0.603
0.789
0.851 0.897 0.913
1.23
0.88
0.64
0.53
0:42
0.30
neutral score real-valued
0.322
0.728
0.868
0 0.898 0,912 0:915
1.16
0.73
0.50
0.40
0.30
0.22
SemEval
Eng
valence
assigned NA
catagorical
0.448
0.851
0.949
0.964 0.972 0.986
1.06
0.57
0.34
0.27
0.21
0.15
2018 (V-OC)
roalvvalued
0.502
0.879
0.970
0.967
1.01
0.29
0.17
0.12
assignad
categorical
0.427
0.850
0.944
0.964 0.976 0.980
1.08
0.58
0.28
0.22
0:16
neutral
real-valued
0.476
0.880
0.954
0.970
0.979 0.984
1.03
0.52
0.32
0.25
0.18
0.1.3
SemEval 2018 (V-Reg)
Eng
valence
assigned NA
categorical
0.458
0.839
0.936
0.956
0.978 0.994
1.05
0.59
0.38
0.31
0.22
0.13
realivalued
0.513
0.867
0:945
0.963
0.980
0.997
1:00
0.53
0.34
0.28
0.20
0.13
assigned
categorical
0.845
0.959
0.997
1.07
0.22
0.12
neutral score real-valued
0.488
0.874
0.948
0.964
0.979
0.997
1.02
0. 34
0.28
0.20
0.31
Corr 0.000
1.000
RMSE 0.09
1.28
Figure I: Valence Ares: Spearman correlation and RMSE values between LexO and gold arcs of English datasets.
datasets. The difference marked for very small bin sizes (such as 1 and 10) but progressively smaller for higher bin sizes. Entries from real- valued lexicons carry more fine-grand fine-nrained emotion information, and it is likely that this extra informa- tion is especially helpful when ares are determined from very little text (as in the case of small bins). O0V Terms: Results with the two OOV-handling methods were comparable (no clear winner).
ML Ares: Emotion Ares Generated from Counting ML-Labeled Sentences
This section explores how the accuracy of instance- level (sentence- or tweet-level) emotion labeling impacts the quality of the generated emotion arcs. We approached this by creating an 'oracle' sys- tem, which has access to the gold instance emotion labels.
There are several metrics for evaluating senti- ment analysis at the instance level such as accuracy, correlation, or F-score. However, we focus on ac- curacy as it is a simple intuitive metric. Inputs to the Oracle system are a dataset of text (for emotion labelling) and level of accuracy (e.g., 90% accu- racy) to perform instance-level emotion labelling at. Then, the system goes through each instance and predicts the correct emotion label with probability corresponding to the pre-chosen accuracy. In the case where the system decides to assign an incor- rect label, it chooses one of the possible incorrect labels at random.
Discussion: For many social media applications, one has access to tens of thousands, if not millions of posts. There, it is not uncommon to have time- steps (bins) that include thousands of instances. Thus, it is remarkable that even with relatively small bin sizes of few hundred, the simple lexicon approach able obtain very high correlations. Of course, the point not that the lexicon approach is somehow special, but rather that aggregation of information can very quickly generate high qual- ity arcs, even if the constituent individual emotion signals are somewhat weak.

---

Instance Accuracy
Test data
Emotion
10
50
100
200
300
SemEval
-0.125
-0.390
-0.703
-0.801
-0.872
-0.893
2018 (V-OC)
20
0.092
0.262
0.489
0.607
0.783
0. 8.84
40
0.320
0.715
0 0:927
0.359
0.979
965
0.505
0.861
0.965
0.986
0.730
0.963
0.984
0.988
0.995
100
1.000
1,000
1. 0.00
1.030
0.000
1.000
Figure 2: Valence Ares: Spearman correlations of the ares generated using the valence-classifiation Oracle System
(OS) with the gold ares English tweets. OS (Accuracy): Accuracy at instance-level sentiment classification.
Source
Model
Instance-Level
Bin Size
Accuracy
50
100
300
Socher al. (2013)
RNTN
85.4%
0.829
0.972
0.980
0.983
0.986
0.992
Devlin et al. (2019)
BFRRT-base
93.5%
0.921
0.981
0.984
0.988
0.993
0.996
Devlin et al. (2019)
BERT-large
94.9% 94.9%
0.932
0.986
0.984
0.988
0.993
0.996
Yang et al. (2019)
XLNet
97.0%
0.959
0.990
0.985
0.988
0.993
0,996
Liu et al. (2019)
RoBERTa
96.4%
0.958
0.989
0.986
0.989
0.994
0.997
Table 3: Valence Ares: Spearman correlations between ares generated using neural models for instance-level sentiment classification and gold ares of English tweet streams.
We use the Oracle to generate emotion labels pertaining to various levels of accuracy, for the same dataset. We then generate emotion ares just described in the previous section (by taking the average of the scores for each instance in bin), and evaluate the generated arcs just as before (by determining correlation with the gold arc). This Oracle System allows us to see how accurate an in- stance level emotion labelling upproach needs to be to obtain various levels of quality when generating emotion arcs.
at bin sizes 200 and 300. However, for a given application scenario, these results can only be ob- tained when the machine learning system is able to train sufficient domain-specine training data (which is often scarce), the computer power, and knowledge work with these systems is accessi- ble. For applications where simple, interpretable, low-cost, and low-carbon-foopprint systems are de- sired, the lexicon-based based systems described the previous section, are often more suitable.
6 Lex0 Ares: Emotion Ares Generated from Translated Emotion Lexicons
Figure 2 shows the correlations of the valence ares generated using the Oracle System with the gold valence ares created from the SemEval 2018 OC test set (that has possible labels: -3 to 3).7 We observe that, as expected the Oracle Systems with instance-level accuracy greater than approximately the random baseline (14.3% for this dataset) obtain positive correlations; whereas those with less than 14% accuracy obtain negative correlations. As seen with the results of the previous section, correlations increase markedly with increase in bin size. Even with an instance-leved accuracy of 60%, correlation approaches at larger bin sizes. Overall, we again observe high quality emotion ares with bin sizes of just few hundred instances.
Given the competitive performance of the Lexicon- Only (LexO) method, and the many benefits of the LexO method such their simplicity and inter- pretability,
explore whether high-quality emotion ares can be created for low-resourced languages using automatic translations of English emotion lexicons. Specifically, we make use of translations of the NRC lexicons from English into the language of interest and perform similar exper- iments as described in the previous section." 8 We explore bin sizes of 400 and 500 well, as we expect that instance-level accuracy will be lower in the wranslttedllexice case.
Table shows the correlations obtained on the same dataset when using various deep neural net- work and transformer-base systems. Observe that the recent systems obtain nearly perfect correlation
We would like note that automatic translations may not be available for many very low resource languages. Currently, Google Translate has func- tionality to translate across about 120 languages.
FFigure 5 (Appendix) shows similar Oracle System results for other datasets.
"The NRC Emotion and VAD Lexicon packages come with translations into over 100 languages.

---

Language
Source
Domain
Dimension
Label Type
Instances
Amharic
Yimam et al. (2020)
tweets
valence
categorical (-1.0.1)
5,984
Hausa
Muhammad et al. (2022)
tweets
valence
categorical (-1,0,1
14,172
Igbo
Muhammad et al. (2022)
tweets
valence
categorical (-1,0,
10,192
Kinyarwanda
Muhammad et al. (2023)
tweets
valence
categorical (-1,0,
3,302
Swahili
Muhammad et al. (2023)
tweets
valence
categorical (-1,0,
1,810
Yoruba
Muhammad et al. (2022)
tweets
valence
categorical (-1,0,1 1)
8,522
Table 4: Key information pertaining to the African language emotion-labeled tweets datasets we use our experiments, The No. of instances includes the train set for the SemEval 2023 Task 12.
This may seem large (and it is indeed a massive im- provement from just a decade back), but thousands of languages still remain on the other side of the digital divide. Thus, even requiring word transla- tions is a significant limitation for many languages.
and assign label NA to OOV words (which was comparable to the "assign neutral score' method), Figure 3 shows the results for generating emotion ares using translations of English lexicons into Arabic, Spanish, and the African languages for both categorically (c.g.. V-OC and SemEval 2023) and continuously (e.g., V-Reg) labelled valence datasets. We also provide the English results on the corresponding datasets for easy comparison.
We generated and evaluated emotion ares in six African languages: Amharic, Hausa, Igbo, Kin- yarwanda, Swahili, and Yoruba (using the datasets shown in Table 4). These are some of the most widely spoken indigenous African languages." 9 Ta- ble 6 in the Appendix presents details about each, including their language family and the primary regions where they are spoken. Note that these languages themselves are diverse covering three different language families: Afroasiatic, Bantu, and Niger-Congo. Swahili, more influenced by the Indo-Furopean languages than Hausa -about 40% of its vocabulary is made up of Arabic loan words. We contrast results on these languages to Arabic, a commonly spoken language in North- ern Africa which has somewhat more resources than indigenous African languages. Arabic still has much fewer NLP resources than English. We also contrast our results to Spanish, which has fewer resources than English; and is more similar to English than Arabic. (The Ar and És valence- labeled datasets used are shown in Table 7 in the Appendix.) High fidelity of the predicted ares with the gold ares will unlock the potential for research in affect-related psychology, health science, and digital humanities for vast number of languages.
For Arabic and Spanish (V-OC and V-Reg datasets): Correlations start at 0.40-0.50 at the instance-level (bin size 1), and reach 0.98 from bin sizes 200 and onwards. Performance us- ing translations of English lexicons does surpris- ingly well. At bin size 1 English does performs better by about 10 points, however this difference quickly dissipates bin size 50 and onwards.
For the African language texts (the SemEval 2023 datasets): Correlations start at lower values at the instance-level, than for Arabic and Spanish. At bin size 1, correlations are about 0.1-0.2 with the exception of Hausa at 0.374. As seen previously, with increasing bin size we are able to gain substantial improvements in performance by aggregating information. With a bin size of 200 and onwards correlations reach mid 0.90's which is large gain in performance compared to instance-level. Among the African languages, at bin size 1-50 ares for Hausa are by far the best, followed by ares for Amharie; then ares for Igho, Kinyarwanda, and Swahili perform similarly; followed by Yoruba. For bin sizes 100 and onwards, Hausa and Kinyarwanda become the best performing, followed by Igbo and Amharic performing similarly, followed by Yoruba.
6.1 Valence Ares: Using Translated Lexicons
We ran a wide range of experiments in Arabic, Spanish, and the African languages with the pa- rameter variations discussed carlier. We found the same trends for the OOV handling and lexicon gran- ularity parameters for these languages as for En- glish. Therefore, for brevity, Figure 3 only shows the results with using real-valued lexicons (which performed better than categorical score lexicons)
Discussion: Using translations of English lexicons allows us to create high-quality emotion ares in African languages. The variation in performance across the African languages could be because of the variation in quality of automatic translations of the emotion lexicons across languages and different antrt-annttaoor agreements for the instance-level emotion labels in the different datasets.
/ httpp/ggw...oaaallllo/////////// 10-most O--most-popile wwtpppplllaarrrria--aaaggggageee

---

Bin
Test Data
Test Lang.
Emotion
Test Type
10
50
100
200
300
400
500
SemEval
Ar
Valence
categorical
0.453
0.828
0.942
0.957
0.969
0.979
0.996
0.997
2018 (V-Oc)
Eng
Valence
categorical
0.502
0.879
0.970
0.978
0.984
0.987
0.991
0.998
Es
Valence
categorical
0.419
0.802
0.939
0.970
0.982
0.983
0,992
0.999
SemEval
Valence
continuous
0.457
0.806
0.939
0.969
0.991
0.998
0.996
0.999
2018 (V-Reg)
Eng
Valence
continuous
0.513
0.867
0.933
0.980
0.991
0.997
0.939
0.999
Valence
continuous
0.419
0.814
0.960
0.978
0.990
0.996
0.999
0.999
SemEval
Valence
categorical
0.259
0.638
0.878
0.912
0.918
0.920
0.927
0.935
2023
Val Valence
categorical
0.374
0.728
0.851
0.896
0.928
0.936
0.940
0.943
(AfriSenti)
Valence
categorical
0,209
0.561
0.819
0.880
0.913
0.923
0.930
0.934
kr
Valence
categorical
0.231
0.569
0.842
0.911
0.940
0.943
0.946
0.953
SW
Valence
categorical
0.231
0.576
0.835
0.855
0.868
0.871
0.898
0.893
yo
Valence
categorical
0.118
0.238
0.419
0. 0.497
0.601
0.647
0.671
0.704
Figure 3: Valence Ares: Spearman correlations between arcs generated using translated emotion lexicons and gold ares created from categorically and continuously labeled test data in Arabic, Spanish, and the African languages. The untranslated English results are shown for comparison.
Although the results for Arabic and the African languages are on different datasets, it is of inter- est to contrast the performance of Arabic to the African languages because Google's translations of words in the lexicon to Arabic is expected be of higher quality than to the African languages (due to the relatively higher amounts of English-Arabic parallel text). Overall, is interesting to note that the translation method creates relatively better ares when the target language is closer to the source lan- guage (e.g., English to Spanish). However, even for distant languages, performance can be improved by increasing bin size (e.g., some African languages). Of course, increasing bin size means lower gran- ularity, but many application scenarios, and for drawing broad trends, it makes sense to aggregate hundreds or thousands of instances in each bin, leading to more reliable inferences about emotion trends over time.
robustness of the LexO method, when dealing with dynamic emotion changes, we created new text streams by reordering the tweets in the emotion- labeled datasets in more random and dramatic ways. We created these dynamic text streams by sampling tweets from the chosen dataset with replacement until 200 crests and 200 troughs of various ampli- tudes and spikiness were obtained." 10 The gold are is then standardized as before. We will refer to this new text stream generated from a dataset as piattasetnamet J-aume--dymiic. and the gold ares created with this process as dynamic gold arcs.
The gold line in Figure 4 shows the beginning portion of the gold valence are produced from the SemEval 2018 (V-Reg) dataset (bin size = 100). Observe that some peaks are rather small in am- plitude whereas some others are greater than two standard deviations. Also, some emotion changes occur across wider span of data (x-axis) whereas some others show large emotion change and back in short span.
7
Are Quality under Varying Emotion Amplitudes and Spikiness
We then generated the predicted are for this dataset using the LexO method, the NRC VAD lexicon, and by ignoring terms not found in the lexicon when determining bin scores. The green line in Figure 4 shows the beginning portion of the gold valence are when using a rolling window with bin size 100. Observe that the green line traces the gold line quite closely. There are also occa- sions where it overshoots or fails to reach the gold arc. Figure shows gold and predicted ares for the Hausa dataset (bin size 300). Observe that here
Aspects of how emotions change across the bins also impacts the quality of generated arcs. For ex- ample, for a given surge or dip of emotion, the change of emotion strength is too small in magni- tude (small amplitude) or occurs for too short of a time period (high spikiness), then the automatic method may fail to register the surge/dip. We will refer to ares with many such emotion changes as more dynamic than those with more gradual and longer-duration emotion changes.
Recall that in Section we used text streams ordered by increasing gold score. Now, to test the
"Sampling with replacement allowed reuse of the limited labeled data to produce a long wave.

---

a) Initial portion of the SemEval 2018 Dynamic Gold Arc (yellow) and predicted are using LoxO mothod (greon).
Bin 100, Spearman correlation 0.917. RMSE 0.35
Are
gold
buboo
velence
2
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
test stream
b) Initial portion the SemEval 2023 (AfriSenti) Dynamic Gold Arc for Hausa (yellow) and predicted arc using the LexO method (green).
Bin size 300, Spearman correlation 0.743, RMSE 0.692
gorld DUTO
1000
2000
3000
4000
5000
5000
7000
8000
5000
10000
11000
12000
13000
streamm
Figure 4: Dynamic and predicted ares for English (a) and Hausa (b) using the LexO method.
the predicted are still follows the gold are closely (but expected, not closely a.)
with the abundance of textual data available from social media, for many applications where one is aggregating hundreds or thousands of instances in each bin, the gains of ML methods in terms of correlation scores are miniscule.
In experiments modifying bin size, the predicted ares obtain high correlation (above 0.9) with a bin size 100 (and above), even for the dynamic text stream. However, these scores are (as expected) consistently lower than what were observed for the corresponding non-dynamic text stream. Note that the difference in scores is more pronounced for the smaller bin sizes than for larger ones. Therefore, in scenarios with smaller amounts of data per bin, judicious use of parameters described in Section (e.g., treating OOVs as neutrals, using real-valued lexicons, etc.) can have marked influence on are quality. Overall, though, these results show that the Lex0 method performs rather well even in the case of highly dynamic emotion changes.
With the cross-lingual experiments, we created emotion ares from texts in six indigenous African languages, well Arabic and Spanish. For all of them, emotion ares obtained high correlations the bin size (aggregation level) was few hundred instances. Correlations between pre- dicted and gold emotion ares were in general higher when the target language was closer to English.
least a
In the last part of the paper, explored how depending the data available and the desired granularity of emotion ares interacts with perfor- mance for various are amplitudes and are spikiness. We showed that the LexO method performs rather well even in the case of highly dynamic emotion changes. However, for smaller bin sizes, the pre- dicted are may markedly overshoot or not fully capture the gold peak or trough.
8 Concluding Remarks
This work made contributions in two broad di- rections: First, showed how methods for pre- dicting emotion ares can be evaluated, and used it to systematically and quantitatively evaluate both Lexicon-Only and Machine Learning meth- ods. Second, we showed that using translations of English lexicons into the language of interest can generate high quality emotion ares, even for languages that are very different from English. Both ML and LexO methods produced high- quality emotion ares when using bin size 50 and above (c.g., with bin size 100: obtaining correla- tions with gold valence arcs exceeding 0.98). MI.
In all, we conducted experiments with 18 senti- ment-labeled datasets from languages, Thus the conclusions drawn are rather robust. The results on individual datasets also establish key benchmarks; useful to practitioners for estimating the quality of the ares under various settings. Finally, since this work shows that simple lexicon-only approaches produce accurate arcs, practitioners from all fields, those working in low-resource languages, those interested in interpretability, or those without the resources to deploy neural models, can easily gen- erate high-quality emotion ares for their data.
methods are able to obtain markedly higher cor- relations at very low bin sizes (<50). However,

---

9 Limitations
Our work on studying emotion word usage should not be construed detecting how peo- ple feel; rather, we draw inferences on the emotions that are conveyed by users via the language that they use.
It is challenging to determine the true emotion are of text stream or story through data anno- tation. This is because usually data annotation is performed on smaller pieces of text such sen- tences or paragraphs. It is difficult for people to read large body of text, say from novel, and produce consistent annotations for how emotions have changed as the text has progressed from start to finish. In the context of tweets, taking the av- erage emotion scores of individual sentences is reasonable option (as we have done here); however, that may not capture true emotions when dealing with large pieces of text written by the same person, for example, emotion ares of stories in novel. We hope future work will determine how to create gold arcs such cases, and how different such ares are from the ares created by averaging the scores of individual sentences.
The language used in an utterance may con- vey information about the emotional state (or perceived emotional state) of the speaker, lis- tener, or someone mentioned in the utterance. However, it is not sufficient for accurately de- termining any of their momentary emotional states. Deciphering the true momentary emo- tional state of an individual requires extra- linguistic context and world knowledge. Even then, one can be easily mistaken.
The inferences we draw in this paper are based on aggregate trends across large populations. We do not draw conclusions about specific individuals or momentary emotional states.
Constructing emotion lexicons from human annotations of emotion scores takes time and resources. Thus, these resources exist only for a handful of languages. As a feasible approach in the meanwhile, we uses automatic translations of emotion lexicons from English to desired target language to generate emotion ares in various languages. While this approach produces highly accurate emotion ares, there several considerations: different languages and cultures express emotions differently, there may be lexical gaps among languages, and characteristies and meaning can be lost in the translation. Therefore, one would expect that using human annotated lexicons would lead to higher emotion are quality.
Acknowledgments
Thank you to Krishnapriya Vishnubhotla for the Emotion Dynamics code-base, which set the ground work for computing instance average emo- tion scores for instances. This research was sup- ported by NSERC, Digital Research Alliance of Canada (alliancecan.ca) Alberta Innovates, and DeepMind. This research project funded by the Bavarian Research Institute for Digital Transforma- tion (bidt), an institute of the Bavarian Academy of Sciences and Humanities. The author is responsi- ble for the content of this publication.
References
10 Ethics
Cecilia Ovesdotter Alm and Richard Sproat. 2005. Emo- tional sequencing and development in fairy tales. In Proceedings of First International Conference on Affective Computing and Intelligent Interaction, ACII'S, page 668-674, Berlin, Heidelberg. Springer- Verlag.
Our research interest is to study emotions at an aggregate/group level. This has applications in de- termining public policy (e.g., pandemic-responss initiatives) and commerce (understanding attitudes towards products). However, emotions are com- plex, private, and central to an individual's 's experi- ence. Additionally, each individual expresses emo- tion differently through language, which results in large amounts of variation. Therefore, several eth- ical considerations should be accounted for when performing any textual analysis of emotions (Mo- hammad, 2022, 2020). The ones we would particu- larly like to highlight are listed below:
Atsushi Ashida, Masataka Tokumaru, and Tomoko Ko- jiri. 2021. Characters' emotion design support sys- tem for writing novels based story of target readers. Information and Technology in Education and Learning, I:Reg-p005.
Sriharsh Bhyravajjula, Ujwal Narayan, and Manish Shri- vastava. 2022. Marcus: An event-centric nlp pipeline that generates character ares from narratives. Tex12Slor@@ECIn ECIR.

---

Mummun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via so- cial media. In Seventh international AAAI conference on weblogs and social media, pages 128-137.
Saif Mohammad. 2023. Best practices in the creation and use of emotion lexicons. In Findings of the Asso- ciation for Computational Linguistics: FACI, 2023, pages 1825-1836, Dubrovnik, Croatia, Association for Computational Linguisties.
Marco Del Vecchio, Alexander Kharlamov, Glenn Parry, and Ganma Pogrebna. 2018. The Data Science of Hol- lywood: Using Emotional Ares of Movies to Drive Business Model Innovation in Entertainment Indus- trics. arXiv e-prints. page arXiv:1807.0222
Saif M. Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings The Annual Confer- ence the Association for Computational Linguistics (ACL), Melbourne, Australia.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Saif M. Mohammad. 2020. Practical and ethical consid- erations in the effective use of emotion and sentiment lexicons.
Saif M. Mohammad. 2021. Sentiment analysis: Au- tomatically detecting valence, emotions, and other affectual states from text. In Herb Meiselman, editor, Emotion Measarement (Second Edition). Elsevier.
Sharath Chandra Guntuku, Rachelle Schneider, Arthur Pelullo, Jami Young. Vivien Wong, Lyle Ungar, Daniel Polsky, Kevin G Volpp, and Raina Merchant. 2019. Studying expressions of loneliness in individu- als using Twitter: observational study. BMJ open, 9(1 9(1)::033355
Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. To Ap- pear in Computational Linguisties.
Saif M. Mohammad, Felipe Bravo-Marquez, Mo-
hammad Salameh, and Svetlana Kiritchenko. 2018.
Semeval-2018 Task I: Affect tweets. Proceed- ings of International Workshop on Semantic Evalua- tion 1SemEval-20188 New Orleans. LA, USA.
Will E. Hipson and Saif M. Mohammad. 2021. Emotion dynamics in movie dialogues. PL.O ONE, 16(9):1- 19.
Saif M. Mohammad, Mohammad Salameh, and Svet- lana Kiritchenko. 2016. How translation alters senti- ment. J. Artif. Int. Res., 55(1):95-130
Alexander Hogenboom, Bas Heerschop, Flavius Fras- incar, Uzay Kaymak, and Franciska de Jong. 2014. Multi-lingual support for lexicon-basee sentiment analysis guided by semantics. Decision Support Sys- tems, 62:43-53.
Saif M. Mohammad and Peter D. Turney. 2013. Crowd- sourcing a word-emotion association lexicon. Com- putational Intelligence, 29(3):436 465.
Tao Hu, Siqin Wang. Wei Luo, Mengxi Zhang, Xiao Huang. Yingwei Yan, Regina Liu, Kelly Ly, Viraj Kacker, Bing She, et al. 2021. Revealing public opin- ion towards covid-19 19 vaccines with twitter data in the united states: spatiotemporal perspective. Journal of Medical Internet Research, 23(9):30085.
Shamsuddeen Muhammad, Seid Yimam, Idris Ab- dulmumin, Ibrahim Ahmad, David Adelani, Se- bastian Ruder, Nedjma Ousidhoum, Saminu Aliyu. Abinew Ayele, Meriem Beloucif, Bello Bello, and Saif M. Mohammad, 2023. SemEval 2023 task 12: Sentiment analysis for african languages (afrisenti- semeval). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-20233) Toronto, Canada.
Evgeny Kim, Sehastian padó, and Roman Klinger. 2017. Investigating the relationship between literary genres and emotional plot development. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 17-26, Vancouver, Canada. Association for Computational Linguistics.
Shamsuddeen Hassan Muhammad, David Adelani, Se- bastian Ruder, Ibrahim Sa'id Ahmad, Idris Abdul- mumin. Shehu Bello Bello, Monojit Choudhury, Chris Chinenye Emezue. Saheed Salahuddeen Abdul- lahi, Anuoluwapo Aremu, Alipio Jeorge, and Pavel Brazdil. 2022. Najjasenti: A nigerian twitter senti- ment corpus for multilingual sentiment analysis. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 590-602, Marseille, France. European Language Resources Association.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach.
Saif Mohammad. 2011. From once upon time to happily ever after: Tracking emotions in novels and fairy tales. In Proceedings of the 5th ACL-HLT Work- shop on Language Technology for Cultrall Heritage, Social Sciences, and Humaniries, pages 105-1 14, Portland, OR, USA. Association for Computational Linguistics.
Emily Öhman. 2021. The validity of lexicon- sentiment analysis in interdisciplinary research. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pages 7-12, NIT Silchar, India. NLP Association of India (NLPAI).

---

Andrew J. Reagan, Lewis Mitchell, Dilan Kiley, Christo- pher M. Danforth, and Peter S. Dodds. 2016. The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science, 5(1):1-12. Copyright- EPJ Data Science is a copyright of Springer, 2016; Last updated 2017-02-06.
Amharic sentiment analysis from social media texts: Building annotation tools and classification mod- els. In Proceedings of the 28th International Con- ference on Computational Linguisties, pages 1048- 1060, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of the 8th Interna- lional Workshop on Semantic Evaluation (SemEval pages 73-80, Dublin, Ireland. Association for Computational Linguistics.
APPENDIX
A Lex0 vs. ML Methods for Emotion Labelling
In Table 5 we show the characteristics of the Lexicon-Only (LexO) and Machine Learning (ML) based methods for emotion labelling instances.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642. Seattle, Washington, USA. Association for Computational Linguistics.
B Impact of Selectively Using the Lexicon
The continuously labeled emotion lexicons include words that may be very mildly associated with emotion category or dimension. It is possible that the very low emotion association entries may fact mislead the system, resulting in poor emotion ares. We therefore also investigated the quality of emotion arcs by generating them only from terms with an emotion association score greater than a pre-chosen threshold; thereby using the lexicon en- tries more selectively. We systematically varied the threshold to study what patterns of threshold lead to better ares across the emotion
Swapna Somasundaran, Xianyang Chen, and Michael Flor. 2020. Eomotion ares of student narratives. In Proceedings of the First Joins Workshop on Narra- tive Understanding. Storylines, and Events, pages 97-107, Online. Association for Computational Lin- guistics.
Charles Spearman. 1987. The proof and measurement of association between two things. The American journal of psychology, 100(344)441-4711
datasets."
Daniela Teodorescu, Tiffany Cheng, Alona Fyshe, and Saif M. Mohammad. 2023a. Language and mental health: Measures of emotion dynamics from text as linguistic biosocial markers. In The 2023 Conference on Empirical Methods in Natural Language Process- ing.
Overall, we observed that valence benefits from including all terms, even lowly associated emotion words, as the optimal threshold across continuous and categorically datasets is O with few notable exceptions (SemEval 2014 LiveJournal, SemEval 2014 tweets, and V-OC). Generally, including only terms with emotion scores above 0.33 to 0.5 im- proves the quality of emotion ares.
Daniela Teodorescu, Alona Fyshe, and Saif Mohammad. 2023b. Utterance emotion dynamics in children's po- ems: Emotional changes across age. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 401-415 Toronto, Canada. Association for Computational Linguistics.
C Impact of the Quality of Instance-Level Emotion Labeling on Emotion Arcs
Krishnapriya Vishnubhotla and Saif M. Mohammad. 2022. Tweet emotion dynamics: Emotion word us- age in tweets from us and canada. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
Figure 5 shows the results for the Oracle System on the categorically labeled Semlival 2014 datasets. We observe similar patterns as discussed in the pa- per for the SemEval 2018 V-OC dataset. (Note that the instance-level random-guess baseline is de- pendent on the number of class labels; thus, the minimum Oracle System Accuracy at which posi- tive correlations with gold ares appear different across the datasets.)
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances Neural Infor- mation Processing Systems, volume 32, Curran Asso- ciates, Inc.
Note that our goal is not to determine the predictive power of the system on new unseen test data. For that one would have to determine thresholds from development set, and apply the model on unseen test data.
Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020. Exploring

---

Method
Pros
Cons
Lex0
Interpetaale:: One can casily examine the words the text that are driving the emotion are soores.
Poor Instance Level: Often very rate labelling emotions individual instances (sentences, tweels, etc.). Does not take context and long mingddistace dependencies account.
Simple and Accessible: Does require exten- sive infrastructure programming expertise execute the method.
Lesy Customization: Does not usually capture domain- specific ways of expressing emotions; although, if available, domain-specifee emotion lexicons mitigate this limitation.
Efficient & Environmentully Friendly: Does not require high compute power.
Widely Applicuble: Does not require domain specific training data.
Coptures Context by Aggregation: As more data available per bin, the predicted emotion are gets closer to the true emotion are because with more data is more likely that: If bin x is ex- pressing more of an emotion than hin y, then x has higher average word emotion association score than y.
ML
Great at Instance Level: Achieves state-of-the- art performance on emotion labelling instances.
Hard interpret/explain the output.
Requires extensive compute power and storage.
Better at Capturing Context: Considers context and long-range dependencies effectively.
Requires extensive computational expertise.
Requires domain-specipp training data.
Table Characteristies the Lexicon- Mexicon-Only (LexO) and Machine Learning (ML) based methods.
D African Languages
Information on the African languages used in our study are shown in Table 6.
E
Emotion Labelled Datasets
Table 7, we shown the emotion labelled in- stances for Arabic, and Spanish languages.
F
Code and Resources
The code and resources used made freely avail- able on the project homepage." 12 The code allows for easy generation of high-quality emotion ares for the provided text stream (especially useful for those outside of computer science).

---

Bin Size
Test data
Emotion
Accuracy
1
10
50
100
200
300
SamEval
valance
-0.474
0.953
-0.35
-0.959
-0.97
-0.994
-0.154
00.50
-0.850
40.43
0.978
(LiveJourna)
0.947
0.964
1.000
1,000
1.000
SemEval
volence
-0.361
-0.777
-0.863
-0 -0,850
+0. 0.829
-0.798
2014 (SMS)
20
-0:225
-0.387
-0.555
-0.735
-0.774
-0.006
40
0.080
0.229
0.478
0.621
0.691
0.761
60
0.358
0.786
0.866
0.877
0.886
0.008
80
0.651
0.880
0.694
0.901
0.916
0.932
100
1.000
1.000
1.000
1.000
1.000
1.000
SemEval
valence
0.562
0.797
-0.95
2014
20
-0.160
-0.652
-0.887
(tweets
40
0,180
0.707
0.953
sarcasm)
60
0.502
0.904
0.995
0.764
0.999
1.000
SamEval 2014
valanca
-0.403
-0.923
(tweets)
40
0.092
0.281
0.553
0.654
0.824
0.832
60
0.367
0.783
0.513
0.916
0.917
0.918
80
0.663
0.514
0.913
0.918
0.920
0.922
100
1.00C
1.000
1.000
1,000
1.000
1.000
0.000
1.000
Figure 5: Valence Ares: Spearman correlations of the ares generated using the sentiment-clasifcation Oracle
System with the gold ares. Note: 'OS (Accuracy)" refers to the accuracy the Oracle System instance-level sentiment classification.
Language Amharic
Family
Spoken In
#Speakers
Afroasiatic
Ethiopia
37.5 million
Hausa
Afroasiatic
northern Nigeria, Ghana, Cameroon,
million
Benin. Togo, Ivory Coast
Igbo
Niger-Congo
Southeastern Nigeria
million
Kinyarwanda
Niger-Congo
Rwanda
9.8 million
Swahili
Bantu
Tanzania, Kenya, Mozambique Southwestern and Central Nigeria
200 million
Yoruba
Niger Congo
million
Table 6: Indigenous African languages included in our study.
Dataset
Source
Domain Dimension
Label Type
#Instances
SemEval 2018 Ar (V-OC)
Mohammad et al. (2018)
tweets
valence
categorical (-3.-2....3)
1,800
SemEval 2018 Ar (V-Reg)
Mohammad et al. (2018)
tweets
valence continuous (0 to
1,800
SemEval 2018 Es (V-OC)
Mohammad et al. (2018)
tweets
valence categorical (-3.-2....3)
2,443
SemEval 2018 Es (V-Reg)
Mohammad al. (2018)
tweets
valence
continuous
2,443
Table 7: Key information pertaining to the Arabie (Ar) and Spanish (Es) emotion-labeled tweets datasets we use
in our experiments. #Instances includes the train, development, and sets the SemEval 2018 Task


Reviews:
[{"review_id": "ZGP3MtQNMW", "reviewer": "Reviewer_QjH9", "summary": "", "questions": "In section 4 the authors discuss the different approaches to emotion labeling and how the annotation guidelines are not consistent. The authors' solution is to us a predeterming emotion-association threshold. However, some recent papers by Ohman et al. (2021&2023 in particular) have shown that emotion intensity lexicons work better for real-world congruent results than lexicons than simply list associations. Did the authors consider using emotion intensity over simple association?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThank you for your thoughtful and kind review. Thank you also for providing the valuable references. We have incorporated them in the revised version. Also, as you are suggesting, emotion intensity lexicons should be explored in addition to association lexicons since they contain more granular information. We use both association lexicons which we refer to as categorical lexicons scores, and emotion intensity lexicons which we refer to as continuous lexicon scores. The emotion intensity lexicon we use for anger, fear, joy, and sadness is the NRC Emotion Intensity Lexicon (continuous scores from 0 to 1) and for valence, the NRC Valence, Arousal, Dominance Lexicon which has scores from 0 to 1. In the result tables we use the column ‘Lexicon Scores’ to refer to whether an association or emotion intensity lexicon is being used. In the scenario the ‘Lexicon Scores’ column is not included in the table for space (this occurs once in Figure 4), we use the emotion intensity lexicon. We’ll make sure this is clear in the table caption as well, thank you for pointing this out. Indeed, as you say and like the paper you pointed to “The validity of lexicon-based sentiment analysis in interdisciplinary research” does very well, we hope this paper also encourages more thoughtfulness in recognizing the advantages and disadvantages of different methods, rather than simply focusing SOTA results. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors convincingly present further evidence that lexicon-based methods have their place in computational sentiment analysis and emotion detection in particular and show that in applied task where usefulness is more important than accuracy metrics lexicon-based methods often outperform purely data-driven methods.\n\nIt is unfortunate that many researchers in emotioion detection are forced by a review process slanted towards achieving higher accuracy metrics to adapt machine learning methods in order to get their paper published. The recent influx of papers showing that these state-of-the-art approaches are not necessarily the most appropriate for applied downstream tasks are extremely important to enable a more accurate overviews of the emotional content of textual data in particular.", "reasons_to_reject": "This paper has no weaknesses to mention.", "questions_to_authors": "In section 4 the authors discuss the different approaches to emotion labeling and how the annotation guidelines are not consistent. The authors' solution is to us a predeterming emotion-association threshold. However, some recent papers by Ohman et al. (2021&2023 in particular) have shown that emotion intensity lexicons work better for real-world congruent results than lexicons than simply list associations. Did the authors consider using emotion intensity over simple association?", "paper_topic_and_main_contributions": "This paper contributes to the ongong discussion and commparison between lexicon- and machine learning-based methods in emotion research. Additionally, it provides important cross-linguistic resources and new methods in implementing emotion detection methods for lower-resourced languages.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are no issues with language or presentations. This is a complete and polished paper."}, "extra_text": {"Missing_References": "Öhman, E., 2021, December. The validity of lexicon-based sentiment analysis in interdisciplinary research. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities (pp. 7-12).\n\nÖhman, E. and Rossi, R., 2023. Affect as a proxy for literary mood. Journal of Data Mining and Digital Humanities: NLP4DH.\n\nAlthough Matthew Jockers' work on the syuzhet package has not been used much in academic research, it is still perhaps the most discussed emotion arc generator in DH and perhaps should be mentionded in this paper? At the very least perhaps:\nGao, J., Jockers, M.L., Laudun, J. and Tangherlini, T., 2016, November. A multiscale theory for the dynamical evolution of sentiment in novels. In 2016 International Conference on Behavioral, Economic and Socio-cultural Computing (BESC) (pp. 1-4). IEEE.\n"}}, {"review_id": "orifERvQlZ", "reviewer": "Reviewer_MGk7", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThank you for your thoughtful comments and helping us to improve the description and presentation of the work.\n\nWith the extra page available if the paper is accepted, we will add a figure showing the steps in the construction of the emotion arcs. We now also added a bulleted list of steps in the main text.\n\nBin Sizes: Choice of bin size depends on the application and available data. For example, if a company wants to know how the proportion of angry comments has changed from month to month in the last 10 years, then the bin size to use is month. If instead, they want to know how things changed on a day by day basis over the last 45 days, then the bin size to use is a day. Depending on the volume of relevant data in the bin chosen, the bins may include a larger or smaller number of instances. We did experiments with many different amounts of data per bin to show that using bins with 100+ instances leads to better results. The systematic study and the results show how the quality of the arc changes with bin size so that practitioners can use suitable bin sizes for their particular data and task. Overall, smaller bin sizes provide higher granularity, but may not always have a sufficient amount of data for accurate arcs; therefore, using bigger bins is then preferable.\n\nNeural models: This paper additionally describes results with neural arcs to determine whether they provide markedly higher quality arcs than lexicon-based methods and under what conditions. We show that in most cases the improvement is only slight. The neural models are markedly better however when the bin size is very small (approaching 1 or few instances). However, in most applications of emotion arcs, especially those involving social media data, one has access to thousands, if not, tens or hundreds of thousands of instances, and so lexicon methods with bin sizes of a few hundred provide high quality arcs with little room left for improvement using more complex methods.\n\nExperiments with many diverse datasets: We performed a vast number of experiments across diverse domains in 9 languages for 5 emotions, where instances had both categorical and continuous emotion scores. We systematically generate and evaluate emotion arcs on 42 datasets. We consider each dataset that has been labelled with a different emotion, emotion score type (categorical vs continuous), or in a different language as a separate dataset. As an overview, these included movie reviews with continuous and categorical emotion scores (2 datasets); SemEval 2014 data consisting of 4 domains (4 datasets); SemEval 2018 data in English, Arabic, and Spanish for valence, anger, fear, joy, sadness with both continuous and categorical emotion scores (30 datasets); and SemEval 2013 data for 6 Indigenous African languages (6 datasets). Due to the limited space, we show results in the main paper and appendix for 28 datasets. This means that 14 datasets have been left out due to space however, all result tables are also available on our project website. Generally, there were similar trends across datasets and languages, where increased bin size improved performance.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our author response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This study focuses on the assessment of a challenging task across multiple data sources and languages.", "reasons_to_reject": "- The description of the evaluation and construction of emotion arcs in this paper lacks clarity. It would have been beneficial to include a diagram illustrating the process of constructing emotion arcs from tweets and reviews. Additionally, the justification for using different bin sizes is insufficient. This may be due to the chosen datasets in this paper, which may not fully capture the nuances of storytelling and playwriting commonly found in narratives and stories.\n\n- In this paper, there is a discussion about using different levels of accuracy (line 322) when evaluating neural network models. However, it is unclear why this choice is important or how it contributes to the creation of emotion arcs. Providing further justification would be beneficial in helping readers understand which level of accuracy is most suitable for generating emotion arcs, such as high, medium, or low levels of accuracy.\n\n- There appears to be a lack of experiments in this paper, considering the authors' claim in line 13 that they conducted experiments on 42 diverse datasets. However, only a portion of these datasets is presented in the paper. Furthermore, the experiments included in the paper fail to provide sufficient justification for the quality of the emotion arcs generated by both the lexical-based and machine learning approaches. It is necessary to conduct additional experiments and analyses to demonstrate the strengths and weaknesses of the two approaches employed in generating emotion arcs.\n\n-----------------------------------------------\nI thank the authors for their response to the comments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The purpose of this paper is to evaluate the advantages of automatically generating emotion arcs by exploring lexical-based and machine learning approaches. To achieve that, well-established emotion lexicons are employed to construct emotion arcs using diverse data sources, including tweets, reviews, sentences, and various languages. Subsequently, the constructed emotion arcs are assessed using the two approaches utilized in this study.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m3wPlRMMVp", "reviewer": "Reviewer_ZKef", "summary": "", "questions": "What is the influence of the methods across the languages and domains? \n\nWhat is the motivation of choosing the specific metrics used? Have you considered others?\n\nAbout translation, have you manually validated the accuracy and appropriateness of the translated emotion lexicons (sample size) for different languages?\n\nFurther analysis of the impact of OOV terms on emotion arcs could enhance the paper's insights.\n\nWhat are the main limitations of the method used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your thoughtful comments on the presentation of the work. We have addressed these already in a revised version:\n\n– Added examples, qualitative analysis, limitations of translations: Since this is not a sentence-level task but rather a task about emotion arcs from thousands of sentences/tweets, it is difficult to provide examples of data behind emotion arcs in a paper (for space) and also challenging for qualitative analysis (due to the size of data to be assessed). However, we will make the data available on the project website. And we now include snippets from a few peaks and troughs of the arcs to give information about the context in those sets of sentences/tweets. This also enables qualitative analysis, as now we can explore where the predicted arc is farther away from the gold arc and examine the text in those areas. We will add the qualitative analysis and discussion of where translation quality impacts results in the revised version. \n\n– Experiments with many diverse datasets: We performed a vast number of experiments across diverse domains in 9 languages for 5 emotions, where instances had both categorical and continuous emotion scores. We systematically generate and evaluate emotion arcs on 42 datasets. We consider each dataset that has been labelled with a different emotion, emotion score type (categorical vs continuous), or in a different language as a separate dataset. As an overview, these included movie reviews with continuous and categorical emotion scores (2 datasets); SemEval 2014 data consisting of 4 domains (4 datasets); SemEval 2018 data in English, Arabic, and Spanish for valence, anger, fear, joy, sadness with both continuous and categorical emotion scores (30 datasets); and SemEval 2013 data for 6 Indigenous African languages (6 datasets). Due to the limited space, we show results in the main paper and appendix for 28 datasets. This means that 14 datasets have been left out due to space however, all result tables are also available on our project website. Generally, there were similar trends across datasets and languages, where increased bin size improved performance.\n\n– Impact of OOV terms: We performed experiments where words not found in the lexicon were either assigned a neutral score such as 0 (‘assigned neutral score’), or were disregarded (‘assigned NA’). In Figure 1 (pg. 4) we show the results on the quality of emotion arcs when using these two methods. At smaller bin sizes it was found the OOV method is slightly more impactful and ‘assigned NA’ generally performed better. However, at larger bin sizes (e.g., 200-300) both methods performed better than the other an approximately equal number of times. This is described further in Section 4 (lines 268, 288, 304). In Figure 2, for simplicity we only show the results for handling OOV as ‘assigned NA’ since both methods were comparable and this method performed better more often than not (lines 423-425).\n\nQ1. What is the influence of the methods across the languages and domains?\n\nThe main things that influence results for lexicon-based methods for any language or domain are:\n– Quality and coverage of lexicon for the terms in the test set: the higher the better\n– Size of bins (aggregation): the higher the better\n\nThe main things that influence results for ML methods for any language or domain are:\n– Quality, size, and domain match of labeled training data with test data: the higher the better\n  \nQ2. What is the motivation of choosing the specific metrics used? Have you considered others?\n\nSpearman correlation and RMSE are the most commonly used metrics to determine the closeness of two time series arcs. Spearman correlation determines whether the shapes of the arcs are similar, without regard to the exact values. Thus this metric is ideal for most applications where the goal is to determine shape accurately (where the need is to determine whether emotions increased since the last comparison point or decreased). RMSE requires the shape and values to be close. Mathematically, RMSE is the standard deviation of the residuals. Residuals represent the distance between the regression line and the data points. Examining results with both metrics gives us a better picture of arc quality in different application scenarios.\n\nWe also considered KL Divergence, but that metric is asymmetric and is more suitable for probability distributions.\nPearson correlation largely produced the same results as Spearman.\n\nQ3. About translation, have you manually validated the accuracy and appropriateness of the translated emotion lexicons (sample size) for different languages?\n\nThere are several works that have examined the quality of Google Translate’s translations over the last decade for various language pairs. The resource is also highly used throughout the world. We did a systematic study for some language pairs in our past work and found the suitability of translations to be high (> 90%).\n\nQ4. Further analysis of the impact of OOV terms on emotion arcs could enhance the paper's insights.\n\nWe addressed this above, however we summarize the key takeaways. At smaller bin sizes it was found the OOV method is slightly more impactful and ‘assigned NA’ generally performed better. However, at larger bin sizes (e.g., 200-300) both methods performed better than the other an approximately equal number of times. This is described further in Section 4 (lines 268, 288, 304). \n\nQ5. What are the main limitations of the method used?\n\nSection 9 (Limitations) in the paper provides the limitations.\n\nRe Illustrative example:\nWith the extra page available if the paper is accepted, we will add a figure showing the steps in the construction of the emotion arcs. We now also added a bulleted list of steps for the methodology of creating emotion arcs in the main text.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our author response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. Original Contribution: The paper presents novel research on evaluating emotion arcs, a topic that has received limited attention in previous work. \n\n2. Comparative Analysis: The paper compares two common methods for generating emotion arcs (LexO and ML) and highlights the strengths of the LexO approach in generating high-quality emotion arcs.\n\n3. Evaluation: The paper conducts a systematic and quantitative evaluation of automatically generated emotion arcs using a diverse set of datasets in multiple languages.", "reasons_to_reject": "Weak Presentation and Clarity: The paper's writing and organization may be unclear, making it difficult for readers to understand the research methods and findings. No examples are provided in the paper.\n\nEvaluation of different datasets: The authors assert that they conducted experiments with diverse datasets; however, the paper does not present all of the results obtained from these experiments.\n\nEvaluation Metrics: Although the paper uses correlation and RMSE values for evaluating emotion arcs, more discussion and analysis of the significance of these metrics in the context of emotion arcs would strengthen the research.\n\nNo quality analysis: The paper fails to provide a thorough quality analysis of the experiments, leaving a gap in assessing the reliability and robustness of the results.\n\nLimited Discussion on Language Variations: The paper briefly mentions variations in translation quality across different languages.  Emotions are culturally and linguistically influenced, and direct translations of emotion lexicons may not fully capture the intricacies and subtleties of emotions in different languages and cultures.\n\nNo analysis of the impact of the OOV terms is provided.\n\nThe reproducibility of the results could be challenging. Not enough details are provided.", "questions_to_authors": "What is the influence of the methods across the languages and domains? \n\nWhat is the motivation of choosing the specific metrics used? Have you considered others?\n\nAbout translation, have you manually validated the accuracy and appropriateness of the translated emotion lexicons (sample size) for different languages?\n\nFurther analysis of the impact of OOV terms on emotion arcs could enhance the paper's insights.\n\nWhat are the main limitations of the method used?", "paper_topic_and_main_contributions": "This paper addresses the evaluation and generation of emotion arcs, which represent how an individual or population's emotions change over time. The study compares two common methods for generating emotion arcs: Lexicon-Only (LexO) and Machine-Learning (ML) approaches. The paper systematically evaluates automatically generated emotion arcs using 42 diverse datasets in 9 languages. The authors also experimented with automatic translations of English emotion lexicons to generate emotion arcs in less-resourced languages.\n\nThe main contributions are:\n\n- Evaluation of automatically generated emotion arcs.\n- Comparison of two common methods for generating emotion arcs: LexO and ML approaches.\n- Experiments on 42 diverse datasets in 9 languages, including indigenous African languages, Arabic, and Spanish.\n- Authors claim they will release the data and code for easy generation of high-quality emotion arcs.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "To improve the presentation of the paper, the authors should consider organizing the context more clearly to provide a better overview of the research. Additionally, they can include illustrative examples to guide readers through the steps of the methodology, making it easier for them to understand and follow the experiments."}}]

Author rebuttal:

Meta-review: The paper addresses emotion arcs using lexicon-based and machine learning approaches. The reviewers have praised the paper's originality addressing a topic that was largely unexplored in previous work. 

While one of the reviewers is very positive about the paper, the other two reviewers have identified several points that need improvement. For example, they have pointed to the lack of clarity in the evaluation and the description of experiments.
Decision: Accept-Findings