Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection
Authors: Pranaydeep Singh
Luna De Bruyne
Orphee De Clercq
Els Lefever
Venue: EMNLP 2023 Findings (2023)
Keywords: emotion detection
complexity
readability
TL;DR: Understanding how linguistic complexity correlates with emotions, as observed on social media. When fine-tuning for emotion detection, is a transformer able to understand the linguistically complex samples?

Abstract: Given the omnipresence of social media in our society, thoughts and opinions are being shared online in an unprecedented manner. This means that both positive and negative emotions can be equally and freely expressed. However, the negativity bias posits that human beings are inherently drawn to and more moved by negativity and, as a consequence, negative emotions get more traffic. Correspondingly, when writing about emotions this negativity bias could lead to expressions of negative emotions that are linguistically more complex. In this paper, we attempt to use readability and linguistic complexity metrics to better understand the manifestation of emotions on social media platforms like Reddit based on the widely-used GoEmotions dataset. We demonstrate that according to most metrics, negative emotions indeed tend to generate more complex text than positive emotions. In addition, we examine whether a higher complexity hampers the automatic identification of emotions. To answer this question, we fine-tuned three state-of-the-art transformers (BERT, RoBERTa, and SpanBERT) on the same emotion detection dataset. We demonstrate that these models often fail to predict emotions for the more complex texts. More advanced LLMs like RoBERTa and SpanBERT also fail to improve by significant margins on complex samples. This calls for a more nuanced interpretation of the emotion detection performance of transformer models. We make the automatically annotated data available for further research at: https://huggingface.co/datasets/pranaydeeps/CAMEO

Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection
Pranaydeep Singh', Luna De Bruyne Orphée De Clereqe and Els Lefever LT3, Language and Translation Technology Team, Ghent University
firstname lastname @ugent..eeeeeeeeee
+CLiPS, University of Antwerp
firstname
be
Abstract
easier than ever before. Recent research by Rathje et al. (2021) has shown that publishing negative criticism or insults against political opposition is a highly effective way to acquire engagement on social media platforms. Similar research in the news domain shows that headlines with negative keywords generate more engagement and clicks than headlines with positive keywords (Robert- son al., 2023). While these studies explore the negativity bias from consumer's perspective, in this work we explore the idea of negativity bias from the perspective the author and more specif- ically explore the correlations between negativity and complex language use.
Given the omnipresence social media in our society, thoughts and opinions being shared online in an unprecedented manner. This means that both positive and negative emotions can he equally and freely expressed. However, the neg- ativity bias posits that human beings are inher- ently drawn to and more moved by negativity and, consequence, negative emotions get more traffic. Correspondingly, when writing about emotions this negativity bias could lead to expressions of negative emotions that are lin- guistically more complex. In this paper, we at- tempt to readability and linguistic complex- ity metrics to better understand the manifesta- tion of emotions on social media platforms like Reddit based the widely- used GoEmotions dataset. We demonstrate that according to most metrics, negative emotions indeed tend to gen- erate more complex text than positive emotions. addition, we examine whether a higher com- plexity hampers the automatic identification of emotions. To answer this question, we fine- tuned three state-of-the-an transformers (BERT, RoBERTa, and SpanBERT) the tion detection dataset. We demonstrate that these models often fail to predict emotions for the more complex texts. More advanced LLMs like RoBERTa and SpanBERT also fail to im- prove by significant margins on complex sam- ples. This calls for a more nuanced interpre- tation of the emotion detection performance of transformer models. We make the automat- ically annotated data available for further re- search at hf. fffoodusce//rmmalleps//AAa
The goal of this research thus to further in- vestigate the negativity bias in social media texts by exploring whether there is a link between emo- tion and text complexity. This overarching research goal will be broken down into two specific research questions. (1) Are negative emotions on social media more complex than positive emotions? (2) Does a text's linguistic complexity make it harder to model certain emotions using transformers, and do transformers that are better at understanding more complex text - such as SpanBERT (Joshi et al., 2019) perform better than BERT (Devlin et al., 2019) on linguistically complex emotional text?
Firstly, we attempt to understand, in a strictly lin- guistic sense, the differences in the ways negative and positive emotions are manifested social me- dia. To investigate this we rely on the GoEmotions dataset (Demszky et al., 2020), which comprises English Reddit comments that have been manually annotated with 28 emotion categories. We investi- gate these texts' linguistic complexity by relying on diverse metrics from readability research. More specifically we consider a text more complex ac- cording to both lexical and syntactic complexity metrics.
1 Introduction
The negativity bias, coined by Kanouse et al. (1987), states that human beings are inherently drawn to, and impacted more by negative experi- ences, emotions, and interactions than by their pos- itive counterparts. This cognitive bias may more present than ever because of the evolution of social media which makes sharing thoughts and opinions
In this first stage of our research, we investigate

---

possible correlations between these metrics and the polarity of the gold-standard emotion labels. In the second stage, we fine-tune three state-of-the -art transformers (BERT, RoBERTa, and SpanBERT) on the emotion dataset to perform multi-label clas- sification.
(Averill, 1980). On a similar note, Jing-Schmidt et al. (2007) observed that there is large number of emotive intensifiers that are based on negative emotion words, even to express positive messages (e.g., "He is insanely good at his job." or "This was terribly sweet of you.") ").
We demonstrate that according to most metrics, negative emotions indeed tend to generate more complex text than positive emotions and that the transformer models are generally worse predict- ing emotions for the more complex texts. We be- lieve this calls for a more nuanced interpretation of the emotion detection performance of transformer models, with more detailed evaluations to have a better understanding of performance for complex data.
Besides focusing on the prevalence of negative vocabulary, few studies have dealt with examin- ing the relationship between language complexity and negative emotions. Some studies have exam- ined the language used in personal narratives about positive and negative experiences, but findings re- garding complexity are inconclusive: in the work of Bohanek al. (2005) on narratives of female un- dergraduates, it was found that positive narratives were more coherent and complex, although nega- tive narratives were longer. However, in a study by Fivush et al. (2003), in which children were asked to narrate highly negative and highly positive events, it was found that narratives about negative events were more coherent. Morcover, these sto- ries exhibited more emotion-related and cognition- related words, while the narrations about positive events focused more on descriptions of objects and people and were less coherent. These findings are more in line with the work of Guillory et al. (201 I), in which emotion contagion in online groups was studied and it was found that negative emotions are correlated with more complex language. However, seeing these mixed findings, thorough study us- ing well-designed complexity metrics is desired reliably examine the relationship between negative emotions and linguistic complexity.
The remainder of this paper organized as fol- lows. Section 2 gives a brief overview of relevant related research, whereas Section 3 describes the data and information sources used for the experi- ments and analyses. Section 4 elaborates on the correlations between complexity and emotion po- larity, whereas Scction 5 discusses the machine learning experiments carried out evaluate the performance of transformers on instances with var- ious degrees of complexity. Section ends this paper with concluding remarks and indications for future research.
2 Related Research
2.1 Negativity Bias and Complexity
The negativity bias or negativity effect is a cog- nitive bias that is described as the greater impact that negative stimuli have on humans compared to positive stimuli (Kanouse and Hanson Jr, 1987). This greater impact is not only reflected in obser- vations that humans are more drawn to negative stimuli - which, for example, shows itself in more engagement with negative posts social media (Rathje et al., 2021) in higher consumption of negative news (Robertson et al., 2023)- but also the fact that negative stimuli are associated with a higher cognitive load and more complex cogni- tive representations (Peeters and Czapinski, 1990). One could therefore ask whether this complexity is also reflected in language use.
2.2 Text Complexity
Text complexity can be studied from different an- gles. The first strand of experimental research in- vestigates complexity by measuring processing ef- fort when reading sentences. Effort can then be captured by cognitive proxies, such as for instance eye-tracking metrics (King and Just, 1991).
A second strand of research, more related to the work proposed here, research on measuring text readability. Pioneering work in this respect has been performed by pedagogical researchers who designed traditional readability formulas, such Flesch (1948) or Dale and Chall (1948). These formulas calculate text readability based on shal- low surface text characteristics, such as the average word or sentence length or frequencies according to a predefined list. Thanks to advances both
Some researchers have indeed investigated the negativity bias in the context of language. An early but important observation is that there are more words for referring to negative emotional states than there are for referring to positive emotions

---

NLP and machine learning, we are currently able derive more intricate text characteristics and con- sequently the research evolved towards readabil- ity prediction using both feature-base (Collins- Thompson, 2014; De Clercq and Hoste, 2016) and deep learning approaches, though the latter mostly rely on combination with explicitly derived lin- guistic features (Meng et al., 2020; Lee et al., 2021). Interestingly, among the text characteristics that have been identified as most predictive when comes to assessing text's readability, the super- ficial ones as employed in the traditional formu- las are often retained (François and Miltsakaki, 2012; De Clercq and Hoste, 2016). Additionally, syntactic features derived from parse tree infor- mation have also proven important indicators of text's complexity as first revealed in the work by Schwarm and Ostendorf (2005) relying on constituency parser and later also corroborated us- ing dependency parsing information (De Clereq and Hoste, 2016). Considering the latter, many efforts have been devoted towards universal de- pendencies (De Marneffe et al., 2021), which was employed in recent research to operationalize how cognitive effort can be increased by complex syn- tax during translation through variety of syntactic complexity metries (Zou et al., 2022).
example for quality assurance". The applied emo- tion taxonomy includes 12 positive, 11 negative, ambiguous emotion categories, and "neutral" category.
Demszky et al. (2020) employed a BERT-based model and achieved an average F1-score of .46 on the fine- fine-grained emotion classification task. The best performance was obtained for the positive emo- tions gratitude (.86), amusement (.80), and love (.78), while the model performed worst on grief (0), relief (.15) and realization (.21). Indeed, over- all the performance was better on positive emotions (average F1-score of 0.52) compared to negative emotions (average Fi-score of 0.41). The authors explain this by the presence of overt lexical mark- for the best-predicted emotions, but also by frequency: overall, there are more instances with a positive emotion label than negative label in the dataset. In this paper, we will therefore investigate whether complexity, besides frequency, could be an explaining factor in emotion detection perfor- mance.
Extracting Complexity Metries
To investigate the link between emotions and com- plexity, we extracted set of linguistic text charac- teristics that have shown be good predictors for complexity in readability research (see Section 2.2). In the remainder of this paper, we will refer to these features as complexity metrics. In this section, we will first describe the datasets used for the correla- tion analysis and emotion detection experiments, and subsequently present overview of all metrics that were extracted to measure complexity.
2.3 Emotion Detection
Given the abundance of opinionated data the web, there is significant interest in methods to auto- matically analyze the emotional information within this data, e.g. in the context of market analysis and customer satisfaction for business intelligence (Song et al., 2021) or reputation management (Mo- hammad et al., 2015).
3.1 Datasets
While initially, the aim was to identify the se- mantic polarity (positive, negative, or neutral) of a given instance, the field evolved towards a more fine -reigea paradigm where the goal is to identify specific emotions instead of mere polarity orienta- tions.
In order to investigate the link between emotions and complexity on social media posts, we deliber- ately chose work with high-quality benchmark dataset for emotion detection that has been manu- ally annotated.
The GoEmotions dataset consists of English Reddit comments that were manually labeled with 28 emotion categories (including 12 positive, 11 negative, 4 ambiguous emotion categories, and 1 "neutral" category, allowing one to distinguish be- tween subtle emotion expressions). The dataset consists of 58,009 instances. To create the Reddit corpus, first NLTK's s word tokenizer was applied
In line with this interest in fine-grained emotion detection, datasets labeled with wide range of emotion categories have been developed. One of the largest and most fine-grained emotion datasets is GoEmotions (Demszky al., 2020), which con- sists of English Reddit comments that were manu- ally labeled with 28 emotion categories. As stated by the authors, it is "the largest -nnatttn [emotion] dataset, with multiple annotations per
hhtpss//ai.googllbbble www.leblog.comm/2211/// wttp:coccmmmmcm/c/ http:////////gggmmmoooonnnnnssse for-fine-grained.hhmm

---

and only comments consisting of to 30 tokens were selected. Subsequently, downsampling was performed to create a relatively balanced distribu- tion of comments with varying lengths. We ran- domly selected a subset of 50,000 instances from a total of 58,009 instances in the simplified version of the dataset for automatic annotation of complex- ity information. In this evaluation set, the negative and positive polarities are fairly evenly distributed with 1 1,,935 negative samples and 14,190 positive samples, while there are 23,875 neutral samples. the fine-tuning experiments for Scction 5, we use the entire dataset of 58,009 instances with an 80:10:10 split for training, validation, and testing respectively.
Second, the following syntactic metrics were calculated based on the work of Zou et al. (2022), these metrics can be viewed as proxies to measure the impact of syntactic complexity on cognitive effort:
IDT Incomplete Dependency Theory:
For a given token t,, the IDT metric counts the number of incomplete dependencies between
t, and 1++1-
DLT - Dependency Locality Theory:
For a head token 1s, the DLT metric counts the number of discourse referents (proper nouns, nouns, and verbs) starting from i; ending to its longest lefumost dependent. The boundary- ending words should also be counted if they are discourse referents, For non-head token. DLT defined as zero.
3.2 Complexity Metrics
To measure text complexity, we extracted of linguistic features that range from well-known superficial word shapes and lexical features to more complex syntactic features.
NND Nested Noun Distance:
The distance between tokens and l is their absolute positional distance, li i). A noun tę nested in another noun t; if tj is the ancestor of t; in the dependency tree. any tree structure, node that is connected to some lower-level nodes is called an ancestor. NND is the aggregation (i.e., sum, max or average) of nested noun distances. Following the research of Zou al. (2022) we used sum the aggregating metric.
First, a set of word shape and lexical diversity and frequency features were extracted based on the readability research of De Clercq and Hoste (2016). We refer to this set as the lexical metrics. It comprises two more traditional readability features, namely average word length and percentage of polysyllable words, which have proven successful in readability work (François and Miltsakaki, 2012). Next, lexical complexity was modeled by measuring the type-token ratio and the percentage of words that can be found in the 1995 Dale and Chall list (Chall and Dale, 1995) for English. In addition, two-term weighting features were implemented that are often used to distinguish specialized terms in the text, namely Term Erequency-Inverss Document Frequency, aka tf-idf (Salton and Buckley, 1988) and the Likeeiielddodd ratio (Rayson and Garside, 2000). For both of these features, the average values for all words in the text were calculated. Because connectives serve as an important indication of textual cohesion in text (Graesser et al., 2004), we also counted the average number of connec- tives within the text. Finally, named entity information provides us with a good estimation of the amount of world knowledge required read and understand a particular text (De Clereq and Hoste, 2016), the number of unique entities was calculated as well.
LE
The LE metric counts the number of tokens on the left-hand side of the main verb which are not verbs.
We have made the 50k subset of the GoEmo- tions dataset, automatically annotated with all 12 complexity metrics, available?.
Correlation between Linguistic Complexity and Negative Emotions
In this section, we investigate whether there is positive correlation between linguistic complex- ity and the presence of negative emotions. As explained in Section 3, we rely on syntactic and lexical readability metrics to measure complexity, while dividing the 28 emotion labels into sets: positive (12 labels), negative (11 labels), and neu- tral and ambiguous (5 labels). As preliminary

---

Factor
Factor
Factor 3
IDT
DLT
LE
NND
Avg. Word Length
Polysyliable Words
DALE Word Frequency
Type Token
Avg. Connective
Num. Unique Entities
Avg. TF-IDF
Avg. Log-Likelihood
0.00
0.25
0.50
0.75
1.00
1.25
Figure 1: Factor loading matrix showing the importance of the considered readability metries for three Factors.
attempt to study these correlations, we first perform an Exploratory Factor Analysis (EFA) to justify the use and selection of the complexity metrics before proceeding investigate links between complexity and emotions. Moreover, by classifying the metrics into factors, can further analyze the difficulties transformers modelling certain types of com- plexity.
2.5
Elpevvluue
2.5
First, we perform Barlett's Test of Spheric- ity (Armstrong and Soelberg, 1968) to confirm the validity of the factor analysis. The p-value was found be 0.0 which means the observed cor- relation matrix not an identity matrix and the test is statistically significant. We also perform a Keyser-Meyer-OIie Olkin (KMO) Test which measures the proportion of variance among all the observed variables. The KMO for our data was found to be 0.659. A value higher than 0.6 is usually consid- ered adequate for factor analysis.
0.5
100
Ractors
Figure 2: Scree plot showing the Eigenvalue in corre- spondence the number of Factors.
variance reveals cumulative value of 0.413, which means that cumulative variance 41.3% of the data is explained by the factors. Section we further use the distinction between the factors analyze the difficulties of transformer models modeling certain types of linguistic complexity.
To determine the adequate number of factors for decomposition, we set up a Scree plot, which is shown Figure 2. Since factors seem to have an eigenvalue higher than 1.0, we perform the factor analysis with components. Figure visualizes the factor loading matrix, demonstrating that Factor 1 largely combines syntactic metrics, while Factor focuses on frequency- based metrics like TF-IDE, Log-Likedn Lig-Liielihood, and Type Token Ratio and Factor 3 solely focuses on basic lexical metrics like word length and polysyllable occurrences. The factor
Since the metrics used seem to adequately rep- resent linguistic complexity statistically proven by the EFA, can use these metries further understand the relationship between complexity and emotion polarities. To this end, we divide the complexit-eenriceed GoEmotions dataset into

---

1A
12
23
23
Posttopp
negotthe
nestral
postive
negative
nedtral
posnrie
negatim
neutral
(a) IDT
(b) DLT
(c) Left- SeffeEmbededness
(d) Nested Noun Distance
100
Ang. Word Length
9.45
DALE WordFreuunnc
Frequanyyycyyy
109
94.3
4.4
108
94
4.35
107
26
99.7
94.5
206
24
94.5
4.25
94.4
postiiv
magalive
posilive
Hegatiee
noterrll
posttivd
regative
nestral
postiise
rigottie
neuttel
(c) Average Word Length
%6 Pollysyllable Words
DALE Word Frequency
(h) Type Token Ratio %
288
182
Unique Enities
2.75
19.1
Avt
0.35
0.003
0.34
2.8
0.33
Pooltive
nestral
Positio necatoee
nestral
povitive
negottive
nestral
grastne
negative
nestral
Avg. Connectives
(j Num. Unique Entities
(k) Avg. TF-IDF
Avg. Log- Likelihood
Figure 3: Average value of each complexity metric for the broader emotion categories of positive, negative, and neutral for GoEmotions evaluation set.
three polarities and calculate the average value for each metric for the three polarities individually, Figure 3 displays the averages for all 12 metrics for the three polarities. According to the figure, out of the 12 metrics indeed have higher aver- age score for negative polarity. For some metrics, like Seft-Emmededness, Nested Noun Distance, or DALE Word Frequency, the difference between positive polarity averages is strikingly lower than the negative and neutral polarity averages. While Avg. TF-ID,, Avg. Log-Likelihood and Type To- ken Ratio % anomalies, all three of these met- rics convey correlated information represented by Factor 2 in the EFA (as illustrated in Figure 1).
emotion categories a very fine-grained classifica- tion task. For these experiments, we investigate 3 major transformers, BERT (Devlin et al., 2019), SpanBERT (Joshi et al., 2019), and RoBERTa (Liu et al., 2019). We fine-tune each of these models for multi-label classification for 20 epochs, with an initial learning rate of 2e 5 and weight de- cay of 0.1, on train set of 46,407 samples (80%) from the raw version of the GoEmotions dataset. We then use the best peetfreerorming checkpoint on a validation set of about 5,800 samples (10%). As evaluation set, (10%). All scores reported averaged over runs with different seeds.
the remaining 5,802 samples
5 Emotion Detection Evaluation
To evaluate the output of the three fine-tuned transformer models, we again incorporate the metrics described in Section 3.2. We divide metrics using the EFA described Section 4 into the Factors, each representing a different aspect of linguistic complexity, i.e., syntactic complexity
this section, we attempt to understand how dif- ferent state-of-the--e transformers are affected by complexity when classifying emotions using the GoEmotions datasets, which constitutes with its 28

---

Factor
Factor 2
Factor 3
Sim.
Med.
Comp.
Sim.
Med.
Comp.
Sim.
Med.
Comp.
Negative 275
889
214
29
1280
69
196
929
253
Neutral
656
1733
381
46
2480
244
420
1825
525
Positive 432
1023
199
34
1526
94
210
1138
236
Table 1: Distribution of Complex. Medium and Simple samples in the evaluation set considering the three main factors resulting from the Exploratory Factor Analysis (Section 4).
Factor 1
Factor 2
Factor 3
Sim.
Med.
Comp.
Sim.
Med.
Comp.
Sim.
Med.
Comp.
Negative
0.2195
0.2068
0.2660
0.1034
0.2202
0.2480
0.2486
0.2053
0.2484
BERT
Neutral
0,4875
0.4331
0.3874
0.3544
0.4283
0.5741
0.4656
0.4262
0.4661
Positive
0.4850
0.4151
0.3665
0.3157
0.4312
0.4100
0.4733
0.4110
0.4540
Negative
0.1148
0.1164
0.0732
0.0444
0.1097
0.1272
0.1346
0.1020
0.1173
SpanBERT
Neutral
0,5176
0,4209
0.3793
0.2769
0.4278
0.5751
0.4574
0.4243
0.4736
Positive
0.4494
0.3848
0.3324
0.3235
0.3996
0.3516
0.4319
0.3802
0.4252
Negative
0.3053
0.2508
0.2564
0.3157
0.2610
0.2647
0.3077
0.2500
0.2738
RoBERTa
Neutral
0,4642
0.4275
0.3921
0.3000
0.4211
0.5658
0.4423
0.4199
0.4632
Positive
0.5027
0.4142 0.3644
0.3846
0.4352
0.3814
0.4774
0.4212
0.4248
Table Performance (micro-F1) the three transformer models fine-tuned for emotion detection with respect to the three main factors resulting from the EFA.
(Factor 1), metrics based on frequencies in a background corpus (Factor 2), and simple lexical complexity (Factor 3). For each set, if more than half the metrics are in the top 20 percentile, we label the sample as a complex one (Comp.). If more than half of the metrics are in the bottom 20 percentile, we label the sample as a non-complex one (Sim.). The rest of the samples as medium complex (Med.). An overview of the distribution of the evaluation set according this complexity labeling scheme is presented in Table 1.
findings in psychology research that Negative Emo- tion Differentiation (ED) harder than Positive ED for certain individuals (Starr et al., 2020), or other- wise put, that it is more difficult to distinguish be- tween various fine-graines negative emotions than between different positive emotions, for humans and automatic systems alike.
labeled
A second noteworthy observation is that the Fac- tor 1 Simple subsets tend to frequently outperform the Factor 1 Complex and Medium subsets, as il- lustrated by Figure 4(a). This is again verified by a two-tailed sample T-test, which proves statisti- cally significant at p 0.05. This is in accordance with the presumption that complex text samples require higher degree of Natural Language Un- derstanding (NLU), which the current generation of transformers is not (yet) capable of. This obser- vation seems contrasted by the better performance on most Factor 2 Complex subsets, which often even surpasses the results obtained on the Factor 2 Simple subsets. This, however, might be linked to the fact that lexical complexity also comes with lexical richness, i.c., the presence of distinguish- ing keywords seldom used, which was one of the complexity metries (type-token ratio) we used categorize the evaluation set according to complex- ity, and therefore potentially a key factor in the decision miision-making for ED. Whereas, for Factor 3,
The detailed emotion classification results, which are summarized in Table 2, provide some interesting findings. First and foremost, it is im- portant draw attention the performance of the negative subclass of emotions which is consistently worse than the neutral or positive instances. This proven by a two-tailed sample T-test, which proves the findings are statistically significant at p < 0.05. This is also apparent in Figure 4 where the per- formance for negative subsets for each factor is consistently poorer than for the positive subsets. The difference can be quite staggering: the Fac- tor Simple Positive set, for instance, has a micro F1 F1-score of 0.4850 for the BERT model, while the Factor Simple Negative set obtains a micro Fl- score of 0.2195. This is in line with experimental

---

point to the pre-training data lacking emotional richness. SpanBERT was trained on the same datasets as BERT, but the data was restricted samples exceeding 512 sub-tokens only from Book- Corpus and Wikipedia, potentially inhibiting the ability of the model to understand non-standard short-form social media communication.
S
02
norr Jea
maanerr Jrop
Rudentz Heg
RiPt Pr
aeenterr Pa
6
Conclusion
In this work, we attempted to answer two research questions on the relationship between linguistic complexity and emotion detection. For this pur- pose, we relied on the GoEmotions dataset, which is a high-quality benchmark dataset for emotion detection that has been manually annotated with 28 emotion categories.
Factor I
Cemphe m
0.5
03
02
In the first part of this research, we investigated whether negative emotions are more linguistically complex than positive ones. With the help of twelve complexity metrics sub-divided into three factors, we discovered that negative emotions indeed have higher average degree of complexity than posi- tive neutral emotions. Secondly, we attempted to dive deeper into the evaluation of fine-tuned transformers for emotion detection, by dividing our evaluation set from the GoEmotions dataset into complex, medium, and simple samples. Through this nuanced evaluation of three state-of-the-arr transformers (BERT, SpanBERT and RoBERTa), we find that syntactically complex samples are of- ten misclassified. We also notice that irrespective of difficulty, all models tend to perform poorly negative emotions. Finally, also discover that even though SpanBERT encodes higher degree of complexity, this does not extend to emotion detec- tion of complex text, as it either performs worse or is comparable to transformers trained with standard masking (BERT and RoBERTa).
0-
BEAT 3ig
Brasigrr reg
ReBEnT' Hog
USRT Pot
Speatter de
Factor 2
Cerrete nor
02
0-
RUBERT Jigg
HoYr Ph
Sponserr.r Pa
Randiaa Pd
Factor 3
Figure 4: Classification performance of BERT, Span- BERT and RoBERTa (in micro-F1) for Complex vs. Simple subsets for the three Factors (Figure (a), (b), (c)) for Negative and Positive emotions.
To conclude, we aim to encourage more de- tailed evaluation of current emotion detection sys- tems, by extending the evaluation setup to better understand the performance for more linguistically complex instances, rather than the reporting of aggregated scores which may only paint a half- formed picture. In future work, we would like to demonstrate the validity of the results datasets from other domains and annotation strategies, well as languages other than English. Further, since linguistic complexity does have an impact on emo- tion detection performance, we would like to in- vestigate more intricate approaches to model more
the difference in Simple vs. Complex subsets is inconclusive. This suggests that modeling lexically complex samples is not pertinent issue of these sets of transformers.
The final, rather disappointing, observation is that SpanBERT, which is designed to better repre- sent and predict spans of text, and thus assumed better model more complex syntax, does not obtain better classification scores than BERT or RoBERTa on the complex subsets. In fact, Span- BERT only outperforms BERT and RoBERTa for some emotionally neutral subsets, which might

---

complex samples.
Dorottya Demszky, Dana Movshovitz-Attias Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emo- tions. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4040 4054, Online. Association for Computational Linguisties.
Limitations
As a first and primary limitation, we want to point out that the present study has only been validated on an English dataset, whereas emotions are global. Thus, more work needs to be done before the con- clusions can be extended to other languages and domains. The study was also conducted using the GoEmotions dataset. Even though can be consid- ered the largest in English, the results should be validated on more datasets and source domains. Lastly, the inferences in this work assume that the selected metrics are exhaustive and adequately represent the complexity of text sample, however, assessing linguistic com- plexity is complex and ongoing multidisciplinary area of research, and the selected metries may be missing some information to properly distinguish complexity.
Jacob Devlin, Ming- Wei. Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ming-Wei.
emotion dataset
Robyn Fivush, Ann Hazzard, Jessica McDermott Sales, Deborah Sarfati, and Trina Brown. 2003. Creating coherence out of chaos? children's narratives of emo- tionally positive and negative events. Applied Cog- nitive Psychology: The Official Journal of the Soci- ety for Applied Research in Memory and Cognition. 17(1):1-19.
Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221 233.
References
Thomas François and Eleni Miltsakaki. 2012. Do NLP and machine learning improve traditional readability formulas? In Proceedings of the 1st Workshop Predicting and Improving Text Readability for Target Reader Populations (PITR2012), pages 49-57.
J. Scott Armstrong and Peer Soelberg. 1968. On the interpretation of factor analysis. Psychological Bul- letin, 70:361-364
James R Averill. 1980. On the paucity of positive emo- tions. Assessment and modification of emotional behavior, pages 7-45.
Arthur C. Graesser, Danielle S. McNamara, Max M. Louwerse, and Zhiqiang Cai. 2004. Coh-metrix: Analysis of text on cohesion and language. Behav- ior Research Methods, Instruments, and Computers, 36(2):193-20:
Jennifer G Bohanek, Robyn Fivush, and Elaine Walker. 2005. Memories of positive and negative emotional events. Applied Cognitive Psychology, 19(1)551-66
Jamie Guillory, Jason Spiegel, Molly Drislane, Ben- jamin Weiss, Walter Donner, and Jeffrey Hancock. 2011 I. Upset now? emotion contagion in distributed groups. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 745-748.
Jeanne Sternlicht Chall and Edgar Dale. 1995. Readabil- ity revisited: The new Dale- Chall readability formula. Brookline Books
Kevin Collins-Thonnee Thompson. 2014. Computational assess-
ment of text readability: a survey of current and fu-
7huo ling-Schmidt. 2007. Negativity bias in language: A cognitive-affective model of emotive intensifiers.
ture research. Special Issue of the International Jour-
nal of Applied Linguistics, 165(2)997-135.
Mandar Joshi, Danqi Chen. Yinhan Liu, Daniel S. Weld. Luke Zettlemoyer, and Omer Levy. 2019. Span- BERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv: 1907.
Edgar Dale and Jeanne S. Chall. 1948. A formula for predicting readability. Educational research bulletin, 27:11-20.
Orphée De Clereq and Véronique Hoste. 2016. All mixed up? finding the optimal feature set for general readability prediction and its application to English and Dutch. Computational Linguistics, 42(3):457- 490.
David E Kanouse and L Reid Hanson Jr. 1987. Neg- ativity in evaluations. In Preparation of this paper grew out of a workshop on attribution theory held at University of California. Los Angeles, Aug 1969. Lawrence Erlbaum Associates, Inc.
Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255- 308.
Jonathan King and Marcel Adam Just. 1991. Individ- ual differences in syntactic processing: The role of working memory. Journal of Memory and Language. 30(5):580-602.

---

Bruce W. Lee. Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: trans- former meets handcrafted linguistic features. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10669- 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lisa Starr, Zoey A. Shaw, Irina Li. Angela C. Santee, and Rachel Hershenberg. 2020. Negative emotion differentiation through developmental lens: Associations with parental factors and age ado- lescence. Personality and Individual Differences. 152:10
Longhui Zou, Michael Carl, Mehdi Mirzapour, Hélène Jacquenet, and Lucas Numes Vieira. 2022. Ai-based syntactic complexity metrics and sight interpreting performance. In Intelligent Human Computer Inter- action, pages 534-547, Cham, Springer International Publishing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach.
Changping Meng, Muhao Chen, Jie Mao, and Jennifer Neville. 2020. Readnet: A hicrarchical transformer framework for web article readability analysis. In Ad- vances in Information Retrieval, pages 33-49, Cham. Springer International Publishing.
Saif M Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, style in electoral tweets. Information Processing & Management, 51(4):480-499
Guido Peeters and Janusz Czapinski. 1990. Positive- negative asymmetry in evaluations: The distinction between affective and informational negativity ef- fects. European review of social psychology. 1(1):33- 60.
Steve Rathje, Jay J Van Bavel, and Sander Van Der Lin- den. 2021. Out-group animosity drives engage- on social media. Proceedings of the National Academy of Sciences, (1822)):20242221188
Paul Rayson and Roger Garside. 2000. Comparing cor- pora using frequency profiling. In The Workshop on Comparing Corpora, pages 1-6, Hong Kong, China. Association for Computational Linguistics.
Claire E Robertson, Nicolas Pröllochs, Kaoru Schwarzenegger, Philip Pärnamets, Jay J Van Bavel, and Stefan Feuerriegel. 2023. Negativity drives on- line news consumption. Nature Human Behaviour, pages 1-11.
Gerard Salton and Chris Buckley, 1988. Inf. Process. Manag,, 23(5)3513-523.
Sarah E. Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL-2005), pages 523-530.
Shuangyong Song, Chao Wang, Haiqing Chen, and Huan Chen. 2021. An emotional comfort frame- work for improving user satisfaction in E-commerce customer service chatbots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies: Industry Papers, pages 130-137, Online. Association for Computational Lin- guistics.


Reviews:
[{"review_id": "nUehGetlV1", "reviewer": "Reviewer_z7G8", "summary": "", "questions": "A. What are the main differences between the raw and simplified versions of the GoEmotions dataset (L2787-279)? Additionally, why was a subset of 50k instances randomly selected in the simplified version (L285), while the raw version was used for the fine-tuning experiments (L292-293)?\n\nB. Have you considered including other available datasets for emotion detection, such as HurricaneEmo?\n\nC. What is the primary motivation for conducting the factor analysis in Section 4? Initially, I initially thought it was to categorize the metrics into three categories (syntactic metrics, frequency-based metrics, lexical metrics) for subsequent analyses. However, the paper appears to have used either the entire set of features (Figure 3) or the two categories (Table 1 and 2), leaving it unclear what additional insights the factor analysis contributes to the paper other than showing there are three groups of features in the analyses.\n\nD. What was the motivation behind your definition of \"complex\" and \"non-complex\" examples discussed in lines 451-453? How was the specific criteria (i.e., top/bottom 20 percentile for more than half of the metrics) determined?\n\nE. Why do you think the metrics represented by Factor 2 were outliers in Figure 3? Do you have any good explanations?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Respected Reviewer,\n\n\nThank you for your feedback and extremely constructive comments on our work. First of all, we would like to address the questions put forth to us:\n\n\nA. The raw and simplified versions of the data are taken directly from the official versions uploaded on the huggingface datasets hub here: https://huggingface.co/datasets/go_emotions. However, it is unclear from the paper and GitHub repository how the authors selected the reduced 58,009 instances for the simplified version. According to the dataset card: “In the raw data, labels are listed as their own columns with binary 0/1 entries rather than a list of [label] IDs as in the simplified data”. Additionally, the raw version consists of additional meta-data which is left out in the simplified version. Finally, the simplified version is divided into pre-existing train/test/val splits.\n\nB. We have considered using additional datasets to validate our hypothesis, however, at the current stage, we could not identify any other datasets that fulfilled all our requirements, since we wanted to analyze only user-generated content, which was manually annotated, with fine-grained emotions and not the standard coarse Ekman or Plutchik labels. Therefore, datasets like Dair-AI and HurricaneEmo did not fulfill our requirements.\n\nC. The factor analysis was conducted to determine the validity of the selected complexity metrics, to check for information overlap among the metrics, and as pointed out, to demonstrate that the metrics largely belong to 3 broad categories. We indeed really like and appreciate the suggestion of also using these broad factor categories for further analysis in the later section and we will implement this in the updated version of the paper.\n\nD. This was an intuitive decision: taking the top/bottom 20% ensures that we have a high chance of really capturing complex and non-complex instances. However, being in the top/bottom 20% was not required for all metrics, but for at least half of them, because tightening the requirement to all metrics would leave us with too little data to analyze.\n\nE. We hypothesize that TF-IDF and Avg. LL are likely outliers due to a large influence of frequencies in the background corpora used to calculate these metrics. \n\n\nAdditionally, according to suggestions, we would like to make the following changes to the paper:\n\n1. Improve the quality of figures and tables, and make suggested changes to Table 1. \n2. Add more detailed examples, explore the factors contributing to the complexity, discuss anomalies, etc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The research questions are interesting, novel, and well-motivated by prior research.\n- The methods and experiments are well-thought-out.\n- The paper is well-written, easy to read, and clearly presents their findings and contributions.\n", "reasons_to_reject": "- Limited generalizability: All experiments were conducted solely on one data set, GoEmotions, which may limit the broader applicability of the findings.\n- Some parts of the paper were slightly confusing. For instance, the motivation behind conducting factor analysis over the complexity metrics is not clear. Additionally, the reasoning for not grouping the features into three groups based on the factor analysis in the later model performance analyses, and instead using binary syntactic and lexical feature categorization, could use some explanation.\n- The method, findings, and contributions of the paper might be better suited for a shorter publication rather than a long paper. More in-depth and fine-grained analyses of the results, such as exploring the specific factors contributing to the complexity of negative language with examples and differences among the fine-grained negative emotions, could have made the paper stronger. However, the current draft does not provide much detail in this regard.\n", "questions_to_authors": "A. What are the main differences between the raw and simplified versions of the GoEmotions dataset (L2787-279)? Additionally, why was a subset of 50k instances randomly selected in the simplified version (L285), while the raw version was used for the fine-tuning experiments (L292-293)?\n\nB. Have you considered including other available datasets for emotion detection, such as HurricaneEmo?\n\nC. What is the primary motivation for conducting the factor analysis in Section 4? Initially, I initially thought it was to categorize the metrics into three categories (syntactic metrics, frequency-based metrics, lexical metrics) for subsequent analyses. However, the paper appears to have used either the entire set of features (Figure 3) or the two categories (Table 1 and 2), leaving it unclear what additional insights the factor analysis contributes to the paper other than showing there are three groups of features in the analyses.\n\nD. What was the motivation behind your definition of \"complex\" and \"non-complex\" examples discussed in lines 451-453? How was the specific criteria (i.e., top/bottom 20 percentile for more than half of the metrics) determined?\n\nE. Why do you think the metrics represented by Factor 2 were outliers in Figure 3? Do you have any good explanations?\n", "paper_topic_and_main_contributions": "This paper explores the connection between language complexity and emotions, and specifically examines how these factors impact the performance of transformer-based emotion detection models. The findings reveal that negative emotions tend to produce more complex text compared to positive emotions, and transformer models generally struggle more with complex language.\n\nThe main contributions of this research are twofold. First, the study addresses intriguing research questions inspired by previous work in linguistics. By providing empirical evidence through quantitative results, the paper offers valuable insights that support and validate existing theories. Second, the authors present a well-designed set of experiments that incorporate various metrics to capture different aspects of language complexity. This comprehensive approach enhances the understanding of the relationship between language complexity and emotions.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The design, image quality, readability, and legends could be improved for all figures in the paper.\n- Tables 1 and 2 contain an excessive amount of numbers, making them challenging to read. Consider converting Table 1 into a stacked bar graph, and use different number formatting (e.g., 0.4409 → 44.1) for Table 2 to enhance clarity."}}, {"review_id": "TpnR9drruD", "reviewer": "Reviewer_nrYf", "summary": "", "questions": "Did you experiment with different hyperparameters for fine-tuning the individual baseline models? This could potentially help in further understanding and explaining the performance differences between them.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Respected Reviewer,\n\n\nThank you for your feedback and extremely constructive comments on our work. First of all, we would like to address the question put forth to us:\n\n\nA. Yes, we optimized the hyper-parameters of each of the baseline models individually and implemented early stopping to ensure we had the best checkpoint for each transformer.\n\n\nAdditionally, we also want to address some of the criticisms pointed out:\n\n\n1. We indeed acknowledge that the emotion annotations have not been made by the authors themselves and can thus only be considered as a proxy. We do know that the instructions for the annotators were to identify the emotions as expressed by the writer of the text and that good inter-rater agreement was reported. If our paper gets accepted we will definitely add this as one of the limitations of our work.\n\n2. We consciously opted to only evaluate transformers with similar architectures and especially the difference between models like BERT & RoBERTa, designed for generic use versus SpanBERT, designed to understand higher complexity. However, in future work, we would indeed like to explore how the next generation of auto-regressive models deal with the complexity of emotions.\n\n3. We thank you for pointing out the suggestion for Table 2, we will be sure to update it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper provides a detailed introduction to the psychological related works motivating their research objective.\n* The complexity analysis relies on a wide range of features, which speaks to the robustness of the obtained results. \n* Overall, the paper is well-written, and the analysis is technically sound.\n", "reasons_to_reject": "* The paper describes the task as “explor[ing] the idea of a negativity bias from the perspective of the author” (lines 51-52). However, in the context of the GoEmotions dataset, it is important to make the distinction between the emotions experienced by the authors and those identified by the annotators. Since in the GoEmotions dataset texts have not been annotated by the authors themselves, relying on this dataset has its limitations (which should be addressed in the paper).\n* Given recent advancements in large language models, I would be curious to see how zero-shot and few-shot learning with larger models perform as compared to the BERT-based models.\n", "questions_to_authors": "Did you experiment with different hyperparameters for fine-tuning the individual baseline models? This could potentially help in further understanding and explaining the performance differences between them.", "paper_topic_and_main_contributions": "This paper studies negativity bias in social media posts, specifically by asking (i) whether posts conveying negative emotions are more linguistically complex than positive ones and (ii) whether a text’s linguistic complexity increases the difficulty to model the related emotions using Transformers. To do so, the paper analyses the GoEmotions dataset, a collection of over 200,000 Reddit posts manually annotated according to 28 emotion categories (12 positive, 11 negative, 4 ambiguous, and 1 neutral). \nFor the first research question, the authors downsampled from that dataset to obtain a balanced set of 50,000 examples. Linguistic complexity is modeled via lexical metrics (e.g., average word length and percentage of polysyllable words, TF-IDF) and syntactic metrics (based on dependency-tree indicators). Results show that out of the 12 metrics used, 9 show a higher score for negative emotions as compared to positive ones.  \nFor the second one, the authors experiment with the full 200,000 examples dataset, and fine-tune three Transformer models (BERT, SpanBERT, and RoBERTa). To do so, the authors utilize the aforementioned complexity metrics and divide dataset samples into simple, medium, and complex. The authors notice first that classification performance is lower for negative emotions than for positive ones. Furthermore, the simple instances seem to be more easily correctly classified by the models as compared to the medium and complex ones.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "Table 2: I’d recommend mentioning the performance metric (F1) being used here to avoid confusion."}, "extra_text": {}}, {"review_id": "I3YSnrAviu", "reviewer": "Reviewer_hKnV", "summary": "", "questions": "A. Can you provide insights or explanations as to why TF-IDF, Avg. Log-Likelihood, and Type Token Ratio % exhibit higher average scores for positive polarity? Does this suggest that positive emotions have greater lexical complexity, given that all three of these metrics are lexical metrics?\n\nB. Regarding the 120,000 samples from the raw dataset version used to train the classifiers, do they ensure a balanced representation of both positive and negative samples, as well as an even distribution between complex and simple samples?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Respected Reviewer,\n\n\nThank you for your feedback and extremely constructive comments for our work. First of all, we would like to address the question put forth to us:\n\nA. Given that the majority (5 out of the other 8) of lexical complexity metrics point to the opposite, we would still say that negative emotions have a higher lexical complexity. However, it is an interesting line of discussion as to why TF-IDF, Avg. LL and TTR% have a higher average for positive emotions. It is also interesting to note that all three of these features belong to Factor 2 in the EFA and might encode very similar information. For TF-IDF and Avg. LL, the scores could largely be influenced by the frequencies in the background corpora used for calculating these metrics as well.\n\nB. The 120,000 or so samples used for training do indeed have a near identical distribution of positive-negative samples, however the complex/simple distribution cannot be verified because only the evaluation set of 50,000 was annotated with complexity metrics, since some of the complexity metrics are extremely time intensive to calculate and the task would be quite expensive computationally. However, we agree that it would be fairer to use the same set of 50,000 examples for the finetuning experiments (in a cross-validation setting), so that we have complexity metrics for all used and evaluated instances. Therefore, we have already started the finetuning experiments in a 5-fold setting on this smaller dataset. We will report these results in the camera-ready version upon acceptance.\n\n\nAdditionally, we would also like to address some of the criticisms posed:\n\n\n1. Based on the suggestion of other reviewers we agree that the correlation analysis and the emotion detection evaluation can be improved by taking into account the different factors produced during the EFA and analyzing the factors independently, pointing out anomalies, and providing more examples. Significance tests, therefore, could also be an excellent improvement for this section.\n\n2. For now, we have opted to only evaluate BERT, RoBERTa, and SpanBERT. This was a conscious choice and we wanted to focus the evaluation on the comparisons between SpanBERT (designed for higher complexity) versus BERT and RoBERTa (designed for general use cases). However, we do plan to continue this line of research and conduct similar studies on the zero-shot, few-shot emotion detection capabilities of generative models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study provides insights for enhancing existing emotion detection models. It raises the issue that text expressing negative emotions and those with syntactic complexity pose greater challenges for model classification, and thus potentially need to be addressed specifically.", "reasons_to_reject": "The paper's limitations are significant:\n\n1. The correlation analysis is preliminary. Conclusions are made solely based on average scores of metrics for positive and negative samples. Comprehensive testing, including significance tests, is needed.\n2. The emotion detection evaluation lacks depth. Only basic experiments and metrics (like F1 score) were performed and reported.\n3. State-of-the-art emotion and sentiment detection models should also be assessed.\n4. There's a notable absence of detailed analysis explaining anomalies in certain metrics when assessing correlations between negative and positive emotions (refer to Question A).", "questions_to_authors": "A. Can you provide insights or explanations as to why TF-IDF, Avg. Log-Likelihood, and Type Token Ratio % exhibit higher average scores for positive polarity? Does this suggest that positive emotions have greater lexical complexity, given that all three of these metrics are lexical metrics?\n\nB. Regarding the 120,000 samples from the raw dataset version used to train the classifiers, do they ensure a balanced representation of both positive and negative samples, as well as an even distribution between complex and simple samples?", "paper_topic_and_main_contributions": "This paper examines the relationship between linguistic complexity and the detection of negative emotions. The primary insights from this study are:\n\n1. Texts expressing negative emotions tend to have a higher degree of complexity compared to those conveying positive or neutral emotions.\n2. Transformer-based models are more likely to misclassify texts that are syntactically complex and those expressing negative emotions.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the relation between language complexity when expressing emotions, and the performance of transformer-based models on emotion classificaition. The experimental setul and analysis are sound as highlighted by all reviewers. Reviewers found the paper moderately exciting (i.e. interesting research question grounded on linguistic theory).
Decision: Accept-Findings