EMO-KNOW: A Large Scale Dataset on Emotion-Cause
Authors: Mia Huong Nguyen
Yasith Samaradivakara
Prasanth Sasikumar
Chitralekha Gupta
Suranga Chandima Nanayakkara
Venue: EMNLP 2023 Findings (2023)
Keywords: Dataset
Emotion Analysis
Emotion-Cause
Large-scale
TL;DR: A large-scale dataset of emotion and emotion-cause on Twitter

Abstract: Emotion-Cause analysis has attracted the attention of researchers in recent years. However, most existing datasets are limited in size and number of emotion categories. They often focus on extracting parts of the document that contain the emotion cause and fail to provide more abstractive, generalizable root cause.
To bridge this gap, we introduce a large-scale dataset of emotion causes, derived from 9.8 million cleaned tweets over 15 years.
We describe our curation process, which includes a comprehensive pipeline for data gathering, cleaning, labeling, and validation, ensuring the dataset's reliability and richness. We extract emotion labels and provide abstractive summarization of the events causing emotions. 
The final dataset comprises over 700,000 tweets with corresponding emotion-cause pairs spanning 48 emotion classes, validated by human evaluators. 
The novelty of our dataset stems from its broad spectrum of emotion classes and the abstractive emotion cause that facilitates the development of an emotion-cause knowledge graph for nuanced reasoning.
Our dataset will enable the design of emotion-aware systems that account for the diverse emotional responses of different people for the same event.

EMO-KNOW: A Large Scale Dataset on Emotion and Emotion-cause
Mia Huong Nguyen Yasith Samaradivakara
Prasanth Sasikumar
Suranga Nanayakkara A
Chitralekha Gupta
School of Computing, National University of Singapore
School of Computing, University of Colombo {mia, yasith, prasanth, chitra, surranaaJeahlab.. org
Abstract
Tweet: I'm home but doesn't feel like it. And I'm angry because of
hate the feeling of being lonely. They are with me. But foel crapty
Emotion-Cause analysis has attracted the at- tention of researchers in recent years. How- ever, most existing datasets are limited size and number of emotion categories. They of- ten focus on extracting parts of the document that contain the emotion cause and fail pro- vide more abstractive, generalizable root cause. To bridge this gap, we introduce large-scale dataset of emotion causes, derived from 9.8 million cleaned tweets over 15 years. We de- scribe our curation process, which includes comprehensive pipeline for data gathering, cleaning, labeling, and validation, ensuring the dataset's reliability and richness. We extract emotion labels and provide abstractive summa- rization of the events causing emotions. The final dataset comprises over 700,000 tweets with corresponding emotion-cause pairs span- ning 48 emotion classes, validated by human evaluators. The novelty of our dataset stems from its broad spectrum of emotion classes and the abstractive emotion cause that facilitates the development of emotion-cause knowl- edge graph for nuanced reasoning. Our dataset will enable the design of emotion-awee aware sys- tems that account for the diverse emotional re- sponses of different people for the same event.
Emotion; angry
feeling lonely empty despito being
Twest: you're not making mistakes, not doing unything. feel positive because make mistakes and can be better uftr that.
Emotion: positive
Cause: overeoming mistakes und becorning better person
Figure 1: Examples of data points in EMO-KNOW
social because T feel very lonely in real life. My friends suck, my relatives suck more and have nobody in my life. Even my dog was taken away, so if you are wondering about how I am feeling? Yes, am lonely." The second sentence is typically annotated as the emotion's cause in existing datasets, but this fails to highlight the more generalized root cause. This way of annotation, however, fails to highlight the more generalized root cause, which in this case is the lack of fulfilling relationships in the person's life. This level of abstraction is essential for comprehensive emotion- cause understanding and reasoning.
A further limitation of existing datasets lies in the coarse granularity of their emotion categories. Modern theories of emotion suggest that emotions are not simply reducible to a basic set of 6-8 emo- tion categories (Barrett, 2017, Cowen and Keltner, 2017). Therefore, labeling user's 's emotions with a limited set of emotions will either miss out on many other emotions or forcefully label their emotions something else. For this reason, finer granular- ity of emotion categories would better reflect the true diversity and nuance of human emotions. For instance, 'pride" and 'gratitude" both fall under the category of 'joy,' but each carries distinct nuances and potentially different causes.
1 Introduction
Emotion-Cauu Cause analysis is gaining interest due to its potential impact on applications such as empa- thetic dialog systems and mental health support chats. It is more challenging than traditional emo- tion recognition in text because it requires a higher level of semantic understanding (Lee et al., 2010, Gui et al., 2016, Xia and Ding, 2019).
Most existing works (Gui et al., 2016; Bostan et 2020; Kim and Klinger, 2018) model emotion cause analysis an extraction problem, resulting in datasets that provide only specific, descriptive information on emotion causes and lack deeper, abstract causes. For example, in a tweet, a user expresses their emotion by stating, am often on
While previous work by Zhan al. (2022) has made progress in summarizing events that trigger emotions, is limited scale and covers only seven emotions. To enable AI models to reason more

---

Fharigg Censering
Laaiding
Search Fresss
Twitter
Lebeb
Sample
A
EMO - KNOW
Figure 2: Data generation pipeline
diversely and discerningly about emotion-caut cause, comprehensive knowledge base required. How- ever, manually annotating a large-scale dataset is expensive. this work, we present EMO-KNOW, a validated dataset on emotion causes curated from 9.8 million tweets over 15 years. Our dataset con- tains tweets where users describe their emotions and triggering events, spanning 48 emotion types. Unlike other datasets that rely outside label- ing,
of <emotion, emotion experiencer, emotion cause>. Diverging from the extractive approach, Zhan al. (2022) developed a new dataset focusing on emo- tion and triggers by summarizing the events that lead to particular emotions. This dataset has 1883 Reddit posts, annotated with emotions (7 classes) and abstractive summaries of their triggers. As manual annotation often involves the cost of hu- man labor, most annotated datasets are limited in size, and limited in the number of emotion classes they contain.
the users' words, making more genuine. With this data, combined with the emotion-cause details, we can create map that shows how emotions are linked to causes, helping us understand the connections better. (Wang et al.. 2021, Qian et al., 2023). Our contributions are method to create a large-scale, high ecological va- lidity emotion dataset, validation with automatic metrics and human evaluators, and open-sourcing the ! (700K datums) for researchers.
Automatic Data Labeling: an effort to circum- vent the costs of manual annotation, researchers have devised ways to generate large-scale datasets using pre-existing models. For instance, We- livita et al. (2021) produced EDOS by fine-tuning RoBERTa small label set created through crowdsourcing, and then used it to label million conversations in the Opensubtitles dataset (Lison et al., 2018). Similarly, Welivita and Pu (2022) created HEAL a knowledge graph consisting million distress narratives, annotated for emotion, post summarization, and node clustering using a variety of pre-existing models. A prevalent short- coming of emotion labeling with current models is their limited accuracy; the emotion classifiers utilized in both HEAL and EDOS only achieved curacies ranging from 65% to 65. 65.88%. We address this limitation by harnessing users' self-reported emotion labels and rigorously assessing the perfor- mance of our fine-tuned emotion cauue summariza- tion model with human evaluators.
2 Related Work
Annotated Emotion Cause Datasets. The field of emotion analysis has rise in interest recently, leading to the creation of several datasets that pro- vide annotations for both emotion and its causes. (Zheng et al., 2022, Chen et al., 2020) For exam- ple, Lee et al. (2010) introduced a dataset of 5,964 entries with word-level annotations of emotion and emotion causes, covering 6 emotion classes. Gui et al. (2016) annotated 2,105 news articles with clauses that contain 6 emotion classes and emo- tion causes. For English corpora, Bostan al. (2020) annotated dataset of emotion, emotion cause and affect intensity, comprised of 5000 news headlines spanning 15 emotion classes. Kim and Klinger (2018) introduced dataset of 1720 triplets
Methods
3.1 Data Curation
Data was curated by scraping tweets from Twitter using the Twitter API over a period of 15 years,

---

spanning from 2008 to 2022. To extract tweets con- taining emotional expressions and potential emo- tion causes, we refined search phrases iteratively. We explored the popular emotion expressed by users by examining the frequency of emotion words on a sample of 300K tweets and picked a cut-off at 48. We used this list of 48 emotions to iteratively refine and expand our search space to obtain a set of 9.8 million data. We applied data cleaning and refinement pipeline, which include removing expression of sympathy, profanity, and tweets with non-meaningful patterns. The process results in a dataset of 772,863 tweets that follows the following patterns:
do feel e? At first glance, this task appears sim- ple, especially considering tweets are usually short. Therefore, one might expect traditional Q&A mod- els like BERT (Devlin et al., 2019) to perform well. However, most Q&A models are designed to an- swer questions in an extractive way, which prevents them from providing abstractive, generalized an- swers about emotion causes.
Given these limitations of extraction -based mod- els in identifying emotion causes, an alternative approach would be to frame the problem as a gen- erative question-answering problem. This method, however, requires additional training data for pre- trained models. Recent developments in Large Lan- guage Models (LLMs) such GPT-3, InstructGPT (Ouyang et al., 2022) have shown impressive abili- ties in following instructions and performing tasks like summarization.
"I x (e1le21... le4s) because"
"I x (w+) (E1 feile21.. .. le4s) because" where X in {am, feel, am feeling) and e₃ is in the list of 48 emotions.
LLMs have been trained on a vast amount of text and have often demonstrated a remarkable ability to generalize. Previous studies, like those by Bosselut et al. (2019) have used LLMs to generate abstrac- tive common-sense knowledge relations. These models have also been used to generate pseudo- labels for training smaller models on more special- ized tasks, as noted in studies by Ye et al. (2022) and (Taori et al., 2023). This method is very eco- nomical. It climinates the need for human anno- tators while utilizing freely available pre- pre-trained language models without incurring the full cost of labeling millions of tweets with GPTs. Using this approach, we experimented with various prompts before settling on the following prompt to obtain the emotion and cause labels from InstructGPT: "Answer the following questions: emotion expressed in the text? 2. What is the cause of such emotion? Provide the answer in the follow- ing format strictly: 1. E where E is the emotion from this list: [list of 48 emotions]. 2. C where Cs the summary of the cause in 10 words. Use NA for when there is no emotion, and use NA for C when there no cause. Wee selected random sample of 50,000 tweets to feed into InstructGPT to obtain the emotion-cause labels. After filtering out entries with NA for both emotion and causes, we ended up with 42,956 data points.
We detailed our data curation procedure as well as our data statistics in Appendix A.
3.2 Data Labeling
3.2.1
Emotion Labeling
Since dataset consists of tweets that follow the outlined structure, select the emotion words that fit this pattern as the emotion label for each tweet. Although exceptions exist, such as instances of negating statements within tweets, our error anal- ysis found that most cases follow this rule. Thus, we simply extract the emotion label from the word that fits into our identified pattern. This method en- sures that the emotion label accurately reflects the individual's 's true feelings, unlike traditional meth- ods where emotion labels are determined by human annotators, which can introduce biases.
What is the
3.2.2 Labeling with Large Language Models
Model
BLEU-2
BLEU-4
BERTSc
T5-flan
0.40
0.34
0.91
b5-base
0.36
0.31
0.89
BaRg-large
0.35
0.31
0.88
Table 1:
Automatic Evaluation for Candidate Models
After obtaining the emotion label, we proceed to extract the cause of the emotion. We approach this task by treating as a question-answer challenge. Given a tweet and an emotion label e, we train a language model to answer the question: Why
Lastly, we evaluated several language model can- didates on the training dataset identify the most suitable model for labeling the entire dataset. For the task of generative text, we considered three can- didate models: BART-large, T5-base, and T5-flan.

---

Happy
Angry
Sad
Guilty
Proud
Frustrated
Inspired
Alone
Nostalgic
Friends
Betrayal
Inability
Due
Success
Desire
Actions
People
Missing
Life
Inability
Left
Overeating
Achieving
Difficulty
Belief
Connection
Childhood
Love
Frustration
Missing
Studying
Completing
Overwhelming
Desire
Despite
Memories
Success
Lack
Person
Time
Overcoming
Poor
Influence
Interaction
Time
Presence
Loss
Uncertainty|
Work
Victory
Understanding
Support
Missing
Playing
Table 2: Results of topic modeling through LDA(Blei al., 2003). Keywords selected from five most prominent
topies among abstractive summaries of causes of selected few emotion categories in EMO-KNOW.
These models were trained and assessed on the 42k dataset (which was split into train, val, and test sets in 6:2:2 ratio). The models' performance was measured using the BLEU scores (Papineni et al., 2002) and BERT-score (Zhang et al., 2020), which are reported in table As shown in the results, T5-flan outperforms all the other models for all the metrics, scoring 0.91 on BERT-score making it very close to the performance of the responses generated by InstructGPT. To guarantee the quality of our dataset, we sample 100 data points and man- ually inspected the emotion cause generated by the model. We observed that tweets with poor grammar tend to have poor quality of emotion causes. There- fore, utilize TextAttack's grammar checker model to filter out datapoints with poor grammar. This step leaves the dataset with 698800 tweets. Figure 2 shows a summary of our method and table 2 shows the popular keywords of emotion causes that correspond to selected number of emotions.
and its summary, evaluators check that the details align. They verify the emotion label's accuracy and evaluate the emotion cause on a scale of to 5.
We compared data quality from human anno- tators and our model. Specifically, understand the differences in the interpretation of emotions and their causes in tweets. To achieve this, we asked annotators to identify the emotion and describe its cause. We provided four potential emotion label options hased on the tweet's emotion words. If there were fewer than four emotion words, we selected labels with similar affective properties. We used AffectiveSpace (Cambria et al., 2015) to identify emotion words with similar affective prop- erties and provided four pre-generated options for emotion labels. Annotators could provide their own response none of the options represented the expressed emotion. For the cause description, we offered pre-generated summaries and allowed annotators to supply their own response.
want to
4 Evalution
4.2 Evaluation procedure
We sourced annotators from Amazon Mechanical Turk, with selection criteria including being na- tive English speaker and having completed over 500 HITs with 95% above acceptance rate. We conducted qualification task involving annotat- ing 10 tweets and inspected the quality manually before inviting qualified workers to rate our data. To validate our dataset, we randomly selected 300 samples and had three annotators answer four questions about the emotion and summary of the cause. Annotators were paid an average of $12 per hour. Appendix B contains more information about Mechanical Turk task.
4.1 Evaluation criteria
We model emotion-cause identification as an ab- task, but the answers are often short summaries. Traditional metrics fail to capture the quality of these summaries (Kryscin- ski et al., 2020). we use human evaluation cri- teria commonly used in summarization tasks. We adapt the criteria to fit the brevity of tweets and use Relevance, Fluency, and Consistency metrics while omitting Coherence, which is typically used for long text. This approach is similar to that used by Zhan et al. (2022). Relevance assesses whether the emotion cause accurately represents the root cause of the emotion. For tweets that describe mul- tiple events, the principal source of emotion should be the targeted event for the emotion cause. Flu- ency looks at the grammatical correctness of the response. To ensure consistency between the tweet
stractive
4.3 Evaluation Results
METRIC
Relevance
Fluency
Coherence
AVERAGE
4.50
4.47
4.54
Table 3: Emotion cause summarization evaluation score
httn:///nggnngfccccoooocaaaahhhe CoLA
Table 3 show the human evaluation results. The

---

average score on Relevance, Fluency and Consis- teney from human annotators on 300 samples are 4.50, 4.47 and 4.54 respectively. Also, 90% of the emotion causes in our dataset have average rating on all three criteria of 4.0 or above. This shows that the emotion cause generated are of good quality, where the abstractive causes are consistent with the emotion and the cause event mentioned in the tweets. Moreover, annotators agree with our emo- tion label 89.5% of the time, and when asked how would they describe the cause of the emotion, 98% the annotators chose to use one of the causes pro- vided by the model. This showed that our dataset contains high-quality emotion causes and accurate emotion labels.
the Department Information Systems and Ana- lytics, School of Computing, NUS for Suranga Nanayakkara.
References
Lisa Feldman Barrett. 2017. How emotions are made: the secret life of the brain. Houghton Mifflin Har- court, Boston New York.
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz. and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In Pro- ceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 4762- 4779, Florence, Italy. Association for Computational Linguistics.
5 Conclusion
We created novel, comprehensive dataset of 700K datum points, which aims to simultaneously iden- tify emotions in text and determine the underlying causes behind each emotion. We detailed the data curation, label generation and evaluation pipeline so that researchers can use to create their own datasets that cater their needs. Finally, we pub- lished the dataset so that other researchers can make use of the dataset for their tasks.
Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020. GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 1554-1566, Marseille, France. European Language Resources Association.
Erik Cambria, Jie Fu, Federica Bisio, and Soujanya Poria. 2015. AffectiveSpace 2: Enabling Affective Intuition for Concept-Level Sentiment Analysis. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 29(1).
6 Limitations
We believe that our datasct will bring great benefits to the community, however, it is not without limita- tions. First, the number 48 was chosen heuristically as we explored the dataset. Second, we labeled emotion classes with patterns matching. This is not 100% accurate and cannot detect sarcasm or nega- tions. Emotion and emotion-cause pairs will not be exhaustively extracted in tweets with multiple emotions and multiple causes. Third, cannot guarantee that tweets with vulgar languages are completely cleaned. Lastly, although the human rating of the cause are very high, the quality of the abstractive cause depends largely on the capability of the pre-trained model.
Xinhong Chen, Qing Li, and Jianping Wang. 2020. Con- ditional Causal Relationships between Emotions and Causes in Texts. Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 3111-3121, Online. As- sociation for Computational Linguistics.
Alan Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38).
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis. Minnesota. Association for Computational Linguistics.
7 Ethical considerations
This research is approved for Internal Board Re- view exemption, reference code number NUS-IRB- 2023-485
Lin Gui, Dongyin Wu, Ruifeng Xu. Qin Lu, and Yu Zhou. 2016. Drint--riven Emotion Cause Ex- traction with Corpus Construction. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 1639-1649, Austin, Texas. Association for Computational Linguistics.
8
Acknowledgements
This research is supported by SINGA Award for Mia Huong Nguyen and the start-up grant from

---

Evgeny Kim and Roman Klinger. 2018. Who Feels What and Why? Annotation of a Literature Corpus with Semantic Roles of Emotions. In Proceedings of the 27th International Conference on Computa- lional Linguistics, pages 1345-1359, Santa Fe, New Mexico, USA. Association for Computational Lin- guistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto, 2023. Stanford alpaca: An catructioo-following llama model. https:// gi thub. wwt.eoccocccaaaeee http:/tattulabb//om haallllabsstttta http://taffoddaapppe/aa
Jiashuo Wang. Wenjie Li, Peiqin Lin, and Feiteng Mu. 2021. Empathetic Response Generation through Graph-based Multi-hop Reasoning on Emotional Causality, Knowledge-Basee Systems, 233:107547.
Wojciech Kryscinski. Bryan McCann, Caiming Xiong. and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 9332-9346, Online, Association for Computa- tional Linguistics.
Anuradha Welivita and Pearl Pu. 2022. HEAL: A Knowledge Graph for Distress Management Con- versations. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10): 1459-114671
Sophia Yat Mei Lee, Ying Chen, Shoushan Li, and Chu- Ren Huang. 2010. Emotion Cause Events: Corpus Construction and Analysis. In Proceedings of the Seventh International Conference on Language Re- sources and Evaluation.
Anuradba Welivita, Yubo Xie, and Pearl Pu. 2021. A Large-Scale Dataset for Empathetic Response Gen- eration. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251-1264 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubiitles2118: Statistical rescoring of sentence aligmments in large, noisy parallel corpora. Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re- sources Association (ELRA).
Rui Xia and Zixiang Ding. 2019. Emotion-Caute Pair Extraction: New Task to Emotion Analysis Texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1003-1012, Florence, Italy. Association for Compu- tational Linguistics.
Saif Mohammad, Xiaodan Zhu, and Joel Martin. 2014. Semantic Role Labeling of Emotions in Tweets. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Me- dia Analysis. pages 32-41. Baltimore, Maryland. As- sociation for Computational Linguistics.
Jiacheng Ye, Jiahui Gao, Qintong Li. Hang Xu, Jiang- tao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022. ZeroGen: Efficient Zero-shot Learning via Dataset Generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653- 11669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Hongli Zhan, Tiberiu Sosea, Cornelia Caragea, and Junyi Jessy Li. 2022. Why Do You Feel This Way? Summarizing Triggers of Emotions in Social Media Posts. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kil- ian Q. Weinberger. and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. ArXiv: [cs].
Yushan Qian, Bo Wang, Ting-En En Lin, Yinhe Zheng, Ying Zhu, Dongming Zhao, Yuexian Hou, Yuchuan Wu, and Yongbin Li. 2023. Empathetic Response Generation via Emotion Cause Transition Graph. ArXiv:2302. 787 [cs].
Xiaopeng Zheng, Zhiyue Liu, Zizhen Zhang. Zhaoyang Wang, and Jiahai Wang. 2022. UECA-Prompt: Uni- versal Prompt for Emotion Cause Analysis. In Proceedings of the 29th International Conference Computational Linguistics, pages 7031-7041, Gycongju. Republic of Korea. International Com- mittee on Computational Linguistics.
Tiberiu Sosea, Chau Pham, Alexander Tekle, Comelia Caragea, and Junyi Jessy Li. 2022. Emotion analysis and detection during COVID-19. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6938-6947, Marseille, France. Eu- ropean Language Resources Association.

---

Search phrase
mum tweets
"("I feel sad" OR "I feel happy" OR .. OR "I feel drained")"
1,300,928
"("Tm sad" OR "I" m happy" OR OR "I'm drained")"
3,109.289
"("Im feeling" OR "I am feeling" OR "T m feeling" OR "I feel") because" "("T' m feeling sad because" OR OR ("I'm m feeling drained because")"
3,798,830
230,191
"("Tm feeling sad" OR "I'm feeling happy" OR OR "I'm feeling drained")) because "("I'm m sad because" OR "I'm happy because" OR OR "I'm 'm drained because")"
120,218
408,291
"("Im" OR "I am" OR "T'm" OR "iam" OR "i am" OR "i'm") (happy OR... OR drained) because"
930,900
Table 4: Search phrases and their corresponding number of tweets
A Data Curation Pipeline
tify tweets that included phrases like am," feel," "I am feeling," and so on, combination with the list of emotions we had previously iden- tified. In total, we tested eight different search phrases, resulting in an initial dataset of 9.8 million tweets. The table 4 provides additional information about each search phrase and the corresponding number of curated tweets.
Data was curated by scraping tweets from Twitter using the Twitter API over a period of 15 years, spanning from 2008 to 2022. To extract tweets con- taining emotional expressions and potential emo- tion causes, we refined search phrases iteratively. We narrowed down the search by searching for tweets where users explicitly express their emo- tions and the cause of their emotions using the following search phrase: "("Im feeling" OR "I am feeling" OR "I'm feeling"- like) because" This search phrase yields 350,000 tweets. By follow- ing prior works, such as Mohammad al. (2014) and Sosea et al. (2022), we cleaned the tweets by removing emojis, emoticons, hashtags, and dupli- cated letters. The data that had been clcaned was then processed using a heuristic algorithm based on predefined rules. This algorithm aimed to extract emotions and the causes of those emotions. The cleaned tweets were divided into individual tokens and tagged with their respective parts of speech ing the Part-of-Speech (POS)². tagging technique. each tweet, we searched for the phrases "I feel" or "I am feeling" and analyzed the six words that followed those phrases to identify words related to emotions. We assigned the emotion word found the label for the emotion in that tweet. Typ- ically, emotion-case datasets are limited to only six to eight labels. However, upon analyzing the initial results, discovered broad spectrum of words associated with emotions. To address this, we examined the frequency of occurrence for differ- ent emotion words and identified the top 48 most frequently used ones. This list served as a tool to enhance the search process by reducing noise and focusing on relevant data points. The updated search phrases were specifically designed to iden-
To ensure the desired emotion pattern ("I am/feel/am feeling ..."), we applied additional fil- tering rules. We excluded tweets where emotions were not structured in a meaningful order, like "I am," "cmotionX," " and "because" within the same tweet. Phrases like "I feel bad for ..." and "I feel sorry for..." expressed sympathy rather than the emotion "bad," and excluded them from the final dataset. Addi- tionally, we filtered offensive or inappropriate content by comparing the tweets against a list offensive words
marked them sympathy
To further refine the dataset, we excluded tweets where no phrases appeared after "because" since they were less likely to express the cause of an emotion. As result of this filtering process, the dataset reduced to around 3.5 million tweets. To enhance the dataset's quality, we selected data points that adhered to specific patterns, such as
"I X (e1le21. |e50) because"
"I x (w+) (c) |c2 .. |00) because" where X in (am, feel, feeling)
where e; represents the emotion word; (w+) is place holder for exaggerators such as "very", "ex- tremely", etc. These patterns are the most explicit and direct patterns that people can use to express their emotions and the cause behind them. After ap- plying the filter, the dataset was reduced to 800,000 tweets.
"htpss//www.llkkorrg/
hhtps:/wwwwcs.ccm wwt.c/ww.c.c...////cctcmmmmo higlour/ssoureee/hadwwwrr.tt.

---

B Human Evaluation
guilly oollouuum stugid because supposed have project my
Question t: Whis emsuson exprossed teat? onginst word where spolkostree
Qailty
Feei guitty because miko dlaseee leten colloqulum but that shupid because I'n supposed to hawe project and reaearch and do my best for but ettill Feel bad exprossed eneodono?
Diample pia: tweet Entryday tiss lest preas lacause sheaad with not one citt wio lovs. Cacase having move
mixadd conore hoguumm morea of propectreenne
meto no obotao liton to reboquum nostca of working on 9109003
abews
Figure 3: Human Evaluator Questions
Twest: feel gullty becouse mise chasess to leten to colloqutum but that's stupid because I'm supposed have a project and research and my best for but stll feet bad guity Cause: missed classes for colioqulum instead of projcozlescarch
Question Does the emotion label accuratelrefen reflect the use''s expressed emotion?
Figure 4: Human Evaluator Questions
Definitions
cinetyy reproseed
Conadancy
The desolction accuraldy captures the specifis of the use's attution that resulted in expreseed emeotion
Fi Flener
discriction gramenctioaly conuctinn and understand
Question Rata the desoription of the emotion cause based on orteria
Cause- description
Theer:
thet's srgnid
propctreseac
Figure 5: Human Evaluator Questions
Data Distribution

---

Emotion
#tweet
Percentage
Emotion nervous
#tweet 11515
Percentage
bad
166732
23.86%
1.65%
happy
126106
18.05%
good
11231
1.61% %
excited
40010
5.73%
scared
10923
1.56%
upset
37276
5.33%
proud
9356
1.34%
sad
31574
4.52%
grateful
6978
1.00%
sorry
23864
3,41%
frustrated
6405
0.92%
tired
19965
2.86%
terrible
6192
0.89%
depressed
19434
2.78%
emotional
6023
0.86%
sick
17644
2.52%
confident
4944
0.71% %
guilty
17440
2.50%
anxious
4820
0.69%
lonely
15558
2.23%
great
4739
0.68%
blessed
15104
2.16%
awful
4308
0.62%
angry
14274
2.04%
lost
4175
0.60%
stupid
13813
1.98%
down
3473
0.50%
lucky
12650
1.81%
uncomfortable
3146
0.45%
stressed
11836
1.69%
nauscous
2474
0.35%
insecure
2265
0.33%
exhausted
1920
0.28%
helpless
1831
0.27%
overwhelmed
1632
0.24%
hopeful
696
0.10%
optimistic
875
0.13%
motivated
634
0.09%
low
543
0.08%
inspired
542
0.08%
nostalgic
541
0.08%
hopeless
510
0.07%
drained
495
0.07%
positive
449
0.07%
discouraged
389
0.06%
Table 5: Distribution Emotion Classes


Reviews:
[{"review_id": "XT65Ipndww", "reviewer": "Reviewer_XtNV", "summary": "", "questions": "- Why sympathy is not counted as another emotion?\n- Why only emotions are extracted from tweets having specific text patterns, how is it novel from \nother works?\n- How the bias is not introduced the way you have scraped data.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We would like to address your concerns as follows:\n1. Emotion cause extraction done on explicit sample containing emotion and their cause\n - We do not claim a novel method to predict emotions. \n - The use of patterns to extract users' emotion is to capture the authentic emotion of users, in their own words, in contrast with hiring human annotators to annotate what the annotator *think* the users’ emotions to be. **This is to warrant the quality of our dataset**. \n - Emotions are very personal and subjective experiences where expressions of emotions vary among individuals and can be significantly different across cultures[1, 2,3]. What human annotators can provide is their best guesses of users’ emotions, which are often subject to social bias and stereotypes. \n - Though using pattern matching to extract emotion labels appears to be simplistic, this approach provides an effective and elegant solution to the issue of bias in emotion annotations. \n\n2. No novelty on data modelling\n- We believe we should have better highlighted the main contribution of the paper (which we will address in the revised version). Our main contribution is a large-scale dataset of **rich emotion classes** and **abstractive emotion causes**, serving as a knowledge base that enables other researchers to build models that understand the nature of human emotions better. \n- By abstractive emotion cause, we mean a generalizable cause that can be adapted to different situations. Consider these two examples: \n  - “If you are not making mistakes, you’re not doing anything. I feel positive because I make mistakes and I can be better after that.  \nEmotion: positive.  \nCause: overcoming mistakes and becoming a better person.  \nAnd: \n  - 'When setbacks arise, I see them as opportunities for growth. I feel motivated because each mistake I make is a chance to learn and improve.\"  \n  Emotion: motivated.  \n  Cause: overcoming mistakes and becoming a better person.  \n\nIn the two cases, the abstractive cause is the same but expressed in different wordings.\n- Having abstractive emotion-cause enables AI models to learn underlying patterns, enhancing the ability to predict emotional responses across a wide array of contexts. This is the novel contribution of our approach.  In contrast, prior work mainly used extensive annotation to extract parts of the text that contains emotion-cause, making the extracted cause very context-specific and hard to generalize.\n\n3. Number of emotions identified is not properly justified.\nThank you for pointing this out. We briefly discussed our exploration of emotions expressed in Twitter in Appendix A (line 516-518). We examined the frequency of emotion words on a sample of 300K tweets and picked a cut-off at 48. We will move this info from the appendix to the main body of the paper n in our revised manuscript. \n\n4. No analysis on hard examples where there is overlap in emotions but causes separates them.\nThank you for your comment. Table 2 provides a glance of how emotions of related categories can have very different causes. For example, “proud” and “inspired” will be categorized as happiness in all other datasets, but they have very different sets of causes. Given the page limit, we had to prioritize other information to show you that the dataset that we constructed is of high quality and is rigorously validated. We will add more details explaining Table 2.\n\n### For the questions: \n1. Why is sympathy not counted as another emotion? \nWe eliminated “expression of sympathy” for tweets with the pattern “I feel bad/sorry for …” since the person is not really feeling “bad”, and the clause following “for” often describes the situation of someone else and not the users. For the purpose of capturing users’ true emotions and emotion cause, we eliminated these expressions to ensure the quality of our dataset. \n2. Why only emotions are extracted from tweets having specific text patterns, how is it novel from other works? \nAs stated in point 2, the purpose of patterns is to eliminate the need to manually annotate emotion labels (which is costly and not scalable), overcoming the problem of social bias and stereotypes. Other datasets assumed emotions are of only 6-8 categories, overlooking the nuance differences between fine-grained emotions such as proud, gratitude, inspired, lonely, nostalgic, etc. Without patterns, these fine-grained emotion labels need to be manually labeled by annotators - a process that is very costly and prone to biases. \n3. How the bias is not introduced the way you have scraped data? \nThere are several ways that bias can be introduced in the way we scraped the data:\nTweets with emotional expressions that do not follow our defined patterns will not be selected into the dataset.\nTweets with emotion words that are not in our list will also be absent. \n            Although these selection biases exist, it does not affect the quality of our dataset, that is the size of the dataset, the richness of emotion label, the generalizability of emotion causes. \n\n**References**: \n1. Hendrik Schuff, Jeremy Barnes, Julian Mohme, Sebastian Padó, and Roman Klinger. 2017. Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 13–23, Copenhagen, Denmark. Association for Computational Linguistics.\n2. Troiano, E., Padó, S., & Klinger, R. (2021). Emotion Ratings: How Intensity, Annotation Confidence and Agreements are Entangled. Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.\n3. Sven Buechel and Udo Hahn. 2017. Readers vs. Writers vs. Texts: Coping with Different Perspectives of Text Understanding in Emotion Annotation. In Proceedings of the 11th Linguistic Annotation Workshop, pages 1–12, Valencia, Spain. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Large scale data collection approach\n- Human annotation efforts", "reasons_to_reject": "- Emotion cause extraction is done on a sample which is very explicit on containing emotion and their cause. This is because the authors used a particular pattern to extract those.\n- No novelty on data modelling, use of existing models to extract emotion through formulating it to text summarization task\n- Number of emotions identified is not properly justified.\n- No analysis on hard examples where there is overlap in emotions but causes separates them.", "questions_to_authors": "- Why sympathy is not counted as another emotion?\n- Why only emotions are extracted from tweets having specific text patterns, how is it novel from \nother works?\n- How the bias is not introduced the way you have scraped data.", "paper_topic_and_main_contributions": "This paper describes a twitter data scraping pipeline for emotion and emotion cause related tweets collection and the use of SOTA LLMs to find out the emotion cause of a tweet provided an emotion by formulating the task as a prompt to the LLMs as a question; \"provided an emotion e, what is the cause of it?\"", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UoXa92JJYG", "reviewer": "Reviewer_mpV3", "summary": "", "questions": "1.  will the finer granularity of emotion categories bring benefits to the tasks of cause detection and emotion-cause pair detection?\n2.  lines 206-207 mentioned that author(s) adopted prompt to obtain emotion cause label from GPT3.5. So, what is the prompt used in this paper like? can author(s) provide a prompt template used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We appreciate your time and we understand the concerns that you raised. We would like to address them as follows:   \n1. We agree that our introduction has not made it clear the need for having a dataset with rich emotion categories. Prior work largely agree that human emotions are complex, multifaceted, and not reducible to a set of 6 basic emotions (happy, sad, angry, surprise, disgust, fear) as in Ekman’s model[1]. Having finer granularity of emotion labels means richer information of emotion and the context that give rise to such emotions. A model that can distinguish between pride and gratitude would be able to give a more tailored response/prediction/reasoning about the emotional information it is processing than a model that can only distinguish happy and sadness. This is precisely the reason why we see a growing number of datasets that go beyond 6-7 emotions classes.[ 2, 3]\n\n2. We used pattern matching (described in appendix A) and mentioned in line 160, selection 3.2.1 for the precise reason: avoid the need to build a model that classifies tweets in 48 emotion categories. Our models do not predict emotion categories, we select tweets where emotion labels are mostly obvious to extract to ensure the quality of our dataset. This is reflected in our evaluation section.   \n\nAs for your questions, please find the answers in the following:\n1. The finer granularity will bring more nuanced prediction results for models in emotion cause detection and emotion-cause pair prediction.\n Let’s take an example:  \n\t‘This morning I came to work late. There was a car accident so my bus couldn’t move. I got stuck in traffic for 1 hour. When I came to the office, my colleague told me his friend got into an accident and was in a critical condition. I feel sorry for the guy, and I am thankful that death has spared me”.   \nWithout a “grateful” category, it is easy to classify this text as sadness and predict the cause for the emotion as “came to work late”. Doing so would completely miss the point of the text. \n\n2. . Thank you for point this out. The prompt is: \"Answer the following questions: 1. What is the emotion expressed in the text? 2. What is the cause of such emotion? Provide the answer in the following format strictly: 1. E where E is the emotion from this list: [{list of 48 emotions}], 2. C where C is the summary of the cause in 10 words. Use NA for E when there is no emotion, and use NA for C when there is no cause.”\nWe will revise our manuscript and add this to the main pages. \n\n**References**: \n1.  Paul Ekman. 1992b. An argument for basic Emotion. Cognition & Emotion, 6(3-4):169–200.\n2.  Demszky, Dorottya, et al. \"GoEmotions: A Dataset of Fine-Grained Emotions.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n3.  Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. author(s) constructed a large-scale dataset for emotion cause analysis, and it might bring benefits for the further study on the application of pretrained model with the emotion cause analysis.", "reasons_to_reject": "1. the motivation of constructing finer granularity of emotion categories is not clear. I felt confusing about the profits of finer finer granularity of emotion categories in emotion cause analysis.\n2.  the model is required to have strong understanding capacity to predict the finer granularity of emotion categories, which means the model is more possible to make wrong prediction. Further, the model will face bigger challenge based on a wrong emotion prediction.", "questions_to_authors": "1.  will the finer granularity of emotion categories bring benefits to the tasks of cause detection and emotion-cause pair detection?\n2.  lines 206-207 mentioned that author(s) adopted prompt to obtain emotion cause label from GPT3.5. So, what is the prompt used in this paper like? can author(s) provide a prompt template used?", "paper_topic_and_main_contributions": "The paper introduces a large-scale dataset of emotion causes, and describes the pipline of dataset construction. However, author(s) did not  indicate the motivation of constructing this dataset and the advantages of the proposed dataset compared with the existing emotion cause dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PYc60OUPBI", "reviewer": "Reviewer_NfoM", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We understand your concerns and would like to address them as follows: \n1. About the motivation of building the dataset:  \nWe grounded our work in the most recent advances in affective science. Although researchers still debate about the nature of emotions, whether emotions are basic and universal, and how many basic emotions are there, most have agreed that there are a lot more than 6 basic emotions, as laid out in Ekman’s model [1, 2, 3]. Most datasets in emotion cause analysis are built on this outdated view of emotion, hindering scientists to build models that predict emotion and emotion causes more accurately.   \nConsider a scenario:  \n*\"Tim and Henry both receive recognition at work for their exceptional contributions to a project. Tim feels a strong sense of pride, while Henry feels immense gratitude for the recognition. Earlier, Tim had work his heart out, spent days and nights overcoming so many difficulties to achieve his goal; while Henry felt that he was lucky and he got the help from other teammates.\"*  \nWithout knowing the causes behind their emotions, it might be easy to assume that both are simply experiencing positive emotions due to the recognition they received.  \nOur dataset is not only large in size, but it has 2 important  features:  \n   - It is rich in its classes (48 emotion categories), \n   - It provides an abstractive emotion cause (not abstractive emotion). The abstraction of emotion cause allows generalization across contexts (e.g. being more fortunate than other people is a common cause for gratitude, but it can be expressed differently in different contexts). This is shown in Table 2. We will elaborate more on this in the revised paper.\n\nThe motivation of building a dataset with abstractive emotion cause is to facilitate emotional reasoning, which is crucial for building models that have a deep understanding of human emotions and do not make stereotypical assumptions about emotions.\n\n We agree that the second paragraph of our introduction is duplicated with poor readability, while the first paragraph has not motivated well enough the need to have a large dataset with a wide range of emotion classes and abstractive causes. We will update our manuscript to address these concerns. \n\n2. About the number 48 emotion classes:  \nThank you for raising this up. We agree that we should have included the information in the main manuscript.  \nWe mostly ground our work in the Theory of Constructed Emotions, where one’s emotion “nauseous'' can be totally different from their emotion “sick” even when others think they are the same. Following the recommendation by Barrett & Westlin 2021, section 2.4.2, para. 3 [4], to preserve the ecological accuracy of the dataset, we keep the emotion words used by people as they are, and do not follow prior works to group them into more “basic” categories (like 28 categories in GoEmotion[5] or 32 in EmpatheticDialogue[6]). This is a strength of the dataset, as it is ecologically accurate, but can make it difficult to combine with other datasets. We briefly discussed our exploration of emotions expressed in Twitter in Appendix A. We examined the frequency of emotion words on a sample of 300K tweets and picked a cut-off at 48. We will update this information in our revised manuscript. \n3. About Limitations:  \nWe largely agree with the reviewer that the limitation section should focus more on the method that we used to construct the dataset. First, the number 48 was chosen heuristically as we explored the dataset. Second, we labeled emotion classes with patterns matching. This is not 100% accurate and cannot detect sarcasm or negations. Emotion and emotion-cause pairs will not be exhaustively extracted in tweets with multiple emotions and multiple causes. Third, we cannot guarantee that tweets with vulgar languages are completely cleaned. Lastly, although the human rating of the cause are very high, the quality of the abstractive cause depends largely on the capability of the pretrained model. \n\nWe acknowledge that our limitation sections can be improved and will update this part in our manuscript. \n\n\n1. Coppini, S., Lucifora, C., Vicario, C.M. et al. Experiments on real-life emotions challenge Ekman's model. Sci Rep 13, 9511 (2023).\n2. Cowen, Alan S., and Dacher Keltner. \"Self-report captures 27 distinct categories of emotion bridged by continuous gradients.\" Proceedings of the national academy of sciences 114.38 (2017): E7900-E7909.\n3. Ortony, A. (2022). Are All “Basic Emotions” Emotions? A Problem for the (Basic) Emotions Construct. Perspectives on Psychological Science, 17(1), 41–61\n4. Barrett, Lisa Feldman, and Christiana Westlin. \"Navigating the science of emotion.\" Emotion measurement. Woodhead Publishing, 2021. 39-84.\n5. Demszky, Dorottya, et al. \"GoEmotions: A Dataset of Fine-Grained Emotions.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n6. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Large-scale dataset containing 700,000 tweets, each with \n2. 48 emotion categories, which are more refined than the classic 7 emotion categories mostly used in existing datasets, enabling more refined emotion-cause analysis.", "reasons_to_reject": "1. Not strong or clear motivation: While I mainly agree that this new dataset is beneficial, its motivation is not clear. The authors claimed that coarse granularity is not good enough and finer granularity is needed but didn't explicitly analyze how finer granularity is beneficial. Take \"pride\" and \"gratitude\" as examples, why is knowing the causes of these similar but still different categories important? What would without knowing this cause? On the other hand, the authors also claim that abstractive emotions are desired and better, but why? How are they better and more beneficial? Stronger motivations are expected.\n\n2. Not clear description: how did the authors end up with 48 categories? I cannot find any relevant description in the manuscript. If this information is contained in the appendix, then this should be added to the manuscript since the guideline states explicitly that the manuscript is a standalone document with every information needed to understand the paper.\n\n3. The given limitations are not relevant enough: the authors haven't claimed to build a balanced dataset, so the balance problem is not the main limitation in this paper; it is a challenge that should be solved by other researchers if they are using the dataset. In my opinion, the limitation should be analyzed from the determination of the 48 categories, and the process of the data cleaning and labeling process. For example, how can the authors guarantee that 48 categories are enough, or there are no circumstances that sometimes two categories have the same meaning and mostly they are similar but different? More insightful limitation analyses are expected.\n\n4. Somehow poor readability: \n- confusing claims and reference: the authors cited (Barrett, 2017) that concluded emotions are not simply reducible to a basic set of broad categories, but what defines \"broad\" and \"basic\"? The authors use 48 emotion categories. Are these 48 categories not broad and basic? What makes them different from the existing categories and why? \n- repetitive description, such as lines 054-057 and lines 066-068; \n- confusing descriptions: \n  - In lines 157-160, what are the \"outlined structure\" and \"this pattern\"? Am I missing something or is it put in the appendix? If it's the latter, same problem as the second comment\n  - In line 162, \"most cases follow this rule\", exactly what is the ratio here? Please put all necessary information in the manuscript for better readability.", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "The paper presents a new large-scale emotion-cause dataset with over 700,000 tweets and 48 emotion classes. The main contributions are the extensive number of data instances contained and the range of emotion classes enabling subtle differences between emotions within the same category.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "See reasons to reject.", "Justification_for_Ethical_Concerns": "How did the authors handle the privacy issues of tweets carrying personal information?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for collecting Twitter data related to emotions and their causes, resulting in the creation of a large-scale dataset. This dataset, sourced from tweets, is designed to identify the cause of a given emotion by prompting state-of-the-art Language Models (LLMs) with specific questions. The dataset encompasses over 700,000 tweets and is categorized into 48 distinct emotion classes. These classes aim to capture nuanced differences between emotions that may fall under a broader category. However, the paper does not clearly outline the motivation behind the creation of this dataset or how it compares to existing datasets in terms of advantages.

The paper's strengths lie in its ambitious data collection approach, with a significant emphasis on human annotation. The creation of a dataset with 700,000 tweets and 48 emotion categories is commendable, as it offers a more detailed perspective on emotions compared to existing datasets. This refined approach to categorizing emotions could be beneficial for in-depth emotion-cause analysis and might aid in the application of pretrained models in this area.

The method used for emotion cause extraction seems to be based on a specific pattern, which might not capture the full spectrum of emotions in real-world scenarios. The paper's approach lacks novelty, especially in data modeling, and there's an over-reliance on existing models. The authors, during the rebuttal phase, convincely answered some concers of the reviewers.
Decision: Accept-Findings