SlowGuess/ACL_2020_OCR huggingface dataset:
1) Indep: It is an independent multi-task learning method, in which emotion extraction and cause extraction are independently modeled.
2) Inter-CE: It is an interactive multi-task learning method, in which the predictions of cause extraction are used to improve emotion extraction.
3) Inter-EC: It is another interactive multi-task learning method, in which the predictions of emotion extraction are used to enhance cause extraction.
ECPE-2D is a joint framework proposed in this paper, which integrates the 2D emotion-cause pair representation, interaction, and prediction in an
$^{1}$ https://github.com/NUSTM/ECPE-2D
Framework	Approach	Emotion-Cause Pair Ext.	Emotion Ext.	Cause Ext.
P	R	F1	P	R	F1	P	R	F1
ECPE-2Steps	Indep	68.32	50.82	58.18	83.75	80.71	82.10	69.02	56.73	62.05
Inter-CE	69.02	51.35	59.01	84.94	81.22	83.00	68.09	56.34	61.51
Inter-EC	67.21	57.05	61.28	83.64	81.07	82.30	70.41	60.83	65.07
ECPE-2D(Ours)	Indep	-	71.60	55.95	62.63	86.32	81.52	83.80	69.15	59.72	63.97
+WC	69.01	59.58	63.80	85.08	81.82	83.35	71.57	59.08	64.64
+CR	69.12	58.78	63.38	85.27	81.82	83.44	69.73	59.37	63.99
Inter-CE	-	69.35	57.24	62.61	86.12	82.40	84.16	69.77	59.42	63.98
+WC	68.62	58.70	63.18	84.97	82.58	83.70	69.24	59.15	63.65
+CR	69.22	59.04	63.56	84.82	82.88	83.76	69.80	58.78	63.68
Inter-EC	-	71.73	57.54	63.66	85.37	81.97	83.54	71.51	62.74	66.76
+WC	71.18	59.84	64.94	85.11	82.37	83.65	71.33	62.85	66.72
+CR	69.60	61.18	64.96	85.12	82.20	83.58	72.72	62.98	67.38
Inter-EC(BERT)	-	70.73	64.86	67.47	86.22	91.82	88.88	73.46	68.79	70.96
+WC	72.92	65.44	68.89	86.27	92.21	89.10	73.36	69.34	71.23
+CR	69.35	67.85	68.37	85.48	92.44	88.78	72.72	69.27	70.87
Table 2: Performance of our models and baseline models (Xia and Ding 2019) using precision, recall, and F1-measure as metrics on the ECPE task as well as the two sub-tasks.
end-to-end fashion. We explored three individual emotion/cause encoding settings: Indep, Inter-CE and Inter-EC, and three emotion-cause pair interaction settings:
1) “-” indicates that we do not introduce emotion-cause pair interaction;
2) “+WC” indicates that we use the window-constrained 2D transformer for emotion-cause pair interaction;
3) “+CR” indicates that we use the cross-road 2D transformer for emotion-cause pair interaction;
Note that due to the limitations of GPU memory, we have not been able to perform experiments with Standard 2D Transformer.
First of all, it can be seen that our proposed model ECPE-2D (Inter-EC+WC) performs better than ECPE-2Step on all metrics of all tasks, which proves the effectiveness of our method.
On the ECPE task, ECPE-2Steps (Inter-EC) performs best among all the previous methods. Compared with ECPE-2Steps (Indep), the improvement of ECPE-2Steps (Inter-EC) is mainly on the recall rate, while the precision score is slightly reduced. On the basis of ECPE-2Steps (Inter-EC), the recall rate of ECPE-2D (Inter-EC+CR) has been further greatly improved, and the precision score has also been slightly improved, which ultimately leads to better performance on the F1 score.
On the emotion extraction and cause extraction subtasks, ECPE-2Steps (Inter-CE) and ECPE-2Steps (Inter-EC) achieves significant improvements compared to ECPE-2Steps (Indep) on the
former and latter subtask respectively by leveraging the interaction between emotion and cause. While our method ECPE-2D (Inter-EC+CR) outperforms the previous methods on both subtasks. We attribute the improvements to multi-task learning, as compared to the ECPE-2Steps (Inter-EC) model, ECPE-2D (Inter-EC+CR) additionally introduces the emotion-cause pair extraction task and trains the three tasks in a unified framework.
In addition, we also explored the effect of using BERT $^2$  (Devlin et al., 2019) as clause encoder in Inter-EC, which is denoted as Inter-EC (BERT). The experimental results in Table 2 show that the performance on all tasks can be further greatly improved (especially, the state-of-the-art F1 score on the ECPE task is improved from  $61.28\%$  to  $68.89\%$ ) by adopting BERT as clause encoder.
3.4 ECPE-2D vs. ECPE-2Steps
In order to verify the effect of our proposed joint framework ECPE-2D, we discard the emotion-cause pair interaction module and compare ECPE-2D models with ECPE-2Step models based on the same individual encoding setting, the results are shown in Table 2.
By comparing ECPE-2D (Indep) with ECPE-2Step (Indep), we find that the performance of ECPE-2D (Indep) on all the metrics of all tasks (especially the ECPE task) are significantly improved. On the ECPE task, the performance of ECPE-2D (Indep) is even better than ECPE-2D
${}^{2}$  BERT is only used to replace the word-level Bi-LSTM. Specifically, each clause in the document is feed into the BERT model independently, and the final hidden state of "[CLS]" is used as the clause representation. Our model is built based on this implementation: https://github.com/google-research/bert.
(Inter-EC), which is the prior state-of-the-art model. On the two subtasks, the performance has also been improved. We attribute the improvements to multi-task learning, as compared to the ECPE2Step (Indep) model, ECPE-2D (Indep) additionally introduces the emotion-cause pair extraction task.
By comparing ECPE-2D (Inter-CE) and ECPE-2D (Inter-EC) with their two-step pipeline versions (ECPE-2Step (Inter-CE) and ECPE-2Step (Inter-EC)), we can draw similar conclusions. All these results prove that the proposed joint framework ECPE-2D is superior to the two-step pipeline framework ECPE-2Step in solving the ECPE task.
3.5 The Effectiveness of 2D Transformer
Comparing with the ECPE-2D (Indep) model, the ECPE-2D (Indep+WC/CR) models can achieve further improvement on the ECPE task, while the improvement on the two subtasks are not significant. Similar conclusions can be drawn when comparing ECPE-2D (Inter-CE) and ECPE-2D (Inter-CE+WC/CR) as well as ECPE-2D(Inter-EC) and ECPE-2D(Inter-EC+WC/CR). Particularly, compared to the strong baseline ECPE-2D (Inter-EC(BERT)), the performance can still be improved by introducing two kinds of 2D transformers. These results demonstrate that the window-constrained and cross-road 2D transformer can effectively improve the performance on the ECPE task via encoding interactive information between pairs.
In addition, we found that for ECPE-2D (Indep/Inter-CE/Inter-EC/Inter-EC(BERT)), the improvements brought by the introduction of window-constrained and cross-road 2D transformer are similar. These results indicate that the two 2D transformers are comparable.
3.6 The Effectiveness of Auxiliary Supervision
In order to explore the impact of the auxiliary supervision of two subtasks (emotion extraction and cause extraction) on the final performance of the ECPE task, we design the experiments in Table 3. "AS" denotes the auxiliary supervision is removed (in practice, we set  $\lambda_{2}$  in formula (20) to 0).
Compared with ECPE-2D (Indep/Inter-CE/Inter-EC), we find that the F1 score of ECPE-2D (Indep/Inter-CE/Inter-EC)-AS on the ECPE task decreased by about  $1.4\%$ ,  $2.2\%$ , and  $2.6\%$ , respectively, which indicates that the supervisions of emo
Emotion-Cause Pair Ext.
P	R	F1
Indep-AS	67.26	56.46	61.24
Indep+WC-AS	68.87	59.78	63.86
Indep+CR-AS	67.48	60.66	63.76
Inter-CE-AS	68.36	54.40	60.42
Inter-CE+WC-AS	67.12	60.79	63.44
Inter-CE+CR-AS	67.28	61.08	63.85
Inter-EC-AS	66.46	56.69	61.08
Inter-EC+WC-AS	67.79	60.47	63.81
Inter-EC+CR-AS	69.26	60.06	64.17
Table 3: Performance of our models on the ECPE task when the auxiliary supervisions of emotion extraction and cause extraction are removed. For brevity, the prefix "ECPE-2D" of all methods in this table are omitted.
tion extraction and cause extraction are important for the ECPE task. Nevertheless, the results of ECPE-2D (Indep)-AS are still better than ECPE-2Step (Indep) and comparable to the prior state-of-the-art result, which shows that emotion-cause pair extraction can be performed individually and proves the effectiveness of our joint framework.
Compared with ECPE-2D (Inter-EC+WC/+CR), the F1 score of ECPE-2D (Inter-EC+WC/+CR)-AS on the ECPE task decreased by about  $1.1\%$  and  $0.8\%$ , which is much less than the decrease between ECPE-2D (Inter-EC) and ECPE-2D (Inter-EC)-AS (drops  $2.6\%$ ). These results lead to the conclusion that the negative impact of removing auxiliary supervision is reduced when pairwise encoders are introduced. From another perspective, when auxiliary supervisions are removed, the improvement brought by introducing pairwise encoders is greater. Comparing ECPE-2D (Inter-CE+WC/+CR), ECPE-2D (Indep+WC/+CR) and their "-AS" versions leads to similar conclusions. The above results again demonstrate the effectiveness of the proposed 2D transformer.
The emotion-cause pair extraction (ECPE) task was first proposed in our prior work (Xia and Ding, 2019) and is derived from the traditional emotion cause extraction (ECE) task. Since the ECPE task was recently proposed, there is little work on it. We mainly introduce the related work of ECE task.
The emotion cause extraction (ECE) task was first proposed by Lee et al. (2010), with the goal to extract the word-level causes that lead to the given emotions in text. Based on the same task settings, there were some other individual studies that conducted ECE research on their own corpus us
ing rule-based methods (Neviarouskaya and Aono, 2013; Li and Xu, 2014; Gao et al., 2015a,b; Yada et al., 2017) or machine learning methods (Ghazi et al., 2015; Song and Meng, 2015).
Based on the analysis of the corpus in (Lee et al., 2010), Chen et al. (2010) suggested that a clause may be the most appropriate unit to detect causes and transformed the task from word-level to clause-level. There was also some work based on this task setting (Russo et al., 2011; Gui et al., 2014). Recently, a Chinese emotion cause dataset was released by (Gui et al., 2016a,b; Xu et al., 2017), and has received much attention. Based on this corpus, a lot of traditional machine learning methods (Gui et al., 2016a,b; Xu et al., 2017) and deep learning methods (Gui et al., 2017; Li et al., 2018; Yu et al., 2019; Xu et al., 2019; Ding et al., 2019; Xia et al., 2019) were proposed.
In addition, there is also some work focused on cause detection for Chinese microblogs using a multiple-user structure and formalized two cause detection tasks for microblogs (current-subtweet-based cause detection and original-subtweet-based cause detection). (Cheng et al., 2017; Chen et al., 2018b,a).
The traditional ECE tasks suffer from two shortcomings: 1) the emotion must be annotated before cause extraction in ECE, which greatly limits its applications in real-world scenarios; 2) the way to first annotate emotion and then extract the cause ignores the fact that they are mutually indicative. To address this problem, we proposed the new emotion-cause pair extraction task in (Xia and Ding, 2019), which aims to extract the potential pairs of emotions and corresponding causes in a document. We have also proposed a two-step framework, which first extracts individual emotion set and cause set, and then pairs the corresponding emotions and causes. In this paper, we propose a new end-to-end approach to represent the emotion-cause pairs by a 2D representation scheme. Two kinds of 2D transformers, namely window-constrained and cross-road 2D transformers, are further proposed to model the interactions of different emotion-cause pairs. Finally, the 2D representation, interaction, and prediction are integrated into a joint framework.
The emotion-cause pair extraction (ECPE) task has drawn attention recently. However the previous
approach employed a two-step pipeline framework and has some inherent flaws. In this paper, instead of a pipeline of two steps, we propose a joint end-to-end framework, called ECPE-2D, to represent the emotion-cause pairs by a 2D representation scheme, and integrate the 2D emotion-cause pair representation, interaction, and prediction into a joint a framework. We also develop two kinds of 2D Transformers, i.e., Window-constrained and Cross-road 2D Transformers, to further model the interaction of different emotion-cause pairs. The experimental results on the benchmark emotion cause corpus demonstrate that in addition to the advantages of joint modeling, our approach outperforms the state-of-the-art method by 7.6 percentage points in terms of the F1 score on the ECPE task.
We would like to thank three anonymous reviewers for their valuable comments. This work was supported by the Natural Science Foundation of China (No. 61672288). Zixiang Ding and Rui Xia contributed equally to this paper.
Ying Chen, Wenjun Hou, and Xiyao Cheng. 2018a. Hierarchical convolution neural network for emotion cause detection on microblogs. In International Conference on Artificial Neural Networks (ICANN), pages 115-122.
Ying Chen, Wenjun Hou, Xiyao Cheng, and Shoushan Li. 2018b. Joint learning for emotion classification and emotion cause detection. In Empirical Methods in Natural Language Processing (EMNLP), pages 646-651.
Ying Chen, Sophia Yat Mei Lee, Shoushan Li, and ChuRen Huang. 2010. Emotion cause detection with linguistic constructions. In Computational Linguistics (COLING), pages 179-187.
Xiyao Cheng, Ying Chen, Bixiao Cheng, Shoushan Li, and Guodong Zhou. 2017. An emotion cause corpus for chinese microblogs with multiple-user structures. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 17(1):1-19.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), pages 4171-4186.
Zixiang Ding, Huihui He, Mengran Zhang, and Rui Xia. 2019. From independent prediction to re-ordered prediction: Integrating relative position and global label information to emotion cause identification. In AAAI Conference on Artificial Intelligence (AAAI), pages 6343-6350.
Kai Gao, Hua Xu, and Jiushuo Wang. 2015a. Emotion cause detection for chinese micro-blogs based on e-cocc model. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 3-14.
Kai Gao, Hua Xu, and Jiushuo Wang. 2015b. A rule-based approach to emotion cause detection for chinese micro-blogs. Expert Systems with Applications, 42(9):4517-4528.
Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In International Conference on Intelligent Text Processing and Computational Linguistics (CICling), pages 152-165.
Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017. A question answering approach to emotion cause extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 1593-1602.
Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. 2016a. Event-driven emotion cause extraction with corpus construction. In Empirical Methods in Natural Language Processing (EMNLP), pages 1639-1649.
Lin Gui, Ruifeng Xu, Qin Lu, Dongyin Wu, and Yu Zhou. 2016b. Emotion cause extraction, a challenging task with corpus construction. In Chinese National Conference on Social Media Processing, pages 98-109.
Lin Gui, Li Yuan, Ruifeng Xu, Bin Liu, Qin Lu, and Yu Zhou. 2014. Emotion cause detection with linguistic construction in chinese weibo text. In *Natural Language Processing and Chinese Computing (NLPCC)*, pages 457-464.
Sophia Yat Mei Lee, Ying Chen, and Chu-Ren Huang. 2010. A text-driven rule-based system for emotion cause detection. In *NAACL HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text*, pages 45-53.
Weiyuan Li and Hua Xu. 2014. Text-based emotion classification using emotion cause extraction. Expert Systems with Applications, 41(4):1742-1749.
Xiangju Li, Kaisong Song, Shi Feng, Daling Wang, and Yifei Zhang. 2018. A co-attention neural network model for emotion cause analysis with emotional context awareness. In Empirical Methods in Natural Language Processing (EMNLP), pages 4752–4757.
Alena Neviarouskaya and Masaki Aono. 2013. Extracting causes of emotions from text. In International Joint Conference on Natural Language Processing (IJCNLP), pages 932-936.
Irene Russo, Tommaso Caselli, Francesco Rubino, Ester Boldrini, and Patricio Martínez-Barco. 2011. Emocause: an easy-adaptable approach to emotion cause contexts. In Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA), pages 153-160.
Shuangyong Song and Yao Meng. 2015. Detecting concept-level emotion cause in microblogging. In World Wide Web (WWW), pages 119-120.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008.
Rui Xia and Zixiang Ding. 2019. Emotion-cause pair extraction: A new task to emotion analysis in texts. In Association for Computational Linguistics (ACL), pages 1003-1012.
Rui Xia, Mengran Zhang, and Zixiang Ding. 2019. RTHN: A RNN-transformer hierarchical network for emotion cause extraction. In International Joint Conference on Artificial Intelligence (IJCAI), pages 5285-5291.
Bo Xu, Hongfei Lin, Yuan Lin, Yufeng Diao, Liang Yang, and Kan Xu. 2019. Extracting emotion causes using learning to rank methods from an information retrieval perspective. IEEE Access, 7:15573-15583.
Ruifeng Xu, Jiannan Hu, Qin Lu, Dongyin Wu, and Lin Gui. 2017. An ensemble approach for emotion cause detection with event extraction and multikernel svms. Tsinghua Science and Technology, 22(6):646-659.
Shuntaro Yada, Kazushi Ikeda, Keiichiro Hoashi, and Kyo Mageura. 2017. A bootstrap method for automatic rule acquisition on emotion cause extraction. In IEEE International Conference on Data Mining Workshops, pages 414-421.
Xinyi Yu, Wenge Rong, Zhuo Zhang, Yuanxin Ouyang, and Zhang Xiong. 2019. Multiple level hierarchical network-based clause selection for emotion cause extraction. IEEE Access, 7(1):9071-9079.