SlowGuess/ACL_2020_OCR huggingface dataset:
To compare with other joint models, we implement SL-BERT (Zheng et al., 2017) and MT-BERT (Caruana, 1993) for this task. The former aims to joint extract entities and relations based on a novel tagging scheme with multiple labels and the other is a multi-task learning framework by sharing the hidden layers among all tasks. We implement them both based on BERT to be consistent with our experimental setting.
We also evaluate our model by only removing the transition procedure to reveal the effect of the transition-based algorithm, denoted as “-transition”. Besides, for a fair comparison, we use LSTM as the basic encoder of clauses instead of BERT and keep the same experimental setting by following (Xia and Ding, 2019), namely LSTM<sub>based</sub>.
5.2 Main Analysis
Table 4 shows the experimental results. With the transition-based algorithm, our proposed model achieves the best performance over all the three tasks, outperforming a number of competitive baselines by at least  $1.74\%$ ,  $3.30\%$  and  $3.33\%$  in  $F1$  score, respectively. The improvements are significant with  $p < 0.01$  in one sample  $t$ -test.
Regarding pipelined approaches, Indep considers framework individually and ignores the fact that
emotions and causes are usually mutually indicative, leading to the lowest performance. On the contrary, Inter-CE and Inter-EC yield better results by exploiting the relevance between emotions and causes. By comparing Inter-CE and Inter-EC, we find that the improvement of Inter-EC on cause extraction is much more than the improvement of Inter-CE on emotion extraction, thus Inter-EC shows better results. Differently, our model jointly extracts emotion-cause pairs and shows consistent performance improvement over the Indep-CE and Indep-EC, demonstrating the superiority of one-stage model by reducing error propagation.
In comparison with other joint models, our proposed model significantly outperforms SL-BERT by  $12.56\%$ ,  $4.44\%$  and  $5.52\%$  in  $F1$  measure, respectively. We guess that SL-BERT jointly identifies emotion-cause pairs but still follows an emotion  $\rightarrow$  cause pipelined decoding order. In contrast, we achieve fully joint decoding with interleaving actions for all the three tasks, thereby achieving better information interaction. Besides, our model also yields better results than MT-BERT, one possible reason is that the interdependence between the emotions and causes cannot be mined effectively only through parameter sharing.
We also show the results where BERT embeddings are replaced by LSTM from the input. It can be seen that the results still outperform the existing methods by at least  $3.06\%$  in  $F1$  score. Furthermore, when we remove the transition procedure, the performance drops heavily over all the three tasks, especially with a  $7.87\%$  decrease in  $F1$  measure on the ECPE task. These results show that the improvements provided by the proposed transition system are more noticeable than other components.
Method	Emotion extraction	Cause extraction	Emotion-cause pair extraction
P(%)	R(%)	F1(%)	P(%)	R(%)	F1(%)	P(%)	R(%)	F1(%)
Ours	87.16	82.44	84.74	75.62	64.71	69.74	73.74	63.07	67.99
-reversal	85.26	83.63	84.43	76.49	63.08	69.14	74.59	61.35	67.33
-buffer	80.92	86.94	83.82	72.51	65.65	68.91	70.44	63.73	66.91
-action	82.18	86.69	84.34	76.49	61.69	68.30	74.61	60.04	66.53
-distance	81.60	85.05	83.29	75.93	57.89	65.69	74.06	56.29	63.96
-LSTM	81.23	83.37	83.29	72.19	59.20	65.06	70.64	57.82	63.59
Table 5: Feature ablation experiments. The results are average score over 20 runs, and the best scores are in bold.
To further evaluate the contribution of neural components, we conduct feature ablation experiments to study the effects of different parts. As shown in Table 5, the  $F1$  score decreases most heavily without LSTM  $(-4.40\%)$ , which indicates that it is necessary to capture non-local dependencies among input clauses, and our model can benefit from it effectively. Distance is also particularly relevant to the model by capturing the position information between the emotions and causes, which is consistent with our intuition that the closer a clause is to the emotion, the higher probability it should be the cause. Seen from the results, the history of actions stored in action has a crucial influence on predicting the next action. The results also show that reversal, which can be regarded as a data augmentation strategy, is useful by exploring the deep directional information. Without buffer, the  $F1$  score drops  $1.8\%$  over the ECPE task. It may be due to the reason that buffer can provide more valuable information about the succeeding sequence.
5.4 Action Set Validation
To gain more insights into the parsing procedure, we analyze the situations that emotion-cause pairs in an emotion text cannot be extracted entirely by our defined actions, as illustrated in Figure 4. For the pseudo sample in Figure 4(a), it can be parsed by the transition system using computation:
$$
\mathrm {S H} (1); \mathrm {S H} (2); \mathrm {S H} (3); \mathrm {R A} _ {l _ {n}} (2 \xrightarrow {l _ {n}} 3);
$$
$$
\mathrm {R A} _ {l _ {t}} (1 \xrightarrow {l _ {t}} 3); \mathrm {S H} (4); \mathrm {R A} _ {l _ {n}} (3 \xrightarrow {l _ {n}} 4); \mathrm {S H} (\$)
$$
Similarity, for the pseudo sample in Figure 4(b), we get the transition sequence by:
$$
\mathrm {S H} (1); \mathrm {S H} (2); \mathrm {R A} _ {l _ {t}} \left(1 \xrightarrow {l _ {t}} 2\right); \mathrm {S H} (3);
$$
$$
\mathrm {R A} _ {l _ {n}} (2 \xrightarrow {l _ {n}} 3); \mathrm {S H} (4); \mathrm {L A} _ {l _ {n}} (3 \xleftarrow {l _ {n}} 4); \mathrm {S H} (\$)
$$
(a)  $(1\stackrel {l_t}{\to}3)$  and  $(2\stackrel {l_t}{\to}4)$
(b)  $(1\stackrel {l_t}{\to}2)$  and  $(1\stackrel {l_t}{\to}3)$
Figure 4: Pseudo samples that cannot be extracted entirely by our defined actions.
In both situations, our model can only extract one emotion-cause pair (i.e.,  $\mathrm{RA}_{l_t}(1 \xrightarrow{l_t} 3)$  and  $\mathrm{RA}_{l_t}(1 \xrightarrow{l_t} 2)$ , respectively.), because the cause which belongs to another emotion has been popped during the parsing procedure.
Based on this observation, one crucial problem about the proposed model is how many situations involving the emotion-cause transformation can be covered by the action set defined here. Although a formal theoretical proof is beyond the scope of this paper, we can empirically verify that the action set works well from Table 4. Going one step further, to further validate the actions, we input the texts into our transition system to obtain the "pseudo-gold" emotion-cause pairs  $P'$  based on the annotation, which can give us the correct action to take for a given parse state. Then we compare  $P'$  with the gold-standard emotion-cause pairs  $P$  to see how similar they are. On the whole dataset, we obtain an overall  $98.5\%$ $F1$  score for  $\langle P, P' \rangle$ , which indicates the upper bound of our transition system can achieve  $98.5\%$  in  $F1$  score. Thus, the defined action set here is capable of extracting emotion-cause pairs through a sequence of actions.
5.5 Error Analysis
We also perform an experiment to understand the impact of action reversal on the performance. Fig-
(a) Without action reversal.
Figure 5: Confusion matrices on test set. Vertical direction indicates the predicted action type and horizontal direction indicates the gold action type.
(b) With action reversal.
ure 5 shows the confusion matrices that present a comparison between the predicted actions and corrective actions. The results show that SHIFT, LEFT-  $\mathrm{ARCl}_n$  and RIGHT-  $\mathrm{ARCl}_n$  yield higher accuracy on both Figure 5(a) and Figure 5(b) since they are account for a large proportion of the total actions. As expected, our model makes more mistakes involving the RIGHT-  $\mathrm{ARCl}_t$  and LEFT-  $\mathrm{ARCl}_t$ , which play decisive roles in identifying the emotion-cause pairs. Especially for the LEFT-  $\mathrm{ARCl}_t$  action, there is only about  $0.43\%$  in the total actions, turning out to be the most difficult action to learn given the relatively small training samples. Thus, as shown in Figure 5(a), the accuracy for LEFT-  $\mathrm{ARCl}_t$  is 0, which drops the overall performance heavily. However, when we apply the action reversal into our model, boosting the accuracy of LEFT-  $\mathrm{ARCl}_t$  by  $58.8\%$  and further improving the overall performance. We guess that based on action reversal, the original RIGHT- $*$  action can be reversed to LEFT- $*$  and vice versa, so that double the training actions. The results in Figure 5 show that our proposed model can capture this subtlety of emotions effectively by exploiting the deep directional information through action reversal strategy.
Different from the traditional emotion analysis, which aims to identify emotion categories in text. Emotion cause extraction (ECE) reveals the essential information about what causes a certain emotion and why there is an emotional change. It is a more challenging task due to the inherent ambiguity and subtlety of emotion expressions.
Lee et al. (2010) first defined the emotion cause extraction as a word-level extraction task. They
manually constructed a dataset from the Academia Sinica Balanced Chinese Corpus and generalized a series of linguistics rules based on the dataset. Based on this setting, there are some studies have been exploited for this task such as rule-based methods (Li and Xu, 2014; Gao et al., 2015; Yada et al., 2017) and machine learning methods (Ghazi et al., 2015; Song and Meng, 2015). Chen et al. (2010) converted the task from word-level to clause-level due to a clause may be the most appropriate unit to detect causes, and extracted causes using six groups of manually constructed linguistic cues. By following this task setting, Gui et al. (2014) extended the rule-based features to 25 linguistics cues, then trained classifiers on SVM and CRFs to detect causes. Gui et al. (2016) released a new Chinese emotion cause dataset collected from SINA city news  $^{4}$  and proposed a multi-kernel based method to identify emotion causes. Following this corpus, Xu et al. (2019) proposed a learning to re-rank method based on a series of emotion-dependent and emotion-independent features. Recently, inspired by the success of deep learning architecture, some studies focused on identifying emotion causes with well designed neural network and attention mechanism (Gui et al., 2017; Li et al., 2018, 2019; Fan et al., 2019; Xia et al., 2019; Ding et al., 2019).
All of the above studies extracted emotion causes rely on the given emotion annotations, which limits the application in real-world scenarios due to the expensive annotations. Targeting this problem, Xia and Ding (2019) proposed a novel task based on ECE, namely emotion-cause pair extraction (ECPE), which aims at extracting emotions and the corresponding causes from unannotated emotion text. However, they followed a pipelined framework which first detects emotions and causes with individual learning frameworks, then performed emotion-cause pairing to eliminate the unmatched pairs, leading to a drawback of error propagation.
In this work, we design a novel transition-based model to extract emotions and causes simultaneously to maximize the mutual benefits of subtasks, thus alleviating the drawback of error propagation. Transition-based system is usually designed to model the chunk-level relation in a sentence for dependency parsing (Zhang and Nivre, 2011; Wang et al., 2015; Fernandez-Gonzalez and Gomez-Rodriguez, 2018). Apart from its application in dependency parsing, transition-based method has
4http://news.sina.com.cn/society/
also achieved great success in other natural language processing tasks, such as word segmentation (Zhang et al., 2016), information extraction (Wang et al., 2018b; Zhang et al., 2019), disfluency detection (Wang et al., 2017) and nested mention recognition (Wang et al., 2018a). To the best of our knowledge, this is the first work which extracts the emotion-cause pairs in an end-to-end manner.
In this paper, we present a novel transition-based framework to extract emotion-cause pairs as a procedure of directed graph construction. Instead of previous pipelined approaches, the proposed framework incrementally outputs the emotion-cause pairs as a single task, thereby the interdependence between emotions and causes can be exploited more effectively. Experimental results on a standard benchmark demonstrate the superiority and robustness of the proposed model compared to a number of competitive methods.
In the future, one possible direction is creating complete graphs with their nodes being input clauses to achieve full coverage. Besides, graph neural network-based (Kipf and Welling, 2016) methods are also worth investigating to model the relations among nodes for this task.
This work was partially supported by National Natural Science Foundation of China 61632011, 61876053, 61906185, Shenzhen Foundational Research Funding JCYJ20180507183527919, JCYJ20180507183608379, Key Technologies Research and Development Program of Shenzhen JSGG20170817140856618, EU-H2020 (grant no. 794196) and the project AWS13C008.
Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27-29, 1993, pages 41-48.
Ying Chen, Sophia Yat Mei Lee, Shoushan Li, and ChuRen Huang. 2010. Emotion cause detection with linguistic constructions. In COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 179-187.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Zixiang Ding, Huihui He, Mengran Zhang, and Rui Xia. 2019. From independent prediction to reordered prediction: Integrating relative position and global label information to emotion cause identification. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6343-6350. AAAI Press.
Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Li-dong Bing, Min Yang, Ruifeng Xu, and Ruibin Mao. 2019. A knowledge regularized hierarchical approach for emotion cause analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5618-5628.
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2018. Non-projective dependency parsing with non-local transitions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 693-700.
Kai Gao, Hua Xu, and Jiushuo Wang. 2015. A rule-based approach to emotion cause detection for chinese micro-blogs. Expert Syst. Appl., 42(9):4517-4528.
Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In Computational Linguistics and Intelligent Text Processing - 16th International Conference, CICLing 2015, Cairo, Egypt, April 14-20, 2015, Proceedings, Part II, volume 9042 of Lecture Notes in Computer Science, pages 152-165. Springer.
Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017. A question answering approach for emotion cause extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1593-1602.
Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. 2016. Event-driven emotion cause extraction with corpus construction. In Proceedings of the
2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1639-1649.
Lin Gui, Li Yuan, Ruifeng Xu, Bin Liu, Qin Lu, and Yu Zhou. 2014. Emotion cause detection with linguistic construction in chinese weibo text. In *Natural Language Processing and Chinese Computing - Third CCF Conference*, NLPCC 2014, Shenzhen, China, December 5-9, 2014. Proceedings, pages 457-464.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Sophia Yat Mei Lee, Ying Chen, and Chu-Ren Huang. 2010. A text-driven rule-based system for emotion cause detection. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 45-53. Association for Computational Linguistics.
Weiyuan Li and Hua Xu. 2014. Text-based emotion classification using emotion cause extraction. Expert Syst. Appl., 41(4):1742-1749.
Xiangju Li, Shi Feng, Daling Wang, and Yifei Zhang. 2019. Context-aware emotion cause analysis with multi-attention-based neural network. Knowl.-Based Syst., 174:205-218.
Xiangju Li, Kaisong Song, Shi Feng, Daling Wang, and Yifei Zhang. 2018. A co-attention neural network model for emotion cause analysis with emotional context awareness. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4752-4757.
Shuangyong Song and Yao Meng. 2015. Detecting concept-level emotion cause in microblogging. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume, pages 119-120. ACM.
Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018a. A neural transition-based model for nested mention recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1011-1017.
Chuan Wang, Nianwen Xue, and Sameer Pradhan. 2015. A transition-based algorithm for AMR parsing. In *NAACL HLT* 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015, pages 366-375.
Shaolei Wang, Wanxiang Che, Yue Zhang, Meishan Zhang, and Ting Liu. 2017. Transition-based disfluency detection using lstms. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2785-2794.
Shaolei Wang, Yue Zhang, Wanxiang Che, and Ting Liu. 2018b. Joint extraction of entities and relations based on a novel graph scheme. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4461-4467.
Rui Xia and Zixiang Ding. 2019. Emotion-cause pair extraction: A new task to emotion analysis in texts. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1003-1012.
Rui Xia, Mengran Zhang, and Zixiang Ding. 2019. RTHN: A rn-transformer hierarchical network for emotion cause extraction. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5285-5291. ijcai.org.
Bo Xu, Hongfei Lin, Yuan Lin, Yufeng Diao, Liang Yang, and Kan Xu. 2019. Extracting emotion causes using learning to rank methods from an information retrieval perspective. IEEE Access, 7:15573-15583.
S. Yada, K. Ikeda, K. Hoashi, and K. Mageura. 2017. A bootstrap method for automatic rule acquisition on emotion cause extraction. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 414-421.
Junchi Zhang, Yanxia Qin, Yue Zhang, Mengchi Liu, and Donghong Ji. 2019. Extracting entities and events as a single task using a transition-based neural model. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5422-5428.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Transition-based neural word segmentation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In
3716
The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA - Short Papers, pages 188-193.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1227-1236.