SlowGuess/ACL_2020_OCR huggingface dataset:
Zixiang Ding, Rui Xia*, Jianfei Yu
School of Computer Science and Engineering,
Nanjing University of Science and Technology, China
{dingzixiang,rxia,jfyu}@njust.edu.cn
In recent years, a new interesting task, called emotion-cause pair extraction (ECPE), has emerged in the area of text emotion analysis. It aims at extracting the potential pairs of emotions and their corresponding causes in a document. To solve this task, the existing research employed a two-step framework, which first extracts individual emotion set and cause set, and then pair the corresponding emotions and causes. However, such a pipeline of two steps contains some inherent flaws: 1) the modeling does not aim at extracting the final emotion-cause pair directly; 2) the errors from the first step will affect the performance of the second step. To address these shortcomings, in this paper we propose a new end-to-end approach, called ECPE-Two-Dimensional (ECPE-2D), to represent the emotion-cause pairs by a 2D representation scheme. A 2D transformer module and two variants, window-constrained and cross-road 2D transformers, are further proposed to model the interactions of different emotion-cause pairs. The 2D representation, interaction, and prediction are integrated into a joint framework. In addition to the advantages of joint modeling, the experimental results on the benchmark emotion cause corpus show that our approach improves the F1 score of the state-of-the-art from  $61.28\%$  to  $68.89\%$ .
Emotion cause extraction (ECE), as a sub-task of emotion analysis, aims at extracting the potential causes of certain emotion expressions in text. The ECE task was first proposed by Lee et al. (2010) and defined as a word-level sequence labeling problem. Gui et al. (2016a) released a new corpus and re-formalized the ECE task as a clause-level extraction problem. Given an emotion annotation,
the goal of ECE is to predict for each clause in a document if the clause is an emotion cause. This framework has received much attention in the following studies in this direction. Although the ECE task was well defined, it has two problems: Firstly, the emotion must be annotated manually before cause extraction, which greatly limits its practical application; Secondly, the way to first annotate the emotion and then extract the causes ignores the fact that emotions and causes are mutually indicative. To address this problem, we have proposed a new task named emotion-cause pair extraction (ECPE), aiming to extract the potential pairs of emotions and their corresponding causes together in our previous work (Xia and Ding, 2019).
Specifically, ECPE is defined as a fine-grained emotion analysis task, where the goal is to extract a set of valid emotion-cause pairs, given a document consisting of multiple clauses as the input. Figure 1 (a) shows an example of the ECPE task. The input in this example is a document consisting of six clauses. Clause c4 contains a "happy" emotion and it has two corresponding causes: clause c2 ("a policeman visited the old man with the lost money"), and clause c3 ("told him that the thief was caught"). Clause c5 contains a "worried" emotion and the corresponding cause is clause c6 ("as he doesn't know how to keep so much money"). The final output is a set of valid emotion-cause pairs defined at clause level:  $\{\mathrm{c4 - c2},\mathrm{c4 - c3},\mathrm{c5 - c6}\}$ . We have also proposed a two-step approach (ECPE-2Steps) to address the ECPE task (Xia and Ding, 2019). ECPE-2Steps is a pipeline of two steps: Step 1 extracts an emotion set and a cause set individually. For example in Figure 1 (a), the emotion set is  $\{\mathrm{c4},$  c5} and the cause set is  $\{\mathrm{c2},\mathrm{c3},\mathrm{c6}\}$ ; Step 2 conducts emotion-cause pairing and filtering based on the outputs of Step 1. As shown in Figure 1 (a), it first gets the candidate emotion-cause pairs by applying a Cartesian product to the emotion set and
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3161-3170 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An example showing two frameworks for solving the emotion-cause pair extraction (ECPE) task.
cause set, and then train an independent filter to remove the invalid pairs.
Although the ECPE-2Steps approach seems reasonable and performs well, it still has the following shortcomings: (1) as a pipeline of two separate steps, ECPE-2Steps requires two prediction steps to get the final emotion-cause pair. The training of the model is also not directly aimed at extracting the final emotion-cause pair. (2) The errors from Step 1 will affect the performance of Step 2. For one thing, the upper bound of the recall in Step 2 is determined by the recall in Step 1, because Step 2 cannot produce emotion-cause pairs from the emotions or causes that were not extracted by Step 1; for another, if Step 1 predicts too many incorrect emotions or causes, the precision of Step 2 will be reduced.
To address these problems, in this work we propose a new end-to-end ECPE solution, called ECPE-Two-Dimensional (ECPE-2D), to represent the emotion-cause pairs by a 2D representation scheme, and integrate the emotion-cause pair representation, interaction and prediction into a joint framework. As shown in Figure 1 (b), firstly, we design a 2D representation scheme to represent the emotion-cause pairs in forms of a square matrix, where each item represents an emotion-cause pair. Secondly, a 2D Transformer framework and its two variants, window-constrained and cross-road 2D transformers, are further proposed to capture the interaction between different emotion-cause pairs. Finally, we extract the valid emotion-cause pairs based on the 2D representation by conducting a binary classification on each emotion-cause pair. These three parts are integrated into a unified framework and trained simultaneously.
We evaluate our ECPE-2D approach on the
benchmark emotion cause corpus. The experimental results prove that ECPE-2D can obtain overwhelmingly better results than the state-of-the-art methods on the emotion-cause pair extraction task and two auxiliary tasks (emotion extraction and cause extraction).
2.1 Overall Architecture
Following our prior work (Xia and Ding, 2019), we formalize the emotion-cause pair extraction (ECPE) task as follows. The input is a document consisting of multiple clauses  $d = [c_{1}, c_{2}, \dots, c_{|d|}]$ , the goal of ECPE is to extract a set of emotion-cause pairs in  $d$ :
$$
P = \{\dots , c ^ {\mathrm {e m o}} - c ^ {\mathrm {c a u}}, \dots \}, \tag {1}
$$
where  $c^{\mathrm{emo}}$  is an emotion clause and  $c^{\mathrm{cau}}$  is the corresponding cause clause.
The overall architecture of the proposed method is shown in Figure 2. It consists of three parts: 1) 2D Emotion-Cause Pair Representation; 2) 2D Emotion-Cause Pair Interaction; 3) 2D Emotion-Cause Pair Prediction. Firstly, an individual emotion/cause encoding component is firstly employed to obtain the emotion-specific representation vectors and cause-specific representation vectors. A full pairing component is applied to pair the two representation vectors into a 2D representation matrix. Then a 2D transformer module is proposed to model the interactions between different emotion-cause pairs. For each emotion-cause pair in the matrix, the updated representation is finally fed to a softmax layer to predict if the pair is valid or not. The three modules are integrated into a unified framework and trained simultaneously.
Figure 2: Overview of the proposed joint framework for emotion-cause pair extraction.
2.2 2D Emotion-Cause Pair Representation
2.2.1 Individual Emotion/Cause Encoding
The purpose of the clause encoder layer is to generate an emotion-specific representation and a cause-specific representation for each clause in a document. The input is a document contains multiple clauses:  $d = [c_{1}, c_{2}, \dots, c_{|d|}]$ , and each clause also contains multiple words  $c_{i} = [w_{i,1}, w_{i,2}, \dots, w_{i,|c_{i}|}]$ . A hierarchical neural network which contains two layers is employed to capture such a word-clause-document structure.
The lower layer consists of a set of word-level Bi-LSTM modules, each of which corresponds to one clause and accumulate the context information for each word of the clause. The hidden state of the  $j$ -th word in the  $i$ -th clause  $\mathbf{h}_{i,j}$  is obtained based on a bi-directional LSTM. An attention mechanism is then adopted to get the clause representation  $\mathbf{s}_i$ .
The upper layer is composed of two independent components, with the goal to generate an emotion-specific representation  $\mathbf{r}_i^{\mathrm{emo}}$  and a cause-specific representation  $\mathbf{r}_i^{\mathrm{cau}}$  for each clause, respectively. Both components take the clause representation  $(\mathbf{s}_1,\mathbf{s}_2,.,\mathbf{s}_{|d|})$  as input and use two clause-level BiLSTMs to obtain  $\mathbf{r}_i^{\mathrm{emo}}$  and  $\mathbf{r}_i^{\mathrm{cau}}$ , respectively. Finally,  $\mathbf{r}_i^{\mathrm{emo}}$  and  $\mathbf{r}_i^{\mathrm{cau}}$  are respectively feed into two softmax layers to get the emotion prediction  $\hat{\mathbf{y}}_i^{\mathrm{emo}}$  and cause prediction  $\hat{\mathbf{y}}_i^{\mathrm{cau}}$ :
$$
\hat {\mathbf {y}} _ {i} ^ {\mathrm {e m o}} = \operatorname {s o f t m a x} \left(\mathbf {W} ^ {\mathrm {e m o}} \mathbf {r} _ {i} ^ {\mathrm {e m o}} + \mathbf {b} ^ {\mathrm {e m o}}\right), \tag {2}
$$
$$
\hat {\mathbf {y}} _ {i} ^ {\mathrm {c a u}} = \operatorname {s o f t m a x} \left(\mathbf {W} ^ {\mathrm {c a u}} \mathbf {r} _ {i} ^ {\mathrm {c a u}} + \mathbf {b} ^ {\mathrm {c a u}}\right). \tag {3}
$$
It should be noted that the individual emotion/cause encoder here is a compatible module.
Other emotion/cause encoder such as Inter-CE, Inter-EC (Xia and Ding, 2019), and BERT (Devlin et al., 2019) can also be used. We will compare and discuss them in the experiments.
2.2.2 Emotion-Cause Full Pairing
In contrast to the ECPE-2Steps approach (Xia and Ding, 2019) which only extract pairs from the individual emotion set and cause set, we consider all possible pairs of clauses in  $d$  as candidates. Assuming the length of the document is  $|d|$ , then all possible pairs form a matrix  $\pmb{M}$  of the shape  $|d|*|d|$ , where the rows and columns represent the index of the emotion clause and the cause clause in the document, respectively.  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  is the element in the  $i$ -th row and the  $j$ -th column of  $\pmb{M}$  and indicates the emotion-cause pair that consists of the  $i$ -th clause and the  $j$ -th clause, encoded as:
$$
\boldsymbol {M} _ {i, j} = \mathbf {r} _ {i} ^ {\mathrm {e m o}} \oplus \hat {\mathbf {y}} _ {i} ^ {\mathrm {e m o}} \oplus \mathbf {r} _ {j} ^ {\mathrm {c a u}} \oplus \hat {\mathbf {y}} _ {j} ^ {\mathrm {c a u}} \oplus r p e _ {i, j}, \tag {4}
$$
where  $\mathbf{r}_i^{\mathrm{emo}}$  and  $\hat{\mathbf{y}}_i^{\mathrm{emo}}$  are emotion-specific representation and emotion prediction of the  $i$ -th clause  $c_i$ ,  $\mathbf{r}_j^{\mathrm{cau}}$  and  $\hat{\mathbf{y}}_j^{\mathrm{cau}}$  are cause-specific representation and cause prediction of the  $j$ -th clause  $c_j$ .  $\mathbf{rpe}_{i,j}$  is a relative position embedding vector of  $c_j$  relative to  $c_i$ .
2.3 2D Emotion-Cause Pair Interaction
In the previous section, we have obtained a 2D representation matrix consisting of all possible emotion-cause pairs. Each element of the matrix represents a specific emotion-cause pair.
Considering that a document of length  $|d|$  will generate  $|d|*|d|$  possible emotion-cause pairs, a
(a) Window-constrained 2D transformer.
(b) Cross-road 2D transformer.
Figure 3: Two simplified versions of 2D transformer for emotion-cause pair interaction.
mong which only a very small number of pairs are positive samples. Using the independent pair representation for emotion-cause pair prediction will not take advantage of this global information. Therefore, we further designed a 2D transformer for the ECPE task to effectively achieve the interaction between emotion-cause pairs.
2.3.1 Standard 2D Transformer
The standard 2D transformer (Vaswani et al., 2017) consists of a stack of  $N$  layers. Each layer consists of two sublayers: a multi-head 2D self-attention mechanism followed by a position-wise feed forward network.
Multi-head 2D Self-attention. The multi-head 2D self-attention mechanism first calculates the query vector  $\pmb{q}_{i,j}$ , key vector  $\pmb{k}_{i,j}$  and value vector  $\pmb{v}_{i,j}$  for each pair  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  in the document  $d$  as:
$$
\boldsymbol {q} _ {i, j} = \operatorname {R e l u} \left(\boldsymbol {M} _ {i, j} \boldsymbol {W} _ {Q}\right), \tag {5}
$$
$$
\boldsymbol {k} _ {i, j} = \operatorname {R e l u} \left(\boldsymbol {M} _ {i, j} \boldsymbol {W} _ {K}\right), \tag {6}
$$
$$
\boldsymbol {v} _ {i, j} = \operatorname {R e l u} \left(\boldsymbol {M} _ {i, j} \boldsymbol {W} _ {V}\right), \tag {7}
$$
where  $W_{Q}\in R^{n\times n}$ $\pmb{W}_{K}\in R^{n\times n}$ $W_{V}\in R^{n\times n}$  are parameters for queries, keys and values respectively.
For each pair  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$ , a set of weights  $\beta_{i,j} = \{\beta_{i,j,1,1}, \beta_{i,j,1,2}, \dots, \beta_{i,j,|d|,|d|}\}$  are learned:
$$
\beta_ {i, j, a, b} = \frac {\exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {a , b}}{\sqrt {n}}\right)}{\sum_ {a ^ {\prime}} \sum_ {b ^ {\prime}} \exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {a ^ {\prime} , b ^ {\prime}}}{\sqrt {n}}\right)}. \tag {8}
$$
Then the new feature representation of  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  is obtained by considering all the  $|d| * |d|$  pairs in
$$
\hat {\mathbf {z}} _ {i, j} = \sum_ {a = 1} ^ {| d |} \sum_ {b = 1} ^ {| d |} \beta_ {i, j, a, b} \cdot \boldsymbol {v} _ {a, b}. \tag {9}
$$
Position-wise Feed Forward Network. In addition to the attention sublayer, a position-wise feed forward network is applied to each pair separately and identically:
$$
\hat {\mathbf {o}} _ {i, j} = \max  \left(0, z _ {i, j} \boldsymbol {W} _ {1} + \boldsymbol {b} _ {1}\right) \boldsymbol {W} _ {2} + \boldsymbol {b} _ {2}. \tag {10}
$$
It should be noted that both of the above two sublayers use the residual connection followed by normalization layer at its output:
$$
\boldsymbol {z} _ {i, j} = \operatorname {N o r m a l i z e} \left(\hat {\boldsymbol {z}} _ {i, j} + \boldsymbol {M} _ {i, j}\right), \tag {11}
$$
$$
\boldsymbol {o} _ {i, j} = \text {N o r m a l i z e} \left(\hat {\boldsymbol {o}} _ {i, j} + \boldsymbol {z} _ {i, j}\right). \tag {12}
$$
As has mentioned, the standard 2D transformer consists of a stack of  $N$  layers. Let  $l$  denotes the index of transformer layers. The output of the previous layer will be used as the input of the next layer:
$$
\boldsymbol {M} _ {i, j} ^ {(l + 1)} = \boldsymbol {o} _ {i, j} ^ {(l)}. \tag {13}
$$
Computational inefficiency. Since the outputs of the standard transformer are  $|d| * |d|$  elements, each element requires the calculation of  $|d| * |d|$  attention weights, and eventually  $(|d| * |d|) * (|d| * |d|)$  weights are needed to be calculated and temporarily stored. To alleviate the computational load, we furthermore propose two variants of the standard 2D Transformer in the following two subsections: 1) window-constrained 2D Transformer and 2) cross-road 2D Transformer, as shown in Figure 3.
2D transformer	Time complexity	Space complexity
Standard Window-constrained Cross-road	O(batch * |d| * |d| * n * (|d| * |d| + n)) O(batch * |d| * w * n * (|d| * w + n)) O(batch * |d| * |d| * n * (|d| + n))	O(batch * |d| * |d| * (|d| * |d| + n)) O(batch * |d| * w * (|d| * w + n)) O(batch * |d| * |d| * (|d| + n))
Table 1: Comparison of three kinds of 2D transformer in resource consumption. batch indicates the batch size during training,  $|d|$  indicates the number of clauses in the document,  $n$  refers to the hidden state size,  $w$  is equal to  $2 * window + 1$ , and window is the window size used in window-constrained 2D transformer.
(c)
Figure 4: Examples of attentions to be calculated in three 2D Transformers: (a) Standard 2D-Transformer, (b) Window-constrained 2D Transformer, and (c) Cross-road 2D Transformer.
2.3.2 Window-constrained 2D Transformer
Considering that most of the cause clauses are around the emotion clauses, we propose the window-constrained 2D transformer, which is a standard 2D transformer while only takes  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  that meets  $j - i \in [-window, window]$  as inputs.
The outputs of the window-constrained 2D transformer are  $|d| * (window * 2 + 1)$  elements, each element requires the calculation of  $|d| * (window * 2 + 1)$  attention weights, and eventually  $(|d| * (window * 2 + 1)) * (|d| * (window * 2 + 1))$  weights are needed to be calculated and temporarily stored.
It should be noted that compared to the standard 2D transformer, the window-constrained transformer not only greatly reduces the resource requirements, but also alleviates the class imbalance problem to some extent since most of the pairs out of the windows are negative samples.
2.3.3 Cross-road 2D Transformer
Since the feature representation of pairs in the same row or column tends to be closer, we believe that pairs in the same row and column with the current pair have a greater impact on the current pair. Therefore, we propose the cross-road 2D transformer, in which the multi-head 2D self-attention mechanism is replaced by the cross-road 2D self-attention, and the other parts remain the same.
In the cross-road 2D self-attention, we calculate a set of row-wise weights  $\beta_{i,j}^{\mathrm{row}} = \{\beta_{i,j,1}^{\mathrm{row}},\beta_{i,j,2}^{\mathrm{row}},\dots ,\beta_{i,j,|d|}^{\mathrm{row}}\}$  and a set of column
wise weights  $\beta_{i,j}^{\mathrm{col}} = \{\beta_{i,j,1}^{\mathrm{col}},\beta_{i,j,2}^{\mathrm{col}},\dots ,\beta_{i,j,|d|}^{\mathrm{col}}\}$  for each pair  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$
$$
\beta_ {i, j, b} ^ {\text {r o w}} = \frac {\exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {i , b}}{\sqrt {n}}\right)}{\sum_ {b ^ {\prime}} \exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {i , b ^ {\prime}}}{\sqrt {n}}\right)}, \tag {14}
$$
$$
\beta_ {i, j, a} ^ {\text {c o l}} = \frac {\exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {a , j}}{\sqrt {n}}\right)}{\sum_ {a ^ {\prime}} \exp \left(\frac {\boldsymbol {q} _ {i , j} \cdot \boldsymbol {k} _ {a ^ {\prime} , j}}{\sqrt {n}}\right)}. \tag {15}
$$
Then the new feature representation of  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  is obtained by considering the pairs in the same row and column with it:
$$
\hat {\mathbf {z}} _ {i, j} = \left(\sum_ {b = 1} ^ {| d |} \beta_ {i, j, b} ^ {\text {r o w}} \cdot \boldsymbol {v} _ {i, b} + \sum_ {a = 1} ^ {| d |} \beta_ {i, j, a} ^ {\text {c o l}} \cdot \boldsymbol {v} _ {a, j}\right) / 2. \tag {16}
$$
The outputs of the cross-road 2D transformer are  $|d| * |d|$  elements, each element requires the calculation of  $(|d| + |d|)$  attention weights, and eventually  $(|d| * |d|) * (|d| * 2)$  weights are needed to be calculated and temporarily stored.
In this way, the new representation of each pair  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$  can encode the information on all the pairs in the same row and column. In addition, if the cross-road 2D transformer is performed twice or more, the feature representation of each pair can encode the global information on all the pairs in  $\pmb{M}$  while standard 2D transformer requires much more resource to achieve this.
We show an example of attentions to be calculated for standard, window-constrained, and cross-road 2D transformer in Figure 4 (a), (b), and (c), respectively, and summarize their resource consumption in Table 1.
2.4 2D Emotion-Cause Pair Prediction
After a stack of  $N$  2D transformer layers, we can get the final representation  $\pmb{o}_{i,j}^{(N)}$  for each pair  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$ , and predict the emotion-cause pair distribution  $\hat{\mathbf{y}}_{i,j}^{\mathrm{pair}}$  as follows:
$$
\hat {\mathbf {y}} _ {i, j} ^ {\text {p a i r}} = \operatorname {s o f t m a x} \left(W ^ {\text {p a i r}} o _ {i, j} ^ {(N)} + b ^ {\text {p a i r}}\right). \tag {17}
$$
The loss of emotion-cause pair classification for a document  $d$  is:
$$
L ^ {\text {p a i r}} = - \sum_ {i = 1} ^ {| d |} \sum_ {j = 1} ^ {| d |} \mathbf {y} _ {i, j} ^ {\text {p a i r}} \cdot \log \left(\hat {\mathbf {y}} _ {i, j} ^ {\text {p a i r}}\right), \tag {18}
$$
where  $\mathbf{y}_{i,j}^{\mathrm{pair}}$  is the ground truth distribution of emotion-cause pair of  $c_{i}^{\mathrm{emo}} - c_{j}^{\mathrm{cau}}$ .
In order to get better emotion-specific representation and cause-specific representation, we introduce the auxiliary loss for emotion prediction and cause prediction:
$$
L ^ {\text {a u x}} = - \sum_ {i = 1} ^ {| d |} \mathbf {y} _ {i} ^ {\text {e m o}} \cdot \log \left(\hat {\mathbf {y}} _ {i} ^ {\text {e m o}}\right) - \sum_ {i = 1} ^ {| d |} \mathbf {y} _ {i} ^ {\text {c a u}} \cdot \log \left(\hat {\mathbf {y}} _ {i} ^ {\text {c a u}}\right) \tag {19}
$$
where  $\mathbf{y}_i^{\mathrm{emo}}$  and  $\mathbf{y}_i^{\mathrm{cau}}$  are emotion and cause annotation of clause  $c_{i}$ , respectively. The final loss of our model for a document  $d$  is a weighted sum of  $L^{\mathrm{pair}}$  and  $L^{\mathrm{aux}}$  with L2-regularization term as follows:
$$
L = \lambda_ {1} L ^ {\text {p a i r}} + \lambda_ {2} L ^ {\text {a u x}} + \lambda_ {3} | | \theta | | ^ {2}, \tag {20}
$$
where  $\lambda_1, \lambda_2, \lambda_3 \in (0, 1)$  are weights,  $\theta$  denotes all the parameters in this model.
3.1 Dataset and Metrics
We evaluated our proposed model on an ECPE corpus from (Xia and Ding, 2019), which was constructed based on a Chinese emotion cause corpus (Gui et al., 2016a). The same as (Xia and Ding, 2019), we stochastically select  $90\%$  of the data as training data and the remaining  $10\%$  as testing data. In order to obtain statistically credible results, we repeat the experiments 20 times and report the average result. The precision, recall, and F1 score defined in (Xia and Ding, 2019) are used as the metrics for evaluation.
In addition, we also evaluated the performance of two sub-tasks: emotion extraction and cause extraction, using the precision, recall, and F1 score defined in (Gui et al., 2016a) as the metrics.
We use word vectors provided by (Xia and Ding, 2019) that were pre-trained on a corpora from Chinese Weibo. The dimensions of word embedding and relative position embedding are set to 200 and 50, respectively. The number of hidden units in BiLSTM for all our models is set to 100. The dimension of the hidden states, query, key, and value in the transformer are all set to 30. The window size in the window-constrained 2D transformer is set to 3. All weight matrixes and bias are randomly initialized by a uniform distribution U(0.01, 0.01).
For training details, we use the stochastic gradient descent (SGD) algorithm and Adam update rule with shuffled minibatch. The batch size and learning rate are set to 32 and 0.005, respectively. As for regularization, dropout is applied for word embeddings and the dropout rate is set to 0.7. The weights  $\lambda_{1},\lambda_{2},\lambda_{3}$  in formula 20 are set to 1, 1, 1e-5, respectively. The code has been made publicly available on Github<sup>1</sup>.
3.3 Overall Performance
Table 2 shows the experimental results of our models and baseline methods on the ECPE task as well as two subtasks (emotion extraction and cause extraction).
ECPE-2Steps is a set of two-step pipeline methods proposed in our prior work (Xia and Ding, 2019), which first perform individual emotion extraction and cause extraction via multi-task learning, and then conduct emotion-cause pairing and filtering. Specifically, there are three kinds of multitask learning settings: