SlowGuess/ACL_2020_OCR huggingface dataset:
He Zhao, Longtao Huang *, Rong Zhang, Quan Lu, Hui Xue Alibaba Group
{sicheng.zh,kaiyang.hlt,stone.zhangr,luquan.lq,hui.xueh}@alibaba-inc.com
Aspect terms extraction and opinion terms extraction are two key problems of fine-grained Aspect Based Sentiment Analysis (ABSA). The aspect-opinion pairs can provide a global profile about a product or service for consumers and opinion mining systems. However, traditional methods can not directly output aspect-opinion pairs without given aspect terms or opinion terms. Although some recent co-extraction methods have been proposed to extract both terms jointly, they fail to extract them as pairs. To this end, this paper proposes an end-to-end method to solve the task of Pair-wise Aspect and Opinion Terms Extraction (PAOTE). Furthermore, this paper treats the problem from a perspective of joint term and relation extraction rather than under the sequence tagging formulation performed in most prior works. We propose a multi-task learning framework based on shared spans, where the terms are extracted under the supervision of span boundaries. Meanwhile, the pair-wise relations are jointly identified using the span representations. Extensive experiments show that our model consistently outperforms state-of-the-art methods.
Fine-grained aspect-based sentiment analysis (ABSA) or opinion mining is a field of study that analyzes people's detailed insights towards a product or service. Aspect terms (AT) extraction and opinion terms (OT) extraction are two fundamental subtasks in ABSA (Pang and Lee., 2008; Liu, 2012). Aspect terms, also named as opinion targets, are the word sequences in the sentence describing attributes or features of the targets. Opinion terms, sometimes called opinion words, are those expressions carrying subjective attitudes. For example,
Figure 1: An example of the difference between coextraction and pair extraction of AT and OT.
in the sentence "Otherwise, this place has great service and prices and a nice friendly atmosphere", the aspect terms are service, prices and atmosphere, and the opinion terms are great and nice friendly.
Recently, a new research focus, which aims at co-extracting the aspect and opinion terms (Wang et al., 2016, 2017; Li and Lam, 2017; Wang and Pan, 2018; Yu et al., 2019), has drawn increasing attention in both academia and industry. Such methods use joint models and have achieved great progress on both subtasks. However, the extracted AT and OT are not in pairs, and the corresponding relations between them are not well extracted. As the example sentence shown in Figure 1, (service, great), (prices, great) and (atmosphere, nice friendly) are three aspect-opinion pairs. In contrast, the co-extraction methods can only output the AT set {service, prices, atmosphere} and the OT set {great, nice friendly} jointly.
The aspect-opinion pairs can deploy more fine-grained sentiment analysis for review text and will benefit many downstream applications, such as opinion summarization and product profiling. By referring to the aspect-opinion pairs in a review sentence, customers can get a glimpse of the pros and cons of a product or service in a short time. Based on the promising results in previous AT and OT extraction, one possible solution for aspect-opinion pair extraction is to decouple the whole task into two subtasks. Firstly, all aspect terms need to be extracted from the sentences. Then, the OT cor
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239-3248 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
responding to each AT can be extracted using a Target-oriented Opinion Words Extraction (TOWE) method (Fan et al., 2019). Though this two-stage pipeline approach can extract aspect-opinion pairs, it will suffer from error propagation and the pairs extracting performance will rely heavily on the accuracy of AT extraction. To this end, an end-to-end method that can automatically extract AT and OT as pairs is essential for fine-grained sentiment analysis and opinion mining.
Considering the significance of the aspect-opinion pairs in review sentences, this paper targets at a new subtask for fine-grained ABSA, named PAOTE (Pair-wise Aspect and Opinion Terms Extraction). Given a review sentence, the objective of PAOTE is to extract all the (AT, OT) pairs. Different from the traditional co-extraction task of AT and OT, PAOTE outputs AT and OT in pairs while the co-extraction task only outputs them in separate sets as shown in Figure 1.
Most of the previous AT and OT extraction methods formulate the task as a sequence tagging problem (Wang et al., 2016, 2017; Wang and Pan, 2018; Yu et al., 2019), specifically using a 5-class tag set: {BA (beginning of aspect), IA (inside of aspect), BP (beginning of opinion), IP (inside of opinion), O (others)}. However, the sequence tagging methods suffer from a huge search space due to the compositionality of labels for extractive ABSA tasks, which has been proven in (Lee et al., 2017b; Hu et al., 2019). And as the example in Figure 1, the sequence tagging methods get into trouble when there exist one-to-many or many-to-one relations between AT and OT in the sentence.
In this paper, we propose a span-based multi-task framework to jointly extract both the AT/OT and the pair-wise relations. Motivated by prior works (Lee et al., 2017a; Luan et al., 2018), the proposed framework firstly learns word-level representations using a base encoder and then enumerates all possible spans on the input sentence. By sharing the generated span representations, the AT/OT can be extracted under the supervision of span boundaries and class labels. Meanwhile, the pair-wise relations can be identified by computing the span-span correspondence. We further design different encoder structures for the framework. To validate the effectiveness of our method, we conduct a serial of experiments based on public datasets. The comparison results show that the proposed framework can efficiently avoid the cascading errors between
tasks and outperforms the state-of-the-art pipeline and joint methods.
In summary, the main contributions of this paper are concluded as follows:
1) We propose an end-to-end model for a new task PAOTE. To the best of our knowledge, it is the first end-to-end model that can jointly extract the AT/OT and the pair-wise relations between them.
2) We design a novel span-based multi-task neural network for PAOTE. It can overcome the drawbacks of sequence tagging methods by taking advantage of the span-level information. And the mutual impact between AT/OT and their pair-wise relations can be identified in this model.
3) We conduct extensive experiments and the results show that our proposed model outperforms the state-of-the-art methods.
2.1 Aspect and Opinion Terms Extraction
For fine-grained ABSA, the aspect terms extraction and opinion terms extraction are two basic subtasks, which has been studied in numerous prior works (Hu and Liu, 2004; Popescu and Etzioni, 2005; Wu et al., 2009; Li et al., 2010; Qiu et al., 2011; Liu et al., 2012, 2013, 2015; Yin et al., 2016; Xu et al., 2019; Devlin et al., 2019). More recently, many works concentrate on co-extracting AT and OT using joint models. Most of the works treat the task as a sequence tagging problem. Wang et al. proposed a joint Recursive Neural Conditional Random Fields (RNCRF) model by using the dependency parse tree to capture dual-propagation among AT and OT (Wang et al., 2016). Then they extended their research and constructed a Recursive Neural Structural Correspondence Network (RN-SCN) for cross-domain aspect and opinion terms co-extraction (Wang and Pan, 2018). Another outstanding work, Coupled Multi-Layer Attentions (CMLA) network, learns attentions for AT and OT (Wang et al., 2017). However, all these co-extraction methods do not consider the AT and OT as pairs.
For the pair-wise aspect and opinion terms extraction, an obvious solution is a two-stage pipeline strategy. The first stage is to extract aspect terms. Li et al. proposed a state-of-the-art model that can extract aspect terms by using the truncated history attention and the selective transformation network (Li et al., 2018). Then in the second stage, the target-oriented opinion terms can be extracted
with the given aspect terms. This subtask has been proposed in a recent work (Fan et al., 2019), where they develop a target-fused sequence tagging method. However, the opinion detection heavily depends on the extracted aspect accuracy, which suffers from error propagation. Our framework is the first to joint perform the two subtasks into an end-to-end model. Moreover, our method does not need any external lexicons or parsers and can effectively deal with multiple relations.
2.2 Joint Entity and Relation Extraction
Joint Entity and Relation Extraction (JERE), which aims to detect entity mentions and their semantic relations simultaneously in text, is an important task in information extraction. The earliest works mostly depend on feature engineering approaches (Kate and Mooney, 2010; Hoffmann et al., 2011; Li and Ji, 2014; Miwa and Sasaki, 2014). In recent studies, neural models for JERE have shown superior performance (Katiyar and Cardie, 2016; Zhang et al., 2017; Miwa and Bansal, 2016; Zheng et al., 2017). Moreover, neural multi-task learning has been shown effective in enhancing the interaction between entities and relations. In this paper, we adopt a JERE paradigm to solve the PAOTE task and develop a multi-task framework by extending previous unified setups (Luan et al., 2018) and end-to-end span-based models (Lee et al., 2017a, 2018).
3 Span-based Multi-task Framework
Given an input sentence  $S = \{w_{1}, w_{2}, \dots, w_{N}\}$  of  $N$  words, the PAOTE task is to extract a set of all the aspect terms  $AT = \{at_{1}, at_{2}, \dots, at_{i}\}$ , a set of all the opinion terms  $OT = \{ot_{1}, ot_{2}, \dots, ot_{j}\}$  and a set of all the (AT, OT) pairs  $P = \{(at_{m}, ot_{n}), \dots\}$  from the sentence. Note that the  $at_{m} \in AT$  and the  $ot_{n} \in OT$  could be a single word or a phrase. Inspired by JERE methods, we process the task in a span-based term-relation joint extraction scheme rather than as a sequence tagging problem. Firstly, all possible spans  $SP = \{s_{1}, s_{2}, \dots, s_{K}\}$  are enumerated from the given sentence, where each span is a slice (up to a reasonable length  $l_{s}$ ) of the input sentence. Based on the candidate spans, the outputs are two folds: 1) the term types  $T$  for all spans  $SP$ , aiming at the AT/OT recognition; 2) the pair-wise relation  $\mathcal{R}$  for all pair of spans  $SP \times SP$ , aiming at the (AT, OT) pair identification. Formally, the two subtasks are defined as follows:
- Term Recognition is to assign a unique term label  $\mathcal{T} \in \{A, O, \text{null}\}$  to each candidate span  $s_c$ , where  $A$  denotes  $s_c \in AT$ ,  $O$  denotes  $s_c \in OT$  and null denotes that the span does not belong to  $AT$  or  $OT$ .
- Pair-wise Relation Identification is to assign a binary label  $\mathcal{R} \in \{\text{True}, \text{False}\}$  to each ordered span pair  $(s_{c1}, s_{c2})$ . Note that the pair-wise relation is defined as a directed relation which always starts from an aspect term and points to an opinion term. So in this formulation,  $s_{c1}$  acts as AT and  $s_{c2}$  acts as OT. True denotes that  $s_{c1}$  and  $s_{c2}$  are correctly associated.
The overall architecture of our span-based multitask framework (SpanMIt) is shown in Figure 2. Given an input sentence, a base encoder is adopted to learn contextualized word representations. Then, a span generator is deployed to enumerate all possible spans, which are represented based on the hidden outputs of the base encoder. For the multitask learning setup, the span representations are shared for two output scorers. The term scorer is to assign the term label with the highest score to each span. And the relation scorer is to evaluate the pair-wise correspondence between every two spans and assign a binary label to each span pair.
3.3 Span Generator
Given an input sentence  $\{w_{1},w_{2},\dots,w_{N}\}$ , a span  $s_i = \{w_{\mathrm{START}(i)},\dots,w_{\mathrm{END}(i)}\}$  is a single word or phrase with a starting index  $\mathrm{START}(i)$  and an ending index  $\mathrm{END}(i)$ . And the maximum length of  $s_i$  is  $l_s$ :
$$
1 \leq \operatorname {S T A R T} (i) \leq \operatorname {E N D} (i) \leq N \tag {1}
$$
$$
\operatorname {E N D} (i) - \operatorname {S T A R T} (i) <   l _ {s} \tag {2}
$$
The span generator is a component enumerating all possible spans to generate the candidates for aspect or opinion terms. Then each span will be represented by using the contextualized word representations learned from various base encoders.
3.4 Base Encoders for Span Representations
Noting that SpanMlt is a general framework, we can potentially leverage any network as the encoder to learn word-level representations, which would be shared by higher-level modules. In this paper, we implement two different encoders. One is the
Figure 2: The overall architecture of the span-based multi-task framework, which alternatively takes a BERT structure or a BiLSTM structure as the base encoder to learn representations for input words and candidate spans.
BiLSTM with pre-trained word embeddings, which has been widely used in numerous neural-based models for NLP tasks. The other is BERT (Devlin et al., 2018), a pre-trained bidirectional transformer encoder which has achieved state-of-the-art performances across a variety of NLP tasks.
3.4.1 BiLSTM Encoder
For the BiLSTM encoder, the input vectors  $\{x_{1}, x_{2}, \ldots, x_{N}\}$  are generated for the word sequence firstly. Motivated by (Lee et al., 2017a; Luan et al., 2018), two strategies are involved in building the vector representations: 1) pre-trained word embeddings and 1-dimension CNN over characters; 2) fixed ELMo embeddings. Then, a bidirectional LSTM network is used to encode each word  $x_{t}$ :
$$
\boldsymbol {h} _ {t} = [ \overrightarrow {\mathrm {L S T M}} (x _ {t}); \overrightarrow {\mathrm {L S T M}} (x _ {t}) ], t \in [ 1, N ] \qquad (3)
$$
where  $h_t$  is the concatenated hidden output of BiLSTM.
To better learn vector representations combined with the syntactic head information for each candidate span, we further employ a self-attention layer over the word vectors in the span. Following previous works (Yang et al., 2016; Zhou et al., 2016), the attention is implemented with a feed forward neural network (FFNN):
$$
u _ {t} = \operatorname {F F N N} _ {\alpha} \left(\boldsymbol {h} _ {t}, \theta_ {\alpha}\right) \tag {4}
$$
$$
\alpha_ {i, t} = \frac {\exp \left(u _ {t}\right)}{\sum_ {k = \text {S T A R T} (i)} ^ {\text {E N D} (i)} \exp \left(u _ {k}\right)} \tag {5}
$$
$$
\hat {\boldsymbol {h}} _ {i} = \sum_ {k = \operatorname {S T A R T} (i)} ^ {\operatorname {E N D} (i)} \alpha_ {i, t} \cdot u _ {t} \tag {6}
$$
where  $\theta_{\alpha}$  is the parameters for FFNN, and  $\hat{\pmb{h}}_i$  is a weighted sum of word vectors in span  $s_i$ . Therefore, based on the BiLSTM encoder, the final representation  $\pmb{p}_i$  for span  $s_i$  can be concatenated as:
$$
\boldsymbol {p} _ {i} = \left[ \boldsymbol {h} _ {\text {S T A R T}} (i); \boldsymbol {h} _ {\text {E N D}} (i); \hat {\boldsymbol {h}} _ {i}; \phi (i) \right] \tag {7}
$$
where  $\phi (i)$  is the feature vector encoding the size of the span  $s_i$ .
3.4.2 BERT Encoder
For the BERT encoder, the input sequence is generated by concatenating a [CLS] token, the original word sequence, and a [SEP] token. Each token is converted into an input vector  $\boldsymbol{x}_t$  by summing the token, segment, and position embeddings. Assume  $\mathrm{BERT}(\cdot)$  is the base (or fine-tuned) BERT model. The hidden representation for each token can be obtained:
$$
\boldsymbol {h} _ {t} = \operatorname {B E R T} (\boldsymbol {x} _ {t}) \tag {8}
$$
Then the span vector representation  $\pmb{p}_i$  is directly generated by  $h_{\mathrm{START}(i)}$  and  $h_{\mathrm{END}(i)}$ :
$$
\boldsymbol {p} _ {i} = \left[ \boldsymbol {h} _ {\text {S T A R T}} (i); \boldsymbol {h} _ {\text {E N D}} (i) \right] \tag {9}
$$
Unlike the BiLSTM encoder, we do not use the self-attention or the feature vector for the BERT encoder. Since the transformer of BERT has already utilized the attention mechanism and can learn sufficient contextualized information. And from our preliminary investigations and experiments, most complicated structures may damage the availability of BERT architecture and increase the training difficulty, which will be discussed in Section 4.
3.5 Objective
To construct the loss function for joint training, we use FFNNs over shared span representations to
compute the scores of how likely a span  $s_i$  has a term label  $y_i^{\mathcal{T}}$ , and how likely a span pair  $(s_i, s_j)$  has a relation label  $y_{i,j}^{\mathcal{R}}$ , respectively.
3.5.1 Term Scorer
For the term score, each span representation  $\pmb{p}_i$  is fed into an FFNN, and then is normalized with the softmax function to output the probability of the term label:
$$
f _ {i} ^ {\mathcal {T}} = \operatorname {F F N N} _ {\mathcal {T}} \left(\boldsymbol {p} _ {i}, \theta_ {\mathcal {T}}\right) \tag {10}
$$
$$
P \left(y _ {i} ^ {\mathcal {T}} \mid s _ {i}\right) = \operatorname {S o f t m a x} \left(f _ {i} ^ {\mathcal {T}}\right) \tag {11}
$$
Thus, the loss function for the term extraction subtask can be formulated using the span-level cross-entropy error between the predicted distribution  $P(y_{i}^{\mathcal{T}}|s_{i})$  and the gold distribution  $P(y_{i}^{\mathcal{T}^{*}}|s_{i})$ :
$$
\operatorname {L o s s} (\mathcal {T}) = - \sum_ {i = 1} ^ {k} P \left(y _ {i} ^ {\mathcal {T} ^ {*}} \mid s _ {i}\right) \log \left(P \left(y _ {i} ^ {\mathcal {T}} \mid s _ {i}\right)\right) \tag {12}
$$
3.5.2 Relation Scorer
For the pair-wise relation score between two spans  $(s_i, s_j)$ , we first compute the probability that a span is in a relation:
$$
f _ {i} ^ {\mathcal {R} _ {s}} = \operatorname {F F N N} _ {\mathcal {R} _ {s}} \left(\boldsymbol {p} _ {i}, \theta_ {\mathcal {R} _ {s}}\right) \tag {13}
$$
In order to reduce the number of generated pairs, we sort the spans according to their scorers  $f_{i}^{\mathcal{R}_{s}}$  and only the top-  $k$  spans are selected to be paired. Then, to measure the correspondence between two spans, the representation  $\pmb{p}_i$  for span  $s_i$ , the representation  $\pmb{p}_j$  for span  $s_j$ , and an element-wise multiplication  $\pmb{p}_i \odot \pmb{p}_j$  are concatenated as the input of FFNN:
$$
f _ {i, j} ^ {\mathcal {R}} = \operatorname {F F N N} _ {\mathcal {R}} \left(\left[ \boldsymbol {p} _ {i}; \boldsymbol {p} _ {j}; \boldsymbol {p} _ {i} \odot \boldsymbol {p} _ {j} \right], \theta_ {\mathcal {R}}\right) \tag {14}
$$
The span scores and the correspondence score are summed and fed into the output softmax function:
$$
P \left(y _ {i, j} ^ {\mathcal {R}} \mid \left(s _ {i}, s _ {j}\right)\right) = \operatorname {S o f t m a x} \left(f _ {i} ^ {\mathcal {R} _ {s}} + f _ {j} ^ {\mathcal {R} _ {s}} + f _ {i, j} ^ {\mathcal {R}}\right) \tag {15}
$$
Thus, the loss function for the pair-wise relation extraction subtask can be formulated using the pair-level cross-entropy error between the predicted distribution  $P(y_{i,j}^{\mathcal{R}}|(s_i,s_j))$  and the gold distribution  $P(y_{i,j}^{\mathcal{R}^{*}}|(s_{i},s_{j}))$ :
$$
L o s s (\mathcal {R}) = - \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} P \left(y _ {i, j} ^ {\mathcal {R} ^ {*}} \mid \left(s _ {i}, s _ {j}\right)\right) \log \left(P \left(y _ {i, j} ^ {\mathcal {R}} \mid \left(s _ {i}, s _ {j}\right)\right)\right) \tag {16}
$$
Finally, losses from the term scorer and the relation scorer are combined as the training objective of the SpanMlt framework:
$$
J (\theta) = \lambda_ {\mathcal {T}} L o s s (\mathcal {T}) + \lambda_ {\mathcal {R}} L o s s (\mathcal {R}) \tag {17}
$$
where  $\lambda_{\mathcal{T}}$  and  $\lambda_{\mathcal{R}}$  are two hyper-parameters to balance the two tasks.
We evaluate our framework on two sets of public datasets, which are both in LAPTOP and RESTAURANT domains from Semeval 2014 Task 4, Semeval 2015 Task 12 and Semeval 2016 Task 5. One is provided by (Fan et al., 2019), where the AT and OT pairs are labeled. The other is provided by (Wang et al., 2017, 2016), where only the aspect terms and opinion terms are labeled.
Since we are the first to study the joint extraction task of pair-wise AT and OT, there is no available end-to-end model in the literature to be compared. To better evaluate our method, we first compare the AT/OT extraction performances with several widely used sequence tagging models which are constructed by different encoder structures. Then we compare with three joint models, which have achieved state-of-the-art results in AT&OT co-extraction. To evaluate the extraction of (AT, OT) pairs, we further implement a pipeline approach HAST+TOWE. Moreover, since we formulate our problem as a joint term and relation extraction task, we also compare with a joint entity and relation extraction method JERE-MHS. These baselines are introduced as follows:
BiLSTM+CRF A sequence tagging method with a BiLSTM network built on top of pre-trained word embeddings, followed by a CRF output layer to perform BIO classification.
BERT+CRF A sequence tagging method based on a BERT encoder. The output hidden states of input words are taken as the features for CRF.
BERT+BiLSTM+CRF A sequence tagging method based on a BERT encoder. The output hidden states of input words are fed into a BiLSTM structure and then followed by an output CRF layer.
RNCRF A joint model of recursive neural network and CRF, proposed by (Wang et al., 2016) for single-domain AT and OT extraction.
CMLA A joint model of multi-layer attentions proposed by (Wang et al., 2017).
GMTCMLA A global inference model based on CMLA proposed by (Yu et al., 2019).
RNSCN A joint model proposed by (Wang and Pan, 2018) for cross-domain aspect and opinion terms extraction.
Models	14lap	14res	15res	16res
AT	OT	Pair	AT	OT	Pair	AT	OT	Pair	AT	OT	Pair
BiLSTM+CRF	69.80	64.96	-	78.03	75.13	-	66.27	64.70	-	70.43	73.33	-
BERT+CRF	56.38	50.14	-	54.37	48.41	-	57.01	45.95	-	55.83	49.38	-
BERT+BiLSTM+CRF	56.99	51.33	-	54.08	51.53	-	55.85	47.79	-	55.18	51.53	-
RNCRF	74.92	67.21	-	75.18	67.95	-	74.14	64.50	-	73.12	65.51	-
CMLA	75.57	66.27	-	76.08	66.32	-	78.31	66.15	-	76.84	65.73	-
RNSCN	73.71	75.89	-	82.12	81.67	-	71.02	69.78	-	75.11	72.18	-
HAST+TOWE (pipeline)	79.14	67.50	53.41	82.56	75.10	62.39	79.84	68.45	58.12	81.44	75.71	63.84
JERE-MHS	74.61	64.02	52.34	79.79	77.44	66.02	75.00	71.38	59.64	76.08	78.02	67.65
SpanMIt (ours)	84.51	80.61	68.66	87.42	83.98	75.60	81.76	78.91	64.68	85.62	85.33	71.78
Table 1: Main results (F1-score) for AT, OT and (AT, OT) pairs extraction on the four datasets from (Fan et al., 2019). State-of-the-art results are marked bold. SpanMlt with the best model setup achieves  $15.25\%$ ,  $9.58\%$ ,  $5.04\%$  and  $4.13\%$  absolute gains compared to the best pair extraction methods.
Models	14lap	14res	15res
AT	OT	AT	OT	AT	OT
RNCRF	78.42	79.44	84.93	84.11	67.47	67.62
CMLA	77.80	80.17	85.29	83.18	70.73	73.68
GMTCMLA	78.69	79.89	84.50	85.20	70.53	72.78
SpanMIt	77.87	80.51	85.24	85.79	71.07	75.02
Table 2: F1-scores for AT/OT extraction on the three datasets from (Wang et al., 2016, 2017).
HAST+TOWE (pipeline) A pipeline approach where the AT are first detected using a model proposed by (Li et al., 2018). Then given the predicted AT, the OT are extracted using a recent TOWE method (Fan et al., 2019). In this way, the pair-wise relation between AT and OT can be established.
JERE-MHS A model for joint entity-relation extraction, proposed by (Bekoulis et al., 2018). Although there are a number of complicated models for JERE, few works can simultaneously classify the entity types and the relation types. This method is the outstanding one which can be appropriate to solve our PAOTE task.
4.3 Hyperparameter Settings