Venue: ACL 2020

# SpanMlt: A Span-based Multi-Task Learning Framework for

Pair-wise Aspect and Opinion Terms Extraction

 He Zhao , Longtao Huang , Rong Zhang, Quan Lu, Hui Xue

Alibaba Group

{sicheng.zh,kaiyang.hlt,stone.zhangr,luquan.lq,hui.xueh}@alibaba-inc.com

Both authors contributed equally to this research.Corresponding author

###### Abstract

Aspect terms extraction and opinion terms extraction are two key problems of fine-grained Aspect Based Sentiment Analysis (ABSA). The aspect-opinion pairs can provide a global profile about a product or service for consumers and opinion mining systems. However, traditional methods can not directly output aspect-opinion pairs without given aspect terms or opinion terms. Although some recent co-extraction methods have been proposed to extract both terms jointly, they fail to extract them as pairs. To this end, this paper proposes an end-to-end method to solve the task of Pair-wise Aspect and Opinion Terms Extraction (PAOTE). Furthermore, this paper treats the problem from a perspective of joint term and relation extraction rather than under the sequence tagging formulation performed in most prior works. We propose a multi-task learning framework based on shared spans, where the terms are extracted under the supervision of span boundaries. Meanwhile, the pair-wise relations are jointly identified using the span representations. Extensive experiments show that our model consistently outperforms state-of-the-art methods.

+
Footnote †: Corresponding author

+
Footnote †: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239–3248

+
Footnote †: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239–3248

## 1 Introduction

Fine-grained aspect-based sentiment analysis (ABSA) or opinion mining is a field of study that analyzes people's detailed insights towards a product or service. Aspect terms (AT) extraction and opinion terms (OT) extraction are two fundamental subtasks in ABSA Pang and Lee. (2008); Liu (2012). Aspect terms, also named as opinion targets, are the word sequences in the sentence describing attributes or features of the targets. Opinion terms, sometimes called opinion words, are those expressions carrying subjective attitudes. For example, in the sentence _"Otherwise, this place has great service and prices and a nice friendly atmosphere"_, the aspect terms are _service_, _prices_ and _atmosphere_, and the opinion terms are _great_ and _nice friendly_.

Recently, a new research focus, which aims at co-extracting the aspect and opinion terms Wang et al. (2016, 2017); Li and Lam (2017); Wang and Pan (2018); Yu et al. (2019), has drawn increasing attention in both academia and industry. Such methods use joint models and have achieved great progress on both subtasks. However, the extracted AT and OT are not in pairs, and the corresponding relations between them are not well extracted. As the example sentence shown in Figure 1, (_Service, great_), (_prices, great_) and (_atmosphere, nice friendly_) are three aspect-opinion pairs. In contrast, the co-extraction methods can only output the AT set {_service, prices, atmosphere_} and the OT set {_great, nice friendly_} jointly.

The aspect-opinion pairs can deploy more fine-grained sentiment analysis for review text and will benefit many downstream applications, such as opinion summarization and product profiling. By referring to the aspect-opinion pairs in a review sentence, customers can get a glimpse of the pros and cons of a product or service in a short time. Based on the promising results in previous AT and OT extraction, one possible solution for aspect-opinion pair extraction is to decouple the whole task into two subtasks. Firstly, all aspect terms need to be extracted from the sentences. Then, the OT cor

Figure 1: An example of the difference between co-extraction and pair extraction of AT and OT.



responding to each AT can be extracted using a Target-oriented Opinion Words Extraction (TOWE) method Fan et al. (2019). Though this two-stage pipeline approach can extract aspect-opinion pairs, it will suffer from error propagation and the pairs extracting performance will rely heavily on the accuracy of AT extraction. To this end, an end-to-end method that can automatically extract AT and OT as pairs is essential for fine-grained sentiment analysis and opinion mining.

Considering the significance of the aspect-opinion pairs in review sentences, this paper targets at a new subtask for fine-grained ABSA, named PAOTE (Pair-wise Aspect and Opinion Terms Extraction). Given a review sentence, the objective of PAOTE is to extract all the (AT, OT) pairs. Different from the traditional co-extraction task of AT and OT, PAOTE outputs AT and OT in pairs while the co-extraction task only outputs them in separate sets as shown in Figure 1.

Most of the previous AT and OT extraction methods formulate the task as a sequence tagging problem Wang et al. (2016, 2017); Wang and Pan (2018); Yu et al. (2019), specifically using a 5-class tag set: {BA (beginning of aspect), IA (inside of aspect), BP (beginning of opinion), IP (inside of opinion), O (others)}. However, the sequence tagging methods suffer from a huge search space due to the compositionality of labels for extractive ABSA tasks, which has been proven in Lee et al. (2017b); Hu et al. (2019). And as the example in Figure 1, the sequence tagging methods get into trouble when there exist _one-to-many_ or _many-to-one_ relations between AT and OT in the sentence.

In this paper, we propose a span-based multi-task framework to jointly extract both the AT/OT and the pair-wise relations. Motivated by prior works Lee et al. (2017a); Luan et al. (2018), the proposed framework firstly learns word-level representations using a base encoder and then enumerates all possible spans on the input sentence. By sharing the generated span representations, the AT/OT can be extracted under the supervision of span boundaries and class labels. Meanwhile, the pair-wise relations can be identified by computing the span-span correspondence. We further design different encoder structures for the framework. To validate the effectiveness of our method, we conduct a serial of experiments based on public datasets. The comparison results show that the proposed framework can efficiently avoid the cascading errors between tasks and outperforms the state-of-the-art pipeline and joint methods.

In summary, the main contributions of this paper are concluded as follows:

1) We propose an end-to-end model for a new task PAOTE. To the best of our knowledge, it is the first end-to-end model that can jointly extract the AT/OT and the pair-wise relations between them.

2) We design a novel span-based multi-task neural network for PAOTE. It can overcome the drawbacks of sequence tagging methods by taking advantage of the span-level information. And the mutual impact between AT/OT and their pair-wise relations can be identified in this model.

3) We conduct extensive experiments and the results show that our proposed model outperforms the state-of-the-art methods.

## 2 Related Works

### Aspect and Opinion Terms Extraction

For fine-grained ABSA, the aspect terms extraction and opinion terms extraction are two basic subtasks, which has been studied in numerous prior works Hu and Liu (2004); Popescu and Etzioni (2005); Wu et al. (2009); Li et al. (2010); Qiu et al. (2011); Liu et al. (2012, 2013, 2015); Yin et al. (2016); Xu et al. (2019); Devlin et al. (2019). More recently, many works concentrate on co-extracting AT and OT using joint models. Most of the works treat the task as a sequence tagging problem. Wang et al. proposed a joint Recursive Neural Conditional Random Fields (RNCRF) model by using the dependency parse tree to capture dual-propagation among AT and OT Wang et al. (2016). Then they extended their research and constructed a Recursive Neural Structural Correspondence Network (RNSCN) for cross-domain aspect and opinion terms co-extraction Wang and Pan (2018). Another outstanding work, Coupled Multi-Layer Attentions (CMLA) network, learns attentions for AT and OT Wang et al. (2017). However, all these co-extraction methods do not consider the AT and OT as pairs.

For the pair-wise aspect and opinion terms extraction, an obvious solution is a two-stage pipeline strategy. The first stage is to extract aspect terms. Li et al. proposed a state-of-the-art model that can extract aspect terms by using the truncated history attention and the selective transformation network Li et al. (2018). Then in the second stage, the target-oriented opinion terms can be extracted 

with the given aspect terms. This subtask has been proposed in a recent work Fan et al. (2019), where they develop a target-fused sequence tagging method. However, the opinion detection heavily depends on the extracted aspect accuracy, which suffers from error propagation. Our framework is the first to joint perform the two subtasks into an end-to-end model. Moreover, our method does not need any external lexicons or parsers and can effectively deal with multiple relations.

### Joint Entity and Relation Extraction

Joint Entity and Relation Extraction (JERE), which aims to detect entity mentions and their semantic relations simultaneously in text, is an important task in information extraction. The earliest works mostly depend on feature engineering approaches Kate and Mooney (2010); Hoffmann et al. (2011); Li and Ji (2014); Miwa and Sasaki (2014). In recent studies, neural models for JERE have shown superior performance Katiyar and Cardie (2016); Zhang et al. (2017); Miwa and Bansal (2016); Zheng et al. (2017). Moreover, neural multi-task learning has been shown effective in enhancing the interaction between entities and relations. In this paper, we adopt a JERE paradigm to solve the PAOTE task and develop a multi-task framework by extending previous unified setups Luan et al. (2018) and end-to-end span-based models Lee et al. (2017a, 2018).

## 3 Span-based Multi-task Framework

### Problem Definition

Given an input sentence \(S=\{w_{1},w_{2},...,w_{N}\}\) of \(N\) words, the PAOTE task is to extract a set of all the aspect terms \(AT=\{at_{1},at_{2},..,at_{i}\}\), a set of all the opinion terms \(OT=\{ot_{1},ot_{2},...,ot_{j}\}\) and a set of all the (AT, OT) pairs \(P=\{(at_{m},ot_{n}),...\}\) from the sentence. Note that the \(at_{m}\in AT\) and the \(ot_{n}\in OT\) could be a single word or a phrase. Inspired by JERE methods, we process the task in a span-based _term-relation_ joint extraction scheme rather than as a sequence tagging problem. Firstly, all possible spans \(SP=\{s_{1},s_{2},...,s_{K}\}\) are enumerated from the given sentence, where each span is a slice (up to a reasonable length \(l_{s}\)) of the input sentence. Based on the candidate spans, the outputs are two folds: 1) the term types \(T\) for all spans \(SP\), aiming at the AT/OT recognition; 2) the pair-wise relation \(\mathcal{R}\) for all pair of spans \(SP\times SP\), aiming at the (AT, OT) pair identification. Formally, the two subtasks are defined as follows:

* **Term Recognition** is to assign a unique term label \(\mathcal{T}\in\{A,O,null\}\) to each candidate span \(s_{c}\), where \(A\) denotes \(s_{c}\in AT\), \(O\) denotes \(s_{c}\in OT\) and \(null\) denotes that the span does not belong to \(AT\) or \(OT\).
* **Pair-wise Relation Identification** is to assign a binary label \(\mathcal{R}\in\{True,False\}\) to each ordered span pair \((s_{c1},s_{c2})\). Note that the pair-wise relation is defined as a directed relation which always starts from an aspect term and points to an opinion term. So in this formulation, \(s_{c1}\) acts as AT and \(s_{c2}\) acts as OT. \(True\) denotes that \(s_{c1}\) and \(s_{c2}\) are correctly associated.

### Framework

The overall architecture of our span-based multi-task framework (**SpanMlt**) is shown in Figure 2. Given an input sentence, a base encoder is adopted to learn contextualized word representations. Then, a span generator is deployed to enumerate all possible spans, which are represented based on the hidden outputs of the base encoder. For the multi-task learning setup, the span representations are shared for two output scorers. The term scorer is to assign the term label with the highest score to each span. And the relation scorer is to evaluate the pair-wise correspondence between every two spans and assign a binary label to each span pair.

### Span Generator

Given an input sentence \(\{w_{1},w_{2},...,w_{N}\}\), a span \(s_{i}=\{w_{\mathrm{START}(i)},...,w_{\mathrm{END}(i)}\}\) is a single word or phrase with a starting index \(\mathrm{START}(i)\) and an ending index \(\mathrm{END}(i)\). And the maximum length of \(s_{i}\) is \(l_{s}\):

\[1\leq\mathrm{START}(i)\leq\mathrm{END}(i)\leq N \tag{1}\]

\[\mathrm{END}(i)-\mathrm{START}(i)<l_{s} \tag{2}\]

The span generator is a component enumerating all possible spans to generate the candidates for aspect or opinion terms. Then each span will be represented by using the contextualized word representations learned from various base encoders.

### Base Encoders for Span Representations

Noting that SpanMlt is a general framework, we can potentially leverage any network as the encoder to learn word-level representations, which would be shared by higher-level modules. In this paper, we implement two different encoders. One is the 

BiLSTM with pre-trained word embeddings, which has been widely used in numerous neural-based models for NLP tasks. The other is BERT (Devlin et al., 2018), a pre-trained bidirectional transformer encoder which has achieved state-of-the-art performances across a variety of NLP tasks.

#### 3.4.1 BiLSTM Encoder

For the BiLSTM encoder, the input vectors are generated for the word sequence firstly. Motivated by (Lee et al., 2017; Luan et al., 2018), two strategies are involved in building the vector representations: 1) pre-trained word embeddings and 1-dimension CNN over characters; 2) fixed ELMo embeddings. Then, a bidirectional LSTM network is used to encode each word :

(3)

where is the concatenated hidden output of BiLSTM.

To better learn vector representations combined with the syntactic head information for each candidate span, we further employ a self-attention layer over the word vectors in the span. Following previous works (Yang et al., 2016; Zhou et al., 2016), the attention is implemented with a feed forward neural network (FFNN):

(4) (5) (6)

where is the parameters for FFNN, and is a weighted sum of word vectors in span . Therefore, based on the BiLSTM encoder, the final representation for span can be concatenated as:

(7)

where is the feature vector encoding the size of the span .

#### 3.4.2 BERT Encoder

For the BERT encoder, the input sequence is generated by concatenating a [CLS] token, the original word sequence, and a [SEP] token. Each token is converted into an input vector by summing the token, segment, and position embeddings. Assume is the base (or fine-tuned) BERT model. The hidden representation for each token can be obtained:

(8)

Then the span vector representation is directly generated by and :

(9)

Unlike the BiLSTM encoder, we do not use the self-attention or the feature vector for the BERT encoder. Since the transformer of BERT has already utilized the attention mechanism and can learn sufficient contextualized information. And from our preliminary investigations and experiments, most complicated structures may damage the availability of BERT architecture and increase the training difficulty, which will be discussed in Section 4.

### Objective

To construct the loss function for joint training, we use FFNNs over shared span representations to

Figure 2: The overall architecture of the span-based multi-task framework, which alternatively takes a BERT structure or a BiLSTM structure as the base encoder to learn representations for input words and candidate spans.

 

compute the scores of how likely a span \(s_{i}\) has a term label \(y^{\mathcal{T}}_{i}\), and how likely a span pair \((s_{i},s_{j})\) has a relation label \(y^{\mathcal{R}}_{i,j}\), respectively.

#### 3.5.1 Term Scorer

For the term score, each span representation \(\mathbf{p}_{i}\) is fed into an FFNN, and then is normalized with the softmax function to output the probability of the term label:

\[f^{\mathcal{T}}_{i}=\textsc{FFNN}_{\mathcal{T}}(\mathbf{p}_{i}, \theta_{\mathcal{T}}) \tag{10}\] \[P(y^{\mathcal{T}}_{i}|s_{i})=\mathrm{Softmax}(f^{\mathcal{T}}_{ i}) \tag{11}\]

Thus, the loss function for the term extraction subtask can be formulated using the span-level cross-entropy error between the predicted distribution \(P(y^{\mathcal{T}}_{i}|s_{i})\) and the gold distribution \(P(y^{\mathcal{T}*}_{i}|s_{i})\):

\[Loss(\mathcal{T})=-\sum_{i=1}^{k}P(y^{\mathcal{T}*}_{i}|s_{i})log(P(y^{ \mathcal{T}}_{i}|s_{i})) \tag{12}\]

#### 3.5.2 Relation Scorer

For the pair-wise relation score between two spans \((s_{i},s_{j})\), we first compute the probability that a span is in a relation:

\[f^{\mathcal{R}_{s}}_{i}=\textsc{FFNN}_{\mathcal{R}_{s}}(\mathbf{p}_{i},\theta_{ \mathcal{R}_{s}}) \tag{13}\]

In order to reduce the number of generated pairs, we sort the spans according to their scorers \(f^{\mathcal{R}_{s}}_{i}\) and only the top-\(k\) spans are selected to be paired. Then, to measure the correspondence between two spans, the representation \(\mathbf{p}_{i}\) for span \(s_{i}\), the representation \(\mathbf{p}_{j}\) for span \(s_{j}\), and an element-wise multiplication \(\mathbf{p}_{i}\odot\mathbf{p}_{j}\) are concatenated as the input of FFNN:

\[f^{\mathcal{R}_{s}}_{i,j}=\textsc{FFNN}_{\mathcal{R}}([\mathbf{p}_{i};\mathbf{p}_{j}; \mathbf{p}_{i}\odot\mathbf{p}_{j}],\theta_{\mathcal{R}}) \tag{14}\]

The span scores and the correspondence score are summed and fed into the output softmax function:

\[P(y^{\mathcal{R}}_{i,j}|(s_{i},s_{j}))=\mathrm{Softmax}(f^{\mathcal{R}_{s}}_{ i}+f^{\mathcal{R}_{s}}_{j}+f^{\mathcal{R}_{s}}_{i,j}) \tag{15}\]

Thus, the loss function for the pair-wise relation extraction subtask can be formulated using the pair-level cross-entropy error between the predicted distribution \(P(y^{\mathcal{R}}_{i,j}|(s_{i},s_{j}))\) and the gold distribution \(P(y^{\mathcal{R}*}_{i,j}|(s_{i},s_{j}))\):

\[Loss(\mathcal{R})=-\sum_{i=1}^{k}\sum_{j=1}^{k}P(y^{\mathcal{R}*}_{i,j}|(s_{i} ,s_{j}))log(P(y^{\mathcal{R}}_{i,j}|(s_{i},s_{j}))) \tag{16}\]

Finally, losses from the term scorer and the relation scorerer are combined as the training objective of the SpanMlt framework:

\[J(\theta)=\lambda_{\mathcal{T}}Loss(\mathcal{T})+\lambda_{\mathcal{R}}Loss( \mathcal{R}) \tag{17}\]

where \(\lambda_{\mathcal{T}}\) and \(\lambda_{\mathcal{R}}\) are two hyper-parameters to balance the two tasks.

## 4 Experiments

### Datasets

We evaluate our framework on two sets of public datasets, which are both in LAPTOP and RESTAURANT domains from Semeval 2014 Task 4, Semeval 2015 Task 12 and Semeval 2016 Task 5. One is provided by Fan et al. (2019), where the AT and OT pairs are labeled. The other is provided by Wang et al. (2017, 2016), where only the aspect terms and opinion terms are labeled.

### Baselines

Since we are the first to study the joint extraction task of pair-wise AT and OT, there is no available end-to-end model in the literature to be compared. To better evaluate our method, we first compare the AT/OT extraction performances with several widely used sequence tagging models which are constructed by different encoder structures. Then we compare with three joint models, which have achieved state-of-the-art results in AT&OT co-extraction. To evaluate the extraction of (AT, OT) pairs, we further implement a pipeline approach HAST+TOWE. Moreover, since we formulate our problem as a joint term and relation extraction task, we also compare with a joint entity and relation extraction method JERE-MHS. These baselines are introduced as follows:

**BiLSTM+CRF** A sequence tagging method with a BiLSTM network built on top of pre-trained word embeddings, followed by a CRF output layer to perform BIO classification.

**BERT+CRF** A sequence tagging method based on a BERT encoder. The output hidden states of input words are taken as the features for CRF.

**BERT+BiLSTM+CRF** A sequence tagging method based on a BERT encoder. The output hidden states of input words are fed into a BiLSTM structure and then followed by an output CRF layer.

**RNCRF** A joint model of recursive neural network and CRF, proposed by Wang et al. (2016) for single-domain AT and OT extraction.

**CMLA** A joint model of multi-layer attentions proposed by Wang et al. (2017).

**GMTCMLA** A global inference model based on CMLA proposed by Yu et al. (2019).

**RNSCN** A joint model proposed by Wang and Pan (2018) for cross-domain aspect and opinion terms extraction 



**HAST+TOWE (pipeline)** A pipeline approach where the AT are first detected using a model proposed by Li et al. (2018). Then given the predicted AT, the OT are extracted using a recent TOWE method Fan et al. (2019). In this way, the pair-wise relation between AT and OT can be established.

**Jere-MHS** A model for joint entity-relation extraction, proposed by Bekoulis et al. (2018). Although there are a number of complicated models for JERE, few works can simultaneously classify the entity types and the relation types. This method is the outstanding one which can be appropriate to solve our PAOTE task.

### Hyperparameter Settings

For the BiLSTM encoder, we use the 300d GloVe word embeddings pre-trained on unlabeled data of 840 billion tokens1. We use a 3-layer BiLSTM with 100-dimension hidden states. The 8-dimensional char embeddings are randomly initialized. For the character CNN, the filter size is 50 with window sizes of 3, 4 and 5. The ELMo embeddings, pretrained by a 3-layer BiLSTM with 1024 hidden states are fixed and not fine-tuned during the training stage. We use 0.4 dropout for the BiLSTMs and 0.5 dropout for the embeddings. The FFNNs are 50-dimensional with 2 hidden layers. The learning rate is set to be 0.005 for Adam optimizer.

Footnote 1: [https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)

For the BERT encoder, we use the pre-trained uncased \({\rm BERT}_{base}\) model2, and run pre-training on 14lap train set and on the sum of 14res, 15res and 16res train set to get the domain-specific \({\rm BERT}_{finetune}\) models, for LAPTOP and RESTAURANT respectively. The maximum sequence length is 512 with a batch size of 8. The FFNNs are 512-dimensional with a single hidden layer. The learning rate is set to 2e-5 for Adam optimizer.

Footnote 2: [https://github.com/google-research/bert](https://github.com/google-research/bert)

The maximum length of generated spans is set to 8 and top 40% are candidate for pairs. \(\lambda_{\mathcal{T}}\) and \(\lambda_{\mathcal{R}}\) are both set to 1.0. We randomly split 10% of the train sets as dev sets for tuning the hyperparameters. Note that, all the baseline methods are implemented using their publicly released source codes. All the compared models are trained with best settings and the results for test sets are reported when it achieves the best performances on the dev sets.

### Evaluation Metrics

We report F1 scores that measure the performance of our model and all the compared methods respectively for the three subtasks: AT extraction, OT extraction, and pair-wise relation extraction. An extracted AT or OT is regarded as a correct prediction when the boundaries of the span are identical to the ground-truth, and the term label is accurately assigned. An extracted pair-wise relation is correct only when both AT and OT are accurately identified and the relation label is accurately predicted.

### Main Results

The main results are shown in Table 1. Our SpanMLt framework consistently achieves the best scores, both for the AT/OT extraction task and the pair-wise relation extraction task. For AT/OT extraction, the performance of sequence tagging methods is not satisfactory and the BERT-based models perform worst among all these methods. This suggests that BERT may not work well when the dataset for fine-tuning is small. The AT and OT co-extraction models perform much better than sequence tagging methods, indicating that the inter

\begin{table}
\begin{tabular}{l|c c c|c c|c c|c c c} \hline \multirow{2}{*}{Models} & \multicolumn{2}{c|}{14lap} & \multicolumn{2}{c|}{14res} & \multicolumn{2}{c}{15res} \\  & AT & OT & Pair & AT & OT & Pair & AT & OT & Pair & AT & OT & Pair \\ \hline BiLSTM+CRF & 69.80 & 64.96 & - & 78.03 & 75.13 & - & 66.27 & 64.70 & - & 70.43 & 73.33 & - \\ BERT+CRF & 56.38 & 50.14 & - & 54.37 & 48.41 & - & 57.01 & 45.95 & - & 55.83 & 49.38 & - \\ BERT+BiLSTM+CRF & 56.99 & 51.13 & - & 54.08 & 51.58 & - & 55.85 & 47.79 & - & 55.81 & 51.53 & - \\ RNCRF & 74.92 & 67.21 & - & 75.18 & 67.95 & - & 74.14 & 64.50 & - & 73.12 & 65.51 & - \\ CMLA & 75.57 & 66.27 & - & 76.08 & 66.32 & - & 78.81 & 66.15 & - & 76.84 & 65.73 & - \\ RNNCNN & 73.71 & 75.89 & - & 82.12 & 81.67 & - & 71.02 & 69.78 & - & 75.11 & 72.18 & - \\ HAST+TOWE (pipeline) & 79.14 & 67.50 & 53.41 & 82.56 & 71.06 & 23.90 & 79.84 & 68.45 & 58.12 & 81.44 & 75.71 & 63.84 \\ JERE-MIHS & 74.61 & 64.02 & 52.34 & 79.79 & 77.44 & 66.02 & 75.00 & 71.38 & 59.64 & 76.08 & 78.02 & 67.65 \\ SpanMLt (ours) & **84.51** & **80.61** & **68.66** & **87.42** & **83.98** & **75.60** & **81.76** & **78.91** & **64.68** & **85.62** & **85.33** & **71.78** \\ \hline \end{tabular}
\end{table}
Table 1: Main results (F1-score) for AT, OT and (AT, OT) pairs extraction on the four datasets from Fan et al. (2019). State-of-the-art results are marked bold. SpanMLt with the best model setup achieves 15.25%, 9.58%, 5.04% and 4.13% absolute gains compared to the best pair extraction methods.

\begin{table}
\begin{tabular}{l|r r r r r r} \hline \multirow{2}{*}{Models} & \multicolumn{2}{c|}{14lap} & \multicolumn{2}{c|}{14res} & \multicolumn{2}{c}{15res} \\  & AT & OT & AT & OT & AT & OT \\ \hline RNCRF & 78.42 & 79.44 & 84.93 & 84.11 & 67.47 & 67.62 \\ CMLA & 77.80 & 80.17 & 85.29 & 83.18 & 70.73 & 73.68 \\ GMTCMA & **78.69** & 79.89 & 84.50 & 85.20 & 70.53 & 72.78 \\ SpanMit & 77.87 & **80.51** & 85.24 & **85.79** & **71.07** & **75.02** \\ \hline \end{tabular}
\end{table}
Table 2: F1-scores for AT/OT extraction on the three datasets from Wang et al. (2016, 2017).



actions between AT and OT are significant for term extraction. However, all these joint models fail to associate AT and OT as pairs. For pair-wise AT/OT extraction, the HAST+TOWE pipeline method outperforms most other models on aspect detection, but the F1 scores of opinion extraction and pair extraction is much lower than that of SpanMlt, which is primarily due to the error propagation. Another joint entity and relation extraction method, namely JERE-MHS, performs worse than HAST for aspect extraction, but better than TOWE for opinion extraction.

To evaluate the efficacy of SpanMlt on separate AT or OT extraction more intuitively, we further compare with two state-of-the-art models on the larger public datasets from Wang et al. (2016); Zhou et al. (2017), which has no (AT, OT) pair labeled. Table 2 shows that our SpanMlt also achieves comparable results. The minor gap is because there exist some sentences only with AT or OT and without pair-wise relations in this dataset. Thus leads our method to fail to involve the impact of pair-wise relations.

### Framework Analysis

**Base Encoders.** To further investigate the efficacy of different base encoders for our framework, namely, BiLSTM encoder and BERT encoder, we do experiments as shown in Table 3. The BiLSTM encoder with ELMo embeddings performs the best, which indicates the importance of initialized input embeddings. When using pre-trained Glove embeddings for BiLSTM encoder, the results are also satisfactory. An ablation study for the two key components, attention mechanism and char embeddings for BiLSTM encoder, suggests that both components are helpful for improving the performance. The \({\rm BERT}_{base}\) encoder performs better in OT extraction but is inferior to the BiLSTM without ELMo in AT extraction. By using the \({\rm BERT}_{finetune}\) model, the performance is improved, which indicates that introducing domain-specific information can help BERT to learn better contextualized word presentations. Figure 3 shows F1 curves with increasing training steps for fine-tuning BERT on our 14lap train set. We can see that the score first increases and achieves the highest at 5000-6000 steps. But then it decreases as the steps increasing. This result demonstrates that despite the domain-specific information is useful, too many steps on fine-tuning the pre-trained BERT models may not benefit the downstream tasks.

**Multi-task Setup.** We evaluate the effect of multi-task learning for the term extraction subtask and the pair-wise relation extraction subtask defined in our SpanMlt framework. Table 4 reports the F1 scores for an ablation study on 14lap test set. It is observed that the performance improves when learning the two tasks jointly compared with each single task. In addition, to investigate the balance between the two subtasks for multi-task learning, we also draw the F1 curves when adjusting the loss weights \(\lambda_{\mathcal{T}}\) and \(\lambda_{\mathcal{R}}\), as shown in Figure 4. By varying \(\lambda_{\mathcal{T}}/\lambda_{\mathcal{R}}\), we can see that the model attains the best performance at \(1.00\) for AT/OT extraction and \(1.25\) for pair-wise relation extraction. Nevertheless, our multi-task framework is relatively robust when varying the weight settings for the two subtasks.

Figure 3: F1 curves on 14lap dataset for the two tasks, using the base BERT model or fine-tuned BERT models with increasing training steps.

\begin{table}
\begin{tabular}{l c c c} \hline  & AT & OT & Pair \\ \hline Multi-task (SpanMlt) & 84.51 & 80.61 & 68.66 \\ Single-task Term & 83.70 & 79.09 & - \\ Single-task Relation & - & - & 64.19 \\ \hline \end{tabular}
\end{table}
Table 4: Ablation study for multi-task learning on 14lap test set.

 



**Parameter Sensitivity.** Figure 5 shows F1 scores with different maximum span length and different top of candidate spans to generate pairs on 14lap test set. We can see that F1 scores first increases as becomes larger. But it slows the growth when the maximum span length is larger than 8. This indicates that too small could not include all the useful words to generate the spans with accurate boundaries. Nevertheless, the extraction performance is not sensitive to maximum span length. For example, the difference between 8 and 20 are not statistically significant. For the number of candidate spans to generate pairs, top , we can observe similar trends as that of span length. Too small may cause that many correct AT and OT are not included in the candidate set, while large will not improve extraction performance and may cost more training time.

### Case Study

As mentioned previously, SpanMlt is able to identify _one-to-many_ or _many-to-one_ relationships between aspect and opinion terms. To verify that, we pick some examples from the test set of 14lap and show the prediction results of SpanMlt and the pipeline approach HAST+TOWE, as presented in Table 5. In the first two cases, we can see that SpanMlt can correctly assign the same opinion term for two appositive aspect terms. While the pipeline method is less effective when dealing the _one-to-many_ relations either by missing the correct AT (e.g. "_updates_") or assigning the incorrect OT (e.g. "_problems_"). Moreover, we find that our method may sometimes fail to recognize term boundaries (e.g., "_log into the system_" in case 3). There are also some bad cases due to the fact that our method fails to extract all pairs (e.g. "_Windows8_" and "_not want_" in case 4 are missed).

## 5 Conclusion

In this paper, we study a novel task Pair-wise Aspect and Opinion Terms Extraction (PAOTE). We treat this task as a joint term and relation extraction problem and develop a span-based multi-task learning framework (SpanMlt). Our framework can effectively learn contextualized information with various base encoders. Specifically, we try two different encoders (BiLSTM encoder and BERT encoder). Then a span generator enumerates all possible spans and each span is represented based on the outputs of the encoders. For joint optimizing the objectives of term extraction and pair-wise relation extraction, the two subtasks share the span representations and the losses are combined. The experimental results demonstrate that our SpanMlt significantly outperforms all the compared methods. For future works, we will explore pair-wise AT and OT extraction together with aspect category and sentiment polarity classification.

Figure 4: F1 curves on 14lap test set for the two tasks using the best model setup when adjusting the loss balance, .

\begin{table}
\begin{tabular}{l|p{176.9pt}|p{176.9pt}} \hline
**Sentence** & **HAST+TOWE** & **SpanMlt** \\ \hline _I’ve had it for about 2 months now and found no issues with software or updates._ & (_software, no issues_)✓ & (_software, no issues_)✓ \\ \hline _I seem to be having repeat problems as the Mother Board in this one is diagnosed as faulty, related to the graphics card._ & (_Mother Board, problems_)✓ & (_Mother Board, faulty_)✓ \\ \hline _Every time I log into the system after a few hours , there is this endlessly frustrating process that I have to go through. My laptop with Windows ✓ crashed and I did not warnt windows 8._ & (_Windows 8, crashed_)✓ & (_Windows 7, crashed_)✓ \\ \hline \end{tabular}
\end{table}
Table 5: Case study. The golden AT and OT in the sentences are colored as blue and red respectively. And the correct predictions are marked with and incorrect predictions are marked with .

Figure 5: Effect of the maximum span length and the top of candidate spans with highest scores to be paired for our framework.





## Acknowledgments

This research is supported in part by the National Natural Science Foundation of China under Grant 61702500.

## References

* Bekoulis et al. (2018) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Joint entity recognition and relation extraction as a multi-head selection problem. _Expert Syst. Appl._, 114:34-45.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _ArXiv_, abs/1810.04805.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fan et al. (2019) Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2509-2518.
* Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke S. Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In _ACL_.
* Hu et al. (2019) Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. _ArXiv_, abs/1906.03820.
* Hu and Liu (2004) Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In _KDD_.
* Kate and Mooney (2010) Rohit J. Kate and Raymond J. Mooney. 2010. Joint entity and relation extraction using card-pyramid parsing. In _CoNLL_.
* Katiyar and Cardiie (2016) Arzoo Katiyar and Claire Cardiie. 2016. Investigating lstms for joint extraction of opinion entities and relations. In _ACL_.
* Lee et al. (2017a) Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017a. End-to-end neural coreference resolution. _ArXiv_, abs/1707.07045.
* Lee et al. (2018) Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In _NAACL-HLT_.
* Lee et al. (2017b) Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2017b. Learning recurrent span representations for extractive question answering. _ArXiv_, abs/1611.01436.
* Li et al. (2010) Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Yingju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In _COLING_.
* Li and Ji (2014) Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In _ACL_.
* Li et al. (2018) Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. _ArXiv_, abs/1805.00760.
* Li and Lam (2017) Xin Li and William W Y Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In _EMNLP_.
* Liu (2012) Bing Liu. 2012. Sentiment analysis and opinion mining. In _Synthesis Lectures on Human Language Technologies_.
* Liu et al. (2013) Kang Liu, Heng Li Xu, Yang Liu, and Jun Zhao. 2013. Opinion target extraction using partially-supervised word alignment model. In _IJCAI_.
* Liu et al. (2012) Kang Liu, Liheng Xu, and Jun Zhao. 2012. Opinion target extraction using word-based translation model. In _EMNLP-CoNLL_.
* Liu et al. (2015) Pengfei Liu, Shafiq R. Joty, and Helen M. Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In _EMNLP_.
* Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In _EMNLP_.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. _ArXiv_, abs/1601.00770.
* Miwa and Sasaki (2014) Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In _EMNLP_.
* Pang and Lee (2008) Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. In _Foundations and Trends in Information Retrieval_.
* Popescu and Etzioni (2005) Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In _HLT/EMNLP_.
* Qiu et al. (2011) Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. _Computational Linguistics_, 37:9-27.
* Wang and Pan (2018) Wenya Wang and Sinno Jialin Pan. 2018. Recursive neural structural correspondence network for cross-domain aspect and opinion co-extraction. In _ACL_

Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In _EMNLP_.
* Wang et al. (2017) Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In _AAAI_.
* Wu et al. (2009) Yuanbin Wu, Qi Zhang, Xuanjing Huang, and Lide Wu. 2009. Phrase dependency parsing for opinion mining. In _EMNLP_.
* Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2324-2335, Minneapolis, Minnesota. Association for Computational Linguistics.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. In _HLT-NAACL_.
* Yin et al. (2016) Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Mingjie Zhang, and Mengchu Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In _IJCAI_.
* Yu et al. (2019) Jianfei Yu, Jing Jiang, and Ruiping Xia. 2019. Global inference for aspect and opinion terms co-extraction based on multi-task neural networks. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 27:168-177.
* Zhang et al. (2017) Meishan Zhang, Yue Zhang, and Guohong Fu. 2017. End-to-end neural relation extraction with global optimization. In _EMNLP_.
* Zheng et al. (2017) Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. _ArXiv_, abs/1706.05075.
* Zhou et al. (2016) Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Attention-based lstm network for cross-lingual sentiment classification. In _EMNLP_.