SlowGuess/ACL_2020_OCR huggingface dataset:
Kai Wang $^{1}$ , Weizhou Shen $^{1}$ , Yunyi Yang $^{1}$ , Xiaojun Quan $^{1*}$ , Rui Wang $^{2}$
{wangk73, shenwzh3, yangyy37}@mail2.sysu.edu.cn
quanxj3@mail.sysu.edu.cn,masi.wr@alibaba-inc.com
Aspect-based sentiment analysis aims to determine the sentiment polarity towards a specific aspect in online reviews. Most recent efforts adopt attention-based neural network models to implicitly connect aspects with opinion words. However, due to the complexity of language and the existence of multiple aspects in a single sentence, these models often confuse the connections. In this paper, we address this problem by means of effective encoding of syntax information. Firstly, we define a unified aspect-oriented dependency tree structure rooted at a target aspect by reshaping and pruning an ordinary dependency parse tree. Then, we propose a relational graph attention network (R-GAT) to encode the new tree structure for sentiment prediction. Extensive experiments are conducted on the SemEval 2014 and Twitter datasets, and the experimental results confirm that the connections between aspects and opinion words can be better established with our approach, and the performance of the graph attention network (GAT) is significantly improved as a consequence.
Aspect-based sentiment analysis (ABSA) aims at fine-grained sentiment analysis of online affective texts such as product reviews. Specifically, its objective is to determine the sentiment polarities towards one or more aspects appearing in a single sentence. An example of this task is, given a review great food but the service was dreadful, to determine the polarities towards the aspects food and service. Since the two aspects express quite opposite sentiments, just assigning a sentence-level sentiment polarity is inappropriate. In this regard, ABSA can provide better insights into user reviews compared with sentence-level sentiment analysis.
Intuitively, connecting aspects with their respective opinion words lies at the heart of this task. Most recent efforts (Wang et al., 2016b; Li et al., 2017; Ma et al., 2017; Fan et al., 2018) resort to assorted attention mechanisms to achieve this goal and have reported appealing results. However, due to the complexity of language morphology and syntax, these mechanisms fail occasionally. We illustrate this problem with a real review So delicious was the noodles but terrible vegetables, in which the opinion word terrible is closer to the aspect noodles than delicious, and there could be terrible noodles appearing in some other reviews which makes these two words closely associated. Therefore, the attention mechanisms could attend to terrible with a high weight when evaluating the aspect noodles.
Some other efforts explicitly leverage the syntactic structure of a sentence to establish the connections. Among them, early attempts rely on handcrafted syntactic rules (Qiu et al., 2011; Liu et al., 2013), though they are subject to the quantity and quality of the rules. Dependency-based parse trees are then used to provide more comprehensive syntactic information. For this purpose, a whole dependency tree can be encoded from leaves to root by a recursive neural network (RNN) (Lakkaraju et al., 2014; Dong et al., 2014; Nguyen and Shirai, 2015; Wang et al., 2016a), or the internal node distance can be computed and used for attention weight decay (He et al., 2018a). Recently, graph neural networks (GNNs) are explored to learn representations from the dependency trees (Zhang et al., 2019; Sun et al., 2019b; Huang and Carley, 2019). The shortcomings of these approaches should not be overlooked. First, the dependency relations, which may indicate the connections between aspects and opinion words, are ignored. Second, empirically, only a small part of the parse tree is related to this
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3229-3238 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
task and it is unnecessary to encode the whole tree (Zhang et al., 2018; He et al., 2018b). Finally, the encoding process is tree-dependent, making the batch operation inconvenient during optimization.
In this paper, we re-examine the syntax information and claim that revealing task-related syntactic structures is the key to address the above issues. We propose a novel aspect-oriented dependency tree structure constructed in three steps. Firstly, we obtain the dependency tree of a sentence using an ordinary parser. Secondly, we reshape the dependency tree to root it at a target aspect in question. Lastly, pruning of the tree is performed to retain only edges with direct dependency relations with the aspect. Such a unified tree structure not only enables us to focus on the connections between aspects and potential opinion words but also facilitates both batch and parallel operations. Then we propose a relational graph attention network (R-GAT) model to encode the new dependency trees. R-GAT generalizes graph attention network (GAT) to encode graphs with labeled edges. Extensive evaluations are conducted on the SemEval 2014 and Twitter datasets, and experimental results show that R-GAT significantly improves the performance of GAT. It also achieves superior performance to the baseline methods.
- We propose an aspect-oriented tree structure by reshaping and pruning ordinary dependency trees to focus on the target aspects.
- We propose a new GAT model to encode the dependency relations and to establish the connections between aspects and opinion words.
- The source code of this work is released for future research.
Most recent research work on aspect-based sentiment analysis (ABSA) utilizes attention-based neural models to examine words surrounding a target aspect. They can be considered an implicit approach to exploiting sentence structure, since opinion words usually appear not far from aspects. Such approaches have led to promising progress. Among them, Wang et al. (2016b) proposed to use an attention-based LSTM to identify important sentiment information relating to a target aspect.
Chen et al. (2017) introduced a multi-layer attention mechanism to capture long-distance opinion words for aspects. For a similar purpose, Tang et al. (2016) employed Memory Network with multi-hop attention and external memory. Fan et al. (2018) proposed a multi-grained attention network with both fine-grained and coarse-grained attentions. The pre-trained language model BERT (Devlin et al., 2018) has made successes in many classification tasks including ABSA. For example, Xu et al. (2019) used an additional corpus to posttrain BERT and proved its effectiveness in both aspect extraction and ABSA. Sun et al. (2019a) converted ABSA to a sentence-pair classification task by constructing auxiliary sentences.
Some other efforts try to directly include the syntactic information in ABSA. Since aspects are generally assumed to lie at the heart of this task, establishing the syntactic connections between each target aspect and the other words are crucial. Qiu et al. (2011) manually defined some syntactic rules to identify the relations between aspects and potential opinion words. Liu et al. (2013) obtained partial alignment links with these syntactic rules and proposed a partially supervised word alignment model to extract opinion targets. Afterward, neural network models were explored for this task. Lakkaraju et al. (2014) used a recursive neural network (RNN) to hierarchically encode word representations and to jointly extract aspects and sentiments. In another work, Wang et al. (2016a) combined the recursive neural network with conditional random fields (CRF). Moreover, Dong et al. (2014) proposed an adaptive recursive neural network (AdaRNN) to adaptively propagate the sentiments of words to the target aspect via semantic composition over a dependency tree. Nguyen et al. (2015) further combined the dependency and constituent trees of a sentence with a phrase recursive neural network (PhraseRNN). In a simpler approach, He et al. (2018a) used the relative distance in a dependency tree for attention weight decay. They also showed that selectively focusing on a small subset of context words can lead to satisfactory results.
Recently, graph neural networks combined with dependency trees have shown appealing effectiveness in ABSA. Zhang et al. (2019) and Sun et al. (2019b) proposed to use graph convolutional networks (GCN) to learn node representations from a dependency tree and used them together with
<sup>1</sup>https://github.com/shenwzh3/RGAT-ABSA
Figure 1: Three examples from restaurant reviews to illustrate the relationships among aspect, attention, and syntax in ABSA. Labeled edges indicate dependency relations, and scores under each word represent attention weights assigned by the attention-equipped LSTM. Words with high attention weights are highlighted in red boxes, and words in brackets are the target aspects followed by their sentiment labels.
other features for sentiment classification. For a similar purpose, Huang and Carley (2019) used graph attention networks (GAT) to explicitly establish the dependency relationships between words. However, these approaches generally ignore the dependency relations which might identify the connections between aspects and opinion words.
3 Aspect-Oriented Dependency Tree
In this section, we elaborate on the details of constructing an aspect-oriented dependency tree.
3.1 Aspect, Attention and Syntax
The syntactic structure of a sentence can be uncovered by dependency parsing, a task to generate a dependency tree to represent the grammatical structure. The relationships between words can be denoted with directed edges and labels. We use three examples to illustrate the relationships among aspect, attention and syntax in ABSA, as shown in Figure 1. In the first example, the word like is used as a verb and it expresses a positive sentiment towards the aspect recipe, which is successfully attended by the attention-based LSTM model. However, when it is used as a preposition in the second example, the model still attends to it with a high weight, resulting in a wrong prediction. The third example shows a case where there are two aspects in a single sentence with different sentiment polarities. For the aspect chicken, the LSTM model mistakenly assigns high attention weights to the words but and dried, which leads
to another prediction mistake. These examples demonstrate the limitations of the attention-based model in this task. Such mistakes are likely to be avoided by introducing explicit syntactic relations between aspects and other words. For example, it might be different if the model noticed the direct dependency relationship between chicken and fine in the third example, rather than with but.
3.2 Aspect-Oriented Dependency Tree
The above analysis suggests that dependency relations with direct connections to an aspect may assist a model to focus more on related opinion words, and therefore should be more important than other relations. Also, as shown in Figure 1, a dependency tree contains abundant grammar information, and is usually not rooted at a target aspect. Nevertheless, the focus of ABSA is a target aspect rather than the root of the tree. Motivated by the above observations, we propose a novel aspect-oriented dependency tree structure by reshaping an original dependency tree to root it at a target aspect, followed by pruning of the tree so as to discard unnecessary relations.
Algorithm 1 describes the above process. For an input sentence, we first apply a dependency parser to obtain its dependency tree, where  $r_{ij}$  is the dependency relation from node  $i$  to  $j$ . Then, we build an aspect-oriented dependency tree in three steps. Firstly, we place the target aspect at the root, where multiple-word aspects are treated as entities. Secondly, we set the nodes with direct connections to the aspect as the children, for which the original
Figure 2: Construction of an aspect-oriented dependency tree (bottom) from an ordinary dependency tree (top).
Algorithm 1 Aspect-Oriented Dependency Tree
Input: aspect  $a = \{w_i^a, w_{i+1}^a, \dots, w_k^a\}$ , sentence  $s = \{w_1^s, w_2^s, \dots, w_n^s\}$ , dependency tree  $T$ , and dependency relations  $r$ .
Output: aspect-oriented dependency tree  $\hat{T}$ .
1: Construct the root  $R$  for  $\hat{T}$
2: for  $i$  to  $k$  do
3: for  $j = 1$  to  $n$  do
4: if  $w_{j}^{s} \xrightarrow{r_{ji}} w_{i}^{a}$  then
5:  $w_{j}^{s}\xrightarrow{r_{ji}}R$
6: else if  $w_{j}^{s} \xleftarrow{r_{ij}} w_{i}^{a}$  then
7:  $w_{j}^{s}\xleftarrow{r_{ij}}R$
8: else
9:  $n = \text{distance}(i, j)$
10:  $w_{j}^{s}\xrightarrow{n:con}R$
11: end if
12: end for
13: end for
14: return  $\hat{T}$
dependency relations are retained. Thirdly, other dependency relations are discarded, and instead, we put a virtual relation  $n$ : con ( $n$  connected) from the aspect to each corresponding node, where  $n$  represents the distance between two nodes. If the sentence contains more than one aspect, we construct a unique tree for each aspect. Figure 2 shows an aspect-oriented dependency tree constructed from the ordinary dependency tree. There are at least two advantages with such an aspect-oriented structure. First, each aspect has its own dependency tree and can be less influenced by unrelated nodes and relations. Second, if an aspect contains more than
one word, the dependency relations will be aggregated at the aspect, unlike in (Zhang et al., 2019; Sun et al., 2019b) which require extra pooling or attention operations.
The idea described above is partially inspired by previous findings (He et al., 2018a; Zhang et al., 2018; He et al., 2018b) that it could be sufficient to focus on a small subset of context words syntactically close to the target aspect. Our approach provides a direct way to model the context information. Such a unified tree structure not only enables our model to focus on the connections between aspects and opinion words but also facilitates both batch and parallel operations during training. The motivation we put a new relation  $n: \text{con}$  is that existing parsers may not always parse sentences correctly and may miss important connections to the target aspect. In this situation, the relation  $n: \text{con}$  enables the new tree to be more robust. We evaluate this new relation in the experiment and the results confirm this assumption.
4 Relational Graph Attention Network
To encode the new dependency trees for sentiment analysis, we propose a relational graph attention network (R-GAT) by extending the graph attention network (GAT) (Veličković et al., 2017) to encode graphs with labeled edges.
Dependency tree can be represented by a graph  $\mathcal{G}$  with  $n$  nodes, where each represents a word in the sentence. The edges of  $\mathcal{G}$  denote the dependency between words. The neighborhood nodes of node  $i$  can be represented by  $\mathcal{N}_i$ . GAT iteratively updates each node representation (e.g., word embeddings)
2We set  $n = \infty$  if the distance is longer than 4.
by aggregating neighborhood node representations using multi-head attention:
$$
h _ {a t t i} ^ {l + 1} = \left\| \right\| _ {k = 1} ^ {K} \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} ^ {l k} W _ {k} ^ {l} h _ {j} ^ {l} \tag {1}
$$
$$
\alpha_ {i j} ^ {l k} = \text {a t t e n t i o n} (i, j) \tag {2}
$$
where  $h_{\text{att}_i}^{l+1}$  is the attention head of node  $i$  at layer  $l + 1$ ,  $||_{k=1}^K x_i$  denotes the concatenation of vectors from  $x_1$  to  $x_k$ ,  $\alpha_{ij}^{lk}$  is a normalized attention coefficient computed by the  $k$ -th attention at layer  $l$ ,  $W_k^l$  is an input transformation matrix. In this paper, we adopt dot-product attention for attention  $(i,j)$ .
4.2 Relational Graph Attention Network
GAT aggregates the representations of neighborhood nodes along the dependency paths. However, this process fails to take dependency relations into consideration, which may lose some important dependency information. Intuitively, neighborhood nodes with different dependency relations should have different influences. We propose to extend the original GAT with additional relational heads. We use these relational heads as relation-wise gates to control information flow from neighborhood nodes. The overall architecture of this approach is shown in Figure 3. Specifically, we first map the dependency relations into vector representations, and then compute a relational head as:
$$
h _ {r e l i} ^ {l + 1} = \left\| \right\| _ {m = 1} ^ {M} \sum_ {j \in \mathcal {N} _ {i}} \beta_ {i j} ^ {l m} W _ {m} ^ {l} h _ {j} ^ {l} \tag {3}
$$
$$
g _ {i j} ^ {l m} = \sigma \left(\operatorname {r e l u} \left(r _ {i j} W _ {m 1} + b _ {m 1}\right) W _ {m 2} + b _ {m 2}\right) \tag {4}
$$
$$
\beta_ {i j} ^ {l m} = \frac {\exp \left(g _ {i j} ^ {l m}\right)}{\sum_ {j = 1} ^ {\mathcal {N} _ {i}} \exp \left(g _ {i j} ^ {l m}\right)} \tag {5}
$$
where  $r_{ij}$  represents the relation embedding between nodes  $i$  and  $j$ . R-GAT contains  $K$  attentional heads and  $M$  relational heads. The final representation of each node is computed by:
$$
x _ {i} ^ {l + 1} = h _ {\text {a t t} _ {i}} ^ {l + 1} \mid \mid h _ {\text {r e l} _ {i}} ^ {l + 1} \tag {6}
$$
$$
h _ {i} ^ {l + 1} = \operatorname {r e l u} \left(W _ {l + 1} x _ {i} ^ {l + 1} + b _ {l + 1}\right) \tag {7}
$$
$\longrightarrow$  Relational head  $\longrightarrow$  Attentional head
Figure 3: Structure of the proposed relational graph attention network (R-GAT), which includes two genres of multi-head attention mechanism, i.e., attentional head and relational head.
We use BiLSTM to encode the word embeddings of tree nodes, and obtain its output hidden state  $h_i$  for the initial representation  $h_i^0$  of leaf node  $i$ . Then, another BiLSTM is applied to encode the aspect words, and its average hidden state is used as the initial representation  $h_a^0$  of this root. After applying R-GAT on an aspect-oriented tree, its root representation  $h_a^l$  is passed through a fully connected softmax layer and mapped to probabilities over the different sentiment polarities.
$$
p (a) = \operatorname {s o f t m a x} \left(W _ {p} h _ {a} ^ {l} + b _ {p}\right) \tag {8}
$$
Finally, the standard cross-entropy loss is used as our objective function:
$$
L (\theta) = - \sum_ {(S, A) \in \mathcal {D}} \sum_ {a \in A} \log p (a) \tag {9}
$$
where  $\mathcal{D}$  contains all the sentence-aspects pairs,  $A$  represents the aspects appearing in sentence  $S$ , and  $\theta$  contains all the trainable parameters.
In this section, we first introduce the datasets used for evaluation and the baseline methods employed for comparison. Then, we report the experimental results conducted from different perspectives. Finally, error analysis and discussion are conducted with a few representative examples.
3Dot product has fewer parameters but similar performance with feedforward neural network used in (Veličković et al., 2017).
Dataset	Positive	Neutral	Negative
Train	Test	Train	Test	Train	Test
Laptop	994	341	870	128	464	169
Restaurant	2164	728	807	196	637	196
Twitter	1561	173	3127	346	1560	173
Table 1: Statistics of the three datasets.
Three public sentiment analysis datasets are used in our experiments, two of them are the Laptop and Restaurant review datasets from the SemEval 2014 Task (Maria Pontiki and Manandhar, 2014),<sup>4</sup> and the third is the Twitter dataset used by (Dong et al., 2014). Statistics of the three datasets can be found in Table 1.
5.1.1 Implementation Details
The Biaffine Parser (Dozat and Manning, 2016) is used for dependency parsing. The dimension of the dependency relation embeddings is set to 300. For R-GAT, we use the 300-dimensional word embeddings of GLoVe (Pennington et al., 2014). For R-GAT+BERT, we use the last hidden states of the pre-trained BERT for word representations and fine-tune them on our task. The PyTorch implementation of BERT  $^{5}$  is used in the experiments. R-GAT is shown to prefer a high dropout rate in between [0.6, 0.8]. As for R-GAT+BERT, it works better with a low dropout rate of around 0.2. Our model is trained using the Adam optimizer (Kingma and Ba, 2014) with the default configuration.
5.2 Baseline Methods
A few mainstream models for aspect-based sentiment analysis are used for comparison, including:
- Syntax-aware models: LSTM+SynATT (He et al., 2018a), AdaRNN (Dong et al., 2014), PhraseRNN (Nguyen and Shirai, 2015), ASGCN (Zhang et al., 2019), CDT (Sun et al., 2019b), GAT (Veličković et al., 2017) and TD-GAT (Huang and Carley, 2019).
- Attention-based models: ATAE-LSTM (Wang et al., 2016b), IAN (Ma et al., 2017), RAM (Chen et al., 2017), MGAN (Fan et al., 2018), attention-equipped LSTM, and fine-tuned BERT (Devlin et al., 2018).
- Other recent methods: GCAE (Xue and Li, 2018), JCI (Wang et al., 2018) and TNET (Li
et al., 2018).
- Our methods: R-GAT is our relational graph attention network. R-GAT+BERT is our R-GAT with the BiLSTM replaced by BERT, and the attentional heads of R-GAT will also be replaced by that of BERT.
5.3.1 Overall Performance
The overall performance of all the models are shown in Table 2, from which several observations can be noted. First, the R-GAT model outperforms most of the baseline models. Second, the performance of GAT can be significantly improved when incorporated with relational heads in our aspect-oriented dependency tree structure. It also outperforms the baseline models of ASGCN, and CDT, which also involve syntactic information in different ways. This proves that our R-GAT is better at encoding the syntactic information. Third, the basic BERT can already outperform all the existing ABSA models by significant margins, demonstrating the power of this large pre-trained model in this task. Nevertheless, after incorporating our R-GAT (R-GAT+BERT), this strong model sees further improvement and has achieved a new state of the art. These results have demonstrated the effectiveness of our R-GAT in capturing important syntactic structures for sentiment analysis.
5.3.2 Effect of Multiple Aspects
The appearance of multiple aspects in one single sentence is very typical for ABSA. To study the influence of multiple aspects, we pick out the reviews with more than one aspect in a sentence. Each aspect is represented with its averaged (GloVe) word embeddings, and the distance between any two aspects of a sentence is calculated using the Euclidean distance. If there are more than two aspects, the nearest Euclidean distance is used for each aspect. Then, we select three models (GAT, R-GAT, R-GAT+BERT) for sentiment prediction, and plot the aspect accuracy by different distance ranges in Figure 4. We can observe that the aspects with nearer distances tend to lead to lower accuracy scores, indicating that the aspects with high semantic similarity in a sentence may confuse the models. However, with our R-GAT, both GAT and BERT can be improved across different ranges, showing that our method can alleviate this problem to a certain extent.
<sup>4</sup> http://alt.qcri.org/semeal2014/task4/.
<sup>5</sup>https://github.com/huggingface/transformers
Category	Method	Restaurant	Laptop	Twitter
Accuracy	Macro-F1	Accuracy	Macro-F1	Accuracy	Macro-F1
Syn.	LSTM+SynATT	80.45	71.26	72.57	69.13	-	-
AdaRNN	-	-	-	-	66.30	65.90
PhraseRNN	66.20	59.32	-	-	-	-
ASGCN	80.77	72.02	75.55	71.05	72.15	70.40
CDT	82.30	74.02	77.19	72.99	74.66	73.66
GAT	78.21	67.17	73.04	68.11	71.67	70.13
TD-GAT	80.35	76.13	74.13	72.01	72.68	71.15
Att.	ATAE-LSTM	77.20	-	68.70	-	-	-
IAN	78.60	-	72.10	-	-	-
RAM	80.23	70.80	74.49	71.35	69.36	67.30
MGAN	81.25	71.94	75.39	72.47	72.54	70.81
LSTM	79.10	69.00	71.22	65.75	69.51	67.98
BERT	85.62	78.28	77.58	72.38	75.28	74.11
Others	GCAE	77.28	-	69.14	-	-	-
JCI	-	68.84	-	67.23	-	-
TNET	80.69	71.27	76.54	71.75	74.90	73.60
Ours	R-GAT	83.30	76.08	77.42	73.76	75.57	73.82
Ours	R-GAT+BERT	86.60	81.35	78.21	74.07	76.15	74.88
Figure 4: Results of multiple aspects analysis, which shows that the aspects with nearer distances tend to lead to lower accuracy scores.
5.3.3 Effect of Different Parsers