SlowGuess/NAACL_2022_OCR huggingface dataset:
Zheng Zhang, Zili Zhou*, Yanna Wang
School of Cyber Science and Engineering, Qufu Normal University, China
zhangzhengzz12@gmail.com,
zzl@qqfnu.edu.cn,wyn@qqfnu.edu.cn
Aspect-based Sentiment Analysis (ABSA) aims to predict the sentiment polarity towards a particular aspect in a sentence. Recently, graph neural networks based on dependency tree convey rich structural information which is proven to be utility for ABSA. However, how to effectively harness the semantic and syntactic structure information from the dependency tree remains a challenging research question. In this paper, we propose a novel Syntactic and Semantic Enhanced Graph Convolutional Network (SSEGCN) model for ABSA task. Specifically, we propose an aspect-aware attention mechanism combined with self-attention to obtain attention score matrices of a sentence, which can not only learn the aspect-related semantic correlations, but also learn the global semantics of the sentence. In order to obtain comprehensive syntactic structure information, we construct syntactic mask matrices of the sentence according to the different syntactic distances between words. Furthermore, to combine syntactic structure and semantic information, we equip the attention score matrices by syntactic mask matrices. Finally, we enhance the node representations with graph convolutional network over attention score matrices for ABSA. Experimental results on benchmark datasets illustrate that our proposed model outperforms state-of-the-art methods.
Aspect-based Sentiment Analysis (ABSA) aims to determine the sentiment polarity of a given aspect term in a sentence, where sentiment polarity includes positive, negative and neutral. For example, in Figure 1, ABSA determines the sentiment towards the aspects "food" and "service". For aspect term "food", the sentiment polarity is negative, but "service" is positive. That is, we need to discriminate sentiment polarities according to
Figure 1: An example sentence with its dependency tree where aspect terms (highlighted in red) are connected with other words based on their syntactic dependencies.
different aspects. The main idea of most works is to model the dependency relation between aspects and their associated opinion words.
Prior studies exploit attention mechanism (Wang et al., 2016; Chen et al., 2017; Ma et al., 2017; Liu et al., 2018; Hu et al., 2018; Wang et al., 2018; Huang et al., 2018; Fan et al., 2018) to model the correlations between aspect term and the context. However, attention mechanism is vulnerable to noise in sentences, i.e., the irrelevant words.
Recent works on ABSA leverage Graph Neural Networks (GNNs) over the dependency tree of a sentence to exploit syntactic structure (Sun et al., 2019; Zhang et al., 2019; Liang et al., 2020; Wang et al., 2020; Zhao et al., 2020; Li et al., 2021; Tian et al., 2021). In these works, Zhang et al. (2019) employ Graph Convolutional Network (GCN) to integrate the syntactic information. Sun et al. (2019) propose a GCN model to enhance the feature representations of aspects learned by a Bi-directional Long Short Term Memory (Bi-LSTM). However, these two studies treat all the neighbor nodes of the current node in the graph equally, and are limited in lacking of efficient mechanisms to distinguish the importance of neighbor nodes. Accordingly, noisy nodes may cause the model to misjudge the sentiment polarity. To tackle this problem, Huang and Carley (2019) design a target-dependent graph attention network that updates each node representation by utilizing multi-head attention. The attention mechanism is used to consider semantic correlation of each neighbor node, but it is a lo
Human Language Technologies, pages 4916 - 4925
cal attention by only calculating the importance of neighboring nodes and neglects the global information of the sentence. Following this line, Chen et al. (2020) and Li et al. (2021) combine a syntactic structure graph with a latent semantic graph but the two graphs are constructed independently.
Notably, existing GCN-based approaches do not fully leverage syntactic structure, where only the information of the neighbor nodes is considered. Moreover, some hard cases obscurely express sentiment of aspect term, there is no direct syntactic relationship between aspect term and opinion words. How to capture the information of second-order nodes, third-order nodes and even the global syntactic structure in one shot is still a challenge. Recently, most methods apply multi-layer GCNs to derive the expression of opinion words which brings potential noise. For instance, consider the dependency tree as depicted in Figure 1, a dependency connection exists between the aspect "food" and the opinion word "good". However, the "good" refers to another aspect "service". For aspect term "food", the representation of word "not" is obtained by utilizing two-layer GCN. At the same time, noisy word "not" is also obtained for aspect term "service". As a result, simply considering the syntactic structure of dependency tree might be unsatisfactory. It is necessary to harness the aspect-related semantic information for different aspect terms.
In this paper, we propose a novel Syntactic and Semantic Enhanced Graph Convolutional Network (SSEGCN) model for integrating the syntactic and semantic information of a sentence to solve the above issues. Firstly, SSEGCN captures the contextualized word representations with sentence encoder. Secondly, to model particular semantic correlations for different aspect terms, we propose an aspect-aware attention mechanism to combine with self-attention. The aspect-aware attention learns aspect-related semantic information, while self-attention learns global semantic of the sentence. We take the obtained attention scores as the initial adjacency matrices for GCN. Besides, to fully utilize syntactic structure to complement semantic information, rather than just syntactic first-order neighbor node information, we construct the syntactic mask matrices calculated by the different distances between words in the syntactic dependency structure of the sentence to learn structure information from local to global. Then, we combine adjacency matrices with syntactic mask matrices
to enhance the conventional GCN. Finally, a multilayer graph convolution operation is implemented to obtain aspect-specific features for aspect term sentiment classification. We evaluate our approach on three benchmark datasets. The results show that our model is more effective than a range of baselines and achieves new state-of-the-art performance.
- We propose a SSEGCN model that effectively integrates syntactic structure and semantic correlation for ABSA task.
- We propose an aspect-aware attention mechanism combined with self-attention to learn the aspect-related semantic correlations and the global semantic of the sentence. Meanwhile, we construct syntactic mask matrices to complement with semantic information.
- Experimental results on three benchmark datasets show that the SSEGCN model achieves the state-of-the-art performance. The code and datasets involved in this paper are provided on Github<sup>1</sup>.
Aspect-based sentiment analysis is a fine-grained sentiment analysis task and generally treated as a classification problem. Earlier methods (Jiang et al., 2011; Kiritchenko et al., 2014) manually defined some syntactic rules to predict the sentiment polarity of aspect term.
Most recent researches solve aspect-based sentiment analysis by utilizing attention-based neural network to model semantic correlation between context and aspect term (Wang et al., 2016; Chen et al., 2017; Ma et al., 2017; Liu et al., 2018; Hu et al., 2018; Wang et al., 2018; Huang et al., 2018; Fan et al., 2018). Among them, (Wang et al., 2016) utilized attention mechanism to concentrate on different parts of a sentence to generate an attention vector for aspect sentiment classification. Chen et al. (2017) proposed a multi-layer attention network to infer the sentiment polarity for the aspect. Ma et al. (2017) introduced an interactive attention mechanism to generate the representations for aspects and contexts separately. Wang et al. (2018) designed a hierarchical aspect-specific attention model for aspect sentiment classification. Hu et al. (2018) employed a constrained attention network
'https://github.com/zhangzheng1997/SSEGCN-ABSA
with both orthogonal regularization and sparse regularization.
Another trend explicitly leverages dependency tree. Syntactical information can establish relation connections between aspect and corresponding opinion words, GCN based on dependency tree have achieved impressive performance in ABSA. (Zhang et al., 2019; Sun et al., 2019) stacked a GCN layer to extract rich representations over dependency tree. Liang et al. (2020) build aspect-focused and inter-aspect graphs to learn aspect-specific sentiment features. Zhang and Qian (2020) constructed a global lexical graph to capture the word co-occurrence relation and combined a global lexical graph and a syntactic graph. Liang et al. (2021) constructed a sentiment enhancement graph by integrating the sentiment knowledge from SenticNet to consider the affective information between opinion words and aspect term. Tian et al. (2021) utilized dependency types and distinguished different relations in the dependency tree. However, these approaches generally ignore the effective fusion of syntactic structure and semantic correlation to obtain richer information.
3 Proposed SSEGCN
Figure 2 gives an overview of SSEGCN. In this section, we describe the SSEGCN model which is mainly composed of three components: the Input and Encoding Layer, the Attention Layer, the Syntax-Mask Layer and the GCN Layer. Next, components of SSEGCN will be introduced separately in the rest of the sections.
3.1 Input and Encoding Layer
Given a sentence-aspect pair  $(s,a)$ , where  $s = \{w_1,w_2,\dots ,w_n\}$ .  $a = \{a_{1},a_{2},\dots ,a_{m}\}$  is an aspect and also a sub-sequence of the sentence  $s$ . We utilize BiLSTM or BERT (Devlin et al., 2019) as sentence encoder to extract hidden contextual representations. We first map each word into a low-dimensional real-value vector with embedding matrix  $E\in \mathbb{R}^{|V|\times d_e}$ , where  $|V|$  is the size of vocabulary and  $d_{e}$  denotes the dimensionality of word embeddings. Thus, the sentence  $s$  has corresponding word embeddings  $x = \{x_{1},x_{2},\ldots ,x_{n}\}$ . With the word embeddings of the sentence, BiLSTM is leveraged to produce hidden state vectors  $H = \{h_1,h_2,\dots ,h_n\}$ , where  $h_i\in \mathbb{R}^{2d}$ .  $H$  contains the sub-sequence  $h_a = \{h_{a_1},h_{a_2},\dots ,h_{a_m}\}$  corresponding to the aspect term representation.
Take  $H$  as initial nodes representation in SSEGCN. For the BERT encoder, we adopt "[CLS] sentence [SEP] aspect [SEP]" as input.
3.2 Attention Layer
Attention mechanism is a common way to capture the interactions between the aspect and context words (Fan et al., 2018). In this subsection, we combine aspect-aware attention and self-attention for better semantic features. Figure 2 shows multiple attention adjacency matrices. Here, we construct  $p$  matrices and the  $p$  is a hyper-parameter.
3.2.1 Aspect-aware Attention
Unlike sentence level sentiment classification task, aspect-based sentiment classification aims at judging sentiments of one specific aspect term in its context sentence, and thus calls for modeling particular semantic correlation based on different aspect terms. We propose the aspect-aware attention mechanism, which regards aspect term as query to attention calculation for learning aspect-related features:
$$
A _ {a s p} ^ {i} = \tanh  \left(H _ {a} W ^ {a} \times \left(K W ^ {k}\right) ^ {T} + b\right) \tag {1}
$$
where  $K$  is equal to  $H$  produced by encoding layer.  $W^{a}\in \mathbb{R}^{d\times d}$  and  $W^{k}\in \mathbb{R}^{d\times d}$  are learnable weights. We apply mean pooling on the  $h_a$  and then copy it  $n$  times to obtain  $H_{a}\in \mathbb{R}^{n\times d}$  as aspect representation. Note that we use  $p$ -head aspect-aware attention to obtain attention score matrices for a sentence,  $A_{asp}^{i}$  indicates that it is obtained through the  $i$ -th attention head.
3.2.2 Self-Attention
Similarly, here  $A_{\text{self}}$  can be constructed by utilizing  $p$ -head self-attention (Vaswani et al., 2017) that captures the interaction between two arbitrary words of a single sentence. The calculation involves a query and a key:
$$
A _ {s e l f} ^ {i} = \frac {Q W ^ {Q} \times \left(K W ^ {K}\right) ^ {T}}{\sqrt {d}} \tag {2}
$$
where  $Q$  and  $K$  are both equal to  $H$  produced by encoding layer.  $W^{Q} \in \mathbb{R}^{d \times d}$  and  $W^{K} \in \mathbb{R}^{d \times d}$  are learnable weights. Then, we integrate aspect-aware attention score with self-attention score:
$$
A ^ {i} = A _ {a s p} ^ {i} + A _ {s e l f} ^ {i} \tag {3}
$$
Figure 2: The overall architecture of SSEGCN.
where  $A^i \in \mathbb{R}^{n \times n}$  is used as the input for the computation of the later Syntax-Mask Layer. For each  $A^i$ , it represents a fully connected graph.
3.3 Syntax-Mask Layer
In this section, we first introduce the syntactic mask matrix, and then mask each fully connected graph in terms of different syntactic distances. We treat the syntactic dependency tree as an undirected graph, and each token as a node. Then, we define the distance between node  $v_{i}$  and  $v_{j}$  as  $d(v_{i},v_{j})$ . Since there are multiple paths between nodes on the syntactic dependency tree, we define the distance of the shortest path as  $D$ :
$$
D (i, j) = \min  d \left(v _ {i}, v _ {j}\right) \tag {4}
$$
In the previous part, the  $p$ -head attention mechanism can obtain  $p$  adjacency matrices. Therefore, we set the number of syntactic mask matrices based on different syntactic distances as the same as the number of attention heads. When syntactic distance is relatively small, our model can learn local information; on the contrary, if syntactic distance is relatively large, global structure information will be considered. The calculation of syntactic mask matrix  $M^k$  with threshold  $k$  can be formulated as:
$$
M _ {i j} ^ {k} = \left\{ \begin{array}{c c} 0, & D (i, j) \leq k \\ - \infty , & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
where  $k \in [1, p]$ . To obtain global information and local feature, attention scopes are restricted by different syntactic distances.
$$
M = \left\{M ^ {1}, \dots , M ^ {k}, \dots , M ^ {p} \right\} \tag {6}
$$
$$
A _ {m a s k} ^ {i} = \operatorname {s o f t m a x} \left(A ^ {i} + M ^ {i}\right) \tag {7}
$$
Similarly, syntactic mask matrix based on the distance  $i$  is denoted as  $A_{mask}^{i}\in \mathbb{R}^{n\times n}$
3.4 GCN Layer
Since we have  $p$  different syntactic mask matrices,  $p$  graph convolution operations over  $A_{mask} \in \mathbb{R}^{p \times n \times n}$  are required. If we denote  $h^{l-1}$  as the input state and  $h^l$  as the output state of the  $l$ -th layer,  $h^0$  is the output of sentence encoding layer. Each node in the  $l$ -th GCN layer is updated according to the hidden representations of its neighborhoods:
$$
h _ {i} ^ {l} = \sigma \left(\sum_ {j = 1} ^ {n} A _ {i j} W ^ {l} h _ {j} ^ {l - 1} + b ^ {l}\right) \tag {8}
$$
where  $W^{l}$  is a linear transformation weight,  $b^{l}$  is a bias term, and  $\sigma$  is a nonlinear function. The final output representation of the  $l$ -layer GCN is  $H^{l} = \{h_{1}^{l}, h_{2}^{l}, \dots, h_{n}^{l}\}$ .
After aggregating node representation from each layer of SSEGCN, we obtain the final feature representation. We mask the non-aspect words of the output representation learned by the GCN layer to obtain aspect term representation. Moreover, an average pooling to retain most of the information in the aspect term representation  $h_{a}^{l}$ .
$$
h _ {a} ^ {l} = f \left(h _ {a _ {1}} ^ {l}, h _ {a _ {2}} ^ {l}, \dots , h _ {a _ {m}} ^ {l}\right) \tag {9}
$$
where  $f(\cdot)$  is a mean-pooling function applied over the enhanced aspect representation by GCN layer. Then, we feed  $h_a^l$  into a linear layer, followed by a softmax function to yield a probability distribution over polarity decision space:
$$
p (a) = \operatorname {s o f t m a x} \left(W _ {p} h _ {a} ^ {l} + b _ {p}\right) \tag {10}
$$
where  $W_{p}$  and  $b_{p}$  are the learnable weight and bias.
$$
L (\theta) = - \sum_ {(s, a) \in \mathcal {D}} \sum_ {c \in \mathcal {C}} \log p (a) \tag {11}
$$
where  $\mathcal{D}$  contains all the sentence-aspect pairs and  $a$  represents the aspect appearing in sentence  $s$ .  $\theta$  represents all the trainable parameters and  $\mathcal{C}$  is the collection of sentiment polarities.
We conduct experiments on three benchmark datasets for aspect-based sentiment analysis, including Restaurant and Laptop reviews from SemEval 2014 Task 4 (Pontiki et al., 2014) and Twitter (twitter posts) from Dong et al. (2014). Each aspect is labeled by one of the three sentiment polarities: positive, neutral and negative. The statistics for three datasets are reported in Table 1.
For our experiments, we initialize word embeddings with 300-dimensional Glove vectors provided by Pennington et al. (2014). Additionally, we use 30-dimensional Part-of-Speech (POS) embeddings and 30-dimensional position embeddings which is the relative position of each word with respect to the aspect term in the sentence. Then, word embeddings, POS embeddings and position embeddings are concatenated as input word representations. All sentences are parsed by the Stanford parser<sup>2</sup>. The batch size of all model is set as 16 and the number of GCN layers is 2. Besides, dropout function is applied to the input word representations of the BiLSTM and the dropout rate is set as 0.3. Our model is trained using the Adam optimizer with a learning rate of 0.002 to optimize the parameters. For SSEGCN+BERT, we employ the bert-base-uncased<sup>3</sup> English version.
4.3 Baseline Comparisons
To comprehensively evaluate the performance of our model, we compare with state-of-the-art baselines:
Dataset	Division	# Positive	# Negative	# Neutral
Rest14	Train	2164	807	637
Test	727	196	196
Laptop	Train	976	851	455
Test	337	128	167
Twitter	Train	1507	1528	3016
Test	172	169	336
Table 1: Statistics for the three experimental datasets.
1) IAN (Ma et al., 2017) interactively learns the relationship between aspect and their context.
2) RAM (Chen et al., 2017) proposes a recurrent attention memory network to learn the sentence representation.
3) TNet (Li et al., 2018) employs a CNN model to extract salient features from target-specific embeddings by transformed BiLSTM embeddings.
4) ASGCN (Zhang et al., 2019) proposes to build GCN to learn syntactical information and word dependencies for ABSA.
5) CDT (Sun et al., 2019) utilizes a convolution over a dependency tree model to learn the representations of sentence features.
6) TD-GAT (Huang and Carley, 2019) proposes a target-dependent graph attention network for aspect level sentiment classification, which explicitly utilizes the dependency relationship among words. 7) BiGCN (Zhang and Qian, 2020) builds a concept hierarchy on both the syntactic and lexical graphs for sentiment prediction.
8) kumaGCN (Chen et al., 2020) combines information from a dependency graph and a latent graph to learn syntactic features.
9) R-GAT (Wang et al., 2020) proposes a relational graph attention network to encode the new tree reshaped by an ordinary dependency parse tree.
10) DGEDT (Tang et al., 2020) proposes a dependency graph enhanced dual-transformer network by jointly considering the flat representations from Transformer and graph-based representations from the dependency graph.
11) DualGCN (Li et al., 2021) designs a SynGCN module and a SemGCN module with orthogonal and differential regularizers.
12) BERT (Devlin et al., 2019) is the vanilla BERT model, which adopts "[CLS] sentence [SEP] aspect [SEP]" as input.
13) R-GAT+BERT (Wang et al., 2020) is the R-GAT model based on pre-trained BERT.
14) DGEDT+BERT (Li et al., 2021) is the DGEDT
$^{2}$ https://stanfordnlp.github.io/CoreNLP/  
 $^{3}$ https://github.com/huggingface/transformers
model based on pre-trained BERT.
15) BERT4GCN (Zhang and Qian, 2020) integrates the grammatical sequential features from the PLM of BERT and the syntactic knowledge from dependency graphs.
16) T-GCN (Tian et al., 2021) utilizes dependency types to distinguish different relations in the graph and uses attentive layer ensemble to learn the contextual information from different GCN layers.
To demonstrate the effectiveness of SSEGCN, we compare our model with previous works using accuracy and macro-averaged F1 as evaluation metrics, and report results in Table 2. Experimental results show that our SSEGCN model achieves best performance on the three datasets. In particular, GCN-based models take into account the syntactic structure of a sentence and capture long-term syntactic dependencies between the aspect word and the opinion word, hence outperform all attention-based methods. In GCN-based models, our SSEGCN learns structure information from local to global and considers aspect-related semantic information, and performs significantly better than the previous GCN-based models (i.e., CDT, TD-GAT, BiGCN, KuamGCN, R-GAT, DGEDT and DualGCN) that verifies the effectiveness of fusing syntactic and semantic information. On the other hand, we can observe that the basic BERT has been significantly better than most ABSA models. Combining our SSEGCN with BERT, the results show that the effectiveness of this powerful model is further improved, justifying that SSEGCN learns more syntactic and semantic knowledge can empower ABSA.
As illustrated in Table 3, we further conduct an ablation study to examine the effectiveness of different modules in SSEGCN. The basic SSEGCN is regarded as a baseline model. First, we observe that removal of self-attention degrades the performance, verifying that global semantics of the sentence is necessary for ABSA. We can also notice that model without aspect-aware attention performs unsatisfactory, which indicates that the model lacks of the ability to capture aspect-related semantics, resulting in  $1.70\%$ ,  $1.58\%$  and  $1.47\%$  reductions in accuracy on Restaurant, Laptop and Twitter, respectively. It indicates that aspect-aware attention is essential to capture the correlated semantic informa
tion between aspect and contextual words. Second, removing syntactic mask matrix leads to dropping  $0.90\%$ ,  $1.42\%$  and  $0.88\%$  in accuracy on Restaurant, Laptop and Twitter respectively, which indicates that syntactic mask matrix can assist GCNs to learn better syntactic structure information in original dependency trees. In addition, the removal of syntactic mask matrix and aspect-aware attention leads to a significant performance drop, which further indicates that these two components play crucial roles in SSEGCN for ABSA task. In summary, the ablation experimental results show that each component contributes to our entire model.