SlowGuess/ACL_2020_OCR huggingface dataset:
Xiao Chen $^{1}$ , Changlong Sun $^{2}$ , Jingjing Wang $^{1,*}$ , Shoushan Li $^{1}$ , Luo Si $^{2}$ , Min Zhang $^{1}$ , Guodong Zhou $^{1}$
$^{1}$ School of Computer Science and Technology, Soochow University, China  $^{2}$ Alibaba Group, China
xchen1206@stu.suda.edu.cn, changlong.scl@taobao.com {djingwang,lishoushan,minzhang,gdzhou}@suda.edu.cn luo.si@alibaba-inc.com
In the literature, existing studies always consider Aspect Sentiment Classification (ASC) as an independent sentence-level classification problem aspect by aspect, which largely ignore the document-level sentiment preference information, though obviously such information is crucial for alleviating the information deficiency problem in ASC. In this paper, we explore two kinds of sentiment preference information inside a document, i.e., contextual sentiment consistency w.r.t. the same aspect (namely intra-aspect sentiment consistency) and contextual sentiment tendency w.r.t. all the related aspects (namely inter-aspect sentiment tendency). On the basis, we propose a Cooperative Graph Attention Networks (CoGAN) approach for cooperatively learning the aspect-related sentence representation. Specifically, two graph attention networks are leveraged to model above two kinds of document-level sentiment preference information respectively, followed by an interactive mechanism to integrate the two-fold preference. Detailed evaluation demonstrates the great advantage of the proposed approach to ASC over the state-of-the-art baselines. This justifies the importance of the document-level sentiment preference information to ASC and the effectiveness of our approach capturing such information.
Aspect Sentiment Classification (ASC), a fine-grained sentiment classification task in the field of sentiment analysis (Pang and Lee, 2007; Li et al., 2010), aims to identify the sentiment polarity (e.g., positive, negative or neutral) for each aspect discussed inside a sentence. For example, the sentence "The restaurant has quite low price but the food tastes not good" would be assigned with a positive polarity for the aspect price and with a negative
Intra-Aspect Sentiment Consistency
Document 1:
S1: Excellent food, although the interior could use some help.
- Category = FOOD#QUALITY, polarity
- Category = AMBIENCE#GENERAL, polarity = negative
S2: The space kind of feels like an Alice in Wonderland setting, without it trying to be that.
- Category = AMBIENCE#GENERAL, polarity = negative
S3: I paid just about $60 for a good meal, tough :)
- Category = FOOD#QUALITY, polarity = positive
- Category  $=$  FOOD#PRICES, polarity  $=$  positive
Inter-Aspect Sentiment Tendency
Document 2:
S1: If you've ever been along with the river in Weehawken you have an idea of the top of view the chart house has to offer. - Category  $=$  LOCATION#GENERAL, polarity  $=$  positive
S2: Add to that great service and great food at a reasonable price and you have yourself the beginning of a great evening.
- Category  $=$  SERVICE#GENERAL, polarity  $=$  positive
- Category = FOOD#QUALITY, polarity = positive
- Category = FOOD#PRICES, polarity = positive
S3: The lava cake dessert was incredible and I recommend it.
- Category = FOOD#QUALITY, polarity = positive
Figure 1: Two documents from SemEval 2016 (Pontiki et al. (2016)) datasets, where aspect category is defined as the entity  $E$  and attribute  $A$  pair (i.e.,  $E\#A$ ). Red lines denote the intra-aspect sentiment consistency and blue lines denote the inter-aspect sentiment tendency.
polarity for the aspect food. Over the past decade, the ASC task has been drawing more and more interests (Tang et al., 2016b; Wang et al., 2018) due to its wide applications, such as e-commerce customer service (Jing et al., 2015), public opinion mining (Wang et al., 2019c) and Question Answering (Wang et al., 2019a).
In the literature, given the ASC datasets (Pontiki et al. (2016)) where aspects (i.e., entity and attribute) are manually annotated comprehensively sentence by sentence, previous studies model the aspect sentiment independently sentence by sentence, which suffer from the problem of ignoring the document-level sentiment preference information. In this study, we argue that such document-level sentiment preference information is crucial to
*Corresponding Author: Jingjing Wang.
3667
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3667-3677
remedy the information deficiency issue in ASC. Especially, we explore two kinds of sentiment preference information inside a document.
On one hand, we assume that the sentences in a document involving the same aspect tend to have the same sentiment polarity on this aspect. For instance, in Document 1, both the sentence S1 and S2 involve aspect AMBIENCE#GENERAL. Although it is difficult to infer the negative sentiment for aspect AMBIENCE#GENERAL through the clause "without it trying to be that" in S2, we can infer that the sentiment of aspect AMBIENCE#GENERAL is more likely to be negative according to S1. This is because it is easier to infer negative for aspect AMBIENCE#GENERAL through the clause "interior could use some help" in S1. Therefore, a well-behaved approach should capture the contextual sentiment consistency w.r.t. the same aspect (namely intra-aspect consistency for short) information.
On the other hand, we assume that the sentences in a document tend to have the same sentiment polarity on all the related aspects. For the example of Document 2 where the sentence S2 involves multiple aspects, it is really hard to precisely predict the sentiment polarity for each aspect. However, when taken the context into consideration, the sentiment polarity for each aspect in S2 is largely possible to be positive, since all the neighboring sentences express the positive sentiment polarity for their aspects. Therefore, a well-behaved model should capture the contextual sentiment tendency w.r.t. all the related aspects (namely inter-aspect tendency for short) information.
To well accommodate the above two kinds of document-level sentiment preference information, we propose a Cooperative Graph Attention Networks (CoGAN) approach to ASC. Specifically, two graph attention networks are constructed to model the two-fold sentiment preference with the attention weight to measure the preference-degree. Furthermore, considering that the two-fold preference can jointly influence the sentiment polarities for aspects, we propose an interactive mechanism to jointly model the two-fold preference for obtaining better aspect-related sentence representation. Detailed evaluation shows our proposed CoGAN approach significantly outperforms the state-of-the-art baselines, including the three top-performed systems from SemEval-2015 Task 12 and SemEval-2016 Task 5 (Pontiki et al., 2015, 2016).
In this section, we first review the Aspect Sentiment Classification (ASC) task, and then introduce the related studies on graph-based neural networks.
Aspect Sentiment Classification. The ASC task aims to predict the sentiment polarity for each aspect discussed inside a sentence. Existing studies mainly focus on utilizing various approaches (e.g., attention mechanism and memory network) to align each aspect and the sentence for learning aspect-related sentence representation. Wang et al. (2016) propose an attention-based LSTM in order to explore the potential correlation of aspects and sentiment polarities in ASC. Wang et al. (2018) propose a hierarchical attention network to incorporate both words and clauses information for ASC. He et al. (2018a) propose an attention-based approach to incorporate the aspect-related syntactic information for ASC. Tang et al. (2016b) and Chen et al. (2017) design deep memory networks to align the aspect and sentence for ASC. Lin et al. (2019) propose a semantic and context-aware memory network to integrate aspect-related semantic parsing information for performing ASC. Wang et al. (2019a) and Wang et al. (2019b) leverage reinforcement learning grounded approaches to select aspect-relevant words for ASC. Recently, a few studies have recognized the information deficiency problem in ASC and attempted to using external information to improve the performance of ASC. He et al. (2018b) and Chen and Qian (2019) incorporate the knowledge from document-level sentiment classification to improve the performance of ASC. Ma et al. (2018) propose an extension of LSTM to integrate the commonsense knowledge into the recurrent encoder for improving the performance of ASC. In addition, it is worthwhile to note that Hazarika et al. (2018) also investigate the inter-aspect sentiment dependency for ASC, but is limited to capture this information inside a single sentence.
In summary, all the above studies ignore the document-level sentiment preference information, which can be leveraged to effectively mitigate the information deficiency problem in ASC.
Graph-based Neural Networks. In recent years, graph-based neural networks have received more and more attentions. As a pioneer, Kipf and Welling (2017) present a simplified graph neural network model, called graph convolutional networks (GCN), which has been exported to several tasks such as scene recognition (Yuan et al., 2019),
semi-supervised node classification (Zhang et al., 2019b), text-to-SQL parsing (Bogin et al., 2019) and relation extraction (Sahu et al., 2019). On this basis, some other improved Graph-based Neural Networks are proposed. Morris et al. (2019) propose a generalization of Graph-based Neural Networks, so-called k-dimensional GNNs (k-GNNs), which can take higher-order graph structures at multiple scales into account. Cao et al. (2019) propose a novel Multi-channel Graph Neural Network model to learn alignment-oriented knowledge graph embeddings by robustly encoding two knowledge graphs via multiple channels. More recently, there exist several studies also adopting graph-based neural networks to ASC. For instance, Hou et al. (2019) and Zhang et al. (2019a) build GCN over the dependency tree of a sentence to exploit syntactical information and word dependencies for learning better aspect-related sentence representation for ASC.
Different from all the above studies, this paper proposes a novel Cooperative Graph Attention Networks approach to capture the document-level sentiment preference information in ASC. To our best knowledge, this is the first attempt to incorporate this information for the ASC task.
3 Cooperative Graph Attention Networks (CoGAN)
In this section, we formulate the Aspect Sentiment Classification (ASC) task as follows. In each document  $D$  with sentences  $\{s_1,s_2,\dots,s_I\}$ , given a sentence  $s_i,i\in \{1,2,\dots,I\}$  and its aspect  $a_{k},k\in \{1,2,\dots,K\}$ , the ASC task aims to predict the sentiment polarity  $\ell$  for aspect  $a_{k}$  by automatically learning the aspect-related sentence representation  $r_i$  of sentence  $s_i$ . Here,  $I$  is the number of sentence  $s_i$ , and  $K$  is the number of aspect  $a_{k}$  inside the document.
In this paper, we propose a Cooperative Graph Attention Networks (CoGAN) approach with two types of Graph Attention Networks (GAN) to incorporate the two-fold preference information respectively. Figure 2 shows the overall architecture of the CoGAN approach which consists of five major blocks: 1) Encoding Block; 2) Intra-Aspect Consistency Modeling Block; 3) Inter-Aspect Tendency Modeling Block; 4) Interaction Block. 5) Softmax
Decoding Block. Before introducing our CoGAN approach, we first give an overview of the basic Graph Attention Network (GAN).
3.1 Basic Graph Attention Network
Graph Attention Network (GAN) (Velickovic et al., 2017) is a new graph neural network architecture including attention mechanism, which enables specifying different attention weights to different vertices in a neighborhood. In principle, GAN can aggregate the features of neighboring nodes and also can propagate the information of a vertex to its nearest neighbors. From this regard, GAN is capable of sufficiently modeling local contextual information for learning the representation of each vertex. Formally, given a graph  $G(V,E)$  where  $V$  and  $E$  denote the vertices and edges respectively, GAN updates each new vertex vector  $\hat{h}_i$  of vertex  $v_i$  by considering neighboring vertices' vectors  $\{h_j\}_{j=1}^I$  with the following formulas:
$$
\hat {h} _ {i} = \tanh  \left(\sum_ {j = 1} ^ {I} \alpha_ {i j} W h _ {j} + b\right) \tag {1}
$$
$$
\alpha_ {i j} = \frac {\exp (f (w ^ {\top} [ \pmb {W} h _ {i} ; \pmb {W} h _ {j} ]))}{\sum_ {t = 1} ^ {I} \exp (f (w ^ {\top} [ \pmb {W} h _ {i} ; \pmb {W} h _ {t} ]))}
$$
where  $\alpha_{ij}$  is the attention weight (i.e., the edge weight) between vertex  $v_{i}$  and vertex  $v_{j}$ .  $f(\cdot)$  is a LeakyReLU activation function.  $[\cdot]$  denotes vector concatenation.  $\mathbf{W} \in \mathbb{R}^{d \times d}$  and  $w \in \mathbb{R}^{2d}$  are the trainable parameters.
In the following, we will illustrate the five main components of our CoGAN approach respectively.
3.2 Encoding Block
As a text encoding mechanism, BERT (Devlin et al., 2019) can be fine-tuned to create state-of-the-art models for a range of NLP tasks, e.g., text classification and natural language inference. In our approach, we use BERT-base $^{2}$  (uncased) model to encode both the aspect and the sentence as follows.
- Aspect Encoding. Since an aspect  $a_k$  consists of an entity  $e_{entity}$  and an attribute  $e_{attribute}$  (Pontiki et al., 2015), we process the entity-attribute pair  $(e_{entity}, e_{attribute})$  into the input pair format of BERT as:
$$
[ \mathrm {C L S} ] e _ {e n t i t y} [ \mathrm {S E P} ] e _ {a t t r i b u t e} [ \mathrm {S E P} ]
$$
Then, we feed the entity-attribute pair into BERT and regard the mark "[CLS]" representation as the aspect vector  $e_k \in \mathbb{R}^d$  of the aspect  $a_k$ .
<sup>1</sup>Like Pontiki et al. (2015), all aspects of every sentence are unrolled in a document. For instance, a sentence with two aspects occurs twice in succession, once with each aspect.
Figure 2: The overall framework of our proposed Cooperative Graph Attention Networks (CoGAN).
- Sentence Encoding. We borrow the approach proposed by Sun et al. (2019) to generate the aspect-related sentence representation, which has achieved promising performance for the ASC task. Following Sun et al. (2019), we first process the sentence  $s_i$  and its corresponding aspect  $a_k$  into the input pair format of BERT as:
$$
[ \mathrm {C L S} ] s _ {i} [ \mathrm {S E P} ] q u e s t i o n (a _ {k}) [ \mathrm {S E P} ]
$$
where  $question(\cdot)$  denotes the construction of auxiliary question sentence for aspect  $a_{k}$  proposed by Sun et al. (2019). For example, the auxiliary sentence for aspect  $FOOD\# PRICE$  is constructed as "what do you think of the food and price?" Then, we similarly feed the above pair into BERT (shared with aspect encoding) and obtain the aspect-related sentence vector  $v_{i} \in \mathbb{R}^{d}$  of the sentence  $s_{i}$ . Further, we fine-tune BERT and update both the aspect vector  $e_{k}$  and sentence vector  $v_{i}$  according to Eq.(8).
3.3 Intra-Aspect Consistency Modeling Block
In our approach, we propose a consistency-aware GAN to model the intra-aspect consistency. Given a document  $\mathcal{D}$  with sentences  $\{s_1,s_2,\dots ,s_I\}$ , the consistency-aware GAN is denoted as a bipartite graph  $G(S\bigcup A,E_{sa})$ . Here,  $S$  and  $A$  are two disjoint sets of vertices, denoting the sentence vertices and the aspect vertices respectively.  $E_{sa}$  is the set of the edge between the sentence  $s_i\in S$  and its corresponding aspect  $a_{k}\in A$  in the document  $\mathcal{D}$ .
On the basis, the intra-aspect consistency is formulated as that sentence vertices  $\{s_i\}_{i = 1}^{I'}$  sharing the same neighboring aspect vertex  $a_{k}$  and located in the same document tend to have the same sentiment for this aspect  $a_{k}$ . Here,  $I^{\prime}$  denotes the
number of sentences sharing the same aspect  $a_{k}$ . Nevertheless, there still possibly exist some sentiment inconsistency cases.
Considering all the scenarios above, we use the graph attention mechanism (Velickovic et al., 2017) to measure the preference-degree, where the attention weight (preference-degree) is computed as the edge weight between the sentence vertex  $s_i$  and the aspect vertex  $a_k$  in a document. Specifically, according to Eq.(1), the attention weight  $\alpha_{ik}$  between sentence  $s_i$  and aspect  $a_k$  is computed as follows:
$$
\alpha_ {i k} = \frac {\exp (f (w ^ {\top} [ \boldsymbol {W} _ {\boldsymbol {v}} v _ {i} ; \boldsymbol {W} _ {\boldsymbol {e}} e _ {k} ]))}{\sum_ {t = 1} ^ {I ^ {\prime}} \exp (f (w ^ {\top} [ \boldsymbol {W} _ {\boldsymbol {v}} v _ {t} ; \boldsymbol {W} _ {\boldsymbol {e}} e _ {k} ])))} \tag {2}
$$
where  $W_{v}, W_{e} \in \mathbb{R}^{d \times d}$  and  $w \in \mathbb{R}^{2d}$  are the trainable parameters.
As a vertex in  $G(S \bigcup A, E_{sa})$ , the sentence  $s_i$  is then encoded as the aspect-related sentence representation  $\hat{v}_i^{(intra)}$  according to the following proposed formula by modifying Eq.(1).
$$
\hat {v} _ {i} ^ {(i n t r a)} = \tanh  \left(\left(v _ {i} + \alpha_ {i k} \left(\sum_ {j = 1} ^ {I ^ {\prime}} \alpha_ {j k} W v _ {j}\right)\right) + b\right) \tag {3}
$$
where  $\sum_{j=1}^{I'} \alpha_{jk} W v_j$  is the vector representation of the aspect vertex  $a_k$ , which is weighted added to the sentence vector  $v_i$  for enhancing the aspect-related sentence representation.  $W \in \mathbb{R}^{d \times d}$ ,  $b \in \mathbb{R}^d$  are trainable parameters.
3.4 Inter-Aspect Tendency Modeling Block
In our approach, we leverage a tendency-aware GAN to model the inter-aspect tendency. Given a document  $\mathcal{D}$  with sentences  $\{s_1,s_2,\dots s_I\}$ , the
tendency-aware GAN is denoted as an undirected graph  $G(S, E_{ss})$ . Here,  $S$  is the set of sentence vertices.  $E_{ss}$  is the set of the edge between sentence  $s_i \in S$  and sentence  $s_j \in S$  in the document  $\mathcal{D}$ .
On the basis, the inter-aspect tendency assumption is formulated as that the sentence vertex  $s_i$  tends to have the same sentiment with the neighboring sentence vertices  $\{s_j\}_{j=1}^I$  inside a same document.
Similar to intra-aspect consistency modeling block, according to Eq.(1), the following formula is applied to compute the attention weight  $\alpha_{ij}$  between the sentence vertex  $s_i$  and the sentence vertex  $s_j$  from the same document:
$$
\alpha_ {i j} = \frac {\exp \left(f \left(w ^ {\top} \left[ \boldsymbol {W _ {1}} v _ {i} ; \boldsymbol {W _ {2}} v _ {j} \right]\right)\right)}{\sum_ {t = 1} ^ {I} \exp \left(f \left(w ^ {\top} \left[ \boldsymbol {W _ {1}} v _ {i}; \boldsymbol {W _ {2}} v _ {t}\right]\right)\right)} \tag {4}
$$
where  $W_{1}, W_{2} \in \mathbb{R}^{d \times d}$  and  $w \in \mathbb{R}^{2d}$  are the trainable parameters.
As a vertex in  $G(S,E_{ss})$ , according to Eq.(1), sentence  $s_i$  is encoded as the new sentence representation  $\hat{v}_i^{(inter)}$ , i.e.,
$$
\hat {v} _ {i} ^ {(i n t e r)} = \tanh (\sum_ {j = 1} ^ {I} \alpha_ {i j} W _ {\boldsymbol {\alpha}} v _ {j} + b _ {\boldsymbol {\alpha}}) \quad (5)
$$
where  $W_{\alpha}\in \mathbb{R}^{d\times d},b_{\alpha}\in \mathbb{R}^{d}$  are the parameters.
3.5 Interaction Block
Since the above two-fold preference can jointly affect the sentiment for aspect  $a_{k}$  in  $s_{i}$ , we make the two-fold preference pairwisely interact with each other for cooperatively boosting the performance. Especially, after obtaining the two sentence representations  $\hat{v}_{i}^{(inter)}$  and  $\hat{v}_{i}^{(intra)}$  of sentence  $s_{i}$  from the above two-fold preference modeling blocks, we propose an interactive mechanism to make an interaction between the two vectors instead of simply concatenating them. This is because a simple vector concatenation does not account for any interactions between the latent features of the two-fold preference, which is insufficient for cooperatively modeling the two-fold preference. In detail, this interactive mechanism leverages two strategies to learn the sentence representation.
- Pyramid Layers. As proposed in He et al. (2016), the model using a small number of hidden units for higher layers can learn more abstractive features. Inspired by this, we add pyramid hidden layers (see Figure 2) on the concatenated vector for interacting the latent features of the two-fold preference, where the bottom layer is the widest
and each successive layer has a smaller number of neurons. More specifically, the sentence vector  $\hat{v}_i^l\in \mathbb{R}^{2d\cdot (\frac{1}{2})^{l - 1}}$  of the  $l$  -th layer is defined as:
$$
\hat {v} _ {i} ^ {l} = \tanh  (\boldsymbol {W} ^ {l} \hat {v} _ {i} ^ {l - 1} + b ^ {l}) \tag {6}
$$
where  $\hat{v}_i^1 = [\hat{v}_i^{(inter)};\hat{v}_i^{(intra)}]$  and adding one layer will make the dimension of the sentence vector half.  $W^{l}\in \mathbb{R}^{2d\cdot (\frac{1}{2})^{l - 1}\times 2d\cdot (\frac{1}{2})^{l - 2}}$  and  $b^{l}$  are the trainable parameters.  $l\in [1,L]$  denotes the layer index.
- Adaptive Layer-Fusion. To sufficiently fuse the sentence representations at different levels of abstractions, an adaptive fusion mechanism is proposed to fuse the representations of all layers for computing the final sentence vector  $r_i \in \mathbb{R}^d$  of vertex  $s_i$  as follows:
$$
r _ {i} = \tanh  \left(\boldsymbol {W} _ {\boldsymbol {r}} \left(\prod_ {l = 1} ^ {L} \alpha_ {i} \hat {v} _ {i} ^ {l}\right) + b _ {r}\right) \tag {7}
$$
where  $\prod$  denotes the concatenation of multiple vectors.  $W_{r}$  and  $b_{r} \in \mathbb{R}^{d}$  are the trainable parameters.  $L$  is the number of added layers and set to be 4 fine-tuned according to the development data.  $\alpha = [\alpha_{1}, \dots, \alpha_{L}]$  is a normalized weights vector to weigh each layer, which is learned during training.
3.6 Softmax Decoding Block
After obtaining the final sentence vector  $r_i$  of sentence  $s_i$ , we feed it to a softmax classifier  $m = W r_i + b$ , where  $m \in \mathbb{R}^C$  is output vector;  $W$  and  $b$  are the trainable parameters.
Then, the probability of labeling sentence with sentiment polarity  $\ell \in [1,C]$  is computed by  $p_{\theta}(\ell |r_i) = \frac{\exp(m_\ell)}{\sum_{\eta = 1}^{C}\exp(m_\eta)}$  Finally, the label with the highest probability stands for the predicted sentiment polarity for the aspect  $a_{k}$
We use cross-entropy loss function to train our model end-to-end given a set of training data  $(s_i, a_k, y_i)$  from corpus  $\mathcal{C}$ , where  $s_i$  is the  $i$ -th sentence to be predicted,  $a_k$  is its corresponding aspect and  $y_i$  is the ground-truth sentiment polarity for aspect  $a_k$ . The objective of learning  $\theta$  is to minimize the loss function as follows:
$$
J (\theta) = \mathbb {E} _ {(s _ {i}, a _ {k}, y _ {i}) \sim \mathcal {C}} [ - \log p _ {\theta} (y _ {i} | r _ {i}) ] + \frac {\delta}{2} | | \theta | | _ {2} ^ {2} \quad (8)
$$
where  $\mathbb{E}$  denotes the expectation-maximization.  $\sim$  denotes the sampling operation.  $\theta$  denotes all the trainable parameters of our CoGAN approach.  $\delta$  is a  $L_{2}$  regularization.
4 Experimentation
Data Settings. We conduct experiments on four datasets $^3$ , i.e., two datasets (restaurant15 and laptop15) released by SemEval-2015 Task 12 (Pontiki et al., 2015) and the other two datasets (restaurant16 and laptop16) released by SemEval-2016 Task 5 (Pontiki et al., 2016), to verify the effectiveness of our proposed approach. Wherein, each dataset averagely consists of about 442 documents and one document averagely contains 4.9 sentences. Moreover, each sentence is annotated with one or multiple aspects and a sentiment polarity (i.e., positive, negative or neutral) for each aspect. Additionally, we set aside  $10\%$  from the training set as the development data to tune the hyper-parameters.