Venue: ACL 2020

# Aspect Sentiment Classification with Document-level Sentiment Preference Modeling

Xiao Chen\({}^{1}\), Changlong Sun\({}^{2}\), Jingjing Wang\({}^{1,*}\), Shoushan Li\({}^{1}\),

**Luo Si\({}^{2}\), Min Zhang\({}^{1}\), Guodong Zhou\({}^{1}\)**

\({}^{1}\)School of Computer Science and Technology, Soochow University, China

\({}^{2}\)Alibaba Group, China

xchen1206@stu.suda.edu.cn, changlong.scl@taobao.com

{djingwang, lishoushan, minzhang, gdzhou}@suda.edu.cn

luo.si@alibaba-inc.com  Corresponding Author: Jingjing Wang.

###### Abstract

In the literature, existing studies always consider Aspect Sentiment Classification (ASC) as an independent sentence-level classification problem aspect by aspect, which largely ignore the document-level sentiment preference information, though obviously such information is crucial for alleviating the information deficiency problem in ASC. In this paper, we explore two kinds of sentiment preference information inside a document, i.e., contextual sentiment consistency w.r.t. the same aspect (namely intra-aspect sentiment consistency) and contextual sentiment tendency w.r.t. all the related aspects (namely inter-aspect sentiment tendency). On the basis, we propose a Cooperative Graph Attention Networks (CoGAN) approach for cooperatively learning the aspect-related sentence representation. Specifically, two graph attention networks are leveraged to model above two kinds of document-level sentiment preference information respectively, followed by an interactive mechanism to integrate the two-fold preference. Detailed evaluation demonstrates the great advantage of the proposed approach to ASC over the state-of-the-art baselines. This justifies the importance of the document-level sentiment preference information to ASC and the effectiveness of our approach capturing such information.

+
Footnote †: July 5 – 10, 2020 Association for Computational Linguistics

+
Footnote †: July 5 – 10, 2020 Association for Computational Linguistics

+
Footnote †: July 5– 10, 2020 Association for Computational Linguistics

+
Footnote †: July 5 – 10, 2020 Association for Computational Linguistics


remedy the information deficiency issue in ASC. Especially, we explore two kinds of sentiment preference information inside a document.

On one hand, we assume that the sentences in a document involving the same aspect tend to have the same sentiment polarity on this aspect. For instance, in **Document 1**, both the sentence **S1** and **S2** involve aspect _AMBIENCE#GENERAL_. Although it is difficult to infer the _negative_ sentiment for aspect _AMBIENCE#GENERAL_ through the clause _"without it trying to be that"_ in **S2**, we can infer that the sentiment of aspect _AMBIENCE#GENERAL_ is more likely to be _negative_ according to **S1**. This is because it is easier to infer _negative_ for aspect _AMBIENCE#GENERAL_ through the clause _"interior could use some help"_ in **S1**. Therefore, a well-behaved approach should capture the contextual sentiment consistency w.r.t. the same aspect (namely intra-aspect consistency for short) information.

On the other hand, we assume that the sentences in a document tend to have the same sentiment polarity on all the related aspects. For the example of **Document 2** where the sentence **S2** involves multiple aspects, it is really hard to precisely predict the sentiment polarity for each aspect. However, when taken the context into consideration, the sentiment polarity for each aspect in **S2** is largely possible to be _positive_, since all the neighboring sentences express the _positive_ sentiment polarity for their aspects. Therefore, a well-behaved model should capture the contextual sentiment tendency w.r.t. all the related aspects (namely inter-aspect tendency for short) information.

To well accommodate the above two kinds of document-level sentiment preference information, we propose a Cooperative Graph Attention Networks (CoGAN) approach to ASC. Specifically, two graph attention networks are constructed to model the two-fold sentiment preference with the attention weight to measure the preference-degree. Furthermore, considering that the two-fold preference can jointly influence the sentiment polarities for aspects, we propose an interactive mechanism to jointly model the two-fold preference for obtaining better aspect-related sentence representation. Detailed evaluation shows our proposed CoGAN approach significantly outperforms the state-of-the-art baselines, including the three top-performed systems from SemEval-2015 Task 12 and SemEval-2016 Task 5 [43, 8].

## 2 Related Work

In this section, we first review the Aspect Sentiment Classification (ASC) task, and then introduce the related studies on graph-based neural networks.

**Aspect Sentiment Classification.** The ASC task aims to predict the sentiment polarity for each aspect discussed inside a sentence. Existing studies mainly focus on utilizing various approaches (e.g., attention mechanism and memory network) to align each aspect and the sentence for learning aspect-related sentence representation. Wang et al.wang2016deep propose an attention-based LSTM in order to explore the potential correlation of aspects and sentiment polarities in ASC. Wang et al.wang2017deep propose a hierarchical attention network to incorporate both words and clauses information for ASC. He et al.he2018deep propose an attention-based approach to incorporate the aspect-related syntactic information for ASC. Tang et al.tang2016deep and Chen et al.chen2017deep design deep memory networks to align the aspect and sentence for ASC. Lin et al.lin2019deep propose a semantic and context-aware memory network to integrate aspect-related semantic parsing information for performing ASC. Wang et al.wang2019deep and Wang et al.wang2019deep leverage reinforcement learning grounded approaches to select aspect-relevant words for ASC. Recently, a few studies have recognized the information deficiency problem in ASC and attempted to using external information to improve the performance of ASC. He et al.hen2018deep and Chen and Qian2019deep incorporate the knowledge from document-level sentiment classification to improve the performance of ASC. Ma et al.ma2018deep propose an extension of LSTM to integrate the commonsense knowledge into the recurrent encoder for improving the performance of ASC. In addition, it is worthwhile to note that Hazarika et al.hazarika2018deep also investigate the inter-aspect sentiment dependency for ASC, but is limited to capture this information inside a single sentence.

In summary, all the above studies ignore the document-level sentiment preference information, which can be leveraged to effectively mitigate the information deficiency problem in ASC.

**Graph-based Neural Networks.** In recent years, graph-based neural networks have received more and more attentions. As a pioneer, Kipf and Wellinging2017deep present a simplified graph neural network model, called graph convolutional networks (GCN), which has been exported to several tasks such as scene recognition [29], 

semi-supervised node classification (Zhang et al., 2019), text-to-SQL parsing (Bogin et al., 2019) and relation extraction Sahu et al. (2019). On this basis, some other improved Graph-based Neural Networks are proposed. Morris et al. (2019) propose a generalization of Graph-based Neural Networks, so-called k-dimensional GNNs (k-GNNs), which can take higher-order graph structures at multiple scales into account. Cao et al. (2019) propose a novel Multi-channel Graph Neural Network model to learn alignment-oriented knowledge graph embeddings by robustly encoding two knowledge graphs via multiple channels. More recently, there exist several studies also adopting graph-based neural networks to ASC. For instance, Hou et al. (2019) and Zhang et al. (2019) build GCN over the dependency tree of a sentence to exploit syntactical information and word dependencies for learning better aspect-related sentence representation for ASC.

Different from all the above studies, this paper proposes a novel Cooperative Graph Attention Networks approach to capture the document-level sentiment preference information in ASC. To our best knowledge, this is the first attempt to incorporate this information for the ASC task.

## 3 Cooperative Graph Attention Networks (CoGAN)

In this section, we formulate the Aspect Sentiment Classification (ASC) task as follows. In each document \(D\) with sentences1\(\{s_{1},s_{2},...,s_{I}\}\), given a sentence \(s_{i},i\in\{1,2,...,I\}\) and its aspect \(a_{k},k\in\{1,2,...,K\}\), the ASC task aims to predict the sentiment polarity \(\ell\) for aspect \(a_{k}\) by automatically learning the aspect-related sentence representation \(r_{i}\) of sentence \(s_{i}\). Here, \(I\) is the number of sentence \(s_{i}\) , and \(K\) is the number of aspect \(a_{k}\) inside the document.

Footnote 1: Like Pontiki et al. (2015), all aspects of every sentence are unrolled in a document. For instance, a sentence with two aspects occurs twice in succession, once with each aspect.

In this paper, we propose a Cooperative Graph Attention Networks (CoGAN) approach with two types of Graph Attention Networks (GAN) to incorporate the two-fold preference information respectively. Figure 2 shows the overall architecture of the CoGAN approach which consists of five major blocks: **1)** Encoding Block; **2)** Intra-Aspect Consistency Modeling Block; **3)** Inter-Aspect Tendency Modeling Block; **4)** Interaction Block. **5)** Softmax Decoding Block. Before introducing our CoGAN approach, we first give an overview of the basic Graph Attention Network (GAN).

### Basic Graph Attention Network

Graph Attention Network (GAN) (Velickovic et al., 2017) is a new graph neural network architecture including attention mechanism, which enables specifying different attention weights to different vertices in a neighborhood. In principle, GAN can aggregate the features of neighboring nodes and also can propagate the information of a vertex to its nearest neighbors. From this regard, GAN is capable of sufficiently modeling local contextual information for learning the representation of each vertex. Formally, given a graph \(G(V,E)\) where \(V\) and \(E\) denote the vertices and edges respectively, GAN updates each new vertex vector \(\hat{h}_{i}\) of vertex \(v_{i}\) by considering neighboring vertices' vectors \(\{h_{j}\}^{I}_{j=1}\) with the following formulas:

\[\begin{split}&\hat{h}_{i}=\tanh(\sum_{j=1}^{I}\alpha_{ij}\mathbf{W} h_{j}+b)\\ &\alpha_{ij}=\frac{\exp(f(w^{\top}[\mathbf{W}h_{i};\mathbf{W}h_{j}]))}{ \sum_{t=1}^{I}\exp(f(w^{\top}[\mathbf{W}h_{i};\mathbf{W}h_{t}]))}\end{split} \tag{1}\]

where \(\alpha_{ij}\) is the attention weight (i.e., the edge weight) between vertex \(v_{i}\) and vertex \(v_{j}\). \(f(\cdot)\) is a LeakyReLU activation function. \([;]\) denotes vector concatenation. \(\mathbf{W}\in\mathbb{R}^{d\times d}\) and \(w\in\mathbb{R}^{2d}\) are the trainable parameters.

In the following, we will illustrate the five main components of our CoGAN approach respectively.

### Encoding Block

As a text encoding mechanism, BERT Devlin et al. (2019) can be fine-tuned to create state-of-the-art models for a range of NLP tasks, e.g., text classification and natural language inference. In our approach, we use BERT-base2 (uncased) model to encode both the aspect and the sentence as follows.

Footnote 2: [https://github.com/google-research/bert](https://github.com/google-research/bert)

\(\bullet\)**Aspect Encoding.** Since an aspect \(a_{k}\) consists of an entity \(e_{entity}\) and an attribute \(e_{attribute}\)Pontiski et al. (2015), we process the entity-attribute pair (\(e_{entity}\), \(e_{attribute}\)) into the input pair format of BERT as:

Then, we feed the entity-attribute pair into BERT and regard the mark "[CLS]" representation as the aspect vector \(e_{k}\in\mathbb{R}^{d}\) of the aspect \(a_{k}\).

 

\(\bullet\)**Sentence Encoding.** We borrow the approach proposed by Sun et al. (2019) to generate the aspect-related sentence representation, which has achieved promising performance for the ASC task. Following Sun et al. (2019), we first process the sentence \(s_{i}\) and its corresponding aspect \(a_{k}\) into the input pair format of BERT as:

\[\begin{array}{cc}\left[\mathrm{CLS}\right]&s_{i}~{}\left[\mathrm{SEP}\right] &question(a_{k})~{}\left[\mathrm{SEP}\right]\\ \end{array}\]

where \(question(\cdot)\) denotes the construction of auxiliary question sentence for aspect \(a_{k}\) proposed by Sun et al. (2019). For example, the auxiliary sentence for aspect _FOOD#PRICE_ is constructed as "_what do you think of the food and price?_". Then, we similarly feed the above pair into BERT (shared with aspect encoding) and obtain the aspect-related sentence vector \(v_{i}\in\mathbb{R}^{d}\) of the sentence \(s_{i}\). Further, we fine-tune BERT and update both the aspect vector \(e_{k}\) and sentence vector \(v_{i}\) according to Eq.(8).

### Intra-Aspect Consistency Modeling Block

In our approach, we propose a consistency-aware GAN to model the intra-aspect consistency. Given a document \(\mathcal{D}\) with sentences \(\{s_{1},s_{2},...,s_{I}\}\), the consistency-aware GAN is denoted as a bipartite graph \(G(S\bigcup A,E_{sa})\). Here, \(S\) and \(A\) are two disjoint sets of vertices, denoting the sentence vertices and the aspect vertices respectively. \(E_{sa}\) is the set of the edge between the sentence \(s_{i}\in S\) and its corresponding aspect \(a_{k}\in A\) in the document \(\mathcal{D}\).

On the basis, the intra-aspect consistency is formulated as that sentence vertices \(\{s_{i}\}_{i=1}^{I^{\prime}}\) sharing the same neighboring aspect vertex \(a_{k}\) and located in the same document tend to have the same sentiment for this aspect \(a_{k}\). Here, \(I^{\prime}\) denotes the number of sentences sharing the same aspect \(a_{k}\). Nevertheless, there still possibly exist some sentiment inconsistency cases.

Considering all the scenarios above, we use the graph attention mechanism (Velickovic et al., 2017) to measure the preference-degree, where the attention weight (preference-degree) is computed as the edge weight between the sentence vertex \(s_{i}\) and the aspect vertex \(a_{k}\) in a document. Specifically, according to Eq.(1), the attention weight \(\alpha_{ik}\) between sentence \(s_{i}\) and aspect \(a_{k}\) is computed as follows:

\[\alpha_{ik}=\frac{\exp(f(w^{\top}[\mathbf{W_{v}}v_{i};\mathbf{W_{e}}e_{k}]))}{\sum_{t= 1}^{I^{\prime}}\exp(f(w^{\top}[\mathbf{W_{v}}v_{t};\mathbf{W_{e}}e_{k}]))} \tag{2}\]

where \(\mathbf{W_{v}},\mathbf{W_{e}}\in\mathbb{R}^{d\times d}\) and \(w\in\mathbb{R}^{2d}\) are the trainable parameters.

As a vertex in \(G(S\bigcup A,E_{sa})\), the sentence \(s_{i}\) is then encoded as the aspect-related sentence representation \(\hat{v}_{i}^{(intra)}\) according to the following proposed formula by modifying Eq.(1).

\[\hat{v}_{i}^{(intra)}=\tanh((v_{i}+\alpha_{ik}(\sum_{j=1}^{I^{\prime}}\alpha_ {jk}\mathbf{W}v_{j}))+b) \tag{3}\]

where \(\sum_{j=1}^{I^{\prime}}\alpha_{jk}\mathbf{W}v_{j}\) is the vector representation of the aspect vertex \(a_{k}\), which is weighted added to the sentence vector \(v_{i}\) for enhancing the aspect-related sentence representation. \(\mathbf{W}\in\mathbb{R}^{d\times d}\), \(b\in\mathbb{R}^{d}\) are trainable parameters.

### Inter-Aspect Tendency Modeling Block

In our approach, we leverage a tendency-aware GAN to model the inter-aspect tendency. Given a document \(\mathcal{D}\) with sentences \(\{s_{1},s_{2},...s_{I}\}\), t

Figure 2: The overall framework of our proposed Cooperative Graph Attention Networks (CoGAN).

 

tendency-aware GAN is denoted as an undirected graph \(G(S,E_{ss})\). Here, \(S\) is the set of sentence vertices. \(E_{ss}\) is the set of the edge between sentence \(s_{i}\in S\) and sentence \(s_{j}\in S\) in the document \(\mathcal{D}\).

On the basis, the inter-aspect tendency assumption is formulated as that the sentence vertex \(s_{i}\) tend to have the same sentiment with the neighboring sentence vertices \(\{s_{j}\}^{I}_{j=1}\) inside a same document.

Similar to intra-aspect consistency modeling block, according to Eq.(1), the following formula is applied to compute the attention weight \(\alpha_{ij}\) between the sentence vertex \(s_{i}\) and the sentence vertex \(s_{j}\) from the same document:

\[\alpha_{ij}=\frac{\exp(f(w^{\top}|\mathbf{W_{1}v_{i};\mathbf{W_{2}v_{j}}}]))}{\sum_{l=1 }^{I}\exp(f(w^{\top}|\mathbf{W_{1}v_{i};\mathbf{W_{2}v_{l}}}]))} \tag{4}\]

where \(\mathbf{W_{1},W_{2}}\in\mathbb{R}^{d\times d}\) and \(w\in\mathbb{R}^{2d}\) are the trainable parameters.

As a vertex in \(G(S,E_{ss})\), according to Eq.(1), sentence \(s_{i}\) is encoded as the new sentence representation \(\hat{v}_{i}^{(inter)}\), i.e.,

\[\hat{v}_{i}^{(inter)}=\tanh(\sum_{j=1}^{I}\alpha_{ij}\mathbf{W_{\alpha}v_{j}+b_{ \alpha}}) \tag{5}\]

where \(\mathbf{W_{\alpha}}\in\mathbb{R}^{d\times d}\), \(b_{\alpha}\in\mathbb{R}^{d}\) are the parameters.

### Interaction Block

Since the above two-fold preference can jointly affect the sentiment for aspect \(a_{k}\) in \(s_{i}\), we make the two-fold preference pairwisely interact with each other for cooperatively boosting the performance. Especially, after obtaining the two sentence representations \(\hat{v}_{i}^{(inter)}\) and \(\hat{v}_{i}^{(intra)}\) of sentence \(s_{i}\) from the above two-fold preference modeling blocks, we propose an interactive mechanism to make an interaction between the two vectors instead of simply concatenating them. This is because a simple vector concatenation does not account for any interactions between the latent features of the two-fold preference, which is insufficient for cooperatively modeling the two-fold preference. In detail, this interactive mechanism leverages two strategies to learn the sentence representation.

\(\bullet\)**Pyramid Layers.** As proposed in He et al. (2016), the model using a small number of hidden units for higher layers can learn more abstractive features. Inspired by this, we add pyramid hidden layers (see Figure 2) on the concatenated vector for interacting the latent features of the two-fold preference, where the bottom layer is the widest and each successive layer has a smaller number of neurons. More specifically, the sentence vector \(\hat{v}_{i}^{l}\in\mathbb{R}^{2d\cdot(\frac{l}{2})^{l-1}}\) of the \(l\)-th layer is defined as:

\[\hat{v}_{i}^{l}=\tanh(\mathbf{W^{l}}\hat{v}_{i}^{l-1}+b^{l}) \tag{6}\]

where \(\hat{v}_{i}^{1}=[\hat{v}_{i}^{(inter)};\hat{v}_{i}^{(intra)}]\) and adding one layer will make the dimension of the sentence vector half. \(\mathbf{W^{l}}\in\mathbb{R}^{2d(\cdot(\frac{l}{2})^{l-1}\times 2d(\cdot(\frac{ l}{2})^{l-2}}\) and \(b^{l}\) are the trainable parameters. \(l\in[1,L]\) denotes the layer index.

\(\bullet\)**Adaptive Layer-Fusion.** To sufficiently fuse the sentence representations at different level of abstractions, an adaptive fusion mechanism is proposed to fuse the representations of all layers for computing the final sentence vector \(r_{i}\in\mathbb{R}^{d}\) of vertex \(s_{i}\) as follows:

\[r_{i}=\tanh(\mathbf{W_{r}}(\prod_{l=1}^{L}\alpha_{i}\hat{v}_{i}^{l})+b_{r}) \tag{7}\]

where \(\prod\) denotes the concatenation of multiple vectors. \(W_{r}\) and \(b_{r}\in\mathbb{R}^{d}\) are the trainable parameters. \(L\) is the number of added layers and set to be 4 fine-tuned according to the development data. \(\mathbf{\alpha}=[\alpha_{1},...,\alpha_{L}]\) is a normalized weights vector to weigh each layer, which is learned during training.

### Softmax Decoding Block

After obtaining the final sentence vector \(r_{i}\) of sentence \(s_{i}\), we feed it to a softmax classifier \(m=\mathbf{W}r_{i}+b\), where \(m\in\mathbb{R}^{C}\) is output vector; \(\mathbf{W}\) and \(b\) are the trainable parameters.

Then, the probability of labeling sentence with sentiment polarity \(\ell\in[1,C]\) is computed by \(p_{\theta}(\ell|r_{i})=\frac{\exp(m_{\ell})}{\sum_{y=1}^{C}\exp(m_{\eta})}\). Finally, the label with the highest probability stands for the predicted sentiment polarity for the aspect \(a_{k}\).

### Model Training

We use cross-entropy loss function to train our model end-to-end given a set of training data (\(s_{i},a_{k},y_{i}\)) from corpus \(\mathcal{C}\), where \(s_{i}\) is the \(i\)-th sentence to be predicted, \(a_{k}\) is its corresponding aspect and \(y_{i}\) is the ground-truth sentiment polarity for aspect \(a_{k}\). The objective of learning \(\theta\) is to minimize the loss function as follows:

\[J(\theta)=\mathbb{E}_{(s_{i},a_{k},y_{i})\sim\mathcal{C}}[-\log p_{\theta}(y_{ i}|r_{i})]+\frac{\delta}{2}||\theta||_{2}^{2} \tag{8}\]

where \(\mathbb{E}\) denotes the expectation-maximization. \(\sim\) denotes the sampling operation. \(\theta\) denotes all the trainable parameters of our CoGAN approach. \(\delta\) is a \(L_{2}\) regularization 



## 4 Experimentation

### Experimental Settings

**Data Settings.** We conduct experiments on four datasets3, i.e., two datasets (_restaurant15_ and _laptop15_) released by SemEval-2015 Task 12 Pontiki et al. (2015) and the other two datasets (_restaurant16_ and _laptop16_) released by SemEval-2016 Task 5 Pontiki et al. (2016), to verify the effectiveness of our proposed approach. Wherein, each dataset averagely consists of about 442 documents and one document averagely contains 4.9 sentences. Moreover, each sentence is annotated with one or multiple aspects and a sentiment polarity (i.e., _positive_, _negative_ or _neutral_) for each aspect. Additionally, we set aside 10% from the training set as the development data to tune the hyper-parameters.

Footnote 3: Detail statistics can be seen in Pontiki et al. (2015, 2016).

**Implementation Details.** In our experiments, all hyper-parameters are tuned according to the development set. Specifically, BERT is optimized by the Adam optimizer Kingma and Ba (2015), where \(\beta_{1}=0.9\) and the initial learning rate is \(10^{-4}\). Other parameters of BERT are following Devlin et al. (2019). For our CoGAN approach, we adopt another Adam optimizer with an initial learning rate of \(10^{-3}\) and \(\beta_{1}=0.95\) for cross-entropy training. The regularization weight of parameters is \(10^{-5}\). The dropout rate is 0.25. All matrix and vector parameters of the layers are initialized by the Glorot uniform Glorot and Bengio (2010).

**Evaluation Metrics.** The performance is evaluated using standard _Accuracy_ (Acc.) and _Macro-F1_ (F1) Wang et al. (2017). Moreover, \(t\)-test is used to evaluate the significance of the performance difference Yang and Liu (1999).

**Baselines.** We give the following baseline approaches for comparison in order to comprehensively evaluate the performance of our approach. **1) TC-LSTM.** This approach extends LSTM by considering the aspect information where a forward LSTM and a backward one towards the aspect are adopted Tang et al. (2016). **2) ATAE-LSTM.** This approach models the aspect-related context words via attention-based LSTM Wang et al. (2016). **3) RAM.** This approach captures importance of context words for a specific aspect with a deep memory network and the results of multiple attentions are non-linearly combined with a recurrent neural network Chen et al. (2017). **4) IAN.** This approach is an interactive learning approach, which models the contexts and aspects via LSTM and then interactively learns attentions in the contexts and aspects Ma et al. (2017). **5) Clause-Level ATT.** This approach employs hierarchical attention to incorporate the clause information for ASC Wang et al. (2018). **6) LSTM+synATT+TarRep.** This approach employs syntax-aware attention to learn aspect-related representation for ASC. This is a state-of-the-art approach proposed by He et al. (2018a). **7) BERT.** This approach transforms ASC from a single sentence classification task to a sentence pair classification task. In our implementation, we regard the pair of sentence and its aspect as the input pair of BERT-base model Devlin et al. (2018) for performing ASC. **8) CADMN.** This approach employs attention model to attend on relevant aspects for enhancing the aspect representation. This is a state-of-the-art approach proposed by Song et al. (2019). **9) IMN.** This approach is a multi-task learning approach, which employs a novel message passing mechanism to better exploit the correlation among the tasks related to ASC. This is a state-of-the-art approach proposed by He et al. (2019). **10) BERT-QA.** This approach is an extension of the above BERT baseline proposed by Sun et al. (2019). In this study, we adopt BERT-pair-QA-M in our implementation. This is another state-of-the-art approach for ASC. **11) Sentiue.** This is the best-performed system in SemEval-2015 Task 12 Saias (2015), which achieves the best accuracy scores in both the _laptop15_ and _restaurant15_ domains. **12) XRCE.** This is the best-performed system in SemEval-2016 Task 5 Pontiki et al. (2016), which achieves the best accuracy score in the _restaurant16_ domain. **13) IIT-TUDA.** This is also the best-performed system in SemEval-2016 Task 5 Pontiki et al. (2016), while achieving the best accuracy score in the _laptop16_ domain. **15) CoGAN w/o Intra-Aspect Consistency.** Our approach only modeling Inter-Aspect Tendency. **16) CoGAN w/o Inter-Aspect Tendency.** Our approach only modeling Intra-Aspect Consistency. **17) CoGAN w/o Interactive Mechanism.** Our approach only concatenating the two vectors \(\hat{v}_{i}^{(inter)}\) and \(\hat{v}_{i}^{(intra)}\) instead of using interaction block to integrate them.

### Experimental Results

Table 1 shows the performance comparison of different approaches. From the table, we can see that, all state-of-the-art approaches, such as **Clause



**Level ATT**, **CADMN** and **IMN**, perform better than **TC-LSTM**. This result demonstrates the effectiveness of using a proper attention mechanism to learn the aspect-related sentence representation for performing the ASC task.

The BERT-based approaches, i.e., **BERT** and **BERT-QA**, perform better than the above approaches on almost all datasets. This result encourages to utilize the pre-trained BERT model as the aspect and sentence encoder for the ASC task.

Furthermore, our approach **CoGAN w/o Intra-Aspect Consistency** and **CoGAN w/o Inter-Aspect Tendency** outperform most of the above state-of-the-art approaches. This encourages to model the intra-aspect consistency or inter-aspect tendency information for the ASC task.

In comparison, when incorporating both the two-fold sentiment preference information, our approach **CoGAN** outperforms all the above baseline approaches and even significantly outperforms (\(p\)-value \(<\) 0.05) all three top-performed systems from SemEval-2015 Task 12 and SemEval-2016 Task 5, i.e., **Sentiue**, **XRCE** and **IIT-TUDA** on all four datasets. Impressively, compared to **TC-LSTM**, our approach achieves the average improvement of 11.6% (_Accuracy_), 14.3%(_MacroF1_) on the two _restaurant_ datasets and 9.1% (_Accuracy_), 12.6%(_Macro-F1_) on the two _laptop_ datasets. Significance test shows that these improvements are all significant (\(p\)-value \(<\) 0.01). These results highlight the importance of incorporating both the intra-aspect consistency and inter-aspect tendency information in a document for the ASC task.

In addition, it is worthwhile to note that **CoGAN** outperforms **CoGAN w/o Interactive Mechanism**, which encourages to employ the proposed interactive mechanism to cooperatively integrate the two-fold sentiment preference information.

## 5 Analysis and Discussion

### Case Study

We provide a qualitative analysis of our **CoGAN** approach on the test sets of the _restaurant16_ and _laptop16_ datasets respectively. Figure 3 shows two documents, along with their predicted sentiment for each aspect, and probabilities of the groundtruth label by different approaches. From this figure, we can see that: **1)** For the example of Document 1, it is difficult to infer the sentiment for aspect _LAPTOP#MISCELLANEOUS_ (to classify) in **S1** since the long sentence **S1** involves syntactic complications. Despite this, **S8** expresses explicit _negative_ polarity for the same aspect _LAPTOP#MISCELLANEOUS_. Considering this intra-aspect consistency information, our **CoGAN** approach can still give the correct _negative_ for aspect _LAPTOP#MISCELLANEOUS_, while both **BERT

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c} \hline \hline \multirow{2}{*}{**Approaches**} & \multicolumn{2}{c|}{_Restaurant15_} & \multicolumn{2}{c|}{_Laptop15_} & \multicolumn{2}{c|}{_Restaurant16_} & \multicolumn{2}{c}{_laptop16_} \\  & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1 \\ \hline TC-LSTM Tang et al. (2016) & 0.747\({}^{\dagger}\) & 0.634\({}^{\dagger}\) & 0.745\({}^{\dagger}\) & 0.622\({}^{\dagger}\) & 0.813 & 0.629 & 0.766 & 0.578 \\ ATAE-LSTM Wang et al. (2016) & 0.752\({}^{\dagger}\) & 0.641\({}^{\dagger}\) & 0.747\({}^{\dagger}\) & 0.637\({}^{\dagger}\) & 0.821 & 0.644 & 0.781 & 0.591 \\ RAM Chen et al. (2017) & 0.767\({}^{\dagger}\) & 0.645\({}^{\dagger}\) & 0.759\({}^{\dagger}\) & 0.639\({}^{\dagger}\) & 0.839 & 0.661 & 0.802 & 0.627 \\ IAN Ma et al. (2017) & 0.755\({}^{\dagger}\) & 0.639\({}^{\dagger}\) & 0.753\({}^{\dagger}\) & 0.625\({}^{\dagger}\) & 0.836 & 0.652 & 0.794 & 0.622 \\ Clause-Level ATT Wang et al. (2018) & 0.809\({}^{\dagger}\) & 0.685\({}^{\dagger}\) & 0.816\({}^{\dagger}\) & 0.667\({}^{\dagger}\) & 0.841 & 0.667 & 0.809 & 0.634 \\ LSTM+synATT+MarRep He et al. (2018a) & 0.817\({}^{\dagger}\) & 0.661\({}^{\dagger}\) & 0.822 & 0.649 & 0.846\({}^{\ddagger}\) & 0.675\({}^{\dagger}\) & 0.813 & 0.628 \\ BERT Devlin et al. (2018) & 0.811 & 0.647 & 0.809 & 0.683 & 0.884 & 0.729 & 0.811 & 0.670 \\ CADMN Song et al. (2019) & - & - & - & - & 0.879\({}^{\ddagger}\) & 0.700\({}^{\dagger}\) & - & - \\ IMN He et al. (2019) & 0.856\({}^{\ddagger}\) & 0.718\({}^{\ddagger}\) & 0.831 & 0.654 & 0.892 & 0.710 & 0.802 & 0.623 \\ BERT-QA Sun et al. (2019) & 0.824 & 0.650 & 0.827 & 0.595 & 0.896 & 0.715 & 0.812 & 0.596 \\ \hline Sentiue Gasias (2015) & 0.787\({}^{\dagger}\) & 0.660\({}^{\dagger}\) & 0.793\({}^{\dagger}\) & 0.634\({}^{\dagger}\) & - & - & - & - \\ XCRE Brun et al. (2016) & - & - & - & - & 0.881\({}^{\ast}\) & - & - & - \\ IIT-TUDA Kumar et al. (2016) & - & - & - & - & - & 0.828\({}^{\ddagger}\) & - \\ \hline CoGAN w/o Intra-Aspect Consistency & 0.857 & 0.707 & 0.846 & 0.722 & 0.907 & 0.769 & 0.839 & 0.706 \\ CoGAN w/o Inter-Aspect Tendency & 0.854 & 0.716 & 0.841 & 0.708 & 0.915 & 0.770 & 0.811 & 0.676 \\ CoGAN w/o Interactive Mechanism & 0.864 & 0.704 & 0.839 & 0.698 & 0.908 & 0.788 & 0.839 & 0.700 \\
* denotes both the results and codes are not reported by these papers.


and **IMN** give wrong predictions. This justifies the effectiveness of the intra-aspect consistency information for ASC. **2)** For the example of Document 2, it is rather difficult to infer the sentiment for aspect _RESTAURANT#GENERAL_ (to classify) in **S1**, since the sentence **S1** "_hidden little jam._" is too short and can not provide sufficient information to predict the _positive_ polarity for aspect _RESTAURANT#GENERAL_. Despite this, **CoGAN** considering the inter-aspect tendency information can still give the _positive_ for aspect _RESTAURANT#GENERAL_. This is reasonable because take the whole context into consideration, this restaurant has good reputations due to its service and food.

### Effectiveness Study

To better illustrate the effectiveness of modeling the intra-aspect consistency and inter-aspect tendency information, we systematically investigate both sentiment preference phenomena in all the four evaluation datasets respectively. Specifically, we sample 200 sentence4 pairs inside each dataset and calculate the ratio that the two sentences in the pair have the same sentiment for their corresponding aspects. Especially, we propose three sampling strategies as follows. **1)** Randomly Sampling: randomly selecting sentence pairs inside each dataset. **2)** Inter-Aspect Tendency Sampling: randomly selecting the sentence pairs under the premise that each two sentences should be located in the same document. **3)** Intra-Aspect Consistency Sampling: randomly selecting the sentence pairs under the premise that each two sentences should be located in the same document and should have the same aspect. Figure 4 shows the statistical results of the three sampling strategies on all four datasets. From this figure, we can see that **Inter-Aspect Tendency Sampling** and **Intra-Aspect Consistency Sampling** impressively outperform **Randomly Sampling** by 27.9 % and 34% respectively. Moreover, the average ratio of the two highest sampling strategies is up to 84.1%. This is the reason for the effectiveness of our CoGAN approach to ASC, and encourages to leverage CoGAN for incorporating the two-fold sentiment preference information.

Footnote 4: Sentences are repeated in a document according to the unrolled aspects. For instance, a sentence with two aspects will be repeated twice, each sentence with only one aspect.

### Error Analysis

We randomly analyzed 100 error cases and roughly categorized them into 5 classes briefly introduced as follows. (1) 29% of errors are due to the occurrence of negation words, e.g., "_Nothing really came across as outstanding._". CoGAN incorrectly predicts positive polarity, inspiring us to optimize CoGAN for capturing negation scope better. (2) 27% are due to incorrectly recognizing neutral instances. The shortage of neutral training examples makes it hard to recognize neutral instances, inspiring us to use data augmentation to enlarge the scale of neutral data. (3) 24% are due to the implicit sentiment expression, e.g., "_There is definitely more to

Figure 4: Ratios that two sentences have an identical sentiment polarity for their corresponding aspects.

Figure 3: Examples from the test data with their polarities predicted by different approaches (i.e., BERT, IMN and our approach). ✓ (or ✗) denotes that the predicted sentiment polarity is correct (or wrong).

 

_say..."_. CoGAN incorrectly predicts positive polarity instead of negative. (4) 12% are due to too short sentences (e.g. with less than 5 words), inspiring us to incorporate external ConceptNet knowledgebase to enhance the semantic representation. (5) 8% are due to comparative opinions, e.g., "_I've had better frozen pizza_". CoGAN incorrectly predicts positive, inspiring us to investigate whether incorporating syntactic information can remedy this issue.

## 6 Conclusion

In this paper, we propose a novel Cooperative Graph Attention Networks (CoGAN) approach to Aspect Sentiment Classification (ASC). The main idea of the proposed approach is to incorporate two kinds of sentiment preference information (i.e., the intra-aspect consistency and inter-aspect tendency) in a document for remedying the information deficiency problem in ASC. Experimental results on four datasets from SemEval-2015 and 2016 demonstrate that our approach significantly outperforms a number of competitive baselines, including all the three best-performed systems in the shared tasks of both SemEval-2015 and 2016.

In our future work, we would like to improve the performance of the ASC task by using unlabeled data since our graph-based neural network approach is easy to add unlabeled data. Moreover, we would like to apply our approach to other sentiment analysis tasks, e.g., aspect-oriented opinion summarization and multi-label emotion detection.

## Acknowledgments

We thank our anonymous reviewers for their helpful comments. This work was supported by three NSFC grants, i.e., No.61751206, No.61672366 and No.61525205. This work was also supported by a Project funded by China Postdoctoral Science Foundation No.2019M661930, a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD), and a joint research project of Alibaba Group and Soochow University.

## References

* B. Bogin, J. Berant, and M. Gardner (2019)Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of ACL-2019, pp. 4560-4565. Cited by: SS1.
* C. Brun, J. Perez, and C. Roux (2016)XRCE at semeval-2016 task 5: feedback ensemble modeling on syntactico-semantic knowledge for aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pp. 277-281. Cited by: SS1.
* Y. Cao, Z. Liu, C. Li, Z. Liu, J. Li, and T. Chua (2019)Multi-channel graph neural network for entity alignment. In Proceedings of ACL-2019, pp. 1452-1461. Cited by: SS1.
* P. Chen, Z. Sun, L. Bing, and W. Yang (2017)Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of EMNLP-2017, pp. 452-461. Cited by: SS1.
* Z. Chen and T. Qian (2019)Transfer capsule network for aspect level sentiment classification. In Proceedings of ACL-2019, pp. 547-556. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018)BERT: pre-training of deep bidirectional transformers for language understanding. CoRRabs/1810.04805. External Links: Link, 1810.04805 Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT-2019, pp. 4171-4186. Cited by: SS1.
* X. Glorot and Y. Bengio (2010)Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS-2010, pp. 249-256. Cited by: SS1.
* D. Hazarika, S. Poria, P. Vij, G. Krishnamurthy, E. Cambria, and R. Zimmermann (2018)Modeling inter-aspect dependencies for aspect-based sentiment analysis. In Proceedings of NAACL-2018, pp. 266-270. Cited by: SS1.
* K. He, X. Zhang, S. Ren, and J. Sun (2016)Deep residual learning for image recognition. In Proceedings of CVPR-2016, pp. 770-778. Cited by: SS1.
* R. He, W. S. Lee, H. T. Ng, and D. Dahlmeier (2018)Effective attention modeling for aspect-level sentiment classification. In Proceedings of COLING-2018, pp. 1121-1131. Cited by: SS1.
* R. He, W. S. Lee, H. T. Ng, and D. Dahlmeier (2018)Exploiting document knowledge for aspect-level sentiment classification. In Proceedings of ACL-2018, pp. 579-585. Cited by: SS1.
* R. He, W. S. Lee, H. T. Ng, and D. Dahlmeier (2019)An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of ACL-2019 Volume 1: Long Papers, pp. 504-515. Cited by: SS1.
* X. Hou, J. Huang, G. Wang, K. Huang, X. He, and B. Zhou (2019)Selective attention based graph convolutional networks for aspect-level sentiment classification. CoRRabs/1910.10857.

 

Ranzhe Jing, Yang Yu, and Zhangxi Lin. 2015. How service-related factors affect the survival of B2T providers: A sentiment analysis approach. J. Org. Computing and E. Commerce_, 25(3):316-336.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR-2015.
* Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR-2017.
* Kumar et al. (2016) Ayush Kumar, Sarah Kohail, Amit Kumar, Asif Ekbal, and Chris Biemann. 2016. IIT-TUDA at semeval-2016 task 5: Beyond sentiment lexicon: Combining domain dependency and distributional semantics features for aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pp. 1129-1135.
* Li et al. (2010) S. Li, C. Huang, G. Zhou, and Sophia Yat Mei Lee. 2010. Employing personal/impersonal views in supervised and semi-supervised sentiment classification. In Proceedings of ACL.
* Lin et al. (2019) Peiqin Lin, Meng Yang, and Jianhuang Lai. 2019. Deep mask memory network with semantic dependency and context moment for aspect level sentiment classification. In Proceedings of IJCAI-2019, pages 5088-5094.
* Ma et al. (2017) Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of IJCAI-2017, pages 4068-4074.
* Ma et al. (2018) Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In Proceedings of AAAI-2018, pp. 5876-5883.
* Morris et al. (2019) Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. 2019. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of AAAI-2019, pages 4602-4609.
* Pang and Lee (2007) B. Pang and L. Lee. 2007. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2).
* Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia V. Loukachevitch, Evgeniy V. Kotelnikov, Noria Bel, Salud Maria Jimenez Zafra, and Gulsen Erygit. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pages 19-30.
* Pontik et al. (2015) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2015, pages 486-495.
* Sahu et al. (2019) Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In Proceedings of ACL-2019, pages 4309-4316.
* Saas (2015) Jose Saas. 2015. Sentique: Target and aspect based sentiment analysis in semeval-2015 task 12. In Proceedings of SemEval@NAACL-HLT-2015, pages 767-771.
* Song et al. (2019) Kaisong Song, Wei Gao, Lujun Zhao, Jun Lin, Changlong Sun, and Xiaozhong Liu. 2019. Cold-start aware deep memory network for multi-entity aspect-based sentiment analysis. In Proceedings of IJCAI-2019, pages 5197-5203.
* Sun et al. (2019) Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of NAACL-HLT-2019, pages 380-385.
* Tang et al. (2016) Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective lstms for target-dependent sentiment classification. In Proceedings of COLING-2016, pages 3298-3307.
* Tang et al. (2016b) Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. In Proceedings of EMNLP-2016, pages 214-224.
* Velickovic et al. (2017) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. CoRRabs/1710.10903.
* Wang et al. (2018) Jingjing Wang, Jie Li, Shoushan Li, Yangyang Kang, Min Zhang, Luo Si, and Guodong Zhou. 2018. Aspect sentiment classification with both word-level and clause-level attention networks. In Proceedings of IJCAI-2018, pages 4439-4445.
* Wang et al. (2017) Jingjing Wang, Shoushan Li, and Guodong Zhou. 2017. Joint learning on relevant user attributes in microblog. In Proceedings of IJCAI, pages 4130-4136.
* Wang et al. (2019a) Jingjing Wang, Changlong Sun, Shoushan Li, Xiaozhong Liu, Luo Si, Min Zhang, and Guodong Zhou. 2019a. Aspect sentiment classification towards question-answering with reinforced bidirectional attention network. In Proceedings of ACL-2019, pages 3548-3557.
* Wang et al. (2019b) Jingjing Wang, Changlong Sun, Shoushan Li, Jiancheng Wang, Luo Si, Min Zhang, Xiaozhong Liu, and Guodong Zhou. 2019b. Human-like decision making: Document-level aspect sentiment classification via hierarchical reinforcement learning. In Proceedings of EMNLP-2019, pages 5580-5589 

Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In _Proceedings of EMNLP-2016_, pages 606-615.
* Wang et al. (2019c) Yunjie Wang, Hui Li, and Chen Lin. 2019c. Modeling sentiment evolution for social incidents. In _Proceedings of CIKM-2019_, pages 2413-2416.
* Yang and Liu (1999) Yiming Yang and Xin Liu. 1999. A re-examination of text categorization methods. In _Proceedings of SIGIR-1999_, pages 42-49.
* Yuan et al. (2019) Yuan Yuan, Zhitong Xiong, and Qi Wang. 2019. ACM: adaptive cross-modal graph convolutional neural networks for RGB-D scene recognition. In _Proceedings of AAAI-2019_, pages 9176-9184.
* Zhang et al. (2019a) Chen Zhang, Qiuchi Li, and Dawei Song. 2019a. Aspect-based sentiment classification with aspect-specific graph convolutional networks. _CoRR_, abs/1909.03477.
* Zhang et al. (2019b) Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay. 2019b. Bayesian graph convolutional neural networks for semi-supervised classification. In _Proceedings of AAAI-2019_, pages 5829-5836.