Venue: NAACL 2018

# Modeling Inter-Aspect Dependencies for Aspect-Based Sentiment Analysis

Devamanyu Hazarika\({}^{*}\)

School of Computing,

National University of Singapore

devamanyu@comp.nus.edu.sg &Soujanya Poria\({}^{*}\)

Artificial Intelligence Initiative,

A*STAR, Singapore

spora@ihpc.a-star.edu.sg &Prateek Vij

Department of Computer Science

and Engineering, IIT, Guwahati

v.prateek@iitg.ernet.in Gangeshwar Krishnamurthy Artificial Intelligence Initiative,

A*STAR, Singapore

gangeshwark@ihpc.a-star.edu.sg &Erik Cambria School of Computer Science and Engineering, NTU, Singapore

cambria@ntu.edu.sg &Roger Zimmermann School of Computing,

National University of Singapore

rogerz@comp.nus.edu.sg

###### Abstract

Aspect-based Sentiment Analysis is a fine-grained task of sentiment classification for multiple aspects in a sentence. Present neural-based models exploit aspect and its contextual information in the sentence but largely ignore the inter-aspect dependencies. In this paper, we incorporate this pattern by simultaneous classification of all aspects in a sentence along with temporal dependency processing of their corresponding sentence representations using recurrent networks. Results on the benchmark SemEval 2014 dataset suggest the effectiveness of our proposed approach.

## 1 Introduction

Aspect-based Sentiment Analysis (ABSA) is a fine-grained task of sentiment classification. Sentimentally involved sentences in reviews, debates, etc., often comprise of multiple aspects that have varied sentiment polarities. An important sub-task of ABSA is aspect or aspect-term classification which involves predicting sentiment of aspects embodied in a sentence Young et al. (2017). Present works in the literature approach this task by analyzing associations between aspects and their contexts provided in the sentence. In this work, we argue that to classify an aspect into sentiment categories, knowledge of surrounding aspects, their sentiment orientation, and resulting inter-dependencies, is beneficial.

Inter-aspect dependencies abound in sentences with multiple aspects. Largely ignored in present literature, these dependencies may reveal themselves in many forms, such as a) _Incomplete information_, where a certain aspect does not contain enough contextual information to convey the sentiment. In such cases, the surrounding aspects and their sentiment tone become crucial to fill the contextual gap. As an example, in the sentence _The menu is very limited - I think we counted 4 or 5 entries._, the subsentence _I think ... entries_ containing aspect _entries_ does not provide the required sentiment unless considered with the aspect _menu_. Here, the negative sentiment of _menu_ induces _entries_ to have the same sentiment. b) _Sentiment influence in conjunctions_, in which, the sentiment of an aspect in a sentence influences the succeeding aspects due to the presence of conjunctions. In particular, for sentences containing conjunctions like _and, not only, also, but, however, though,_ etc., aspects tend to share/contrast their sentiments. In the sentence _Food is usually very good, though I wonder about freshness of raw vegetables_, the aspect _raw vegetables_ does not have any sentiment marker linked to it. However, the positive sentiment of _food_ due to the word _good_ and presence of conjunction _though_ determines the sentiment of _raw vegetables_ to be negative. Thus, aspects when arranged as a sequence, reveal high correlation and interplay of sentiments.

In this paper, we facilitate such phenomena by proposing a neural network where the information is shared among the aspects by means of a Long Short-Term Memory (LSTM) network Hochreiter and Schmidhuber (1997). In other words, we model the sequential relationship between the aspects as per their occurrence in the sentence.



Specifically, our model first takes a sentence along with all of its aspect-terms and then generates the sentential representations relative to each aspect to get better aspect-oriented features [14]. This is done using an attention-based LSTM network, where the attention mechanism enables the model to focus on key parts of the sentence that modulate the sentiment of the aspects. To further guide the attention process the model incorporates aspect information at the word-level by concatenating aspect representations with each word [20]. Finally, to capture the inter-aspect dependencies, the aspect-based sentential representations are ordered as a sequence and temporally modeled using another LSTM. Each timestep of this LSTM corresponds to a particular aspect. The hidden state output for each timestep is then projected to a dense layer and fed to a softmax classifier to predict the polarities of the corresponding aspect. To the best of our knowledge, use of inter-aspect dependencies in neural models is unprecedented and fills a significant gap in the literature.

In the remaining paper, Section 2 first provides a summary of existing works; Section 3 then describes the proposed approach in detail; Section 4 gives training and dataset details followed by results and a qualitative case study. Finally, Section 5 concludes the paper.

## 2 Related Works

Traditional methods in this field leveraged sentiment lexicons to solve this task [9, 8] whereas present methods have transitioned to neural-based approaches. tang2016multi introduced the idea of aspect-based sentential representations which generates a custom representation of the sentence based on the aspect. This approach has been heavily adapted by modern works. wang2016multi built on this framework and introduced attention mechanism for generating these sentential features. They also incorporated aspect information into the attention module by concatenating them with the words. More recently, ma2017attention proposed a model where both context and aspect representations interact with each other's attention mechanism to generate the overall representation. tay2017attention proposed word-aspect associations using circular correlation as an improvement over wang2017attention's work. ABSA has also been approached from a question-answering perspective where memory networks have played a major role [14, 15]. Our work is different from all these works since we train all aspects of a particular sentence together and capitalize on inter-aspect dependency modeling which they ignore.

## 3 Proposed Approach

Let us take a sentence \(S=[w^{1},...,w^{n}]\) having \(n\) words. Each word is represented as a low-dimensional real-valued vector of size \(d_{em}\), called word embedding. To get the embeddings, we use the pre-trained Glove vectors [2] having \(d_{em}=300\). We can thus represent \(S\) as a matrix of dimensions \(\mathbb{R}^{d_{em}\times n}\).

The sentence \(S\) also contains \(m\) aspect-terms (or aspects), where for each \(i\in[1,m]\), aspect \(A_{i}\) is a multi-word subsequence of \(S\), i.e., \(\exists j\in[1,n]\), such that, \(A_{i}=[w^{j},...,w^{j+|A_{i}|-1}]\in\mathbb{R}^{d_{em}\times|A_{i}|}\). All the aspects \(A_{1},...,A_{m}\) are enumerated as per their order of occurrence in the sentence. The goal is to determine the sentiment label for each of these \(m\) aspects belonging to \(S\).

The proposed model comprises two distinct phases (Figure 1). The first phase involves the generation of aspect-based sentential representations \(s_{1},...,s_{m}\), where, vector \(s_{i}\) is created by coupling aspect \(A_{i}\) with sentence \(S\). The second phase models the inter-aspect dependencies in a sentence using an LSTM which is followed by the sentiment prediction for all the aspects.

### Phase 1: Aspect-based sentential representations

Below, we describe the methodology to generate the \(i^{th}\) aspect-based sentential representation \(s_{i}\) for aspect \(A_{i}\) and sentence \(S\).

Given sentence \(S\) and aspect-term \(A_{i}\), the model first generates the aspect representation \(t_{i}\). This is done by passing \(A_{i}\) through an LSTM, named _LSTM\({}_{a}\)_, having internal dimension \(d_{a}\). _LSTM\({}_{a}\)_'s final hidden state vector \(h_{a}^{|A_{i}|}\in\mathbb{R}^{d_{a}}\) is taken to be this representation, i.e., \(t_{i}=h_{a}^{|A_{i}|}\).

Following this, an attention-based LSTM model is used to create \(s_{i}\) using \(S\) and \(t_{i}\)[20]. First, each word vector \(w^{j}\) in \(S\) is concatenated with aspect \(t_{i}\) to create a comprehensive feature vector \(x_{i}^{j}=(w^{j};t_{i})\in\mathbb{R}^{(d_{em}+d_{a})}\), where \(;\) is the concatenation operator. We then take this new sequence representation \(X_{i}=[x_{i}^{1},...,x_{i}^{n}]\) and 

apply an LSTM, named \(LSTM_{s}\) with dimension \(d_{s}\), to model the long-term temporal dependencies within the sentence. The hidden state memory vectors across all \(n\) timesteps result in matrix \(H_{i}=[h^{1}_{s},...,h^{n}_{s}]\in\mathbb{R}^{d_{s}xn}\).

Attention:Attention mechanism is applied on \(H_{i}\) to get an attention vector \(\alpha\), which is in turn used to generate a weighted representation of \(H_{i}\). We use this weighted representation to be the \(i^{th}\) aspect's sentential representation \(s_{i}\). Previous concatenations of words with the aspect-representations infuse aspect information into the attention process. This enables the attention mechanism to focus on relevant segments in the sentence with respect to the aspect. The overall attention mechanism to generate \(s_{i}\) is summarized as:

\[M=tanh(H_{i}.W_{h}) \tag{1}\] \[\alpha=softmax(M^{T}.W_{b})\] (2) \[s_{i}=H.\alpha^{T} \tag{3}\]

where, \(W_{h}\in\mathbb{R}^{nx1}\) and \(W_{b}\in\mathbb{R}^{d_{s}xn}\) are projection parameters to be learnt during training and \(d_{s}\) is the dimension of the final sentence vector, i.e., \(s_{i}\in\mathbb{R}^{d_{s}}\).

The overall process described above is individually applied to all \(m\) aspects to get sentential representations \(s_{1},...,s_{m}\).

### Phase 2: Inter-aspect relationship

To capture the implicit inter-aspect dependencies, we model the sentential representations as a sequence \([s_{1},...,s_{m}]\), following the order of occurrence of their corresponding aspect-terms in sentence \(S\). An LSTM, named LSTM\({}_{ad}\) with dimension \(d_{ad}\) is then applied on this sequence and at each of the \(i^{th}\) timestep, its hidden state is projected to another vector having dimensions equal to the number of classes to predict. Finally, softmax operation is applied on this vector to get the prediction probabilities for the sentiment of this \(i^{th}\) aspect-term for sentence \(S\). The transitions are as follows:

\[[h_{adj_{1}},...,h_{adm_{m}}]=LSTM_{ad}([s_{1},...,s_{m}]) \tag{4}\] \[\hat{y_{i}}=softmax(W_{ad}.h_{adj_{i}}) \tag{5}\]

Here, \(\hat{y_{i}}\in\mathbb{R}^{C}\) is the predicted probability distribution for the \(i^{th}\) aspect of sentence \(S\) where \(C\) is the number of sentiment classes. \(W_{ad}\in\mathbb{R}^{C\times d_{ad}}\) is a parameter and \(softmax(x_{i})=e^{x_{i}}/\sum_{j}e^{x_{j}}\).

Loss Function:We use categorical cross-entropy as the loss function which is averaged over all aspects for a sentence. Thus, stochastic loss for sentence \(S\) is calculated as:

\[Loss=\frac{-1}{m}\underset{i=1}{\overset{m}{\sum}}\underset{j=1}{\overset{C}{ \sum}}y_{i,j}~{}log_{2}(\hat{y}_{i,j})+\lambda\|~{}\theta~{}\|^{2} \tag{6}\]

Here, \(m\) is the number of aspects for a sentence and \(C\) is the number of sentiment categories. \(y_{i}\) is the one-hot vector ground truth of \(i^{th}\) aspect of sentence \(S\) and \(\hat{y}_{i,j}\) is its predicted probability of belonging to sentiment class \(j\). \(\lambda\) is the L\({}_{2}\) - regularization term and \(\theta\) is the parameter set, i.e., \(\theta=\{W_{[l,b,ad]},LSTM_{[t,sa,dd]}\}\), where \(LSTM_{[]}\) represents the internal parameters of that LSTM.

## 4 Experimentation

Training details:To perform experiments and subsequent hyperparameter tuning, we first split

Figure 1: Overall architecture of the proposed method. The _aspect-based sentential representation generator_ is described in the right end of the figure.

 

the training set randomly in the ratio \(9:1\) to get a held-out validation set. For optimization, we use the Adam optimizer Kingma and Ba (2014) having learning rate \(0.01\). Embedding dimensions are set as follows, \(d_{a}=100,d_{s}\) and \(d_{ad}=300\). To facilitate batch processing, we attach dummy aspects in sentences with lesser aspects and also provide masking schemes. For termination, we use the early-stopping procedure with a patience value of \(10\) that is monitored on the validation loss.

Dataset:We conduct our experiments using the dataset for SemEval 2014 Task 4 containing customer reviews on restaurants and laptops. Each review has one or more aspects with their corresponding polarities. The polarity of an aspect can be positive, negative, neutral or conflict; however, we consider the first three labels for classification. Table 1 contains the statistics for the dataset.

### Results

Table 2 presents the results of our proposed model along with state-of-the-art methods. Our model significantly surpasses the performance of ATAE-LSTM Wang et al. (2016). Given that ATAE's architecture has a strong correlation to our aspect-based sentential generator (see Figure 1), their work can be categorized as a baseline to our model. This reinforces our hypothesis that a model capable of capturing inter-aspect dependencies indeed performs better. We also compare our model to the recently proposed IAN Ma et al. (2017). On both datasets, our model performs competitively with IAN and produces nominal improvement. Given that IAN explores the inter-dependencies of aspects with their contexts, while we try to model inter-dependencies between aspects, an interesting direction would be to explore the IAN modeled in our proposed setting (Phase 2 of Figure 1). We set this path as an option for future research.

Table 1 also presents variations of our proposed model. Specifically, we try out variants (a) _Without attention:_ in this setting, we omit the attention mechanism while generating aspect-based sentential representation \(s_{i}\) (Equation 1-3). Instead, we define \(s_{i}\) to be \(h^{n}_{s}\), i.e., the last hidden state vector of LSTM\({}_{s}\) with input \(S\) and \(A_{i}\). However, removing attention brings degradation in the performance of our model on the Restaurant and Laptop dataset by \(4\%\) and \(3\%\), respectively. This signifies the importance of an attention mechanism to derive the aspect-based sentential representations.

(b) _With hadamard fusion:_ instead of concatenation of \(w^{j}\) and \(t_{i}\), we use the hadamard product which is the element wise multiplication of the vectors. Although this variation reduces the total parameter sizes of the network, it still does not benefit the model and gives a poorer performance to simple concatenation. Numerous other fusion methods such as tensor fusion Zadeh et al. (2017), compact bilinear pooling Gao et al. (2016), attention-based fusion Poria et al. (2017); Hazarika et al. (2018), etc. are applicable, whose analyses, however, is not the focus of this paper.

### Case Study

A qualitative study on the test set classifications by our model reveals its capability to learn inter-aspect dependencies (Section 1). For the sentence _I love the keyboard and the screen_, the model correctly identifies the sentiment of _screen_ as positive which is hinted by positive aspect _keyboard_ and conjunction _and_. In another case, for the sentence _The best thing about this laptop is the price along with some of the newer features_, aspect _features_ is correctly classified as positive which is influenced by aspect _price_ and positive word _best_. This shows that our model is performing well in clas

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|} \hline \multirow{2}{*}{Data} & \multicolumn{5}{c|}{Aspect Labels} & No. of \\ \cline{2-5}  & Positive & Negative & Neutral & reviews \\ \hline \multirow{2}{*}{Rest.} & Train & 2148 & 790 & 628 & 1977 \\ \cline{2-5}  & Test & 725 & 195 & 196 & 600 \\ \hline \multirow{2}{*}{Laptop} & Train & 974 & 839 & 450 & 1462 \\ \cline{2-5}  & Test & 340 & 125 & 169 & 411 \\ \hline \end{tabular}

* Rest. = Restaurant

\end{table}
Table 1: Labels and review statistics for the dataset SemEval 2014.

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline \multirow{2}{*}{Models} & Attn. & Fusion & \multicolumn{2}{c|}{3-way classification} \\ \cline{3-5}  & & & Rest. & Laptop \\ \hline LSTM & ✗ & - & 74.3 & 66.5 \\ AE-LSTM & ✓ & Concat & 76.6 & 68.9 \\ ATAE-LSTM & ✓ & Concat & 77.2 & 68.7 \\ IAN & ✓ & - & 78.6 & 72.1 \\ \hline \hline Our Model & ✓ & Hadamard & 73.42 & 63.7 \\ Our Model & ✗ & Concat & 74.5 & 69.6 \\ Our Model & ✓ & Concat & **79.0** & **72.5** \\ \hline \end{tabular}

* Atn. = Attention, Rest. = Restaurant

\end{table}
Table 2: Accuracies for three-way classification on the Restaurant and Laptop SemEval 2014 dataset.

 

sifying joint aspects having conjunctions. For the slightly harder case of tackling incomplete information, our model fares well in sentences having this pattern. For example, one of the sentence _Boot up slowed significantly after all windows updates were installed_ has aspect _windows update_ which does not have a clear sentiment orientation but is implicitly dependent on the aspect _boot up_ having a negative sentiment. This was also correctly classified by our model. Moreover, the above examples were incorrectly classified by ATAE. This reaffirms our hypothesis that the ability to learn inter-aspect dependencies is a crucial factor in the task of ABSA.

## 5 Conclusion

In this paper, we present a way to incorporate inter-aspect dependencies in the task of Aspect-based Sentiment Analysis. Our results suggest that capturing such information indeed improves the task of prediction. Through this work, we hope that future attempts by researchers include this idea in their methods.

## 6 Acknowledgement

This research was supported in part by the National Natural Science Foundation of China under Grant no. 61472266 and by the National University of Singapore (Suzhou) Research Institute, 377 Lin Quan Street, Suzhou Industrial Park, Jiang Su, People's Republic of China, 215123. We would also like to thank the anonymous reviewers for their valuable feedback.

## References

* Gao et al. (2016) Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. 2016. Compact bilinear pooling. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 317-326.
* Hazarika et al. (2018) Devamanyu Hazarika, Sruthi Gorantla, Soujanya Poria, and Roger Zimmermann. 2018. Self-attentive feature-level fusion for multimodal emotion detection.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Li et al. (2017) Cheng Li, Xiaoxiao Guo, and Qiaozhu Mei. 2017. Deep memory networks for attitude identification. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_, pages 671-680. ACM.
* Ma et al. (2017) Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. _arXiv preprint arXiv:1709.00893_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Perez-Rosas et al. (2012) Veronica Perez-Rosas, Carmen Banea, and Rada Mihalcea. 2012. Learning sentiment lexicons in spanish. In _LREC_, volume 12, page 73.
* Poria et al. (2017) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Multi-level multiple attentions for contextual multimodal sentiment analysis. In _2017 IEEE International Conference on Data Mining (ICDM)_, pages 1033-1038. IEEE.
* Rao and Ravichandran (2009) Delip Rao and Deepak Ravichandran. 2009. Semi-supervised polarity lexicon induction. In _Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics_, pages 675-682. Association for Computational Linguistics.
* Tang et al. (2016a) Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a. Effective lstms for target-dependent sentiment classification. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 3298-3307.
* Tang et al. (2016b) Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. _arXiv preprint arXiv:1605.08900_.
* Tay et al. (2017) Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2017. Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. _arXiv preprint arXiv:1712.05403_.
* Wang et al. (2016) Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based lstm for aspect-level sentiment classification. In _EMNLP_, pages 606-615.
* Young et al. (2017) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2017. Recent trends in deep learning based natural language processing. _arXiv preprint arXiv:1708.02709_.
* Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1103-1114