Aspect-based Sentiment Classification via Reinforcement Learning

1 INTRODUCTION . The goal of aspect-based ( also known as aspect-level ) sentiment classification is to predict the sentiment polarities of individual aspects . As shown in Figure 1 , given a sentence “ I like this computer but do not like the screen ” , the sentiment of the aspect “ computer ” is positive because of “ like ” . Meanwhile , the sentiment of the aspect “ screen ” is negative for ” do not like ” . Aspect-based sentiment classification is challenging , where the core problem is to correctly align aspects with their sentiment descriptions . State-of-the-art methods rely on supervision signals to automatically learn such alignment . By leveraging textual context and word-level attention learned from deep models ( Vo & Zhang , 2015 ; Dong et al. , 2014 ; Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Xu et al. , 2015 ; Wang et al. , 2016 ; Tang et al. , 2016b ; Ma et al. , 2017a ; He et al. , 2018 ; Zhang et al. , 2018 ; 2019 ; Gao et al. , 2019 ; Tang et al. , 2020 ) , existing methods have made great progress on discovering aspect-specific sentimental statements . Meanwhile , the existing methods could suffer serious overfitting problems , as natural language inevitably includes a large proportion of task-irrelevant texts , or noise from the perspective of machine learning . Ideally , with a sufficient amount of training labels , the existing methods could effectively contain the negative impact of such task-irrelevant information . In practice , because of the high variance in language expression , it is costly to collect a large number of task-specific labels , and it is difficult to guarantee the expected label sufficiency . With limited labels , the existing approaches could easily include task-irrelevant information into decision processes , overfit training data , and end up with inferior generalization performance to unseen data . To effectively reduce the impact of task-irrelevant information , we propose SentRL , a reinforcement learning based framework for aspect-level sentiment classification . In our approach , input texts are firstly transformed into graph objects ( e.g. , dependency graphs ( Covington , 2001 ) ) , where nodes are words and edges indicate syntactic dependencies/relations between them . Next , we deploy a policy-based agent to discover aspect-related sentiment descriptions in the graphs . This agent is geared with a language understanding module so that it is able to update exploration states and make sentiment decisions for individual aspects . Unlike existing methods that aggregate potential sentiment information from all possible textual contexts or words , our agent strives to leverage the most relevant exploration paths under a limited budget . This strategy not only requires the agent to focus on the most effective paths but also encourages the agent to skip task-irrelevant regions . Using standard back-propagation methods , the policy network and the language understanding module are jointly trained . From public benchmark datasets , we observe our method could achieve up to 3.6 % improvement compared with competitive state-of-the-art methods . The main contributions of our work are listed below . • A novel reinforcement learning framework for aspect-based sentiment classification is proposed . It accurately pinpoints the most effective path between sentiment descriptions and the target aspects , and effectively avoids the impact of the task-irrelevant regions . • A policy network is developed to provide an agent with exploration guidance . This network iteratively provides suggestions on next-hop selection . In particular , the framework is permutation invariant and guarantees the consistency and reliability of the model . • A language understanding module is developed to help an agent “ remember ” its exploration history and make the final sentiment prediction . • Extensive experiments on representative benchmark datasets are evaluated . The results demonstrate the effectiveness , efficiency , and robustness of our approach . 2 RELATED WORK . 2.1 ASPECT-BASED SENTIMENT CLASSIFICATIONS . Aspect-based sentiment classification is to identify sentiment polarities of one or more aspects in given texts ( Thet et al. , 2010 ) . Aspects could be either substantial objects ( e.g. , computer and car ) or conceptional objects ( e.g. , service and atmosphere ) . There are usually three sentiment categories including positive , neutral , and negative , while more sophisticated categories could be explored . Conventional approaches ( Kiritchenko et al. , 2014 ) treat input texts as word sequences , and deploy separate feature extraction modules as well as classification modules . Deep learning-based sentiment analysis methods ( Tang et al. , 2016a ) take contextual information regarding the word order into consideration by using LSTM ( Hochreiter & Schmidhuber , 1997 ; Liu et al. , 2018 ) . Attentionbased approaches are proposed ( Tang et al. , 2016c ; Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Xu et al. , 2015 ; Wang et al. , 2016 ; Tang et al. , 2016b ; Ma et al. , 2017a ; Huang & Carley , 2019 ; Ma et al. , 2017b ; Huang et al. , 2018 ; Li et al. , 2018 ) to improve the effectiveness of contextual feature extraction . While such approaches utilize sequential models and attention mechanisms to learn features from word sequences , they could require a large amount of training labels to be well generalized for natural language expressions with non-trivial variance ( e.g. , long sentences with majority of irrelevant contextual words ) . 2.2 SENTIMENT ANALYSIS ON GRAPHS . Syntactic dependency tree ( Covington , 2001 ) is a widely adopted data structure that encodes syntactic dependencies between words in input texts . To this end , aspect-based sentiment classification problems can be cast into node classification problems on syntactic dependency graphs . Existing methods develop variants of graph convolutional neural networks ( Yao et al. , 2019 ; Linmei et al. , 2019 ; Zhang et al. , 2019 ; Ghosal et al. , 2019 ; Bai et al. , 2020 ; Wang et al. , 2019 ) to extract contextual features that represent aspect nodes . However , such methods still suffer the following issues . First , due to the complexity of natural language , these models require a large amount of training data to be well generalized . Second , such methods ( Kipf & Welling , 2016 ; Yao et al. , 2019 ; Linmei et al. , 2019 ) usually perform under transductive settings with limited application scopes . 2.3 REINFORCEMENT LEARNING IN NLP . In the domain of NLP , reinforcement learning ( RL ) is usually explored in interactive tasks ( Wang et al. , 2018 ) , such as text-based games ( Narasimhan et al. , 2015b ) . Recently , RL methods are also developed for complex tasks , such as relation extraction ( Narasimhan et al. , 2015a ; Qin et al. , 2018 ) , image captioning ( Pasunuru & Bansal , 2017 ) , popularity prediction ( He et al. , 2016 ; Zhou & Wang , 2018 ) , coreference resolution ( Yin et al. , 2018 ) , and reasoning in question answering ( Xiong et al. , 2017 ; Wu et al. , 2018 ) . Unlike previous works , we are the first to study RL methods that intelligently collect information from syntactic dependency graphs for aspect-based sentiment analysis . 3 OUR APPROACH . In this work , we propose a reinforcement learning-based framework , SentRL , for solving the challenges . The details are introduced below : 1 © NEC Corporation 2016 NEC Group Internal Use Only 3.1 PRELIMINARY . Given the target texts c = { wc1 , wc2 , wc3 , ... , wcn } , where wci represents the i-th word in c. n is the total number of the words . The target aspect of c is denoted as ac = { wcr+1 , ... , wcr+m } , where r + 1 indicates the start location of the aspect and m is the length of the aspect . The aspect is either a single-word format ( e.g. , “ computer ” , “ service ” , and “ screen ” ) or multiple-word format ( e.g. , “ HDMI port ” and “ sport mode ” ) . There could be one or more aspects in c , and different aspects could have different sentimental categories . The goal of aspect-based sentiment classification is to predict sentiment polarity ( i.e. , “ positive ” , “ neutral ” , and “ negative ” ) for each given aspect . 3.2 DEPENDENCY GRAPH EXTRACTION . A dependency graph Gc = ( V ( Gc ) , E ( Gc ) ) is extracted from c. V ( Gc ) indicates all vertices/nodes inGc , and each vertice corresponds to a word in c. E ( Gc ) denotes the edges ofGc . Each edge inGc represents the syntactical relation categories between a pair of words . By deploying the dependency graph , we transfer the sequential text data into graph-structured format . In Gc , the long-distance pair of words in texts could be close in the graph . This provides extra syntactical knowledge and makes down-stream algorithms easier to pinpoint the sentimental words . 3.3 PATH SEARCHING VIA REINFORCEMENT LEARNING . Compared with state-of-the-art approaches utilizing graph learning algorithms ( e.g. , GCN , attention , and RNN ) , we proposed a reinforcement learning framework to explore the most effective aspectsentiment path on the dependency graph . First , an agent starts walking from the target aspect node . Then , a policy network selects the most effective walk based on the previous walk history and all feasible walk candidates . Next , a semantic understanding module is deployed to aggregate the path state and a sentiment classifier is used to obtain the final sentiment prediction . In summary , there are three components in our framework , 1 ) A Semantic Understanding Module which aggregates the comprehensive walk state , 2 ) A Policy Network which makes the walk decision , and 3 ) A Sentiment Classifier which obtains the final prediction of the sentiment categories . The details of each component are introduced in the following sections . 3.4 SEMANTIC UNDERSTANDING MODULE . The semantic understanding module aims to provide a comprehensive state of the walk for 1 ) allowing the policy network to make the effective walk action and 2 ) letting the sentiment classifier obtain a final prediction . There is a requirement that the state updating mechanism should make the walking procedure as a Markov Decision Process ( MDP ) . The expression is shown below : P ( St+1|S0 , A0 , ... , St , At ) = P ( St+1|St , At ) , ( 1 ) where Si , Ai , and Ri are the state , action , and reward of the i-th move respectively . Eq . equation 1 indicates that the path state of ( t+ 1 ) -th move , St+1 , should be only relevant to the current state St and action At , and irrelevant to earlier states . While , the information of the entire path is crucial and necessary for sentimental analysis tasks . To this end , St is required to preserve both the current and all previous walk information . To achieve this goal , we deploy the general Long Short-Term Memory ( LSTM ) Hochreiter & Schmidhuber ( 1997 ) structure in our framework , as LSTM and its varieties have been well validated as an effective way to capture both the feature and the sequential knowledge in a given sample . In our model , the function is shown below : St = LSTM ( At , St−1 ) , ( 2 ) where St is the current state , which could also be considered as the hidden state updated in each loop . At is the node feature of the corresponding t-th action ( i.e. , word embedding ) which is walked though by agent in the t-th move . Semantic understanding module keeps updating St for each walk . We consider St is effective enough to aggregate and preserve the entire path information .

Summary: The paper addresses aspect-based sentiment analysis by running reinforcement learning on the dependency parse of input sentences. The agent learns a policy network to select the most effective walk along edges in the dependency graphs, starting from the target aspect in the input sentences. The state representation is learned with an LSTM. At the end of the path, a sentiment classifier predicts the distribution of the polarity. The reward is the mean squared error between the class label (i.e. sentiment polarity) and the probability predicted by the model. The paper claims that by limiting the agent's budget, the approach forces the agent to discard irrelevant information and focus on the effective paths, enabling the approach to perform well with a small number of training examples.